Augmented reality processing method and device and communication equipment

Abstract

The invention relates to an augmented reality processing method and device and communication equipment. The method comprises the following steps: receiving a service request sent by a terminal, wherein the service request is sent by the terminal under the condition of acquiring live-action data; obtaining display information of the terminal, and determining virtual scene data corresponding to the service request according to the display information of the terminal and the network transmission condition; wherein the display information represents the display capability of the terminal for the virtual scene data; and sending the virtual scene data to the terminal. According to the method, the display capability of the terminal is used as input and guidance, the virtual scene processing service quality of the server side is determined in combination with the network transmission condition, high-quality cloud virtual scene processing output is achieved, and efficient fusion of the real scene and the virtual scene is achieved.

Description

Technical Field

[0001] This application relates to the field of network technology, and in particular to an augmented reality processing method, apparatus, and communication device. Background Technology

[0002] Cloud AR (Augmented Reality) is an AR method that leverages cloud computing capabilities. It primarily involves capturing real-world scene data on the terminal, rendering / encoding the virtual scene in the cloud, and then sending the processed virtual scene to the terminal for fusion and display. However, current cloud AR processing mainly employs fixed strategies, resulting in a lack of flexibility in the output cloud AR virtual scene. This makes it difficult to achieve accurate alignment with the real-world scene, impacting service quality. Summary of the Invention

[0003] This application provides an augmented reality processing method, apparatus, and communication device, which can bring about the beneficial effect of improving service quality.

[0004] Firstly, an augmented reality processing method is provided, applied to the virtual scene rendering function in cloud AR, the method including:

[0005] Receive service requests sent by the terminal, which are sent by the terminal when acquiring real-world data;

[0006] The terminal's display information is obtained, and the corresponding virtual scene data for the service request is determined based on the terminal's display information and network transmission conditions; whereby the display information represents the terminal's display capability for virtual scene data.

[0007] Send virtual scene data to the terminal.

[0008] In one embodiment, the displayed information is determined based on the terminal's screen display parameters and the data acquisition parameters for the terminal to obtain real-world data;

[0009] Data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; screen display parameters include terminal display resolution, terminal display frame rate, and terminal display depth of field; display information includes target resolution, target frame rate, and target depth of field; the target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate; the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field;

[0010] In cases where the real-scene data is obtained by the terminal through perspective mode, the displayed information is the screen display parameters.

[0011] In one embodiment, the virtual scene data corresponding to the service request is determined based on the terminal's display information and network transmission conditions, including:

[0012] The virtual scene rendering function obtains the network parameters used for communication between the current network and the terminal; the network parameters are parameters that characterize the network transmission performance of the current network.

[0013] Based on whether the network parameters meet the normal quality transmission requirements for displaying information and whether the network parameters meet the basic quality transmission requirements for service capability information, the display information or service capability information is determined as the target capability information; among them, the basic quality transmission requirements are lower than the normal quality transmission requirements; the service capability information represents the minimum capability that the virtual scene rendering function supports for augmented reality services.

[0014] The virtual scene data is obtained by rendering and encoding according to the target capability information.

[0015] In one embodiment, the display information or service capability information is determined as the target capability information based on whether the network parameters meet the normal quality transmission requirements for display information and whether the network parameters meet the basic quality transmission requirements for service capability information, including:

[0016] If all parameters in the network parameters meet the requirements for normal quality transmission, the displayed information will be identified as the target capability information.

[0017] If some parameters in the network parameters meet the requirements for normal quality transmission, then if it is confirmed that all parameters in the network parameters after network enhancement can meet the requirements for normal quality transmission, the displayed information will be identified as the target capability information.

[0018] If all parameters in the network parameters do not meet the normal quality transmission requirements, but if all parameters in the network parameters meet the basic quality transmission requirements, then the service capability information is determined as the target capability information.

[0019] If all or some of the network parameters do not meet the basic quality transmission requirements, the service request will not be responded to.

[0020] In one embodiment, network parameters include network transmission bandwidth and network transmission latency; display information includes target resolution, target frame rate, and target depth of field; wherein:

[0021] If the network transmission bandwidth is greater than or equal to the first transmission bandwidth, then the network transmission bandwidth is confirmed to meet the normal quality transmission requirements.

[0022] If the network transmission latency is less than or equal to the recommended latency for augmented reality services, then the network transmission latency is confirmed to meet the normal quality transmission requirements.

[0023] Wherein, the first transmission bandwidth is the quotient of the first product and the first compression ratio; the first product is the product of the target resolution, the target frame rate and the target depth of field; the first compression ratio represents the compression ratio of the server performing normal encoding.

[0024] In one embodiment, rendering and encoding are performed according to the target capability information to obtain virtual scene data, including:

[0025] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission latency is less than or equal to the recommended latency, the virtual scene data is obtained by rendering according to the display information and encoding according to the first compression ratio.

[0026] In one embodiment, rendering and encoding are performed according to the target capability information to obtain virtual scene data, including:

[0027] When the network transmission latency is less than or equal to the recommended latency, and the network transmission bandwidth is less than the first transmission bandwidth, network enhancement is performed through bandwidth adjustment, and rendering is performed according to the display information and encoding is performed according to the first compression ratio to obtain virtual scene data.

[0028] In one embodiment, bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

[0029] In one embodiment, rendering and encoding are performed according to the target capability information to obtain virtual scene data, including:

[0030] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and less than the maximum delay corresponding to the augmented reality service, then a delay adjustment strategy is selected based on the network transmission bandwidth to perform network enhancement.

[0031] Based on the displayed information, a processing strategy that is linked with the latency adjustment strategy is used for rendering and encoding to obtain virtual scene data.

[0032] In one embodiment, the latency adjustment strategy includes network link selection and network link optimization;

[0033] Based on the displayed information, a processing strategy linked to the latency adjustment strategy is used for rendering and encoding to obtain virtual scene data, including:

[0034] When the network transmission bandwidth is greater than or equal to the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the second compression ratio; wherein, the second compression ratio is less than the first compression ratio.

[0035] When the network transmission bandwidth is less than the redundancy threshold, the virtual scene data is obtained by rendering according to the displayed information and encoding according to the first compression ratio.

[0036] In one embodiment, the redundancy threshold is a first transmission bandwidth that is a preset multiple; the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0037] In one embodiment, the preset multiple is 2 times; the preset coefficient is the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0038] In one embodiment, network parameters include network transmission bandwidth and network transmission latency; service capability information includes minimum resolution, minimum frame rate, and minimum depth of field; wherein:

[0039] If the network transmission bandwidth is greater than or equal to the second transmission bandwidth and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, then it is confirmed that all parameters in the network parameters meet the basic quality transmission requirements.

[0040] The second transmission bandwidth is the quotient of the second product and the first compression ratio; the second product is the product of the minimum resolution, the minimum frame rate, and the minimum depth of field; the first compression ratio represents the compression ratio used by the server for normal encoding.

[0041] In one embodiment, rendering and encoding are performed according to the target capability information to obtain virtual scene data, including:

[0042] The virtual scene data is obtained by rendering based on the service capability information and encoding according to the first compression ratio.

[0043] Secondly, an augmented reality processing method is provided for application on a terminal, the method including:

[0044] When real-world data is acquired, a service request is sent to the cloud AR cloud, and the acquired display information is also sent to the cloud AR cloud. The service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request based on the display information and network transmission conditions. The display information represents the terminal's display capability for virtual scene data.

[0045] Receive virtual scene data sent from the cloud AR cloud.

[0046] In one embodiment, the method further includes:

[0047] Based on the screen display parameters and the data acquisition parameters for obtaining real-scene data, the display information is determined;

[0048] Data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; screen display parameters include terminal display resolution, terminal display frame rate, and terminal display depth of field; display information includes target resolution, target frame rate, and target depth of field; the target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate; the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field;

[0049] When real-world data is obtained through perspective mode, the screen display parameters are determined as the display information.

[0050] In one embodiment, the method further includes:

[0051] Based on virtual scene data and real-world data, the output is a virtual-real fusion result.

[0052] Thirdly, an augmented reality processing device is provided for virtual scene rendering in cloud AR, the device comprising:

[0053] The request receiving module is used to receive service requests sent by the terminal. The service requests are sent by the terminal when it is acquiring real-world data.

[0054] The virtual data determination module is used to acquire the terminal's display information and determine the virtual scene data corresponding to the service request based on the terminal's display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0055] The data sending module is used to send virtual scene data to the terminal.

[0056] Fourthly, an augmented reality processing device is provided for use on a terminal, the device comprising:

[0057] The information sending module is used to send service requests to the cloud AR cloud when real-world data is acquired, and to send the acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request based on the display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0058] The data receiving module is used to receive virtual scene data sent from the cloud AR cloud.

[0059] Fifthly, a communication device is provided, comprising: a transmitter, a processor, and a receiver;

[0060] The receiver is used to receive service requests sent by the terminal, which are sent by the terminal when acquiring real-time data.

[0061] The processor is used to acquire the display information of the terminal and determine the virtual scene data corresponding to the service request based on the display information of the terminal and the network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0062] A transmitter is used to send virtual scene data to the terminal.

[0063] Sixthly, a communication device is provided, comprising: a transmitter and a receiver;

[0064] The transmitter is used to send service requests to the cloud AR cloud when real-world data is acquired, and to send the acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request based on the display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0065] The receiver is used to receive virtual scene data sent from the cloud AR cloud.

[0066] In a seventh aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in the above aspects.

[0067] Eighthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the methods described in the above aspects.

[0068] The aforementioned augmented reality processing method, apparatus, and communication equipment, when applied to the virtual scene rendering function in cloud AR, can receive service requests issued by the terminal after acquiring real-scene data. Based on the terminal's display information and network transmission conditions, the system determines the virtual scene data corresponding to the service request and sends the virtual scene data to the terminal. The display information can represent the terminal's display capability for the virtual scene data. This application uses the terminal's display capability as input and guide to determine the quality of the cloud-side virtual scene processing service, achieving high-quality cloud-based virtual scene processing output, realizing efficient fusion of real and virtual scenes, and improving user experience. Attached Figure Description

[0069] Figure 1 This is a diagram illustrating the application environment of an augmented reality processing method in one embodiment.

[0070] Figure 2 This is a flowchart illustrating an augmented reality processing method in one embodiment;

[0071] Figure 3 This is a flowchart illustrating the steps for obtaining virtual scene data in one embodiment;

[0072] Figure 4This is a flowchart illustrating the process of determining target capability information in one embodiment;

[0073] Figure 5 This is a flowchart illustrating the augmented reality processing method in another embodiment;

[0074] Figure 6 This is a schematic diagram of the functional architecture of an augmented reality processing method in one embodiment;

[0075] Figure 7 This is a schematic diagram of the cloud-AR cloud-network-end service collaboration process in one embodiment.

[0076] Figure 8 This is a schematic diagram of the cloud AR service operation phase in one embodiment;

[0077] Figure 9 This is a structural block diagram of an augmented reality processing device in one embodiment;

[0078] Figure 10 This is a structural block diagram of the augmented reality processing device in another embodiment;

[0079] Figure 11 This is an internal structure diagram of a communication device in one embodiment;

[0080] Figure 12 This is a diagram of the internal structure of a communication device in another embodiment. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0082] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0083] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more.

[0084] Figure 1 This is a schematic diagram illustrating an application scenario of an augmented reality processing method provided in an embodiment of this application. For example... Figure 1 As shown, in this scenario, terminal 102 communicates with server 104 via a network. For example, terminal 102 can be a cloud AR terminal, acting as the request initiator of cloud AR services and the final display presenter of the services; optionally, server 104 can be a cloud AR cloud platform, acting as the executor of cloud AR virtual scene processing, mainly completing the rendering, compression encoding, and other processing of the virtual scene based on the network linkage capability.

[0085] Optionally, terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services (also known as a cloud server, or simply cloud).

[0086] In traditional technologies, current cloud AR processing primarily employs a fixed strategy in the cloud, such as fixed rendering methods and fixed deep compression ratios. This leads to either fixed rendering and compression encoding mechanisms for processing cloud AR virtual scenes, or processing of virtual scenes on the AR terminal (i.e., locally on the terminal). Consequently, the cloud-output AR virtual scenes lack flexibility and struggle to match the accuracy of the real-world scene seamlessly. This is particularly problematic for fragmented AR terminals, whose capabilities vary significantly, primarily in terms of real-world scene acquisition accuracy, display quality, and processing power. This lack of effective and flexible interaction results in a disconnect between the real and virtual scenes, leading to user experience issues. Furthermore, with the widespread adoption of fiber-to-the-home (FTTH), network transmission capabilities have significantly improved. Deep compression, originally intended to save bandwidth, now often increases the burden on cloud processing and extends latency, diminishing the bandwidth savings. Moreover, the network's enhanced capabilities, such as acceleration, link optimization, and bandwidth expansion, still require full utilization.

[0087] Traditional technologies suffer from at least one problem: the lack of a linkage mechanism between the cloud and AR service networks, resulting in ineffective network capabilities to support the cloud AR business experience. Current cloud AR virtual scene processing typically employs a single graphics rendering and fixed encoding mechanism in the cloud, failing to integrate with network bandwidth, latency, and other transmission attributes. Furthermore, it lacks the ability to leverage network bandwidth improvements, accelerated transmission, and link optimization, hindering cloud AR-network integration and limiting user experience enhancement. Traditional technologies also suffer from a lack of end-to-end latency synchronization in cloud AR, leading to inconsistent user experiences. Current cloud AR services encompass various stages, from terminal-side real-scene acquisition and localized digitization, to cloud-based virtual scene rendering, encoding, and streaming, culminating in terminal-side virtual-real fusion output. The latency of cloud-based virtual scene rendering and encoding is closely related to streaming bandwidth, and while latency can be reduced through network transmission acceleration and link optimization, a lack of linkage mechanisms with network capabilities generally results in inconsistent user experiences.

[0088] Based on the aforementioned traditional technologies, this application proposes to use the real-scene acquisition accuracy of the terminal (hereinafter referred to as real-scene accuracy) as a benchmark, and by integrating terminal and network performance, adopt a more flexible multi-dimensional processing mechanism for cloud-based AR virtual scenes, thereby achieving a high-quality end-to-end user experience and high-quality service output for cloud AR. Specifically, this application adopts the integrated cloud-network-terminal business linkage capabilities, using the terminal's display information as a guide, and employing business-network capability integration and optimization to achieve end-to-end cloud AR virtual scene accuracy enhancement.

[0089] Optionally, embodiments of this application can integrate metropolitan area network technologies and existing network trials, such as end-to-end service-oriented DCI (Data Center Interconnect) networking research. For cloud AR business scenarios, based on the optimal display accuracy of the terminal virtual scene as input and guidance, and through multi-dimensional linkage optimization of cloud, network, and terminal services, high-quality cloud virtual scene processing output is achieved, realizing efficient fusion of cloud AR real and virtual scenes and improving user experience.

[0090] It should be noted that the beneficial effects or technical problems solved by the embodiments of this application are not limited to this one, but may also be other implicit or related problems. For details, please refer to the description of the embodiments below.

[0091] Before introducing the specific embodiments of this application, the technical terms involved in this application will be explained:

[0092] Real-world accuracy: This can refer to the clarity of a cloud AR terminal, such as resolution, frame rate, and depth of field, which it can acquire or perceive through its camera and perspective optical devices.

[0093] Converged Terminal-Network Performance: Terminal and network can refer to the terminal and network respectively. Terminal performance corresponds to terminal performance and network performance. Terminal performance refers to the display clarity, refresh rate, and other capabilities related to cloud AR terminals. Network performance refers to the comprehensive transmission and enhancement capabilities related to cloud AR, including network bandwidth, latency, and link optimization. Converged terminal-network performance refers to the comprehensive transmission capabilities based on and combining terminal display clarity, refresh rate, network bandwidth, latency, and link optimization to provide better service quality and user experience.

[0094] Cloud AR: This refers to a method of realizing AR services based on cloud processing capabilities. Specifically, it can refer to the accuracy analysis of the actual scene acquired on the terminal side, combined with the terminal's processing performance, network transmission, and enhancement performance, to determine the service quality of the virtual scene. The AR virtual scene is then flexibly rendered and encoded in the cloud and sent to the terminal. The terminal, based on the real scene acquired locally, integrates the virtual scene sent from the terminal side to achieve a local virtual-real fusion display.

[0095] Cloud AR rendering: refers to the process of rendering corresponding AR virtual scenes based on the cloud. It completes cloud rendering processing according to the business and network linkage strategy and the determined cloud output virtual scene precision.

[0096] Cloud AR encoding: This refers to the process of encoding and compressing corresponding AR virtual scenes based on the cloud. It involves encoding based on the rendering output results and network transmission conditions, combined with network optimization capabilities, and in accordance with business and network coordination strategies.

[0097] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0098] In one exemplary embodiment, such as Figure 2 As shown, an augmented reality processing method is provided, which is applied to... Figure 1 Taking a server (such as a cloud server) as an example, it can be understood that this method can also be applied to the virtual scene rendering function of cloud AR, including the following steps 202 to 206. Among them:

[0099] Step 202: Receive a service request sent by the terminal. The service request is sent by the terminal when it obtains real-world data.

[0100] Among them, service request can refer to AR service request; taking the terminal as a cloud AR terminal as an example, the cloud AR terminal, as the request initiator of cloud AR service and the final display presenter of the service, can initiate relevant service requests and simultaneously collect and process relevant real-world data.

[0101] For example, a terminal can initiate a service request to the server while acquiring real-world data. Taking a cloud AR service request as an example, the terminal can initiate a cloud AR service request and collect real-world data simultaneously.

[0102] Specifically, when the server receives a service request from the terminal, it determines that virtual scene processing is required. The service request can indicate that the current phase of cloud AR service operation has begun. Optionally, taking the server as the cloud AR cloud as an example, the cloud AR cloud, as the executor of cloud AR virtual scene processing, can perform processing including but not limited to, flexibly adjusting the rendering and compression encoding of the virtual scene based on network linkage capabilities.

[0103] Step 204: Obtain the terminal's display information, and determine the virtual scene data corresponding to the service request based on the terminal's display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data.

[0104] Specifically, the server can obtain the terminal's display information and use it as a benchmark to determine the cloud-based virtual scene processing; wherein, the server connects to the terminal side to obtain the terminal's display information. For example, the terminal can obtain the display information and send the obtained display information to the server in order to achieve optimal virtual-real fusion output.

[0105] In response to receiving a service request, the server can determine the virtual scene data corresponding to the service request based on the terminal's display information and network transmission conditions. The display information can be used to represent the terminal's display capability for the virtual scene data, and the network transmission conditions can be the transmission conditions of the current network for communication between the server and the terminal. For example, the terminal's display information can be obtained during the cloud AR cloud-network-terminal business collaboration stage. Optionally, the cloud AR cloud-network-terminal business collaboration stage can serve as a preparatory stage for the cloud AR business operation stage, and the network transmission conditions can be obtained during the cloud AR business operation stage.

[0106] In some examples, the displayed information is used to characterize the optimal display accuracy of the virtual scene on the terminal. This optimal display accuracy can be understood as the optimal display accuracy of the virtual scene on the terminal side. For example, this optimal display accuracy can refer to the virtual scene output result that the terminal can achieve with the best user experience, determined based on the real-scene acquisition accuracy as a benchmark and target, and according to the terminal's processing capabilities. Further, the process by which the server determines the virtual scene data based on the terminal's display information can be understood as the server aiming for the optimal virtual scene output for the terminal user experience, combining network transmission conditions, and using flexible virtual scene rendering and adaptive encoding in the cloud to obtain the optimal virtual scene result for the cloud user experience. This leverages cloud-network-terminal capabilities to solve the problem of seamless high-precision integration between cloud AR and reality. Based on the embodiments of this application, the optimal virtual-real fusion result with end-to-end optimization guarantee for cloud AR can ultimately be obtained.

[0107] Step 206: Send the virtual scene data to the terminal.

[0108] Specifically, after obtaining the virtual scene data, the server can send the virtual scene data to the terminal, so that the terminal can output a virtual-real fusion result based on the virtual scene data and the collected real scene data.

[0109] It should be noted that outputting the virtual-real fusion result can refer to empowering the captured real-world scene with relevant virtual digital capabilities to facilitate the subsequent addition and interaction of other virtual avatars. For example, based on virtual scene data and real-world data, the terminal can overlay the real-world scene as the bottom layer and the virtual scene as the top layer to form an augmented reality image. It should be understood that the embodiments disclosed herein are not limited to this; in other implementations, the terminal can also overlay the real-world scene as the bottom layer and the real-world scene as the top layer.

[0110] Furthermore, taking a cloud server as an example, the cloud server sends the processed virtual scene results (virtual scene data) to the terminal. The terminal receives the virtual scene results and adjusts the lighting (e.g., brightness adjustment) according to the real-world lighting conditions to complete the cloud AR virtual-real fusion output. Based on the virtual-real fusion result, the terminal performs corresponding interactive operations. Alternatively, taking an AR terminal as an example, cloud AR virtual-real fusion output and interaction can be achieved. The AR terminal receives the virtual scene sent from the cloud and, based on the locally acquired real-world scene, adjusts the corresponding highlights to combine with the locally collected real-world scene, completing the virtual-real fusion cloud AR service terminal display output, and performing relevant cloud AR interactions based on the output results.

[0111] In the aforementioned augmented reality processing method, the cloud AR virtual scene rendering function can include: receiving a cloud AR service request issued after acquiring real-world data; acquiring the terminal's display information, which indicates the terminal's display capability for virtual scene data; determining the corresponding virtual scene data for the service request based on the terminal's display information and network transmission conditions; and finally sending the virtual scene data to the terminal. This application embodiment uses the terminal's display capability as a benchmark and combines it with network transmission conditions to adopt a more flexible virtual scene processing mechanism, achieving high-quality cloud virtual scene processing output and realizing a high-quality end-to-end user experience and high-quality service output for cloud AR.

[0112] In some embodiments, the display information is determined based on the terminal's screen display parameters and the data acquisition parameters for the terminal to acquire real-world data.

[0113] Specifically, in this embodiment, the display information can be determined based on the terminal's screen display parameters and the data acquisition parameters for acquiring real-world data. For example, the terminal can determine the display information based on the data acquisition parameters for acquiring real-world data and the screen display parameters, and then transmit the display information to the server.

[0114] Optionally, the data acquisition parameters can refer to the relevant parameters of the terminal's real-scene acquisition device, used to characterize the real-scene acquisition accuracy (hereinafter referred to as real-scene accuracy); for example, the terminal can obtain the real-scene accuracy through the real-scene acquisition device, including but not limited to resolution, frame rate, etc. The real-scene acquisition device can include, but is not limited to, a camera, a light-transmitting lens, etc.

[0115] Furthermore, screen display parameters can represent the terminal's display capabilities (which can be understood as the terminal's display precision), including but not limited to resolution, frame rate, and refresh rate. Taking the display information representing the terminal's optimal virtual scene display precision as an example, the terminal can combine the real scene acquisition precision and the terminal's display capabilities to obtain the optimal virtual scene display precision, so as to achieve the best virtual-real fusion output.

[0116] For example, based on the accuracy of real-scene data acquisition on the terminal side, and combined with the terminal side's display capabilities, the optimal display accuracy of the virtual scene on the terminal side can be determined. For instance, the terminal side can compare the accuracy of real-scene data acquisition with the terminal's display capabilities to determine the optimal display accuracy of the virtual scene: ① When the accuracy of real-scene data acquisition is greater than or equal to the terminal's display capabilities, the terminal's display capabilities are determined as the optimal display accuracy of the virtual scene; ② When the accuracy of real-scene data acquisition is less than the terminal's display capabilities, the accuracy of real-scene data acquisition is determined as the optimal display accuracy of the virtual scene. By selecting the accuracy of real-scene data acquisition as the optimal display accuracy of the virtual scene, it is possible to seamlessly connect with the corresponding real scene and achieve the best virtual-real fusion output.

[0117] In one embodiment, the data acquisition parameters may include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; the screen display parameters may include terminal display resolution, terminal display frame rate, and terminal display depth of field.

[0118] The displayed information includes target resolution, target frame rate, and target depth of field; where the target resolution is the minimum of the real-scene capture resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene capture frame rate and the terminal display frame rate; and the target depth of field is the minimum of the real-scene capture depth of field and the terminal display depth of field.

[0119] Specifically, the terminal can acquire data acquisition parameters that characterize the accuracy of real-scene acquisition, as well as screen display parameters that characterize the display capability. The data acquisition parameters may include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; the screen display parameters may include terminal display resolution, terminal display frame rate, and terminal display depth of field, and the displayed information may include target resolution, target frame rate, and target depth of field.

[0120] For example, the terminal can obtain the relevant resolution, frame rate, depth of field, etc. of the terminal's current real-scene acquisition device as the real-scene acquisition accuracy, such as the resolution of the camera and the frame rate of real-scene shooting; if the real-scene perspective mode is used for acquisition, the real-scene acquisition accuracy can be assumed to be the real-scene accuracy obtained by the human eye; furthermore, the terminal can obtain the screen display capability, that is, obtain the relevant resolution, frame rate, and other indicators of the terminal screen display.

[0121] Optionally, the target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution, the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate, and the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field.

[0122] In practical applications, taking a cloud AR terminal as an example, during the cloud AR cloud-network-end business collaboration phase, the real-scene acquisition accuracy of the cloud AR terminal is obtained. This includes the camera or light-transmitting lens used by the cloud AR terminal to acquire the real scene. The camera directly acquires its pixel resolution, the corresponding frame rate, and the depth of field. The light-transmitting lens is assumed to be the accuracy observed by the human eye. Assuming the real-scene acquisition resolution of the cloud AR terminal is r1 (real-scene acquisition resolution), its frame rate is f1 (real-scene acquisition frame rate), and its depth of field is d1 (real-scene acquisition depth of field), the terminal display accuracy of the cloud AR terminal includes the resolution, frame rate, and depth of field displayed on the cloud AR terminal screen, which are assumed to be r2 (terminal display resolution), f2 (terminal display frame rate), and d2 (terminal display depth of field), respectively.

[0123] By comparing the real-scene capture accuracy and the screen display accuracy of the cloud AR terminal, the optimal display accuracy of the virtual scene on the cloud AR terminal was obtained: r t (target resolution), f t(Target frame rate), d t (Target Depth of Field), the relevant rules are as follows: the optimal display accuracy of the virtual scene on the cloud AR terminal, where the resolution r t =Min(r1,r2), frame rate f t =Min(f1,f2), depth of field d t =Min(d1,d2).

[0124] It is understandable that when using perspective mode to obtain real-world data, in one embodiment, when the real-world data is obtained by the terminal through perspective mode, the displayed information is the screen display parameters.

[0125] Specifically, perspective mode can refer to the terminal using a real-scene perspective mode to obtain real-scene data; it should be noted that real-scene perspective mode includes, but is not limited to, perspective lens mode, such as using a light-transmitting lens to obtain the corresponding real-time scene.

[0126] In cases where the real-world data is obtained by the terminal through a perspective mode, the terminal can determine the screen display parameters as the displayed information. Taking a cloud AR terminal as an example, if the cloud AR real-world data acquisition uses a perspective lens mode, then by default, the real-world accuracy of the cloud AR terminal is greater than the terminal's screen display accuracy. That is, in this case, the resolution, frame rate, and depth of field related to the optimal display accuracy of the cloud AR terminal's virtual scene are equal to r², f², and d², respectively. Furthermore, the server can acquire and analyze the optimal display accuracy of the terminal's virtual scene: the server obtains the optimal display accuracy of the terminal's virtual scene by connecting to the terminal side, and correspondingly obtains its resolution, frame rate, and depth of field, and uses this as a benchmark to determine the cloud-based virtual scene processing.

[0127] In the above-mentioned augmented reality processing method, the accuracy of the real scene is used as the standard. The terminal screen display clarity, refresh rate performance and other capabilities are comprehensively considered to determine the best virtual scene output that the terminal can achieve. With the best virtual scene output of the terminal as the goal, the service quality of cloud AR is unified to obtain the best virtual scene result for cloud user experience. By obtaining the virtual scene result, the terminal can make light and shadow adjustments based on the real scene (such as real-time brightness adjustment), and finally obtain the best virtual-real fusion result with cloud AR end-to-end optimization guarantee. The embodiments of this application integrate the capabilities of cloud network and terminal to achieve high-precision seamless connection between cloud AR virtual and real, and integrate the multi-dimensional performance of terminal network to provide high-quality cloud AR services.

[0128] In practical applications, this application focuses on cloud AR virtual scene processing. By assessing the terminal's real-scene capture accuracy and display capabilities, it determines the optimal display accuracy of the virtual scene on the terminal and aims to improve the output accuracy of the virtual scene. Furthermore, it enhances the cloud-based processing effect of the cloud AR virtual scene by integrating network performance and business network capabilities, thereby further improving the end-to-end user experience. In one embodiment, such as... Figure 3As shown, step 204 may include steps 302 to 306. Wherein:

[0129] Step 302: Obtain the network parameters for communication between the current network and the terminal through the virtual scene rendering function; the network parameters are parameters that characterize the network transmission performance of the current network.

[0130] Specifically, embodiments of this application can obtain network parameters for communication between the current network and the terminal through virtual scene rendering functionality. Taking a cloud-based AR server as an example, the cloud-based AR server can acquire and analyze network performance, and then integrate terminal and network performance to achieve flexible virtual scene processing, thereby obtaining the best virtual-real fusion result. Exemplarily, network performance can include network transmission performance, which can be obtained through collaboration between the cloud and the terminal. Optionally, the network parameters can be parameters used during normal transmission between the cloud and the terminal.

[0131] The server can obtain network parameters for communication with the terminal using the current network. These parameters characterize the network transmission performance, such as bandwidth and latency. The terminal can also acquire and analyze network performance, working with the server (e.g., a cloud-based AR platform) to obtain the bandwidth, latency, and other performance parameters of the current network transmission. In some examples, using a cloud server as an example, bandwidth and latency testing tools related to both the cloud and the terminal can be used to obtain the bandwidth and latency of normal transmission between the cloud and the terminal as network parameters.

[0132] Step 304: Based on whether the network parameters meet the normal quality transmission requirements for displaying information and whether the network parameters meet the basic quality transmission requirements for service capability information, the display information or service capability information is determined as the target capability information; wherein, the basic quality transmission requirements are lower than the normal quality transmission requirements; the service capability information represents the minimum capability that the virtual scene rendering function supports for augmented reality services.

[0133] Specifically, upon obtaining network parameters, the server can determine the target capability information, either display information or service capability information, based on the network transmission conditions satisfied by the network parameters. The network transmission conditions can refer to the transmission bandwidth and latency required for displaying the information, or the transmission bandwidth and latency required for service capability information, to ensure a better interactive latency experience for the user. Optionally, the network transmission conditions may include, but are not limited to, the normal quality transmission requirements for displaying the information and the basic quality transmission requirements for service capability information. This service capability information can represent the minimum capability required for the virtual scene rendering function to support augmented reality services, and the basic quality transmission requirements are lower than the normal quality transmission requirements.

[0134] Normal quality transmission requirements can represent the transmission requirements of display information (such as the optimal display precision of the terminal's virtual scene). The server can obtain and analyze the terminal's display information to determine the normal quality transmission requirements, such as bandwidth requirements and latency requirements. Taking the optimal display precision of the terminal's virtual scene as an example, the server can obtain the optimal display precision of the virtual scene sent by the terminal and decompose it into resolution, frame rate, depth of field, etc., and simultaneously calculate the required bandwidth requirements based on the compression rate corresponding to commonly used encoding protocols.

[0135] Basic quality transmission requirements can represent the transmission requirements of service information. Servers can acquire and analyze service capability information to determine these basic quality transmission requirements, such as bandwidth and latency requirements. For example, service capability information can characterize the minimum service precision of the virtual scene. For instance, when the optimal display precision of the virtual scene cannot be supported by the terminal, it is necessary to determine the minimum service precision that can meet the user experience. This minimum service precision can include resolution, frame rate, and depth of field. In some examples, the server can determine the minimum service quality of the cloud AR service based on meeting the minimum user experience requirements, specifying the minimum resolution, frame rate, and depth of field indicators output by the cloud. For example, the server can determine the minimum service precision by combining the optimal display precision of the terminal's virtual scene with the precision range of the cloud AR service's virtual scene processing. Alternatively, the server can select the precision output of the basic configuration of the cloud AR application service, such as disabling corresponding lighting effects and reducing the frame rate, but with a certain level of user experience assurance, as the minimum service precision.

[0136] Furthermore, the target capability information in this application embodiment can characterize the cloud processing accuracy to represent different cloud processing requirements. For example, based on whether the network parameters meet the normal quality transmission requirements and the basic quality transmission requirements, the display information or service capability information is determined as the target capability information, so that the server can determine the corresponding cloud processing accuracy according to different combinations based on the optimal display accuracy of the virtual scene of the terminal corresponding to the cloud AR and the user's better interactive latency experience. Optionally, taking the server as a cloud server as an example, combined with the current network transmission performance, the current network bandwidth and latency are used to determine whether the corresponding cloud processing requirements are met, so that the cloud determines the corresponding processing accuracy.

[0137] Step 306: Render and encode according to the target capability information to obtain virtual scene data.

[0138] Specifically, when it is determined that display information or service capability information is the target capability information, the server can render and encode according to the target capability information to obtain virtual scene data.

[0139] The aforementioned augmented reality processing method can complete network capability matching by obtaining network parameters for communication between the current network and the terminal through the virtual scene rendering function. These network parameters characterize the network transmission performance of the current network. Based on whether the network parameters meet the normal quality transmission requirements for displaying information and whether the network parameters meet the basic quality transmission requirements for service capability information, the display information or service capability information is determined as the target capability information. Among them, the basic quality transmission requirements are lower than the normal quality transmission requirements. The service capability information represents the minimum capability that the virtual scene rendering function supports for augmented reality services.

[0140] This application aims to achieve optimal display accuracy for the virtual scene on the terminal. The cloud performs virtual scene rendering and encoding based on network transmission performance to obtain the virtual scene result. By incorporating network transmission performance as part of the cloud AR service's virtual scene processing, the cloud can integrate rendering and encoding with network capabilities, thus integrating network transmission into the cloud AR service processing. The terminal side, by obtaining the virtual scene result, adjusts the brightness in real time based on the real scene, ultimately achieving high-quality cloud AR virtual-real fusion output, effectively improving the user experience.

[0141] In one exemplary embodiment, such as Figure 4 As shown, step 304 includes steps 402 to 408. Wherein:

[0142] Step 402: If all parameters in the network parameters meet the requirements for normal quality transmission, the displayed information will be determined as the target capability information.

[0143] Specifically, when all network parameters meet the requirements for normal quality transmission, the server can determine the display information as the target capability information. Taking network parameters including bandwidth and latency, display information representing the optimal display accuracy of the terminal's virtual scene, and target capability information representing the cloud processing accuracy as an example, if the current network bandwidth and latency both meet the transmission requirements for the optimal display accuracy of the terminal's virtual scene (i.e., the requirements for normal quality transmission), then the optimal display accuracy of the terminal's virtual scene is determined as the cloud processing accuracy.

[0144] Step 404: If some parameters in the network parameters meet the normal quality transmission requirements, then if it is confirmed that all parameters in the network parameters after network enhancement can meet the normal quality transmission requirements, the displayed information will be determined as the target capability information.

[0145] Specifically, if some network parameters meet the requirements for normal quality transmission, it can be determined whether all network parameters can meet the requirements for normal quality transmission after network enhancement. If the requirements for normal quality transmission are met, the displayed information can be identified as the target capability information. For example, network enhancement methods include, but are not limited to, bandwidth enhancement, network acceleration, and link optimization.

[0146] Taking network parameters including bandwidth and latency, display information representing the optimal display accuracy of the terminal's virtual scene, and target capability information representing the cloud processing accuracy as an example, if one of the current network's bandwidth and latency indicators cannot meet the transmission requirements for the optimal display accuracy of the terminal's virtual scene, if network enhancement enables the network parameters that do not meet the transmission requirements to meet the transmission requirements, then the optimal display accuracy of the terminal's virtual scene will be determined as the cloud processing accuracy, and related business network capabilities can be linked for processing.

[0147] This application's embodiments enable real-time fusion and enhancement of network capabilities to meet the needs of cloud AR services. Specifically, by incorporating network transmission performance and network enhancement capabilities as pre-processing factors for cloud AR virtual scene processing, it facilitates the cloud in developing more flexible rendering and encoding strategies based on network capabilities. Furthermore, it allows for flexible post-processing of enhanced network transmission capabilities to be invoked for the cloud AR virtual scene output results, integrating network transmission and enhancement capabilities into the cloud AR service processing.

[0148] Regarding steps 402 and 404 above, in an exemplary embodiment, network parameters may include network transmission bandwidth and network transmission latency; display information may include target resolution, target frame rate, and target depth of field; wherein:

[0149] If the network transmission bandwidth is greater than or equal to the first transmission bandwidth, then the network transmission bandwidth is confirmed to meet the normal quality transmission requirements.

[0150] If the network transmission latency is less than or equal to the recommended latency for augmented reality services, then the network transmission latency is confirmed to meet the normal quality transmission requirements.

[0151] Wherein, the first transmission bandwidth is the quotient of the first product and the first compression ratio; the first product is the product of the target resolution, the target frame rate and the target depth of field; the first compression ratio represents the compression ratio of the server performing normal encoding.

[0152] Specifically, network parameters may include network transmission bandwidth and network transmission latency, and display information may include target resolution, target frame rate, and target depth of field. When the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission latency is less than or equal to the recommended latency, it is confirmed that all parameters in the network parameters meet the normal quality transmission requirements (i.e., the network transmission bandwidth meets the normal quality transmission requirements, and the network transmission latency meets the normal quality transmission requirements); when the network transmission bandwidth is less than the first transmission bandwidth, or the network transmission latency is greater than the recommended latency, it is confirmed that some parameters in the network parameters meet the normal quality transmission requirements (i.e., the network transmission bandwidth meets the normal quality transmission requirements, or the network transmission latency meets the normal quality transmission requirements).

[0153] For example, the recommended latency can refer to the latency corresponding to augmented reality (AR) services, such as determining the corresponding recommended latency based on the attributes of the corresponding cloud AR service. Further, the first transmission bandwidth can represent the transmission bandwidth required by the server to output the virtual scene with the best display accuracy in the cloud and on the terminal, wherein the first transmission bandwidth is the quotient of a first product and a first compression ratio; the first product is the product of the target resolution, the target frame rate, and the target depth of field; and the first compression ratio represents the compression ratio used by the server for normal encoding.

[0154] In some examples, the first compression ratio can be the normal encoding compression ratio ct1 determined by combining the encoding compression algorithm used in the cloud. With a target resolution of r... t The target frame rate is f t The target depth of field is d t And the recommended latency is l q2 For example, the first transmission bandwidth b b =r t *f t *d t / ct1, where the first product is r t *f t *d t Furthermore, by using bandwidth speed testing and latency detection tools related to both the cloud and the terminal, the normal transmission bandwidth b between the cloud and the terminal can be obtained. n and delay l n That is, the network transmission bandwidth is b n Network transmission latency is l n For example, the cloud processing accuracy can be determined by comparing the bandwidth and latency required for optimal display resolution of the virtual scene on the terminal from the cloud perspective, and the relationship between the network transmission capabilities between the cloud and the terminal. This is possible when simultaneously satisfying b... n >=b b ,l n <=l q2 If the network transmission bandwidth and network transmission latency both meet the normal quality transmission requirements, then the optimal display accuracy of the virtual scene on the terminal is taken as the cloud processing accuracy.

[0155] Furthermore, if b cannot be satisfied simultaneously n >=b b ,l n <=l q2 If only one of these conditions is met, then it is determined whether network transmission capacity can be improved through network enhancement. For example, if b n >=b b ,l n >l q2 (but l n <l q1 , l q1To enhance the maximum latency required for real-world services, network enhancements should be implemented to meet the required latency. n <=l q2 Then, the cloud processing accuracy is determined to be the optimal display accuracy of the virtual scene on the terminal. If b n b ,l n <=l q2 When enhanced by the network, it reaches b n >=b b Therefore, the cloud processing accuracy is determined by using the best display accuracy of the virtual scene on the terminal.

[0156] Step 406: If all parameters in the network parameters do not meet the normal quality transmission requirements, or if all parameters in the network parameters meet the basic quality transmission requirements, then the service capability information is determined as the target capability information.

[0157] Specifically, if all network parameters do not meet the requirements for normal quality transmission, but all network parameters meet the basic quality transmission requirements, the server will determine the service capability information as the target capability information to ensure the provision of cloud AR services with guaranteed user experience.

[0158] Taking network parameters including bandwidth and latency, display information representing the optimal display accuracy of the terminal's virtual scene, service capability information representing the minimum service accuracy, and target capability information representing the cloud processing accuracy as an example, if the current network bandwidth and latency cannot simultaneously meet the transmission requirements for the optimal display accuracy of the terminal's virtual scene (i.e., normal quality transmission requirements), but if the current network bandwidth and latency can meet the transmission requirements for the minimum service accuracy (i.e., basic quality transmission requirements), then the minimum service accuracy is determined as the cloud processing accuracy. Subsequently, the server can determine the corresponding cloud processing based on the determined cloud processing accuracy, according to different situations.

[0159] It should be noted that, regarding step 406 above, in an exemplary embodiment, network parameters may include network transmission bandwidth and network transmission latency; service capability information may include minimum resolution, minimum frame rate, and minimum depth of field; wherein:

[0160] If the network transmission bandwidth is greater than or equal to the second transmission bandwidth and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, then it is confirmed that all parameters in the network parameters meet the basic quality transmission requirements.

[0161] The second transmission bandwidth is the quotient of the second product and the first compression ratio; the second product is the product of the minimum resolution, the minimum frame rate, and the minimum depth of field; the first compression ratio represents the compression ratio used by the server for normal encoding.

[0162] ​Specifically, network parameters include network transmission bandwidth and network transmission latency, while service capability information may include minimum resolution, minimum frame rate, and minimum depth of field. When the network transmission bandwidth is greater than or equal to the second transmission bandwidth and the network transmission latency is less than or equal to the maximum latency, it is confirmed that all parameters in the network parameters meet the basic quality transmission requirements (i.e., both network transmission bandwidth and network transmission latency meet the basic quality transmission requirements).

[0163] The second transmission bandwidth can represent the bandwidth required for the cloud to output with the lowest service precision; for example, the second transmission bandwidth is the quotient of the second product and the first compression ratio, the second product is the product of the lowest resolution, the lowest frame rate and the lowest depth of field, and the first compression ratio can represent the compression ratio of the server performing normal encoding.

[0164] In some examples, the first compression ratio can be a normal encoding compression ratio ct1 determined by combining the encoding compression algorithm used in the cloud. For example, the maximum latency corresponds to augmented reality (AR) services, and is determined based on the corresponding cloud AR service attributes. Furthermore, it is recommended that the latency be less than the maximum latency, with the maximum latency set as l. q1 Recommended latency is l q2 For example, l q2 <l q1 .

[0165] Furthermore, with a minimum resolution of r cs The minimum frame rate is f cs The minimum depth of field is d cs And the maximum latency is l q1 For example, the second transmission bandwidth b r =r cs *f cs *d cs / ct1, where the second product is r cs *f cs *d cs Optionally, the normal transmission bandwidth b between the cloud and the terminal can be obtained using bandwidth speed testing and latency detection tools related to both the cloud and the terminal. n Delay l n For example, the network transmission bandwidth is b n Network transmission latency is l n When the network transmission bandwidth b n and network transmission latency n If none of them meet the optimal display resolution output conditions for virtual scenes on the terminal in the cloud (normal quality transmission requirements), then it is necessary to determine whether they simultaneously meet b. n >=b r ,l n <=l q1(That is, all parameters in the network parameters meet the basic quality transmission requirements). If they are met, the cloud will output the minimum service quality of the virtual scene (that is, the minimum service accuracy will be used as the cloud processing accuracy).

[0166] It is understandable that, since the virtual scene processed in cloud AR does not occupy the entire screen display area, the bandwidth comparison calculated in this application embodiment can have a certain degree of redundancy. This redundancy refers to the fact that the bandwidth calculation process for virtual scene transmission in this application is based on the assumption that the virtual scene occupies the entire screen display area. However, in practical applications, the virtual scene in most AR processing only occupies a portion of the screen display. Therefore, the calculation method in this application can have a certain degree of redundancy.

[0167] Step 408: If all or some of the network parameters do not meet the basic quality transmission requirements, then the service request will not be responded to.

[0168] Specifically, if all or some of the network parameters do not meet the basic quality transmission requirements, the server will not respond to the service request. That is, if it is confirmed that the current network does not meet the minimum service accuracy transmission requirements, the cloud AR-related services will be deemed unavailable and the process will terminate.

[0169] Taking network parameters including bandwidth and latency, display information representing the optimal display accuracy of the terminal's virtual scene, service capability information representing the minimum service accuracy, and target capability information representing the cloud processing accuracy as an example, when the current network bandwidth and latency cannot simultaneously meet the transmission requirements for the optimal display accuracy of the terminal's virtual scene (i.e., normal quality transmission requirements), the cloud processing accuracy can be determined according to different situations: ① If the current network bandwidth and latency do not meet the minimum service accuracy of the cloud virtual scene (i.e., basic quality transmission requirements), then cloud AR services that guarantee user experience cannot be provided (i.e., service requests are not responded to). ② If the current network bandwidth and latency meet the minimum service accuracy of the cloud virtual scene, then the minimum service accuracy is determined as the cloud processing accuracy.

[0170] For example, with network transmission bandwidth as b n Network transmission latency is l n The minimum resolution is r cs The minimum frame rate is f cs The minimum depth of field is d cs The first compression ratio is ct1 and the maximum latency is l. q1 For example, the second transmission bandwidth b r =r cs *f cs *d cs / ct1, where the second product is r cs *f cs *d csIf, during network transmission, network bandwidth and network latency cannot simultaneously meet the requirements of b... n >=b r ,l n <=l q1 If so, it is determined that the current network transmission cannot meet the requirements for providing cloud AR services.

[0171] The augmented reality processing method described above can determine the target capability. If all network parameters meet the normal quality transmission requirements, the display information is determined as the target capability information. If some network parameters meet the normal quality transmission requirements, the display information is determined as the target capability information, provided that all network parameters after network enhancement meet the normal quality transmission requirements. If all network parameters do not meet the normal quality transmission requirements, but all network parameters meet the basic quality transmission requirements, the service capability information is determined as the target capability information. If all or some network parameters do not meet the basic quality transmission requirements, the service request is not responded to.

[0172] This application embodiment takes real-world accuracy as the standard, integrates terminal screen display clarity, refresh rate performance, network bandwidth, latency, and network performance enhancement capabilities, and is based on the unified cloud AR service quality. It adopts cloud-based cloud AR virtual scene rendering, encoding adjustment, and terminal lighting and shadow adjustment mechanisms to integrate multi-dimensional terminal and network performance to provide high-quality cloud AR services.

[0173] Regarding the implementation of server-side rendering and encoding to obtain virtual scene data, this application proposes a business-network linkage strategy: based on the determined cloud processing accuracy and considering the instability of network transmission and enhancement, corresponding cloud rendering, encoding strategies, and network transmission enhancement strategies are formulated to achieve real-time fusion and enhancement of network capabilities for cloud AR business needs, while optimizing cloud AR business based on the cloud and releasing the advantages of network transmission.

[0174] In the AR virtual scene rendering process, this application embodiment can comprehensively consider the determined cloud processing accuracy and industry network linkage strategy, and flexibly adjust the rendering processing of the virtual scene according to the adjustability of the cloud AR relative application. This may include, but is not limited to, two rendering methods: outputting the virtual scene with the best display accuracy on the terminal and the lowest service accuracy. In the cloud AR virtual scene encoding process, except when bandwidth capacity has redundancy (indicating sufficient bandwidth redundancy) and latency is insufficient, shallow compression is used to reduce cloud processing latency and optimize overall service latency; in other cases, such as when latency meets certain conditions, conventional encoding processing (e.g., normal encoding strategy) can be used.

[0175] In one embodiment, if all parameters in the network parameters meet the requirements for normal quality transmission, step 306 may include:

[0176] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission latency is less than or equal to the recommended latency, the virtual scene data is obtained by rendering according to the display information and encoding according to the first compression ratio.

[0177] Specifically, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission delay is less than or equal to the recommended delay, it can be determined that all parameters in the network parameters meet the normal quality transmission requirements. Then, the display information can be determined as the target capability information, and the rendering can be performed according to the display information. Furthermore, the encoding can be performed according to the first compression ratio (i.e., the normal encoding strategy) to obtain virtual scene data.

[0178] It is understandable that, taking a cloud server as an example, if the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission latency is less than or equal to the recommended latency, it can be determined that the current network transmission bandwidth and network transmission latency can meet the terminal's optimal display accuracy transmission requirements for virtual scenes. Therefore, there is no need to use network enhancement (e.g., calling network bandwidth enhancement, network acceleration, and link optimization capabilities). The terminal's optimal display accuracy requirements for virtual scenes are determined to be the cloud's processing accuracy. That is, when it is clear that no network enhancement is needed, the cloud uses normal terminal virtual scene optimal display accuracy rendering and normal compression ratio (i.e., the compression ratio used by the server for normal encoding).

[0179] In one embodiment, if some parameters in the network parameters meet the requirements for normal quality transmission, step 306 may include:

[0180] When the network transmission latency is less than or equal to the recommended latency, and the network transmission bandwidth is less than the first transmission bandwidth, network enhancement is performed through bandwidth adjustment, and rendering is performed according to the display information and encoding is performed according to the first compression ratio to obtain virtual scene data.

[0181] Specifically, when the network transmission latency is less than or equal to the recommended latency, the latency is considered to meet the requirements by default. Then, the bandwidth situation can be determined. When the network transmission bandwidth is less than the first transmission bandwidth, it is determined that the bandwidth does not meet the requirements. Network enhancement can be performed by bandwidth adjustment (for example, by using network bandwidth expansion, i.e., by calling network capabilities to expand bandwidth) to increase the network bandwidth to meet the requirements. Then, the display information can be determined as the target capability information, and rendered according to the display information and encoded according to the first compression ratio (i.e., normal cloud rendering and encoding) to obtain virtual scene data.

[0182] In some examples, bandwidth adjustment may include increasing the access bandwidth capacity of a terminal and / or increasing the egress bandwidth capacity of a server.

[0183] Specifically, bandwidth adjustment can refer to network enhancement by increasing network bandwidth. This can be achieved by telecom operators increasing their contracted bandwidth or temporarily boosting bandwidth.

[0184] Optionally, bandwidth adjustment may include, but is not limited to, increasing the access bandwidth capacity of the terminal and increasing the egress bandwidth capacity of the server. Specifically, the server can first determine the location of the bandwidth bottleneck and then address it by increasing the corresponding bandwidth capacity. Taking a cloud server as an example, if the bandwidth bottleneck is on the terminal side, it can be resolved by increasing the terminal's access bandwidth capacity; if the bandwidth bottleneck is on the cloud side, it can be resolved by increasing the cloud's egress bandwidth capacity; if bandwidth bottlenecks exist on both the cloud and the terminal, a comprehensive approach of increasing both the terminal's access bandwidth and the cloud's egress bandwidth capacity can be used simultaneously.

[0185] Regarding methods for determining the location of bandwidth bottlenecks, for example, one can compare the bandwidth on the cloud side and the terminal side with the business requirements. Alternatively, one can narrow it down to processing only the terminal side and not submitting to the cloud side. Optionally, if the cloud side is involved, the bandwidth required by a single terminal (user) can be calculated, along with the current terminal usage data for this service, to obtain the corresponding cloud bandwidth requirement. This requirement is then compared with the current actual cloud outbound bandwidth to determine whether the bandwidth requirements are met.

[0186] Furthermore, for cases where some network parameters meet the normal quality transmission requirements, in one embodiment, step 306 may include:

[0187] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and less than the maximum delay corresponding to the augmented reality service, then a delay adjustment strategy is selected based on the network transmission bandwidth to perform network enhancement.

[0188] Based on the displayed information, a processing strategy that is linked with the latency adjustment strategy is used for rendering and encoding to obtain virtual scene data.

[0189] Specifically, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, the server can determine that the bandwidth meets the requirements. If the network transmission latency is greater than the recommended latency but less than the maximum latency corresponding to the augmented reality service, the server defaults to latency not meeting the requirements and can select a latency adjustment strategy for network enhancement based on the network transmission bandwidth, that is, select different processing strategies according to the bandwidth situation. Among these, the displayed information can be determined as the target capability information.

[0190] The latency adjustment strategy can be determined based on whether the network transmission bandwidth has sufficient redundancy (i.e., whether the bandwidth capacity has redundancy). Then, according to the displayed information, a processing strategy linked to the latency adjustment strategy is used for rendering and encoding to obtain virtual scene data. For example, if the bandwidth has sufficient redundancy, the latency adjustment strategy can be determined to first reduce cloud processing latency at the business level, and then optimize and reduce latency at the network transmission level, using high-precision rendering (high-precision rendering can be understood as rendering according to the displayed information) and shallow compression output. If the bandwidth does not have sufficient redundancy, latency is reduced by optimizing the network transmission level, and normal rendering and normal encoding output are used.

[0191] In an exemplary embodiment, the latency adjustment strategy may include network link selection and network link optimization; according to the displayed information, rendering and encoding are performed using a processing strategy linked to the latency adjustment strategy to obtain virtual scene data, including:

[0192] When the network transmission bandwidth is greater than or equal to the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the second compression ratio; wherein, the second compression ratio is less than the first compression ratio.

[0193] When the network transmission bandwidth is less than the redundancy threshold, the virtual scene data is obtained by rendering according to the displayed information and encoding according to the first compression ratio.

[0194] Specifically, a redundancy threshold can be used to determine whether the network transmission bandwidth has sufficient redundancy. When the network transmission bandwidth is greater than or equal to the redundancy threshold, it is determined that the network transmission bandwidth has sufficient redundancy, and thus rendering can be performed according to the display information and encoding according to the second compression ratio to obtain virtual scene data. When the network transmission bandwidth is less than the redundancy threshold, it is determined that the network transmission bandwidth does not have sufficient redundancy, and thus rendering can be performed according to the display information and encoding according to the first compression ratio to obtain virtual scene data. The second compression ratio is less than the first compression ratio.

[0195] Taking a cloud server as an example, if the bandwidth has sufficient redundancy, high-precision cloud rendering combined with shallow compression to reduce the compression ratio is first used to reduce cloud processing latency from the business perspective. Then, network transmission optimization through network acceleration / link optimization is used to further reduce latency. If the bandwidth does not have sufficient redundancy, network transmission acceleration and link optimization are used to reduce network transmission latency, while the cloud renders and encodes normally. In this embodiment, network link selection can refer to application-level network acceleration to effectively shorten actual network transmission latency; alternatively, this embodiment can use network link optimization to effectively shorten actual network transmission latency.

[0196] In one embodiment, the redundancy threshold is a first transmission bandwidth that is a preset multiple; the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0197] Specifically, in this embodiment, the redundancy threshold can be a first transmission bandwidth that is a preset multiple, and the ratio of the first compression ratio to the second compression ratio can be a preset coefficient. The preset multiple and preset coefficient can be customized as needed, and the preset coefficient can be used to characterize the degree of compression ratio reduction.

[0198] In one embodiment, the preset multiple is 2 times; the preset coefficient is the ratio of network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0199] Specifically, the redundancy threshold can be twice the first transmission bandwidth; and the preset coefficient can be the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0200] With network transmission bandwidth as b n The first transmission bandwidth is b b Taking a compression ratio of ct1 and a preset multiplier of 2 as an example, if the current network transmission bandwidth b n >=2b b Therefore, the latency can be reduced by decreasing the cloud encoding compression ratio and reducing bandwidth consumption. The preferred mode is to reduce the latency of cloud virtual scene processing, followed by network acceleration, link optimization and other network latency reduction modes (i.e., network link selection and network link optimization).

[0201] Among them, the degree of compression reduction n (i.e., the preset coefficient) can be determined: based on the relationship between the current network bandwidth and the bandwidth multiple required for the optimal display accuracy of the terminal's virtual scene, n = b n / b b , where n is an integer. Then, compression processing is performed according to the adjusted compression ratio: based on the reduction degree n of the obtained compression ratio, the original normal compression ratio ct1 is reduced by n times to obtain the second compression ratio cr1 = ct1 / n in the implementation process, thereby reducing the complexity of cloud encoding compression and reducing the corresponding latency of cloud processing.

[0202] If the current network bandwidth cannot meet b n <2b b Therefore, it is necessary to reduce transmission latency through network enhancement (i.e., network link selection and network link optimization), which can be achieved through the following methods:

[0203] First, network latency optimization is performed at the application layer (e.g., network link selection). Then, network link optimization is used to further improve latency capabilities. Specifically, at the application layer, several acceleration servers can be used as relays in the intermediate links. By selecting the optimal relay acceleration server, a better network transmission link can be found at the application layer, thereby reducing network transmission latency and achieving network acceleration. For network link optimization, network capabilities can be integrated, and shortest path algorithms or multi-factor routing algorithms can be used to obtain better network paths, thereby reducing the latency of cloud AR virtual scene transmission.

[0204] The above-mentioned augmented reality processing method addresses the problem of insufficient utilization of current network bandwidth. It adopts encoding mechanisms such as lossless or shallow compression based on current network bandwidth conditions, effectively reducing the processing difficulty and latency in the cloud, and giving full play to the advantages of high bandwidth transmission. This ensures high-quality image output for cloud AR while providing a more optimized latency experience.

[0205] If all network parameters meet the basic quality transmission requirements, in one embodiment, rendering and encoding are performed according to the target capability information to obtain virtual scene data, including:

[0206] The virtual scene data is obtained by rendering based on the service capability information and encoding according to the first compression ratio.

[0207] Specifically, if the network transmission bandwidth is greater than or equal to the second transmission bandwidth and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, then it can be determined that all parameters in the network parameters meet the basic quality transmission requirements, and the service capability information can be determined as the target capability information.

[0208] Taking network parameters including bandwidth and latency, display information representing the optimal display precision of the terminal's virtual scene, service capability information representing the minimum service precision, and target capability information representing the cloud processing precision as an example, if the current network bandwidth and latency do not meet the requirements for the optimal display precision of the terminal's virtual scene, but the current network bandwidth and latency can simultaneously meet the network transmission requirements for the minimum service precision output, then the minimum service precision is determined as the cloud processing precision. Subsequently, the cloud adopts virtual scene rendering with the minimum service precision (rendering according to the service capability information) and normal encoding strategy (encoding according to the first compression ratio), and does not perform corresponding network enhancement processing to obtain virtual scene data.

[0209] In summary, once the virtual scene data is obtained, the server can distribute the virtual scene results: after rendering and flexibly encoding / compressing the cloud AR virtual scene in the cloud, the corresponding cloud AR virtual scene processing results are distributed to the cloud AR terminal. The terminal receives the cloud AR virtual scene processing results, adjusts the brightness according to the real-world lighting conditions, completes the cloud AR virtual-real fusion output, and performs corresponding interactive operations based on the virtual-real fusion results.

[0210] The aforementioned augmented reality processing method uses the accuracy of real-scene acquisition as a benchmark and target. Based on the terminal's processing capabilities, it determines the optimal virtual scene output result that the terminal can achieve for user experience (i.e., the optimal display accuracy of the virtual scene on the terminal). With the optimal virtual scene output for user experience on the terminal as the target, the cloud, based on network transmission performance and augmented attributes, combines flexible virtual scene rendering and adaptive encoding to obtain the optimal virtual scene result for user experience on the cloud. The terminal side obtains the virtual scene result and adjusts the brightness in real time based on the real scene, ultimately obtaining the optimal virtual-real fusion result with cloud AR end-to-end optimization guarantee. The integration of cloud, network, and terminal capabilities achieves high-precision seamless connection between cloud AR virtual and real.

[0211] In one exemplary embodiment, such as Figure 5 As shown, an augmented reality processing method is provided, which is applied to... Figure 1 Taking the terminal in the example, the explanation includes the following steps 502 to 504. Wherein:

[0212] Step 502: After acquiring real-world data, send a service request to the cloud AR cloud and send the acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request based on the display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0213] Specifically, a service request can refer to an AR service request; for example, a terminal can initiate a cloud AR service request and collect real-world data; taking a cloud AR terminal as an example, the cloud AR terminal, as the request initiator of the cloud AR service and the final display presenter of the service, can initiate relevant service requests and simultaneously collect and process relevant real-world data.

[0214] Optionally, the terminal can send a service request while acquiring real-world data. For example, the terminal can initiate a service request to the server while acquiring real-world data. Taking a cloud AR service request as an example, the terminal can initiate a cloud AR service request and collect real-world data simultaneously; for instance, the terminal can initiate the relevant service request and collect real-world data at the same time.

[0215] In this process, the server receives a service request from the terminal and determines that virtual scene processing is required. For example, in response to receiving the service request, the server can determine the virtual scene data corresponding to the service request based on the terminal's display information. This display information can represent the terminal's display capability for virtual scene data. For example, the terminal's display information can be obtained by the terminal during the cloud-AR-cloud-network-terminal business collaboration phase.

[0216] For example, the terminal can send the acquired display information to the server, so that the server can use the display information as a reference to determine the cloud-based virtual scene processing; wherein, the server can connect to the terminal and obtain the terminal's display information, such as the optimal display accuracy of the terminal's virtual scene. For example, the terminal can obtain the display information and send the obtained display information to the server in order to achieve optimal virtual-real fusion output.

[0217] In some examples, the displayed information is used to characterize the optimal display accuracy of the virtual scene on the terminal. This optimal display accuracy can be understood as the optimal display accuracy of the virtual scene on the terminal. For example, this optimal display accuracy can refer to the virtual scene output result that the terminal can achieve with the best user experience, determined based on the accuracy of real-scene acquisition and the terminal's processing capabilities. Further, the process by which the server determines the virtual scene data based on the terminal's display information can be understood as the server using the optimal virtual scene output for the terminal's user experience as the goal, performing flexible virtual scene rendering and adaptive encoding in the cloud to obtain the optimal virtual scene result for the cloud user experience. This leverages cloud-network-terminal capabilities to solve the problem of seamless high-precision integration between cloud AR and reality. Based on the embodiments of this application, the optimal virtual-real fusion result with end-to-end optimization guarantee for cloud AR can ultimately be obtained.

[0218] Step 504: Receive virtual scene data sent from the cloud AR cloud.

[0219] Specifically, after the server obtains the virtual scene data, it can send the virtual scene data to the terminal; after the terminal receives the virtual scene data sent by the server, it can output the virtual-real fusion result based on the virtual scene data and the collected real scene data.

[0220] In the above-mentioned augmented reality processing methods, the display capabilities of the terminal are used as a benchmark, and the network transmission conditions enable the cloud to adopt a more flexible virtual scene processing mechanism to achieve high-quality cloud virtual scene processing output, thereby realizing high-quality user experience and high-quality service output of cloud AR end-to-end.

[0221] In one embodiment, the method may further include:

[0222] Based on virtual scene data and real-world data, the output is a virtual-real fusion result.

[0223] Specifically, outputting a virtual-real fusion result can refer to empowering the captured real-world scene with relevant virtual digital capabilities to facilitate the subsequent addition and interaction of other virtual avatars. For example, based on virtual scene data and real-world data, the terminal can overlay the real-world scene as the bottom layer and the virtual scene as the top layer to form an augmented reality image. It should be understood that the embodiments disclosed herein are not limited to this; in other implementations, the terminal can also overlay the real-world scene as the bottom layer and the real-world scene as the top layer.

[0224] Furthermore, taking a cloud server as an example, the cloud server sends the processed virtual scene results (virtual scene data) to the terminal. The terminal receives the virtual scene results and adjusts the lighting (e.g., brightness adjustment) according to the real-world lighting conditions to complete the cloud AR virtual-real fusion output. Based on the virtual-real fusion result, the terminal performs corresponding interactive operations. Alternatively, taking an AR terminal as an example, cloud AR virtual-real fusion output and interaction can be achieved. The AR terminal receives the virtual scene sent from the cloud and, based on the locally acquired real-world scene, adjusts the corresponding highlights to combine with the locally collected real-world scene, completing the virtual-real fusion cloud AR service terminal display output, and performing relevant cloud AR interactions based on the output results.

[0225] In one embodiment, the method may further include:

[0226] The display information is determined based on the screen display parameters and the data acquisition parameters for obtaining real-world data.

[0227] Specifically, the terminal can determine the display information based on the data acquisition parameters of the acquired real-scene data and the screen display parameters, and then transmit the display information to the server. For example, the data acquisition parameters can refer to the relevant parameters of the terminal's real-scene acquisition device, used to characterize the real-scene acquisition accuracy (referred to as real-scene accuracy); for example, the terminal can obtain the real-scene accuracy through the real-scene acquisition device, including but not limited to resolution, frame rate, etc. The real-scene acquisition device can include, but is not limited to, a camera, a light-transmitting lens, etc.

[0228] Furthermore, screen display parameters can represent the terminal's display capabilities (which can be understood as the terminal's display precision), including but not limited to resolution, frame rate, etc. Taking the display information representing the terminal's optimal virtual scene display precision as an example, the terminal can combine the real scene acquisition precision and the terminal's display capabilities to obtain the optimal virtual scene display precision, so as to achieve the best virtual-real fusion output.

[0229] For example, based on the accuracy of the terminal's real-scene acquisition, and combined with the terminal's display capabilities, the optimal display accuracy of the terminal's virtual scene is determined. For instance, the terminal can compare the accuracy of the real-scene acquisition with the terminal's display capabilities to determine the optimal display accuracy of the terminal's virtual scene: ① When the accuracy of the real-scene acquisition is greater than or equal to the terminal's display capabilities, the terminal's display capabilities are determined as the optimal display accuracy of the terminal's virtual scene; ② When the accuracy of the real-scene acquisition is less than the terminal's display capabilities, the real-scene acquisition accuracy is determined as the optimal display accuracy of the terminal's virtual scene. By selecting the accuracy of the real-scene acquisition as the optimal display accuracy of the terminal's virtual scene, it is possible to seamlessly connect with the corresponding real scene and achieve the best virtual-real fusion output.

[0230] In one embodiment, the data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; the screen display parameters include terminal display resolution, terminal display frame rate, and terminal display depth of field.

[0231] The displayed information includes target resolution, target frame rate, and target depth of field; where the target resolution is the minimum of the real-scene capture resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene capture frame rate and the terminal display frame rate; and the target depth of field is the minimum of the real-scene capture depth of field and the terminal display depth of field.

[0232] Specifically, the terminal can acquire data acquisition parameters that characterize the accuracy of real-scene acquisition, as well as screen display parameters that characterize the display capability. The data acquisition parameters may include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; the screen display parameters may include terminal display resolution, terminal display frame rate, and terminal display depth of field. Furthermore, the display information may include target resolution, target frame rate, and target depth of field. The target resolution is the minimum value between the real-scene acquisition resolution and the terminal display resolution, the target frame rate is the minimum value between the real-scene acquisition frame rate and the terminal display frame rate, and the target depth of field is the minimum value between the real-scene acquisition depth of field and the terminal display depth of field.

[0233] For example, the terminal can obtain the relevant resolution, frame rate, depth of field, etc. of the terminal's current real-scene acquisition device as the real-scene acquisition accuracy, such as the resolution of the camera and the frame rate of real-scene shooting; if the real-scene perspective mode is used for acquisition, the real-scene acquisition accuracy can be assumed to be the real-scene accuracy obtained by the human eye; furthermore, the terminal can obtain the screen display capability, that is, obtain the relevant resolution, frame rate, and other indicators of the terminal screen display.

[0234] In practical applications, taking a cloud AR terminal as an example, during the cloud AR cloud-network-end business collaboration phase, the real-scene acquisition accuracy of the cloud AR terminal is obtained. This includes the camera or light-transmitting lens used by the cloud AR terminal to acquire the real scene. The camera directly acquires its pixel resolution, the corresponding frame rate, and the depth of field. The light-transmitting lens is assumed to be the accuracy observed by the human eye. Assuming the real-scene acquisition resolution of the cloud AR terminal is r1 (real-scene acquisition resolution), its frame rate is f1 (real-scene acquisition frame rate), and its depth of field is d1 (real-scene acquisition depth of field), the terminal display accuracy of the cloud AR terminal includes the resolution, frame rate, and depth of field displayed on the cloud AR terminal screen, which are assumed to be r2 (terminal display resolution), f2 (terminal display frame rate), and d2 (terminal display depth of field), respectively.

[0235] By comparing the real-scene capture accuracy and the screen display accuracy of the cloud AR terminal, the optimal display accuracy of the virtual scene on the cloud AR terminal was obtained: r t (target resolution), f t (Target frame rate), d t (Target Depth of Field), the relevant rules are as follows: the optimal display accuracy of the virtual scene on the cloud AR terminal, where the resolution r t =Min(r1,r2), frame rate f t =Min(f1,f2), depth of field d t =Min(d1,d2).

[0236] It is understood that when using perspective mode to acquire real-world data, in one embodiment, the display information is determined based on screen display parameters and data acquisition parameters for acquiring real-world data, including:

[0237] When real-world data is obtained through perspective mode, the screen display parameters are determined as the displayed information.

[0238] Specifically, perspective mode can refer to the terminal using a real-scene perspective mode to obtain real-scene data; it should be noted that real-scene perspective mode includes, but is not limited to, perspective lens mode, such as using a light-transmitting lens to obtain the corresponding real-time scene.

[0239] In cases where the real-scene data is obtained by the terminal through a perspective mode, the terminal can determine the screen display parameters as the displayed information. Taking a cloud AR terminal as an example, if the cloud AR real-scene acquisition uses a perspective lens mode, then by default, the real-scene accuracy of the cloud AR terminal is greater than the terminal's screen display accuracy. That is, in this case, the resolution, frame rate, and depth of field related to the optimal display accuracy of the cloud AR terminal's virtual scene are equal to r², f², and d², respectively. Furthermore, the server can acquire and analyze the optimal display accuracy of the terminal's virtual scene: the server connects to the terminal to obtain the optimal display accuracy of the terminal's virtual scene, and correspondingly obtains its resolution, frame rate, and depth of field, and uses this as a benchmark to determine the cloud-based virtual scene processing.

[0240] It should be noted that for the augmented reality processing methods executed from the terminal's perspective, the steps implemented from the server's perspective can be found in the previous section on augmented reality processing methods executed from the server's perspective, and will not be repeated here.

[0241] In the above augmented reality processing method, based on the accuracy of the corresponding real scene captured by the terminal side, such as the corresponding clarity, frame rate, and depth of field, combined with the display / processing performance of the cloud AR terminal, and considering factors such as the bandwidth and latency of the current network transmission, as well as the enhancement capabilities such as network acceleration, bandwidth expansion, and link optimization, the service quality of cloud AR virtual scene processing is determined. The virtual scene encoding is completed by flexibly rendering the virtual scene in the cloud and flexibly adjusting the encoding compression ratio, and the processing result is sent to the terminal side. The terminal side receives the cloud AR virtual scene result and, in combination with the brightness adjustment of the local virtual scene, finally realizes the localized output of cloud AR virtual-real fusion.

[0242] To further illustrate the solution of this application, a specific example is provided below, taking a cloud AR terminal as the terminal and a cloud AR cloud platform as the server, such as... Figure 6 As shown, the functions of the cloud AR terminal and the cloud AR cloud platform may include, but are not limited to:

[0243] Regarding cloud AR terminals, as the initiator of cloud AR service requests and the final display provider of the service, they can have the following functions:

[0244] ① Real-scene accuracy acquisition (i.e., real-scene accuracy collection): The terminal acquires the real-scene accuracy that the real-scene collection device can obtain, which can include resolution, frame rate and depth of field, etc.

[0245] ② Acquisition of terminal display capabilities: Collect information on its display capabilities, including resolution, frame rate, and depth of field.

[0246] ③ Obtaining the optimal display accuracy of the virtual scene on the terminal: By combining the accuracy of the real scene and the display capabilities of the terminal screen, the optimal display accuracy of the virtual scene on the terminal is obtained, including resolution, frame rate and depth of field.

[0247] When the real-view accuracy is greater than or equal to the terminal screen display capability: select the terminal screen display capability as the optimal display accuracy of the terminal virtual scene;

[0248] When the real-view accuracy is less than the terminal screen display capability: select the real-view accuracy as the optimal display accuracy of the terminal virtual scene so that it can be seamlessly connected with the corresponding real scene and achieve the best virtual-real fusion output.

[0249] ④ Initiate cloud AR service request and real-scene acquisition: The cloud AR terminal initiates the relevant service request and simultaneously acquires and processes the relevant real-scene data.

[0250] ⑤ Virtual-Real Fusion Output and Interaction: The AR terminal receives virtual scene data from the cloud and adjusts the corresponding highlights based on the local real scene data to combine with the locally collected real scene, thus completing the virtual-real fusion cloud AR service terminal display output, and conducting relevant cloud AR interactions based on the output results.

[0251] ⑥ Network performance acquisition and analysis: In conjunction with cloud AR, obtain the bandwidth, latency and other performance data of the corresponding network.

[0252] Regarding cloud-based AR, as the executor of cloud-based AR virtual scene processing, it mainly completes the rendering, compression encoding, and other processing of virtual scenes based on industry network collaboration capabilities, and can have the following functions:

[0253] ① Acquisition and analysis of optimal display accuracy of virtual scene: Connect to the terminal side to obtain the optimal display accuracy of the virtual scene on the terminal, and obtain its resolution, frame rate and depth of field, etc., and use this as a benchmark to determine the processing of virtual scene in the cloud.

[0254] ② Network performance acquisition and analysis: Acquire relevant performance data such as bandwidth and latency of the corresponding network transmission, as well as enhanced capabilities such as network bandwidth improvement, transmission acceleration, and link optimization.

[0255] ③ Determine the minimum service accuracy of the virtual scene: Based on meeting the minimum user experience requirements of cloud AR services, determine the minimum service quality of the service and clarify the minimum resolution, frame rate, and depth of field indicators of the cloud output.

[0256] ④ Determine the cloud processing accuracy: Based on the optimal display accuracy of the virtual scene on the terminal corresponding to cloud AR and the user's better interactive latency experience, determine the corresponding cloud processing accuracy according to different combination methods.

[0257] If the current network transmission bandwidth and latency can meet the transmission requirements of the terminal's optimal virtual scene display accuracy, then there is no need to consider enhancing network bandwidth, network acceleration, and link optimization capabilities. The terminal's optimal virtual scene display accuracy requirement will be determined as the cloud processing accuracy.

[0258] If either the current network transmission bandwidth or latency cannot meet the transmission requirements for the optimal display accuracy of the virtual scene on the terminal, and the transmission requirements are met through network transmission enhancements such as bandwidth increase, network acceleration, and link optimization, then the optimal display accuracy requirement for the virtual scene on the terminal is determined as the cloud processing accuracy.

[0259] When the current network bandwidth and latency cannot simultaneously meet the transmission requirements of the cloud to output the terminal's virtual scene with optimal display accuracy, the cloud processing accuracy can be determined according to different situations: If the current network bandwidth and latency do not meet the minimum service accuracy of the cloud virtual scene, then cloud AR services that guarantee user experience cannot be provided. If the current network bandwidth and latency meet the minimum service accuracy of the cloud virtual scene, then the minimum service accuracy is determined as the cloud processing accuracy.

[0260] ⑤ Determine the industry-network linkage strategy: Based on the determined cloud processing accuracy, and considering the instability of network transmission and enhancement, formulate corresponding cloud rendering, encoding strategies and network transmission enhancement strategies.

[0261] When the current network transmission bandwidth and latency can meet the transmission requirements of the terminal's optimal virtual scene display accuracy, the cloud performs graphics rendering and encoding according to the normal cloud processing accuracy, that is, the terminal's optimal virtual scene display accuracy.

[0262] When either the network transmission bandwidth or latency of the current network cannot meet the transmission requirements for the optimal display accuracy of the virtual scene on the terminal, the following processing can be performed depending on the situation: the cloud can render and encode according to the determined cloud processing accuracy (i.e., the optimal display accuracy of the virtual scene on the terminal):

[0263] When the bandwidth does not meet the requirements, the bandwidth can be expanded by utilizing network capabilities.

[0264] When latency does not meet the requirements, if bandwidth is sufficient (i.e., bandwidth has sufficient redundancy), the cloud will output more than twice the bandwidth required for the terminal's best virtual scene display precision. First, the cloud will render at the highest precision (i.e., render at the terminal's best virtual scene display precision) and adopt a shallow compression mode to shorten compression latency. At the same time, network acceleration or link optimization will reduce network transmission adjustments and improve the corresponding transmission latency. If bandwidth is insufficient (i.e., bandwidth does not have sufficient redundancy), the cloud will output more than twice the bandwidth required for the terminal's best virtual scene display precision and improve the corresponding transmission latency by reducing network transmission adjustments through network acceleration or link optimization.

[0265] If the current network transmission bandwidth and latency cannot simultaneously meet the transmission requirements of the cloud to output the terminal virtual scene with the best display accuracy, but the current network transmission bandwidth and latency can simultaneously meet the network transmission requirements of the cloud virtual scene to output the minimum service accuracy, then a strategy of rendering and normal encoding processing with the minimum service accuracy through the cloud based on the current network transmission is adopted.

[0266] ⑥ Cloud AR Virtual Scene Rendering: Based on the determined cloud processing precision and business linkage strategies, and considering the adjustability of cloud AR applications, the rendering process of the virtual scene is flexibly adjusted. This includes two rendering methods: outputting the terminal's virtual scene with the best display precision and the minimum service precision. The optimal display precision output corresponds to a situation where the current network's bandwidth and latency fully or partially meet the cloud's transmission requirements for outputting the terminal's virtual scene with the best display precision. The minimum service precision corresponds to a situation where the current network's bandwidth and latency simultaneously meet the network transmission requirements for the cloud's minimum service precision output of the virtual scene.

[0267] ⑦ Cloud AR Virtual Scene Encoding: Except for cases where bandwidth capacity is redundant and latency is insufficient, shallow compression can be used to reduce cloud processing latency and optimize overall service latency; in other cases, conventional encoding can be used if latency meets certain conditions.

[0268] ⑧ Virtual scene result delivery: After rendering, flexibly encoding and compressing the cloud AR virtual scene in the cloud, the corresponding cloud AR virtual scene processing results are delivered to the cloud AR terminal side.

[0269] The aforementioned augmented reality processing method is guided by the optimal display capabilities of the cloud AR terminal, and enhances the quality of cloud AR virtual scene processing in multiple dimensions from the cloud network side. Based on the accuracy of real-scene acquisition on the terminal side, and combined with the terminal's display capabilities, the optimal display accuracy of the virtual scene on the terminal is determined. Based on this optimal display accuracy, and considering the current network bandwidth, latency transmission performance, and enhancements such as network acceleration and link optimization, the cloud-based virtual scene processing accuracy and industry-network collaboration strategy are obtained. Based on these strategies, the cloud-based virtual scene processing accuracy and industry-network collaboration strategy are used to complete the graphics rendering, encoding compression adjustment, and network transmission enhancement of the cloud virtual scene. After receiving the downloaded virtual scene, the terminal side performs virtual scene brightness processing based on the actual acquired lighting conditions, ultimately achieving high-quality cloud AR virtual-real fusion output, effectively improving the user experience quality.

[0270] Furthermore, the implementation process of this application embodiment may include two stages: cloud AR cloud-network-device service collaboration and cloud AR service operation; wherein, the cloud AR cloud-network-device service collaboration stage serves as a preparatory stage for the cloud AR service operation stage, and the specific process can be as follows: Figure 7 As shown, the process of cloud AR service operation can be as follows: Figure 8 As shown.

[0271] like Figure 7 As shown, the cloud-AR cloud-network-device business collaboration phase may include:

[0272] 1) The accuracy of real-scene acquisition and the display capability of the terminal screen are obtained on the terminal side.

[0273] 1.1 The terminal obtains the real-scene acquisition accuracy by acquiring the relevant resolution, frame rate, depth of field, etc. of the terminal's current real-scene acquisition device, such as the resolution of the camera and the frame rate of the real-scene shooting; if the real-scene perspective mode is used for acquisition, its accuracy is assumed to be the real-scene accuracy obtained by the human eye.

[0274] 1.2 The terminal obtains the terminal's screen display capabilities, that is, it obtains the relevant indicators such as the terminal's screen display resolution and frame rate;

[0275] 2) Compare the real-scene acquisition accuracy and display capability of the terminal side, and determine the optimal display accuracy of the virtual scene on the terminal.

[0276] 2.1 When the real-scene acquisition accuracy is greater than or equal to the screen display capability, the terminal screen display capability is determined as the optimal display accuracy of the terminal virtual scene;

[0277] 2.2 When the real-scene acquisition accuracy is less than the screen display capability, the real-scene acquisition accuracy is determined as the optimal display accuracy of the virtual scene on the terminal.

[0278] 3) The cloud simultaneously determines the minimum service precision of the virtual scene, obtains and analyzes the optimal display precision of the virtual scene, and coordinates the cloud and terminal to obtain network transmission performance.

[0279] 3.1 Determine the minimum service precision for virtual scenes: Combine the optimal display precision of virtual scenes on the terminal with the precision range of virtual scene processing in cloud AR services to determine the minimum service precision for virtual scenes.

[0280] 3.2 Obtain and analyze the optimal display precision of the virtual scene: Obtain the optimal display precision of the virtual scene sent by the terminal, and decompose it into resolution, frame rate, depth of field, etc. Simultaneously calculate the required bandwidth based on the compression rate corresponding to commonly used encoding protocols.

[0281] 3.3 Cloud and terminal linkage to obtain network transmission performance and set latency requirements: By connecting the terminal and the cloud network, obtain the corresponding network transmission bandwidth and latency between the two, and set the corresponding latency requirements according to the user experience characteristics of cloud AR services.

[0282] 4) Based on the current network transmission performance, determine whether the current network transmission bandwidth and latency meet the corresponding cloud processing requirements, so that the cloud can determine the corresponding processing accuracy.

[0283] 4.1 If the current network meets the optimal display accuracy requirements of the terminal virtual scene, then the optimal display accuracy of the terminal virtual scene shall be determined as the cloud processing accuracy.

[0284] 4.2 If either the current network bandwidth or latency is not met, but the optimal display accuracy requirement for the terminal's virtual scene can be met after network enhancement, then the optimal display accuracy of the terminal's virtual scene shall be determined as the cloud processing accuracy, and relevant business network capabilities shall be required for coordinated processing.

[0285] 4.3 If the current network bandwidth and latency do not meet the optimal display accuracy requirements of the terminal virtual scene, but the current network can meet the minimum service accuracy requirements, then the minimum service accuracy will be determined as the cloud processing accuracy.

[0286] 4.4 If the current network does not meet the minimum service accuracy transmission requirements, the cloud AR-related services will be deemed unavailable and the process will terminate.

[0287] 5) Based on the predetermined cloud processing precision, determine the corresponding cloud processing and business linkage strategies according to different situations:

[0288] Version 5.1 corresponds to version 4.1, explicitly stating that no network enhancement is required, and the cloud uses normal terminal virtual scene rendering with optimal display precision and normal compression ratio encoding.

[0289] Section 5.2 corresponds to section 4.2. First, determine the bandwidth situation and then optimize the relevant latency aspects based on different bandwidth conditions.

[0290] 5.2.1 If the bandwidth does not meet the requirements (the default latency meets the requirements), then the network bandwidth expansion (i.e., network bandwidth increase) will be used to meet the requirements, and the cloud rendering and encoding will proceed normally.

[0291] 5.2.2 If the bandwidth meets the requirements but the default latency does not, different processing strategies will be selected based on the bandwidth situation.

[0292] 5.2.2.1 If the bandwidth has sufficient redundancy, that is, to meet the requirement of transmitting to the cloud at more than twice the optimal display precision of the terminal virtual scene, then first use cloud high-precision rendering (referred to as high-precision rendering) + shallow compression with reduced compression rate to reduce cloud processing latency from the business level, and then use network transmission optimization through network acceleration / link selection to reduce latency.

[0293] 5.2.2.2 If the bandwidth is not sufficiently redundant, network transmission acceleration and link optimization methods are used to reduce network transmission latency, while cloud rendering and encoding output remain normal.

[0294] Version 5.3 corresponds to version 4.3. It adopts a strategy of rendering virtual scenes with the lowest service precision and normal encoding, without performing corresponding network enhancement processing.

[0295] 6. Determine the corresponding cloud processing accuracy, cloud processing, and business linkage strategies based on the above conditions, thereby completing the corresponding cloud AR cloud network terminal business collaboration processes.

[0296] Furthermore, the cloud AR service operation process starts with the cloud AR terminal initiating a service request and ends with the cloud AR terminal achieving virtual-real fusion output and interaction. This is based on the premise that the optimal display accuracy of the virtual scene is determined on the terminal, the processing accuracy is determined in the cloud, and relevant business network linkage strategies are followed. In other words, it is achieved after the cloud AR cloud network terminal business collaboration processing flow is completed.

[0297] like Figure 8 As shown, the operational phases of cloud AR services may include:

[0298] 1) Terminal initiates cloud AR service request and real-scene acquisition: The terminal initiates cloud AR service request to the cloud and simultaneously obtains the corresponding real-time scene through camera, light-transmitting lens, etc.

[0299] 2) In accordance with the cloud AR cloud network terminal business collaboration stage processing flow, the cloud processing accuracy is clearly defined, and the business network linkage strategy is used to complete the virtual scene rendering.

[0300] 3) The cloud performs coding and network enhancement processing according to the clearly defined business network linkage strategy in the cloud AR cloud network terminal business collaboration stage.

[0301] 4) The cloud will send the processed virtual scene results to the terminal.

[0302] 5) The terminal receives the virtual scene result and adjusts the brightness accordingly based on the real scene lighting conditions to complete the cloud AR virtual-real fusion output, and performs corresponding interactive operations based on the virtual-real fusion result.

[0303] The aforementioned augmented reality processing method uses the resolution, frame rate, and depth of field accuracy of the terminal's real-scene acquisition as a benchmark to clarify the real-scene acquisition accuracy. Simultaneously, it combines the terminal's screen display accuracy, such as resolution and refresh rate, to obtain the optimal display accuracy of the virtual scene that the terminal can process. The cloud, aiming to achieve the optimal display of the virtual scene that the terminal can process, integrates factors such as network transmission bandwidth and latency, and appropriately considers enhanced processing capabilities such as network transmission acceleration and link optimization, as well as the minimum service accuracy of the cloud AR service's real-scene, to determine the cloud processing accuracy and industry-network linkage strategy. Then, based on the cloud processing accuracy and industry-network linkage strategy, it completes the cloud-based graphics rendering and adaptively adjusts the compression ratio of the virtual scene encoding method to obtain the result of the cloud-output virtual scene. Simultaneously, according to the industry-network linkage strategy, it enhances network transmission by expanding network bandwidth, accelerating network applications, and optimizing links for the virtual scene distribution. After obtaining the corresponding virtual scene result, the terminal adjusts the brightness and other aspects of the virtual scene according to the real-time lighting and shadow conditions of the real scene, ultimately completing the corresponding virtual-real fusion output.

[0304] This application aims to enhance the user's cloud AR experience and bridge the gap between virtual and real-world scenes in cloud AR. It employs a multi-dimensional cloud-network-device capability fusion approach, focusing on the final virtual-real fusion effect to construct a high-quality end-to-end cloud AR solution. Through cloud-network-device capability fusion, it ensures high-quality cloud-based virtual scene output. This application uses optimal display accuracy on the terminal side as a benchmark to construct the processing accuracy of the cloud virtual scene, incorporating network transmission conditions and appropriate enhancements when insufficient. It also incorporates methods such as encoding compression ratio optimization in high-bandwidth environments to improve the processing effect of the cloud virtual scene. This application strengthens network capability supply and enhances the service value of typical cloud AR services. By integrating network bandwidth and latency capabilities into cloud AR service optimization, it can more efficiently reflect network value, improve network capability supply, and unlock the potential of new markets and spaces for network services. Furthermore, based on the industry-network linkage strategy, this application can improve the latency experience of cloud AR services in multiple dimensions: on the one hand, it enhances network acceleration at the network transmission application level and improves transmission latency by optimizing network links; on the other hand, it integrates multiple industry-network linkage mechanisms based on high-bandwidth encoding compression rate adjustment during cloud processing to reduce cloud processing time, thereby optimizing cloud AR virtual scene processing and network transmission latency end-to-end and effectively improving the interactive experience of cloud AR services.

[0305] Furthermore, this application is guided by the optimal display accuracy of the virtual scene on the cloud AR terminal, and determines the processing accuracy of the virtual scene in the cloud by combining the current network transmission performance and network capability enhancement attributes. In order to meet the output accuracy of cloud processing, a multi-dimensional collaborative processing mechanism of cloud network terminal services (i.e., business network linkage strategy) is adopted. Based on flexible rendering, encoding processing and network capability enhancement linkage, high-quality output of cloud AR virtual scene is achieved, and the best integration output with the real scene is achieved.

[0306] This application provides a multi-dimensional coordination mechanism for cloud-network-device services, and an end-to-end high-quality cloud AR virtual scene processing mechanism: the optimal display accuracy of the virtual scene on the terminal is determined by the accuracy of the terminal's real-scene acquisition and the terminal's screen display capabilities; the network bandwidth requirements are calculated in the cloud based on the optimal display accuracy of the virtual scene on the terminal, and the cloud virtual scene processing accuracy and cloud-network-device service collaboration mechanism are determined in combination with the current network transmission performance; based on the cloud-network-device service system mechanism, the high-quality output of the cloud virtual scene is ensured through a flexible cloud encoding and enhanced network transmission fusion mode.

[0307] Furthermore, this application achieves the linkage of business and network capabilities, improving the cloud AR virtual scene processing effect: integrating business and network capabilities into the virtual scene enhancement processing process, reducing the encoding compression ratio according to bandwidth redundancy, and reducing cloud processing latency; enhancing network transmission performance through network bandwidth expansion, application-level network acceleration, and network link optimization, maximizing the satisfaction of higher quality output.

[0308] In addition, this application completes end-to-end capability linkage to improve the virtual-real fusion effect of cloud AR: the terminal determines the optimal display accuracy of the virtual scene based on the accuracy of real scene acquisition and the display capability of the terminal screen; the cloud leverages business and network capabilities to improve the processing effect of the virtual scene in order to achieve the best final display accuracy output of the virtual scene; the terminal adjusts the brightness of the virtual scene sent from the cloud based on the actual light and shadow conditions to maximize the high-quality fusion output of cloud AR real scene and virtual scene.

[0309] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0310] Based on the same inventive concept, this application also provides an augmented reality processing apparatus for implementing the augmented reality processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more augmented reality processing apparatus embodiments provided below can be found in the limitations of the augmented reality processing method described above, and will not be repeated here.

[0311] In one exemplary embodiment, such as Figure 9 As shown, an augmented reality processing device is provided for virtual scene rendering in cloud AR, and the device includes:

[0312] The request receiving module 901 is used to receive service requests sent by the terminal, which are sent by the terminal when acquiring real-scene data;

[0313] The virtual data determination module 902 is used to acquire the display information of the terminal and determine the virtual scene data corresponding to the service request based on the display information of the terminal and the network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0314] The data sending module 903 is used to send virtual scene data to the terminal.

[0315] In one embodiment, the display information is determined based on the terminal's screen display parameters and the data acquisition parameters for acquiring real-scene data. The data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field. The screen display parameters include the terminal display resolution, terminal display frame rate, and terminal display depth of field. The display information includes a target resolution, a target frame rate, and a target depth of field. The target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field. When the real-scene data is obtained by the terminal through a perspective mode, the display information is the screen display parameters.

[0316] In one embodiment, the virtual data determination module 902 includes:

[0317] The parameter acquisition module is used to acquire network parameters for communication between the current network and the terminal through the virtual scene rendering function; the network parameters are parameters that characterize the network transmission performance of the current network.

[0318] The target information determination module is used to determine the display information or service capability information as target capability information based on whether the network parameters meet the normal quality transmission requirements required for display information and whether the network parameters meet the basic quality transmission requirements required for service capability information; wherein, the basic quality transmission requirements are lower than the normal quality transmission requirements; the service capability information represents the minimum capability of the virtual scene rendering function to support augmented reality services;

[0319] The rendering and encoding module is used to render and encode according to the target capability information to obtain virtual scene data.

[0320] In one embodiment, the target information determination module is configured to: if all parameters in the network parameters meet the normal quality transmission requirements, determine the displayed information as target capability information; if some parameters in the network parameters meet the normal quality transmission requirements, determine the displayed information as target capability information if it is confirmed that all parameters in the network parameters after network enhancement can meet the normal quality transmission requirements; if all parameters in the network parameters do not meet the normal quality transmission requirements, determine the service capability information as target capability information if all parameters in the network parameters meet the basic quality transmission requirements; and if all or some parameters in the network parameters do not meet the basic quality transmission requirements, not respond to service requests.

[0321] In one embodiment, network parameters include network transmission bandwidth and network transmission latency; display information includes target resolution, target frame rate, and target depth of field; wherein:

[0322] If the network transmission bandwidth is greater than or equal to the first transmission bandwidth, then the network transmission bandwidth is confirmed to meet the normal quality transmission requirements.

[0323] If the network transmission latency is less than or equal to the recommended latency for augmented reality services, then the network transmission latency is confirmed to meet the normal quality transmission requirements.

[0324] Wherein, the first transmission bandwidth is the quotient of the first product and the first compression ratio; the first product is the product of the target resolution, the target frame rate and the target depth of field; the first compression ratio represents the compression ratio of the server performing normal encoding.

[0325] In one embodiment, the rendering encoding module is used to render according to the display information and encode according to the first compression ratio to obtain virtual scene data when the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission latency is less than or equal to the recommended latency.

[0326] In one embodiment, the rendering encoding module is used to perform network enhancement by bandwidth adjustment when the network transmission latency is less than or equal to the recommended latency, and if the network transmission bandwidth is less than the first transmission bandwidth, and to render according to the display information and encode according to the first compression ratio to obtain virtual scene data.

[0327] In one embodiment, bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

[0328] In one embodiment, the rendering and encoding module is configured to perform network enhancement based on a latency adjustment strategy when the network transmission bandwidth is greater than or equal to a first transmission bandwidth, the network transmission latency is greater than a recommended latency, and the network transmission latency is less than the maximum latency corresponding to the augmented reality service; and to perform rendering and encoding according to the displayed information using a processing strategy linked to the latency adjustment strategy to obtain virtual scene data.

[0329] In one embodiment, the latency adjustment strategy includes network link selection and network link optimization;

[0330] The rendering and encoding module is also used to render according to the display information and encode according to the second compression ratio to obtain virtual scene data when the network transmission bandwidth is greater than or equal to the redundancy threshold; wherein the second compression ratio is less than the first compression ratio; and to render according to the display information and encode according to the first compression ratio to obtain virtual scene data when the network transmission bandwidth is less than the redundancy threshold.

[0331] In one embodiment, the redundancy threshold is a first transmission bandwidth that is a preset multiple; the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0332] In one embodiment, the preset multiple is 2 times; the preset coefficient is the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0333] In one embodiment, network parameters include network transmission bandwidth and network transmission latency; service capability information includes minimum resolution, minimum frame rate, and minimum depth of field; wherein:

[0334] If the network transmission bandwidth is greater than or equal to the second transmission bandwidth and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, then it is confirmed that all parameters in the network parameters meet the basic quality transmission requirements.

[0335] The second transmission bandwidth is the quotient of the second product and the first compression ratio; the second product is the product of the minimum resolution, the minimum frame rate, and the minimum depth of field; the first compression ratio represents the compression ratio used by the server for normal encoding.

[0336] In one embodiment, the rendering encoding module is used to render according to service capability information and encode according to a first compression ratio to obtain virtual scene data.

[0337] In one exemplary embodiment, such as Figure 10 As shown, an augmented reality processing device is provided for use in a terminal. The device includes:

[0338] The information sending module 1001 is used to send a service request to the cloud AR cloud when real-scene data is acquired, and to send the acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request based on the display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data;

[0339] The data receiving module 1002 is used to receive virtual scene data sent from the cloud AR cloud.

[0340] In one embodiment, the device further includes:

[0341] The display information determination module is used to determine display information based on screen display parameters and data acquisition parameters for obtaining real-scene data. Data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field. Screen display parameters include terminal display resolution, terminal display frame rate, and terminal display depth of field. Display information includes target resolution, target frame rate, and target depth of field. The target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field. When real-scene data is obtained through perspective mode, the screen display parameters are determined as the display information.

[0342] In one embodiment, the device further includes:

[0343] The virtual-real fusion output module is used to output virtual-real fusion results based on virtual scene data and real scene data.

[0344] Each module in the aforementioned augmented reality processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0345] In one embodiment, a communication device is provided, see [link to previous document]. Figure 11 , Figure 11 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application. The communication device can be a server, which may include a receiver 31, a memory 32, a processor 33, at least one communication bus 34, and a transmitter 35. The communication bus 34 is used to realize communication connections between components. The memory 32 may include high-speed RAM memory, and may also include non-volatile memory (NVM), such as at least one disk storage device. The memory 32 can store various programs for performing various processing functions and implementing the method steps of this embodiment. In this embodiment, the transmitter 35 can be a radio frequency processing module or a baseband processing module in the server, and the receiver 31 can also be a radio frequency processing module or a baseband processing module in the server. The transmitter 35 and the receiver 31 can be integrated together to form a transceiver. Both the transmitter 35 and the receiver 31 can be coupled to the processor 33, and can perform receiving or transmitting actions under the instruction or control of the processor 33.

[0346] In this embodiment, receiver 31 is used to receive service requests sent by the terminal, which are sent by the terminal when acquiring real-scene data;

[0347] The processor 33 is used to acquire the display information of the terminal and determine the virtual scene data corresponding to the service request based on the display information of the terminal and the network transmission conditions; wherein, the display information represents the terminal's display capability for the virtual scene data;

[0348] Transmitter 35 is used to send virtual scene data to the terminal.

[0349] In one embodiment, the display information is determined based on the terminal's screen display parameters and the data acquisition parameters for acquiring real-scene data. The data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field. The screen display parameters include the terminal display resolution, terminal display frame rate, and terminal display depth of field. The display information includes a target resolution, a target frame rate, and a target depth of field. The target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field. Where the real-scene data is obtained by the terminal through perspective mode, the display information is the screen display parameters.

[0350] In one embodiment, the processor 33 is specifically configured to obtain network parameters for communicating with the terminal using the current network through the virtual scene rendering function; the network parameters are parameters characterizing the network transmission performance of the current network; based on whether the network parameters meet the normal quality transmission requirements required for displaying information and whether the network parameters meet the basic quality transmission requirements required for service capability information, the display information or service capability information is determined as target capability information; wherein, the basic quality transmission requirements are lower than the normal quality transmission requirements; the service capability information represents the minimum capability that the virtual scene rendering function supports for augmented reality services; and render and encode according to the target capability information to obtain virtual scene data.

[0351] In one embodiment, the processor 33 is specifically configured to: if all parameters in the network parameters meet the normal quality transmission requirements, determine the displayed information as target capability information; if some parameters in the network parameters meet the normal quality transmission requirements, determine the displayed information as target capability information after confirming that all parameters in the network parameters after network enhancement can meet the normal quality transmission requirements; if all parameters in the network parameters do not meet the normal quality transmission requirements, determine the service capability information as target capability information if all parameters in the network parameters meet the basic quality transmission requirements; and if all or some parameters in the network parameters do not meet the basic quality transmission requirements, not respond to service requests.

[0352] In one embodiment, network parameters include network transmission bandwidth and network transmission latency; display information includes target resolution, target frame rate, and target depth of field; processor 33 is specifically configured to confirm that the network transmission bandwidth meets the normal quality transmission requirements when the network transmission bandwidth is greater than or equal to a first transmission bandwidth; and to confirm that the network transmission latency meets the normal quality transmission requirements when the network transmission latency is less than or equal to the recommended latency corresponding to the augmented reality service; wherein, the first transmission bandwidth is the quotient of a first product and a first compression ratio; the first product is the product of the target resolution, target frame rate, and target depth of field; and the first compression ratio represents the compression ratio used by the server for normal encoding.

[0353] In one embodiment, the processor 33 is specifically configured to render according to the display information and encode according to the first compression ratio to obtain virtual scene data when the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission latency is less than or equal to the recommended latency.

[0354] In one embodiment, the processor 33 is specifically configured to perform network enhancement by bandwidth adjustment when the network transmission latency is less than or equal to the recommended latency and the network transmission bandwidth is less than the first transmission bandwidth, and to render according to the display information and encode according to the first compression ratio to obtain virtual scene data.

[0355] In one embodiment, bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

[0356] In one embodiment, the processor 33 is specifically configured to perform network enhancement based on a latency adjustment strategy when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, the network transmission latency is greater than the recommended latency, and the network transmission latency is less than the maximum latency corresponding to the augmented reality service; and to perform rendering and encoding according to the displayed information using a processing strategy linked to the latency adjustment strategy to obtain virtual scene data.

[0357] In one embodiment, the latency adjustment strategy includes network link selection and network link optimization; the processor 33 is specifically used to render according to the display information and encode according to the second compression ratio to obtain virtual scene data when the network transmission bandwidth is greater than or equal to the redundancy threshold; wherein, the second compression ratio is less than the first compression ratio; when the network transmission bandwidth is less than the redundancy threshold, it renders according to the display information and encodes according to the first compression ratio to obtain virtual scene data.

[0358] In one embodiment, the redundancy threshold is a first transmission bandwidth that is a preset multiple; the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0359] In one embodiment, the preset multiple is 2 times; the preset coefficient is the ratio of network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0360] In one embodiment, network parameters include network transmission bandwidth and network transmission latency; service capability information includes minimum resolution, minimum frame rate, and minimum depth of field; processor 33 is specifically configured to confirm that all parameters in the network parameters meet the basic quality transmission requirements when the network transmission bandwidth is greater than or equal to the second transmission bandwidth and the network transmission latency is less than or equal to the maximum latency corresponding to the augmented reality service; the second transmission bandwidth is the quotient of the second product and the first compression ratio; the second product is the product of the minimum resolution, minimum frame rate, and minimum depth of field; the first compression ratio represents the compression ratio of the server performing normal encoding.

[0361] In one embodiment, processor 33 is specifically configured to render according to service capability information and encode according to a first compression ratio to obtain virtual scene data.

[0362] In one embodiment, a communication device is provided, which may be a terminal device (hereinafter referred to as a terminal); see also Figure 12 , Figure 12 This is a schematic diagram of the structure of the terminal device provided in an embodiment of the present invention. Figure 12 The terminal device 700 shown includes at least one processor 701, a memory 702, at least one network interface 704, and a user interface 703. The various components in the terminal device 700 are coupled together via a bus system 705. It is understood that the bus system 705 is used to implement communication between these components. In addition to a data bus, the bus system 705 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 12 Various buses are designated as bus system 705. Additionally, this embodiment of the invention includes a transceiver 706, which may consist of multiple components, including a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium.

[0363] The user interface 703 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0364] It is understood that the memory 702 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 702 of the systems and methods described in this embodiment is intended to include, but is not limited to, these and any other suitable types of memory.

[0365] In some implementations, memory 702 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 7021 and application program 7022.

[0366] The operating system 7021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 7022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 7022.

[0367] In this embodiment of the invention, by calling the program or instructions stored in the memory 702, specifically the program or instructions stored in the application program 7022, the transmitter is used to send a service request to the cloud AR cloud when real-world data is acquired, and to send the acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request based on the display information and network transmission conditions; wherein, the display information represents the terminal's display capability for virtual scene data; the receiver is used to receive the virtual scene data sent by the cloud AR cloud.

[0368] The methods disclosed in the above embodiments of the present invention, in part or in all of them, can also be applied to processor 701, implemented by processor 701, or implemented by processor 701 in conjunction with other components (e.g., transceivers). Processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above methods can be completed by the integrated logic circuit in the hardware of processor 701 or by instructions in the form of software. The processor 701 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 702, and processor 701 reads the information from memory 702 and, in conjunction with its hardware, completes the steps of the above method.

[0369] It is understood that the embodiments described in this invention can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in this application, or combinations thereof.

[0370] For software implementation, the technology described in the embodiments of the present invention can be implemented by modules (e.g., procedures, functions, etc.) that perform the functions described in the embodiments of the present invention. The software code can be stored in memory and executed by processor 701. The memory can be implemented in processor 701 or external to processor 701.

[0371] In one embodiment, the processor is configured to determine display information based on data acquisition parameters for acquiring real-scene data and screen display parameters. The data acquisition parameters include real-scene acquisition resolution, real-scene acquisition frame rate, and real-scene acquisition depth of field; the screen display parameters include terminal display resolution, terminal display frame rate, and terminal display depth of field; the display information includes target resolution, target frame rate, and target depth of field; wherein the target resolution is the minimum of the real-scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum of the real-scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum of the real-scene acquisition depth of field and the terminal display depth of field.

[0372] In one embodiment, the processor is further configured to determine the screen display parameters as display information when real-world data is obtained through perspective mode.

[0373] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0374] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0375] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0376] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0377] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0378] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An augmented reality processing method characterized by, The method applied to a virtual scene rendering function of a cloud AR cloud end comprises: receiving a service request sent by a terminal, the service request being sent by the terminal in a case of obtaining real scene data; obtaining display information of the terminal, and determining virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information represents display capability of the terminal for virtual scene data; sending the virtual scene data to the terminal.

2. The method of claim 1, wherein: the display information is determined based on screen display parameters of the terminal and data acquisition parameters of the terminal for obtaining the real scene data; the data acquisition parameters comprise real scene acquisition resolution, real scene acquisition frame rate and real scene acquisition depth of field; the screen display parameters comprise terminal display resolution, terminal display frame rate and terminal display depth of field; the display information comprises target resolution, target frame rate and target depth of field; the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field; in a case that the real scene data is obtained by the terminal in a see-through mode, the display information is the screen display parameters.

3. The method of claim 1, wherein, determining the virtual scene data corresponding to the service request according to the display information of the terminal and the network transmission conditions comprises: obtaining network parameters for communication between the terminal and the current network by the virtual scene rendering function; the network parameters are parameters representing network transmission performance of the current network; determining the display information or the service capability information as target capability information according to whether the network parameters meet normal quality transmission requirements required by the display information and whether the network parameters meet basic quality transmission requirements required by the service capability information; wherein the basic quality transmission requirements are lower than the normal quality transmission requirements; the service capability information represents the minimum capability of the virtual scene rendering function supporting augmented reality services; rendering and encoding according to the target capability information to obtain the virtual scene data.

4. The method of claim 3, wherein, determining the display information or the service capability information as target capability information according to whether the network parameters meet normal quality transmission requirements required by the display information and whether the network parameters meet basic quality transmission requirements required by the service capability information comprises: if all the network parameters meet the normal quality transmission requirements, determining the display information as the target capability information; when part of the network parameters meet the normal quality transmission requirements, determining the display information as the target capability information in a case that all the network parameters of the current network after network enhancement meet the normal quality transmission requirements. If all the network parameters do not meet the normal quality transmission requirement, if all or part of the network parameters meet the basic quality transmission requirement, the service capability information is determined as the target capability information; If all or part of the network parameters do not meet the basic quality transmission requirement, the service request is not responded.

5. The method of claim 4, wherein, The network parameters include network transmission bandwidth and network transmission delay; the display information includes target resolution, target frame rate and target depth of field; wherein: When the network transmission bandwidth is greater than or equal to a first transmission bandwidth, it is confirmed that the network transmission bandwidth meets the normal quality transmission requirement; When the network transmission delay is less than or equal to the recommended delay corresponding to the augmented reality service, it is confirmed that the network transmission delay meets the normal quality transmission requirement; The first transmission bandwidth is a quotient of a first product and a first compression ratio; the first product is a product of the target resolution, the target frame rate and the target depth of field; and the first compression ratio represents a compression ratio of normal encoding of the server.

6. The method of claim 5, wherein, The rendering and encoding according to the target capability information to obtain the virtual scene data, comprising: When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, and the network transmission delay is less than or equal to the recommended delay, the rendering is performed according to the display information, and the encoding is performed according to the first compression ratio to obtain the virtual scene data.

7. The method of claim 5, wherein, The rendering and encoding according to the target capability information to obtain the virtual scene data, comprising: When the network transmission delay is less than or equal to the recommended delay, if the network transmission bandwidth is less than the first transmission bandwidth, network enhancement is performed through bandwidth adjustment, and the rendering is performed according to the display information and the encoding is performed according to the first compression ratio to obtain the virtual scene data.

8. The method of claim 7, wherein, The bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

9. The method of claim 5, wherein, The rendering and encoding according to the target capability information to obtain the virtual scene data, comprising: When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and the network transmission delay is less than the maximum delay corresponding to the augmented reality service, network enhancement is performed based on the network transmission bandwidth and a delay adjustment strategy is selected; The rendering and encoding are performed according to the display information and a processing strategy linked with the delay adjustment strategy to obtain the virtual scene data.

10. The method of claim 9, wherein, The delay adjustment strategy includes network link selection and network link optimization; The rendering and encoding according to the display information and the processing strategy linked with the delay adjustment strategy to obtain the virtual scene data, comprising: When the network transmission bandwidth is greater than or equal to a redundancy threshold, the rendering is performed according to the display information and the encoding is performed according to a second compression ratio to obtain the virtual scene data; wherein the second compression ratio is less than the first compression ratio; When the network transmission bandwidth is less than the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the first compression ratio.

11. The method of claim 10, wherein, The redundancy threshold is a preset multiple of the first transmission bandwidth; and a ratio of the first compression ratio to the second compression ratio is a preset coefficient.

12. The method of claim 11, wherein, The preset multiple is 2; and the preset coefficient is a ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

13. The method of claim 4, wherein, The network parameters include network transmission bandwidth and network transmission delay; and the service capability information includes minimum resolution, minimum frame rate and minimum depth of field; wherein: When the network transmission bandwidth is greater than or equal to a second transmission bandwidth, and the network transmission delay is less than or equal to a maximum delay corresponding to the augmented reality service, it is confirmed that all parameters in the network parameters meet the transmission requirement of the basic quality; The second transmission bandwidth is a quotient of a second product and a first compression ratio; the second product is a product of the minimum resolution, the minimum frame rate and the minimum depth of field; and the first compression ratio represents a compression ratio of normal encoding by the server.

14. The method of claim 13, wherein, The virtual scene data is obtained by rendering according to the target capability information and encoding. The virtual scene data is obtained by rendering according to the service capability information and encoding according to the first compression ratio.

15. An augmented reality processing method, characterized by, The method applied to a terminal comprises: sending a service request to a cloud AR cloud end in a case of obtaining real scene data, and sending display information obtained to the cloud AR cloud end; the service request is used for instructing the cloud AR cloud end to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information represents display capability of the terminal for the virtual scene data; receiving the virtual scene data sent by the cloud AR cloud end.

16. The method of claim 15, wherein, The method further comprises: determining the display information based on screen display parameters and data acquisition parameters of the real scene data; the data acquisition parameters include real scene acquisition resolution, real scene acquisition frame rate and real scene acquisition depth of field; the screen display parameters include terminal display resolution, terminal display frame rate and terminal display depth of field; the display information includes target resolution, target frame rate and target depth of field; the target resolution is a minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is a minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is a minimum value of the real scene acquisition depth of field and the terminal display depth of field; when the real scene data is obtained by a perspective mode, the screen display parameters are determined as the display information.

17. The method according to claim 15 or 16, characterized in that, The method further comprises: outputting a virtual-real fusion result based on the virtual scene data and the real scene data.

18. An augmented reality processing apparatus, characterized by comprising: The device applied to a virtual scene rendering function of a cloud AR cloud end comprises: a request receiving module, configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case of obtaining real scene data; The virtual data determining module is configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information indicates display capability of the terminal for virtual scene data. The data sending module is configured to send the virtual scene data to the terminal.

19. An augmented reality processing apparatus, characterized by comprising: The apparatus applied to a terminal comprises: The information sending module is configured to send a service request to a cloud AR cloud in a case of acquiring real scene data, and send acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information indicates display capability of the terminal for the virtual scene data. The data receiving module is configured to receive the virtual scene data sent by the cloud AR cloud.

20. A communications device, comprising: The apparatus comprises: A transmitter, a processor and a receiver; The receiver is configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case of acquiring real scene data; The processor is configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information indicates display capability of the terminal for virtual scene data. The transmitter is configured to send the virtual scene data to the terminal.

21. A communications device, comprising: The apparatus comprises: A transmitter and a receiver; The transmitter is configured to send a service request to a cloud AR cloud in a case of acquiring real scene data, and send acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information indicates display capability of the terminal for the virtual scene data. The receiver is configured to receive the virtual scene data sent by the cloud AR cloud.

22. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement steps of the method in any one of claims 1 to 17.

23. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement steps of the method in any one of claims 1 to 17.

Images