Augmented reality processing method and apparatus, and communication device

Abstract

The present application relates to an augmented reality processing method and apparatus, and a communication device. The method comprises: receiving a service request sent by a terminal, wherein the service request is sent by the terminal when real scene data is acquired; acquiring display information of the terminal, and on the basis of the display information of the terminal and a network transmission condition, determining virtual scene data corresponding to the service request, wherein the display information represents a display capability of the terminal for the virtual scene data; and sending the virtual scene data to the terminal.

Description

Augmented reality processing method, device and communication equipment

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 2024108560857, filed on June 28, 2024, and entitled "Augmented reality processing method, device and communication equipment", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of network technology, and in particular, to an augmented reality processing method, device and communication equipment. BACKGROUND

[0004] Cloud AR (Augmented Reality) is an AR mode realized by means of cloud computing capability, mainly based on actual scene collection on the terminal side, virtual scene rendering / coding processing on the cloud side, and the processed virtual scene result is sent to the terminal, and the terminal side completes virtual-real fusion output and display. However, in the current cloud AR processing process, the cloud side mainly adopts a fixed strategy, resulting in that the cloud AR virtual scene output by the cloud side does not have flexibility, and it is difficult to match the actual scene accuracy, affecting the quality of service. SUMMARY

[0005] In a first aspect, an augmented reality processing method is provided, applied to a virtual scene rendering function of a cloud AR cloud side, and the method comprises:

[0006] receiving a service request sent by a terminal, the service request being sent by the terminal in a case of obtaining actual scene data;

[0007] obtaining display information of the terminal, and determining virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information represents display capability of the terminal for the virtual scene data;

[0008] sending the virtual scene data to the terminal.

[0009] In one of the embodiments, the display information is determined based on screen display parameters of the terminal and data collection parameters of the terminal for obtaining the actual scene data;

[0010] The data collection parameters include actual scene collection resolution, actual scene collection frame rate and actual scene collection depth of field; the screen display parameters include terminal display resolution, terminal display frame rate and terminal display depth of field; the display information includes target resolution, target frame rate and target depth of field; the target resolution is the minimum value of the actual scene collection resolution and the terminal display resolution; the target frame rate is the minimum value of the actual scene collection frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the actual scene collection depth of field and the terminal display depth of field;

[0011] If the real scene data is obtained by the terminal through the perspective mode, the display information is a screen display parameter.

[0012] In one embodiment, the virtual scene data corresponding to the service request is determined according to the display information of the terminal and the network transmission condition, comprising:

[0013] The network parameter for the current network communicating with the terminal is obtained by the virtual scene rendering function; the network parameter is a parameter representing the network transmission performance of the current network;

[0014] The display information or the service capability information is determined as the target capability information according to whether the network parameter meets the normal quality transmission requirement required by the display information and whether the network parameter meets the basic quality transmission requirement required by the service capability information; the basic quality transmission requirement is lower than the normal quality transmission requirement; the service capability information represents the minimum capability supported by the virtual scene rendering function for the augmented reality service;

[0015] The rendering and encoding are performed according to the target capability information to obtain the virtual scene data.

[0016] In one embodiment, the display information or the service capability information is determined as the target capability information according to whether the network parameter meets the normal quality transmission requirement required by the display information and whether the network parameter meets the basic quality transmission requirement required by the service capability information, comprising:

[0017] If all the parameters in the network parameter meet the normal quality transmission requirement, the display information is determined as the target capability information;

[0018] When part of the parameters in the network parameter meet the normal quality transmission requirement, the display information is determined as the target capability information if it is confirmed that all the parameters in the network parameter after network enhancement can meet the normal quality transmission requirement;

[0019] When all the parameters in the network parameter do not meet the normal quality transmission requirement, if all the parameters in the network parameter meet the basic quality transmission requirement, the service capability information is determined as the target capability information;

[0020] If all or part of the parameters in the network parameter do not meet the basic quality transmission requirement, the service request is not responded.

[0021] In one embodiment, the network parameter comprises a network transmission bandwidth and a network transmission delay; the display information comprises a target resolution, a target frame rate and a target depth of field; wherein:

[0022] When the network transmission bandwidth is greater than or equal to a first transmission bandwidth, it is confirmed that the network transmission bandwidth meets the normal quality transmission requirement;

[0023] When the network transmission delay is less than or equal to the recommended delay corresponding to the augmented reality service, it is confirmed that the network transmission delay meets the normal quality transmission requirement;

[0024] The first transmission bandwidth is a quotient of a first product and a first compression ratio; the first product is a product of the target resolution, the target frame rate and the target depth of field; and the first compression ratio represents a compression ratio of normal encoding of the server.

[0025] In one of the embodiments, the rendering and encoding are performed according to the target capability information to obtain the virtual scene data, including:

[0026] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission delay is less than or equal to the recommended delay, the rendering is performed according to the display information and the encoding is performed according to the first compression ratio to obtain the virtual scene data.

[0027] In one of the embodiments, the rendering and encoding are performed according to the target capability information to obtain the virtual scene data, including:

[0028] When the network transmission delay is less than or equal to the recommended delay, if the network transmission bandwidth is less than the first transmission bandwidth, the network enhancement is performed through bandwidth adjustment, and the rendering is performed according to the display information and the encoding is performed according to the first compression ratio to obtain the virtual scene data.

[0029] In one of the embodiments, the bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

[0030] In one of the embodiments, the rendering and encoding are performed according to the target capability information to obtain the virtual scene data, including:

[0031] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and the network transmission delay is less than the maximum delay corresponding to the augmented reality service, the network enhancement is performed based on the network transmission bandwidth to select a delay adjustment strategy;

[0032] The rendering and encoding are performed according to the display information and a processing strategy linked with the delay adjustment strategy to obtain the virtual scene data.

[0033] In one of the embodiments, the delay adjustment strategy includes network link selection and network link optimization.

[0034] The rendering and encoding are performed according to the display information and a processing strategy linked with the delay adjustment strategy to obtain the virtual scene data, including:

[0035] When the network transmission bandwidth is greater than or equal to the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the second compression ratio; wherein the second compression ratio is less than the first compression ratio.

[0036] When the network transmission bandwidth is less than the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the first compression ratio.

[0037] In one of the embodiments, the redundancy threshold is a preset multiple of the first transmission bandwidth; and the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0038] In one of the embodiments, the preset multiple is 2; and the preset coefficient is the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0039] In one of the embodiments, the network parameters include the network transmission bandwidth and the network transmission delay; and the service capability information includes the minimum resolution, the minimum frame rate and the minimum depth of field; wherein:

[0040] When the network transmission bandwidth is greater than or equal to the second transmission bandwidth, and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, it is confirmed that all the parameters in the network parameters meet the transmission requirements of the basic quality.

[0041] The second transmission bandwidth is the quotient of the second product and the first compression ratio; the second product is the product of the minimum resolution, the minimum frame rate and the minimum depth of field; and the first compression ratio represents the compression ratio of the normal encoding of the server.

[0042] In one of the embodiments, the virtual scene data is obtained by rendering and encoding according to the target capability information, including:

[0043] The virtual scene data is obtained by rendering according to the service capability information and encoding according to the first compression ratio.

[0044] In a second aspect, an augmented reality processing method is provided, applied to a terminal, and the method includes:

[0045] In the case of obtaining the real scene data, a service request is sent to a cloud AR cloud, and the obtained display information is sent to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine the virtual scene data corresponding to the service request according to the display information and the network transmission condition; wherein the display information represents the display capability of the terminal for the virtual scene data;

[0046] The virtual scene data sent by the cloud AR cloud is received.

[0047] In one of the embodiments, the method further includes:

[0048] determine the display information based on the screen display parameters and the data acquisition parameters for acquiring the real scene data;

[0049] The data acquisition parameters include a real scene acquisition resolution, a real scene acquisition frame rate, and a real scene acquisition depth of field; the screen display parameters include a terminal display resolution, a terminal display frame rate, and a terminal display depth of field; the display information includes a target resolution, a target frame rate, and a target depth of field; the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field.

[0050] When the real scene data is obtained through the perspective mode, the screen display parameters are determined as the display information.

[0051] In one of the embodiments, the method further includes:

[0052] Outputting a virtual-real fusion result based on the virtual scene data and the real scene data.

[0053] In a third aspect, an augmented reality processing apparatus is provided, which is applied to a virtual scene rendering function of a cloud AR cloud end, and the apparatus includes:

[0054] A request receiving module is configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case of acquiring real scene data;

[0055] A virtual data determining module is configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information represents display capability of the terminal for the virtual scene data;

[0056] A data sending module is configured to send the virtual scene data to the terminal.

[0057] In a fourth aspect, an augmented reality processing apparatus is provided, which is applied to a terminal, and the apparatus includes:

[0058] An information sending module is configured to send a service request to a cloud AR cloud end in a case of acquiring real scene data, and send acquired display information to the cloud AR cloud end; the service request is used to instruct the cloud AR cloud end to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information represents display capability of the terminal for the virtual scene data;

[0059] A data receiving module is configured to receive virtual scene data sent by the cloud AR cloud end.

[0060] In a fifth aspect, a communication device is provided, which includes a transmitter, a processor, and a receiver.

[0061] a receiver configured to receive a service request sent by the terminal, the service request being sent by the terminal in a case where the real scene data is acquired;

[0062] a processor configured to acquire display information of the terminal, and determine the virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information indicates a display capability of the terminal for the virtual scene data;

[0063] a transmitter configured to send the virtual scene data to the terminal.

[0064] In a sixth aspect, a communication device is provided, which includes a transmitter and a receiver.

[0065] a transmitter configured to send a service request to a cloud AR cloud in a case where real scene data is acquired, and send acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information indicates a display capability of the terminal for the virtual scene data;

[0066] a receiver configured to receive the virtual scene data sent by the cloud AR cloud.

[0067] In a seventh aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method in the above aspects.

[0068] In an eighth aspect, a computer program product is provided, which includes a computer program, and the computer program is executed by a processor to implement the method in the above aspects.

[0069] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the application will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0070] FIG. 1 is a diagram of an application environment of an augmented reality processing method in an embodiment;

[0071] FIG. 2 is a flowchart of an augmented reality processing method in an embodiment;

[0072] FIG. 3 is a flowchart of a step of obtaining virtual scene data in an embodiment;

[0073] FIG. 4 is a flowchart of a step of determining target capability information in an embodiment;

[0074] FIG. 5 is a flowchart of an augmented reality processing method in another embodiment;

[0075] FIG. 6 is a schematic diagram of a functional architecture of an augmented reality processing method in an embodiment;

[0076] FIG. 7 is a schematic diagram of a flow of a cloud AR cloud side service cooperation stage in an embodiment;

[0077] FIG. 8 is a schematic diagram of a flow of a cloud AR service running stage in an embodiment;

[0078] FIG. 9 is a structural block diagram of an augmented reality processing apparatus in an embodiment;

[0079] FIG. 10 is a structural block diagram of an augmented reality processing apparatus in another embodiment;

[0080] FIG. 11 is an internal structure diagram of a communication device in an embodiment;

[0081] FIG. 12 is an internal structure diagram of a communication device in another embodiment. DETAILED DESCRIPTION

[0082] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0083] The terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be exchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than that illustrated or described herein, and the objects distinguished by "first", "second" are usually a class, and are not limited to the number of objects, for example, the first object can be one or more. In addition, the term "and / or" is only a description of the associated relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " generally represents an "or" relationship between the associated objects before and after.

[0084] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0085] FIG. 1 is a schematic diagram of an application scenario of an augmented reality processing method according to an embodiment of the present application. As shown in FIG. 1, in the scenario, a terminal 102 communicates with a server 104 through a network. Exemplarily, the terminal 102 can be a cloud AR terminal, serving as a request initiator of a cloud AR service and a display presenter of a final service; optionally, the server 104 can be a cloud AR cloud, serving as an execution party of cloud AR virtual scene processing, mainly completing rendering, compression encoding and other processing of a virtual scene based on industry network linkage capability and flexibly adjusting the rendering, compression encoding and other processing of the virtual scene.

[0086] Optionally, the terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server (also referred to as a cloud server, referred to as a cloud) providing cloud computing services.

[0087] In the prior art, in the current cloud AR processing process, the cloud mainly adopts a fixed strategy, for example, a fixed rendering mode, a fixed deep compression ratio, etc., for example, processing a cloud AR virtual scene based on a fixed rendering and compression encoding mechanism, or completing virtual scene processing based on an AR terminal (i.e., a terminal locally), resulting in that the cloud AR virtual scene output by the cloud does not have flexibility, is difficult to match the precision of a real scene, and realizes seamless connection. Especially for fragmented AR terminals, their capabilities are different, mainly including differences in real scene acquisition precision, screen display and processing capability, and cannot effectively and flexibly link, resulting in a split between a real scene and a virtual scene, and causing a problem of user experience. At the same time, with the popularization of optical fiber into a household, the bandwidth capability has been greatly improved. The deep compression originally performed to save bandwidth not only increases the processing pressure of the cloud and prolongs the processing delay of the cloud, but also the saved bandwidth benefit is less and less obvious. At the same time, the network has acceleration, link optimization and bandwidth expansion enhancement capabilities, and the comprehensive capability of the network needs to be fully utilized.

[0088] The traditional technology at least has the problems of lacking linkage mechanism of industry network, and network capability cannot effectively serve the cloud AR business experience: in the current cloud AR virtual scene processing process, the cloud generally adopts a single graphics rendering and fixed encoding mechanism, which cannot be linked with network bandwidth, delay and other transmission attributes, and at the same time cannot obtain related network bandwidth enhancement, accelerated transmission, link optimization and other enhanced capabilities, so it is difficult to realize the linkage of cloud AR industry network, and limits the improvement of user experience effect. The traditional technology at least has the problem that the end-to-end delay of cloud AR cannot be linked, resulting in different user experience: in the current cloud AR service, including terminal side real scene acquisition, localized digital processing, cloud virtual scene rendering, encoding streaming delivery, to the terminal side virtual-real fusion output, the virtual scene rendering, encoding and delay of cloud processing are closely related to the streaming bandwidth, and can be reduced by network transmission acceleration, link optimization and other modes, but generally lack linkage mechanism with network capability, resulting in different user experience.

[0089] Based on the above-mentioned traditional technology, the embodiment of the present application proposes to take the real scene collection accuracy (referred to as real scene accuracy) of the terminal as the benchmark, to fuse the terminal and network performance, to adopt a more flexible multi-dimensional processing mechanism of cloud AR virtual scene, and to realize high-quality end-to-end user experience and high-quality service output of cloud AR. Among them, the embodiment of the present application adopts the fusion of cloud network end business linkage capability, takes the display information of the terminal as the guide, adopts industry network capability fusion enhancement optimization, and realizes the enhancement of end-to-end cloud virtual scene accuracy.

[0090] Optionally, the embodiment of the present application can fuse the metropolitan area technology and the existing network test, such as end-to-end service DCI (Data Center Interconnect, data center interconnection) networking research. For cloud AR business scenarios, based on the best display accuracy of terminal virtual scene as input and guide, with the help of multi-dimensional linkage optimization of cloud network end business, high-quality cloud virtual scene processing output is achieved, efficient fusion of cloud AR real scene and virtual scene is realized, and user experience is improved.

[0091] It should be noted that the beneficial effects brought by the embodiments of the present application or the technical problems solved are not limited to this, but also other implicit or related problems, which can be referred to the description of the following embodiments.

[0092] Before introducing the specific embodiments of the present application, the professional terms involved in the present application are explained:

[0093] Real scene accuracy: can refer to the resolution, frame rate, depth of field and other clarity conditions of the real scene that can be acquired or perceived by the camera, perspective optical device of the cloud AR terminal.

[0094] Fusion of end-to-end network performance: End-to-end network can refer to terminal and network respectively, and end-to-end network performance corresponds to terminal performance and network performance respectively, wherein the terminal performance can refer to the screen display clarity, refresh rate and other capabilities related to the cloud AR terminal; the network performance refers to the comprehensive transmission and enhancement capabilities such as network bandwidth, delay, link optimization related to cloud AR. The fusion of end-to-end network performance can refer to the transmission comprehensive capabilities based on and combined with the terminal screen display clarity, refresh rate and network bandwidth, delay, link optimization, so as to provide better service quality and user experience.

[0095] Cloud AR: refers to a way of realizing AR service based on cloud processing capability. Specifically, it can refer to the accuracy analysis of the actual scene obtained on the terminal side, the comprehensive terminal processing performance and the network transmission and enhancement performance, the determination of the service quality of the virtual scene, the flexible rendering of the AR virtual scene on the cloud, the encoding and sending of the terminal after the cloud, the terminal based on the real scene obtained locally, and the fusion of the virtual scene sent by the terminal side to realize the local virtual-real fusion display and presentation of the terminal.

[0096] Cloud AR rendering: can refer to the process of completing the rendering of the corresponding AR virtual scene based on the cloud, which completes the cloud rendering processing according to the determined cloud output virtual scene accuracy according to the business and network linkage strategy.

[0097] Cloud AR encoding: can refer to the process of completing the encoding compression of the corresponding AR virtual scene based on the cloud, which performs encoding processing according to the business and network coordination strategy in combination with the network optimization capability according to the rendering output result and the network transmission condition.

[0098] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described in detail in some examples. The embodiments of the present application will be described below with reference to the drawings.

[0099] In one exemplary embodiment, as shown in FIG. 2, an augmented reality processing method is provided, which is applied to the server (such as cloud server) in FIG. 1 for example, it can be understood that the method can also be applied to the virtual scene rendering function of the cloud AR cloud, including the following steps 202 to 206. Among them:

[0100] Step 202, receiving the service request sent by the terminal, the service request being issued by the terminal in the case of obtaining real scene data.

[0101] Among them, the service request can refer to the AR service request; taking the cloud AR terminal as an example, the cloud AR terminal as the request initiator of the cloud AR service and the final service display and presentation party can start the related service request and synchronously collect and process the related real scene data.

[0102] For example, the terminal can initiate a service request to the server in the case of real scene data acquisition. For example, the terminal can initiate a cloud AR service request and real scene collection, such as starting a related service request and synchronously collecting real scene data.

[0103] Specifically, the server receives the service request sent by the terminal, determines that virtual scene processing is needed, and the service request can be used to indicate that the current enters the cloud AR business running stage. Alternatively, the cloud AR cloud end as an execution party of the cloud AR virtual scene processing can complete the processing including but not limited to flexible adjustment of rendering, compression encoding and the like of the virtual scene based on the industry network linkage capability.

[0104] In step 204, display information of the terminal is acquired, and virtual scene data corresponding to the service request is determined according to the display information of the terminal and network transmission conditions; wherein the display information indicates display capability of the terminal for the virtual scene data.

[0105] Specifically, the server can acquire the display information of the terminal and use it as a reference to determine the cloud end virtual scene processing; wherein the server interfaces with the terminal side to acquire the display information of the terminal. For example, the terminal can acquire the display information and send the acquired display information to the server to achieve the best virtual-real fusion output.

[0106] In response to receiving the service request, the server can determine virtual scene data corresponding to the service request according to display information of the terminal and network transmission conditions, the display information can be used to indicate display capability of the terminal for the virtual scene data, and the network transmission conditions can be transmission conditions of a current network in which the server and the terminal communicate; for example, the display information of the terminal can be acquired in the cloud AR cloud network end business cooperation stage; alternatively, the cloud AR cloud network end business cooperation stage can be a preparatory stage of the cloud AR business running stage, and the network transmission conditions can be acquired in the cloud AR business running stage.

[0107] In some examples, the display information is used to represent the terminal virtual scene optimal display precision, where the terminal virtual scene optimal display precision can be understood as a terminal side virtual scene optimal display precision; for example, the terminal virtual scene optimal display precision can refer to determining the virtual scene output result that the terminal can achieve with the best user experience as a benchmark and target based on the real scene collection precision, according to the terminal processing capability. Further, the process of determining the virtual scene data by the server according to the display information of the terminal can be understood as that the server targets the virtual scene output with the best user experience of the terminal, and combines the network transmission conditions to perform flexible virtual scene rendering and adaptive coding by the cloud end, to obtain the virtual scene result with the best user experience of the cloud end, so as to solve the cloud AR virtual-real high-precision seamless connection problem end-to-end by means of the cloud network end capability. Based on the embodiments of the present application, the best virtual-real fusion result can be finally obtained, which is optimized and guaranteed by the cloud AR terminal.

[0108] In step 206, the virtual scene data is sent to the terminal.

[0109] Specifically, after obtaining the virtual scene data, the server can send the virtual scene data to the terminal, so that the terminal can output the virtual-real fusion result based on the virtual scene data and the collected real scene data.

[0110] It should be noted that outputting the virtual-real fusion result can refer to enabling the virtual digital capability related to the photographed real scene to cooperate with the subsequent addition and interaction of other virtual images. For example, based on the virtual scene data and the real scene data, the terminal can superimpose the real scene picture and the virtual scene picture to form a reality augmented picture, with the real scene picture as the bottom picture and the virtual scene picture as the top picture. It should be understood that the embodiments of the present disclosure are not limited thereto, and in other implementation manners, the terminal can also superimpose the real scene picture and the virtual scene picture with the virtual scene picture as the bottom picture and the real scene picture as the top picture.

[0111] Further, taking the server as an example of a cloud server, the cloud server sends the processed virtual scene result (virtual scene data) to the terminal, the terminal receives the virtual scene result, and performs light adjustment (such as brightness corresponding adjustment) according to the real scene lighting condition, completes the cloud AR virtual-real fusion output, and performs corresponding interaction operation based on the virtual-real fusion result. For another example, taking the terminal as an example of an AR terminal, cloud AR virtual-real fusion output and interaction can be realized, the AR terminal receives the virtual scene sent by the cloud end, and performs corresponding highlight adjustment based on the locally acquired real scene, so as to combine with the real scene collected locally, complete the cloud AR service terminal display output of virtual-real fusion, and perform related cloud AR interaction based on the output result.

[0112] In the above enhanced reality processing method, the virtual scene rendering function of the cloud AR cloud end can include: receiving a cloud AR service request issued in the case of obtaining real scene data, obtaining display information of the terminal, the display information indicating the display capability of the terminal for virtual scene data, and determining the virtual scene data corresponding to the service request according to the display information of the terminal and the network transmission condition, and finally sending the virtual scene data to the terminal. The display capability of the terminal is used as the basis, and a more flexible virtual scene processing mechanism is used in combination with the network transmission condition to achieve high-quality cloud virtual scene processing output, and to realize end-to-end high-quality user experience and high-quality service output of the cloud AR end.

[0113] In some embodiments, the display information is determined based on the screen display parameters of the terminal and the data acquisition parameters of the terminal for obtaining real scene data.

[0114] Specifically, in the embodiments of the present application, the display information can be determined based on the screen display parameters of the terminal and the data acquisition parameters of the terminal for obtaining real scene data. For example, the terminal can determine the display information based on the data acquisition parameters for obtaining real scene data and the screen display parameters, and transmit the display information to the server.

[0115] Optionally, the data acquisition parameters can refer to the related parameters of the real scene acquisition device of the terminal, which are used to represent the real scene acquisition accuracy (referred to as real scene accuracy), for example, the terminal can obtain the real scene accuracy through the real scene acquisition device, including but not limited to resolution, frame rate, etc. The real scene acquisition device can include but is not limited to a camera, a light-transmitting lens, etc.

[0116] Further, the screen display parameters can represent the terminal screen display capability (which can be understood as the terminal screen display accuracy), including but not limited to resolution, frame rate, refresh rate, etc. For example, the terminal can obtain the best virtual scene display accuracy of the terminal based on the real scene acquisition accuracy and the terminal screen display capability, so as to realize the best virtual-real fusion output.

[0117] For example, based on the real scene acquisition accuracy of the terminal side and in combination with the screen display capability of the terminal side, the best virtual scene display accuracy of the terminal side is determined, for example, the terminal side can compare the real scene acquisition accuracy and the terminal screen display capability to determine the best virtual scene display accuracy of the terminal: ① when the real scene acquisition accuracy >= the terminal screen display capability, the terminal screen display capability is determined as the best virtual scene display accuracy of the terminal; ② when the real scene acquisition accuracy < the terminal screen display capability, the real scene acquisition accuracy is determined as the best virtual scene display accuracy of the terminal, and the real scene acquisition accuracy is selected as the best virtual scene display accuracy of the terminal, so as to be seamlessly connected with the corresponding real scene, and to realize the best virtual-real fusion output.

[0118] In one of the embodiments, the data acquisition parameters can include live scene acquisition resolution, live scene acquisition frame rate and live scene acquisition depth of field; the screen display parameters include terminal display resolution, terminal display frame rate and terminal display depth of field;

[0119] The display information includes target resolution, target frame rate and target depth of field; wherein the target resolution is the minimum value of the live scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the live scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the live scene acquisition depth of field and the terminal display depth of field.

[0120] Specifically, the terminal side can acquire data acquisition parameters representing live scene acquisition accuracy, and screen display parameters representing screen display capability, wherein the data acquisition parameters can include live scene acquisition resolution, live scene acquisition frame rate and live scene acquisition depth of field; the screen display parameters can include terminal display resolution, terminal display frame rate and terminal display depth of field, and then the display information can include target resolution, target frame rate and target depth of field.

[0121] Exemplarily, the terminal can obtain relevant resolution, frame rate, depth of field, etc. of the current live scene acquisition device of the terminal as the live scene acquisition accuracy, such as the resolution of the camera and the frame rate of live scene shooting, etc.; if the live scene perspective mode is adopted, the live scene acquisition accuracy can be defaulted as the live scene accuracy obtained by the human eye; further, the terminal can acquire the screen display capability, i.e. obtain relevant resolution, frame rate, etc. of the terminal screen display.

[0122] Optionally, the target resolution is the minimum value of the live scene acquisition resolution and the terminal display resolution, the target frame rate is the minimum value of the live scene acquisition frame rate and the terminal display frame rate, and the target depth of field is the minimum value of the live scene acquisition depth of field and the terminal display depth of field.

[0123] In actual application, taking the terminal as a cloud AR terminal for example, the live scene acquisition accuracy of the cloud AR terminal is acquired in the cloud AR cloud network side service cooperation stage, including the camera or the light transmission lens of the cloud AR terminal for acquiring live scene, etc. The camera directly acquires its pixel resolution, shooting corresponding frame rate, depth of field, etc. The light transmission lens is defaulted as the accuracy of human eye observation; assuming that the live scene acquisition resolution of the cloud AR terminal is r1 (live scene acquisition resolution), the frame rate thereof is f1 (live scene acquisition frame rate), and the depth of field thereof is d1 (live scene acquisition depth of field), the terminal screen display accuracy of the cloud AR terminal, including the resolution, frame rate and depth of field of the screen display of the cloud AR terminal, are assumed to be r2 (terminal display resolution), f2 (terminal display frame rate) and d2 (terminal display depth of field) respectively.

[0124] The live scene acquisition accuracy of the cloud AR terminal and the screen display accuracy of the cloud AR terminal are compared to obtain the best display accuracy of the virtual scene of the cloud AR terminal: r t (target resolution), f t(target frame rate), d t (target depth of field), the relevant rules are as follows: cloud AR terminal virtual scene best display precision, wherein the resolution r t = Min(r1, r2), frame rate f t = Min(f1, f2), depth of field d t = Min(d1, d2).

[0125] It can be understood that when the real scene data is obtained by using the perspective mode, in an embodiment, the display information is the screen display parameter in the case that the real scene data is obtained by the terminal through the perspective mode.

[0126] Specifically, the perspective mode can refer to that the terminal obtains the real scene data by using the real scene perspective mode; it should be noted that the real scene perspective mode includes but is not limited to the perspective lens mode, for example, the corresponding real-time real scene is obtained by using the light transmission lens.

[0127] In the case that the real scene data is obtained by the terminal through the perspective mode, the terminal can determine the screen display parameter as the display information; taking the cloud AR terminal as an example, if the cloud AR real scene acquisition adopts the perspective lens mode, the real scene precision of the cloud AR terminal is greater than the terminal screen display precision of the cloud AR terminal by default, that is, in this case, the resolution, frame rate and depth of field related to the best display precision of the virtual scene of the cloud AR terminal are equal to r2, f2 and d2 respectively. Further, the server can obtain and analyze the best display precision of the virtual scene of the terminal: the server obtains the best display precision of the virtual scene of the terminal through the terminal side, and correspondingly obtains the resolution, frame rate and depth of field, and determines the cloud virtual scene processing based on the same.

[0128] In the above-mentioned augmented reality processing method, the real scene precision is used as a criterion, the terminal screen display clarity, refresh rate performance and other capabilities are comprehensively considered to determine the best virtual scene output that can be achieved by the terminal, and the best virtual scene output of the terminal is taken as a target to unify the service quality of the cloud AR, so that the best virtual scene result of the cloud AR user experience is obtained. The terminal side can obtain the virtual scene result based on the real scene to perform light and shadow adjustment (for example, real-time brightness adjustment), and finally obtain the best virtual-real fusion result optimized and guaranteed by the cloud AR terminal. The embodiment of the application integrates the capabilities of the cloud network side to realize seamless connection of the cloud AR virtual-real high precision, and provides high-quality cloud AR services by integrating the multi-dimensional performance of the terminal and the network.

[0129] In actual applications, the present application is oriented to cloud AR virtual scene processing, and the best display precision of a terminal virtual scene is determined through the real scene collection precision of the terminal and the terminal screen display capability, and the virtual scene output precision is improved with the best display precision of the terminal virtual scene as a target. In addition, the cloud AR virtual scene cloud processing effect can be improved through the fusion of network performance and industry network capability linkage, so as to further improve the end-to-end user experience. In one embodiment, as shown in FIG. 3, step 204 can include steps 302 to 306. Wherein:

[0130] In step 302, the network parameter for communication with the terminal by using the current network is obtained through the virtual scene rendering function. The network parameter is a parameter representing the network transmission performance of the current network.

[0131] Specifically, the embodiment of the present application can obtain the network parameter for communication with the terminal by using the current network through the virtual scene rendering function. Taking the server as an example of the cloud AR cloud, the cloud AR cloud can perform network performance acquisition and analysis, and then realize flexible virtual scene processing by fusing the terminal and network performance to obtain the best virtual-real fusion result. Illustratively, the network performance can include network transmission performance, which can be obtained by the cloud and the terminal in linkage. Optionally, the network parameter can be the parameter during normal transmission between the cloud and the terminal.

[0132] The server can obtain the network parameter for communication with the terminal by using the current network, which can represent the network transmission performance of the current network, such as bandwidth, delay, etc. Wherein, the terminal can also perform network performance acquisition and analysis, and cooperate with the server (for example, the cloud AR cloud) to obtain the bandwidth, delay, etc. performance of the current transmission corresponding to the network. In some examples, taking the server as an example of the cloud server, the bandwidth, delay, etc. of the normal transmission between the cloud and the terminal can be obtained as the network parameter according to the bandwidth speed measurement and delay detection tools related to the cloud and the terminal.

[0133] In step 304, the display information or the service capability information is determined as the target capability information according to whether the network parameter meets the normal quality transmission requirement required by the display information, and whether the network parameter meets the basic quality transmission requirement required by the service capability information. The basic quality transmission requirement is lower than the normal quality transmission requirement. The service capability information represents the minimum capability supported by the virtual scene rendering function for augmented reality service.

[0134] Specifically, in the case of obtaining the network parameter, the server can determine the display information or the service capability information as the target capability information according to a network transmission condition met by the network parameter, wherein the network transmission condition can refer to a transmission bandwidth, a time delay, etc. required by the display information, or the network transmission condition can refer to a transmission bandwidth, a time delay, etc. required by the service capability information, so as to ensure a better interactive time delay experience of the user; optionally, the network transmission condition can include, but is not limited to, a normal quality transmission requirement required by the display information and a basic quality transmission requirement required by the service capability information, the service capability information can represent a minimum capability of a virtual scene rendering function supporting an augmented reality service, and the basic quality transmission requirement is lower than the normal quality transmission requirement.

[0135] The normal quality transmission requirement can represent a transmission requirement of the display information (for example, a terminal virtual scene optimal display precision), the server can obtain and analyze the display information of the terminal to determine the normal quality transmission requirement, such as a bandwidth requirement, a time delay requirement, etc.; taking the display information as an example of the terminal virtual scene optimal display precision, the server can obtain the virtual scene optimal display precision sent by the terminal and decompose it into resolution, frame rate, depth of field, etc., and simultaneously convert its required bandwidth requirement according to a compression rate corresponding to a commonly used encoding protocol.

[0136] The basic quality transmission requirement can represent a transmission requirement of the service information, the server can obtain and analyze the service capability information to determine the basic quality transmission requirement, such as a bandwidth requirement, a time delay requirement, etc.; exemplarily, the service capability information can represent a minimum service precision of the virtual scene, for example, when the terminal virtual scene optimal display precision cannot be supported, the minimum service precision capable of meeting the user experience needs to be determined, which can include resolution, frame rate, depth of field, etc.; in some examples, the server can determine the minimum service quality of the cloud AR service according to a minimum requirement of the user experience of the cloud AR service, and explicitly determine the minimum resolution, frame rate, depth of field, etc. of the output of the cloud, for example, the server can determine the minimum service precision by combining the terminal virtual scene optimal display precision and the precision range of the virtual scene processing of the cloud AR service, thereby determining the minimum service precision of the virtual scene. For another example, the server can select the precision output of the basic configuration of the cloud AR application service, for example, turn off the corresponding light and shadow effect and reduce the frame rate, but have a certain user experience guarantee as the minimum service precision.

[0137] Further, the target capability information in the embodiments of the present application can represent cloud processing precision to represent different cloud processing requirements; for example, the display information or the service capability information is determined as the target capability information based on whether the network parameters meet the normal quality transmission requirement or the basic quality transmission requirement, so that the server can determine the corresponding cloud processing precision according to the terminal virtual scene best display precision of the cloud AR and the better interactive delay experience of the user in different combination modes; optionally, taking the server as the cloud server as an example, the current network transmission performance is combined to determine whether the corresponding cloud processing requirement is met through the current network bandwidth and delay, so that the cloud determines the corresponding processing precision.

[0138] In step 306, rendering and encoding are performed according to the target capability information to obtain virtual scene data.

[0139] Specifically, in the case of determining the display information or the service capability information as the target capability information, the server can perform rendering and encoding according to the target capability information to obtain virtual scene data.

[0140] The above-mentioned augmented reality processing method can complete network capability matching, obtain network parameters for communication between the current network and the terminal through virtual scene rendering function; the network parameters are parameters representing network transmission performance of the current network; the display information or the service capability information is determined as the target capability information according to whether the network parameters meet the normal quality transmission requirement required by the display information and whether the network parameters meet the basic quality transmission requirement required by the service capability information; the basic quality transmission requirement is lower than the normal quality transmission requirement; the service capability information represents the minimum capability of the virtual scene rendering function supporting the augmented reality service.

[0141] The embodiments of the present application take the terminal virtual scene best display precision output as the target, and the cloud performs virtual scene rendering and encoding according to the network transmission performance to obtain virtual scene results, wherein the network transmission performance is taken as a part of the virtual scene processing of the cloud AR service, so as to facilitate the cloud to perform rendering and encoding in combination with the network capability, and to integrate the network transmission into the cloud AR service processing process. The terminal side obtains the virtual scene results, performs real-time brightness adjustment based on the real scene, and finally completes high-quality cloud AR virtual-real fusion output, effectively improving the user experience quality.

[0142] In one exemplary embodiment, as shown in FIG. 4, step 304 includes steps 402 to 408. Wherein:

[0143] In step 402, if all parameters in the network parameters meet the normal quality transmission requirement, the display information is determined as the target capability information.

[0144] Specifically, when all parameters in the network parameters meet the transmission requirement of normal quality, the server can determine the display information as the target capability information. Taking the network parameters including bandwidth and latency, the display information representing the best display precision of the terminal virtual scene, and the target capability information representing the cloud processing precision as examples, when the bandwidth and the latency of the current network both meet the transmission requirement of the best display precision of the terminal virtual scene (i.e., the transmission requirement of normal quality), the best display precision of the terminal virtual scene is determined as the cloud processing precision.

[0145] Step 404, when part of the parameters in the network parameters meet the transmission requirement of normal quality, the display information is determined as the target capability information in the case that all parameters in the network parameters of the current network after network enhancement meet the transmission requirement of normal quality.

[0146] Specifically, if part of the parameters in the network parameters meet the transmission requirement of normal quality, it can be judged whether all parameters in the network parameters can meet the transmission requirement of normal quality after network enhancement, and when the transmission requirement of normal quality is met, the display information can be determined as the target capability information. Exemplarily, the network enhancement manner includes but is not limited to bandwidth improvement, network acceleration, and link optimization.

[0147] Taking the network parameters including bandwidth and latency, the display information representing the best display precision of the terminal virtual scene, and the target capability information representing the cloud processing precision as examples, when one of the bandwidth and the latency of the current network cannot meet the transmission requirement of the best display precision of the terminal virtual scene, if the network parameter that does not meet the transmission requirement can meet the transmission requirement through network enhancement, the best display precision of the terminal virtual scene is determined as the cloud processing precision, and related network capability linkage processing can be performed.

[0148] The embodiment of the present application can realize real-time fusion and enhancement of network capability for cloud AR service demand. Among them, the network transmission performance and network enhancement capability are taken as the pre-factors of cloud AR service virtual scene processing, which facilitates the cloud to formulate more flexible rendering and coding strategies combined with network capability, and at the same time, the network transmission capability enhancement post-processing can be flexibly called for the cloud AR cloud virtual scene output result, so as to integrate the network transmission and enhancement capability into the cloud AR service processing process.

[0149] Regarding the above steps 402 and 404, in an exemplary embodiment, the network parameters can include network transmission bandwidth and network transmission latency; the display information can include target resolution, target frame rate, and target depth of field; wherein:

[0150] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, it is confirmed that the network transmission bandwidth meets the transmission requirement of normal quality;

[0151] When the network transmission delay is less than or equal to the recommended delay corresponding to the augmented reality service, it is confirmed that the network transmission delay meets the transmission requirement of normal quality.

[0152] The first transmission bandwidth is a quotient value of a first product and a first compression ratio. The first product is a product of the target resolution, the target frame rate, and the target depth of field. The first compression ratio represents a compression ratio of normal encoding performed by the server.

[0153] Specifically, the network parameters can include a network transmission bandwidth and a network transmission delay, and the display information can include a target resolution, a target frame rate, and a target depth of field. When the network transmission bandwidth is greater than or equal to a first transmission bandwidth, and the network transmission delay is less than or equal to a recommended delay, it is confirmed that all parameters in the network parameters meet the transmission requirement of normal quality (i.e., the network transmission bandwidth meets the transmission requirement of normal quality, and the network transmission delay meets the transmission requirement of normal quality). When the network transmission bandwidth is less than the first transmission bandwidth, or the network transmission delay is greater than the recommended delay, it is confirmed that part of the parameters in the network parameters meet the transmission requirement of normal quality (i.e., the network transmission bandwidth meets the transmission requirement of normal quality, or the network transmission delay meets the transmission requirement of normal quality).

[0154] Exemplarily, the recommended delay can refer to a delay corresponding to an augmented reality service (AR service), for example, the corresponding recommended delay is determined according to the corresponding cloud AR service attribute. Further, the first transmission bandwidth can represent a transmission bandwidth required for the server to output at a virtual scene best display precision in the cloud terminal, wherein the first transmission bandwidth is a quotient value of a first product and a first compression ratio. The first product is a product of the target resolution, the target frame rate, and the target depth of field. The first compression ratio represents a compression ratio of normal encoding performed by the server.

[0155] In some examples, the first compression ratio can be a normal encoding compression rate ct1 determined in combination with an encoding compression algorithm used in the cloud. For example, the target resolution is r t , the target frame rate is f t , the target depth of field is d t , and the recommended delay is l q2 , the first transmission bandwidth b b = r t *f t *d t / ct1, wherein the first product is r t *f t *d t . Further, the normal transmission bandwidth b n and the delay l n between the cloud and the terminal are obtained according to a bandwidth speed and a delay detection tool related to the cloud and the terminal, i.e., the network transmission bandwidth is b n , and the network transmission delay is l nFor example, the cloud processing precision can be determined by comparing the required bandwidth, time delay and the relationship between the network transmission capacity of the cloud and the terminal, and determining the cloud processing precision when b n >=b b ,l n <=l q2 , it is considered that the network transmission bandwidth and network transmission time delay both meet the normal quality transmission requirements, and the terminal virtual scene best display precision is used as the cloud processing precision.

[0156] In addition, if b n >=b b ,l n <=l q2 , only one of them can be met, it is determined whether the network transmission capacity can be improved through network enhancement. For example, if b n >=b b ,l n >l q2 (But l n <l q1 , l q1 is the maximum time delay corresponding to the augmented reality service), after network enhancement, l n <=l q2 can be met, it is determined that the cloud uses the terminal virtual scene best display precision as the cloud processing precision. If b n <b b ,l n <=l q2 , after network enhancement, b n >=b b is reached, it is determined that the cloud uses the terminal virtual scene best display precision as the cloud processing precision.

[0157] Step 406, when all parameters in the network parameters do not meet the normal quality transmission requirements, if all parameters in the network parameters meet the basic quality transmission requirements, the service capability information is determined as the target capability information.

[0158] Specifically, when all parameters in the network parameters do not meet the normal quality transmission requirements, if all parameters in the network parameters meet the basic quality transmission requirements, the server determines the service capability information as the target capability information, to ensure that the cloud AR service with user experience guarantee is provided.

[0159] The network parameters include bandwidth and time delay, the display information represents the best display precision of the terminal virtual scene, the service capability information represents the minimum service precision, and the target capability information represents the cloud processing precision. When the bandwidth and time delay of the current network cannot simultaneously meet the transmission requirements of the best display precision of the terminal virtual scene (i.e., the transmission requirements of normal quality), if the bandwidth and time delay of the current network can meet the transmission requirements of the minimum service precision (i.e., the transmission requirements of basic quality), the minimum service precision is determined as the cloud processing precision. Thereafter, the server can determine the corresponding cloud processing according to the determined cloud processing precision.

[0160] It should be noted that, regarding the above step 406, in an exemplary embodiment, the network parameters can include network transmission bandwidth and network transmission time delay; the service capability information can include minimum resolution, minimum frame rate, and minimum depth of field; wherein:

[0161] When the network transmission bandwidth is greater than or equal to the second transmission bandwidth, and the network transmission time delay is less than or equal to the maximum time delay corresponding to the augmented reality service, it is confirmed that all parameters in the network parameters meet the transmission requirements of basic quality.

[0162] The second transmission bandwidth is the quotient of the second product and the first compression ratio; the second product is the product of the minimum resolution, the minimum frame rate, and the minimum depth of field; and the first compression ratio represents the compression ratio of normal encoding performed by the server.

[0163] Specifically, the network parameters include network transmission bandwidth and network transmission time delay, and the service capability information can include minimum resolution, minimum frame rate, and minimum depth of field. When the network transmission bandwidth is greater than or equal to the second transmission bandwidth, and the network transmission time delay is less than or equal to the maximum time delay, it is confirmed that all parameters in the network parameters meet the transmission requirements of basic quality (i.e., the network transmission bandwidth and the network transmission time delay both meet the transmission requirements of basic quality).

[0164] The second transmission bandwidth can represent the required bandwidth for the cloud to output at the minimum service precision. Exemplarily, the second transmission bandwidth is the quotient of the second product and the first compression ratio, the second product is the product of the minimum resolution, the minimum frame rate, and the minimum depth of field, and the first compression ratio can represent the compression ratio of normal encoding performed by the server.

[0165] In some examples, the first compression ratio can be the determined normal encoding compression rate ct1 in combination with the encoding compression algorithm used by the cloud. Exemplarily, the maximum time delay corresponds to an augmented reality service (AR service), for example, the corresponding maximum time delay is determined according to the corresponding cloud AR service attributes. In addition, the recommended time delay is less than the maximum time delay, and the maximum time delay is l q1 , the recommended time delay is l q2 , for example, l q2 < l q1 .

[0166] Further, the lowest resolution is r cs , the lowest frame rate is f cs , the lowest depth of field is d cs , and the maximum latency is l q1 For example, the second transmission bandwidth b r = r cs * f cs * d cs / ct1, wherein the second product is r cs * f cs * d cs . Alternatively, the normal transmission bandwidth b n between the cloud and the terminal is obtained by using a bandwidth measurement and latency detection tool related to the cloud and the terminal, and the latency l n For example, the network transmission bandwidth is b n , and the network transmission latency is l n , if neither the network transmission bandwidth b n nor the network transmission latency l n satisfies the cloud-terminal virtual scene optimal display precision output condition (normal quality transmission requirement), it is necessary to determine whether they satisfy b n >= b r and l n <= l q1 (i.e., all parameters in the network parameters satisfy the basic quality transmission requirement), if so, the cloud outputs the virtual scene with the lowest service quality (i.e., the lowest service precision is used as the cloud processing precision).

[0167] It can be understood that, since the virtual scene processed by the cloud AR cloud does not occupy the entire screen display range, the bandwidth comparison calculated in the embodiments of the application can have a certain redundancy. The redundancy can refer to that the virtual scene transmission bandwidth calculation process in the application is calculated in the mode that the virtual scene occupies the entire screen display range. However, in actual application, the virtual scene in most AR processing processes can only occupy a part of the screen display. Therefore, the calculation method in the application can have a certain redundancy.

[0168] Step 408, if all or part of the network parameters do not satisfy the basic quality transmission requirement, the service request is not responded.

[0169] Specifically, if all or part of the network parameters do not satisfy the basic quality transmission requirement, the server does not respond to the service request, i.e., if it is confirmed that the current network does not satisfy the minimum service precision transmission requirement, it is determined that the cloud AR related service cannot be provided, and the process is terminated.

[0170] Taking the network parameters including bandwidth and time delay, the display information representing the best display precision of the terminal virtual scene, the service capability information representing the minimum service precision, and the target capability information representing the cloud processing precision as examples, when the bandwidth and time delay of the current network cannot simultaneously meet the transmission requirements of the best display precision of the terminal virtual scene (i.e., the transmission requirements of normal quality), the cloud processing precision can be determined according to different situations: ① the bandwidth and time delay of the current network do not meet the minimum service precision of the cloud virtual scene (i.e., the transmission requirements of basic quality), and thus the cloud AR service that can guarantee the user experience cannot be provided (i.e., the service request is not responded to). ② the bandwidth and time delay of the current network meet the minimum service precision of the cloud virtual scene, and the minimum service precision is determined as the cloud processing precision.

[0171] Exemplarily, taking the network transmission bandwidth as b n , the network transmission time delay as l n , the minimum resolution as r cs , the minimum frame rate as f cs , the minimum depth of field as d cs , the first compression ratio as ct1, and the maximum time delay as l q1 as examples, the second transmission bandwidth b r = r cs * f cs * d cs / ct1, wherein the second product is r cs * f cs * d cs . If the network transmission bandwidth and the network transmission time delay cannot simultaneously meet b n > = b r , l n < = l q1 during the network transmission process, it is determined that the current network transmission cannot meet the provision of the cloud AR service.

[0172] The above-mentioned augmented reality processing method can complete target capability determination. If all parameters in the network parameters meet the transmission requirements of normal quality, the display information is determined as the target capability information. If part of the parameters in the network parameters meet the transmission requirements of normal quality, the display information is determined as the target capability information in the case that all parameters in the network parameters after network enhancement meet the transmission requirements of normal quality. If all parameters in the network parameters do not meet the transmission requirements of normal quality, if all parameters in the network parameters meet the transmission requirements of basic quality, the service capability information is determined as the target capability information. If all or part of the parameters in the network parameters do not meet the transmission requirements of basic quality, the service request is not responded to.

[0173] The embodiment of the application takes real scene precision as the criterion, comprehensively considers terminal screen display definition, refresh rate performance, network bandwidth, time delay, network performance enhancement and other capabilities, adopts cloud AR virtual scene rendering, encoding adjustment and terminal light and shadow adjustment mechanisms based on unified cloud AR service quality, and can provide high-quality cloud AR services by fusing end-to-end multi-dimensional performance.

[0174] Regarding the implementation of server-side rendering and encoding to obtain virtual scene data, the application proposes that an industry-network linkage strategy can be determined: according to the determined cloud processing precision and considering the instability of network transmission and enhancement, corresponding cloud rendering, encoding and network transmission enhancement strategies are formulated to realize real-time fusion and enhancement of network capabilities for cloud AR business requirements and cloud AR business optimization based on the cloud, and release the advantages of network transmission.

[0175] In the process of AR virtual scene rendering, the embodiment of the application can comprehensively determine the cloud processing precision and the industry-network linkage strategy, and flexibly adjust the rendering processing of the virtual scene according to the adjustable conditions of the cloud AR relative application, which can include but is not limited to two rendering methods of terminal virtual scene optimal display precision output and minimum service precision. In the process of cloud AR virtual scene encoding, in addition to the case that the bandwidth capacity has redundancy (indicating that the bandwidth has sufficient redundancy) and the time delay is insufficient, the method of shallow compression is adopted to reduce the cloud processing time delay, so as to optimize the overall service time delay; in other cases, for example, under the condition that the time delay meets certain conditions, the normal encoding processing (such as normal encoding strategy) can be performed.

[0176] For the case that all parameters in the network parameters meet the normal quality transmission requirements, in one embodiment, step 306 can include:

[0177] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, and the network transmission time delay is less than or equal to the recommended time delay, the rendering is performed according to the display information and the encoding is performed according to the first compression ratio to obtain the virtual scene data.

[0178] Specifically, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, and the network transmission time delay is less than or equal to the recommended time delay, it can be determined that all parameters in the network parameters meet the normal quality transmission requirements, and then the display information can be determined as the target capability information, so that the rendering can be performed according to the display information, and the encoding can be performed according to the first compression ratio (i.e. normal encoding strategy) to obtain the virtual scene data.

[0179] It can be understood that, taking the server as an example, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, and the network transmission delay is less than or equal to the recommended delay, it can be determined that the network transmission bandwidth and the network transmission delay of the current network can both meet the terminal virtual scene best display precision transmission requirement, so that network enhancement (such as calling network bandwidth improvement, network acceleration, and link optimization, etc. capability enhancement) is not needed, and the terminal virtual scene best display precision requirement is determined as the cloud processing precision, that is, in the case where network enhancement is not needed, the cloud adopts normal terminal virtual scene best display precision rendering and normal compression ratio (i.e. the compression ratio of normal encoding of the server) encoding.

[0180] For the case where part of the network parameters meet the normal quality transmission requirement, in one of the embodiments, step 306 can include:

[0181] When the network transmission delay is less than or equal to the recommended delay, if the network transmission bandwidth is less than the first transmission bandwidth, network enhancement is performed through bandwidth adjustment, and rendering is performed according to the display information and encoding is performed according to the first compression ratio to obtain virtual scene data.

[0182] Specifically, when the network transmission delay is less than or equal to the recommended delay, it can be determined that the delay meets the requirement by default, and then the bandwidth condition can be judged, when the network transmission bandwidth is less than the first transmission bandwidth, it is determined that the bandwidth does not meet the requirement, network enhancement can be performed through bandwidth adjustment (such as network bandwidth expansion, that is, bandwidth expansion is performed by calling network capability), to improve the network bandwidth to meet the requirement; then, the display information can be determined as the target capability information, and rendering is performed according to the display information, and encoding is performed according to the first compression ratio (i.e. normal rendering and encoding of the cloud), to obtain virtual scene data.

[0183] In some examples, bandwidth adjustment can include improving the access bandwidth capacity of the terminal, and / or improving the egress bandwidth capacity of the server.

[0184] Specifically, bandwidth adjustment can refer to network enhancement through a mode of increasing network bandwidth, wherein increasing network bandwidth can be achieved by modes such as increasing the contracted bandwidth of a telecom operator or temporarily improving bandwidth, etc.

[0185] Optionally, the bandwidth adjustment can include, but is not limited to, increasing the access bandwidth capacity of the terminal and increasing the egress bandwidth capacity of the server. Wherein, the server can first determine the location of the bandwidth bottleneck, and solve the method of increasing the bandwidth capacity. Taking the cloud server as an example, if the bandwidth bottleneck is on the terminal side, it can be solved by increasing the access bandwidth capacity of the terminal; if the bandwidth bottleneck is on the cloud side, it can be solved by increasing the egress bandwidth capacity of the cloud; if the bandwidth bottleneck exists on both the cloud and the terminal, the comprehensive method of increasing the access bandwidth of the terminal and the egress bandwidth capacity of the cloud can be used to solve.

[0186] Regarding the judgment method of the location of the bandwidth bottleneck, exemplarily, it can be judged by comparing the bandwidth on the cloud side and the terminal side with the business demand, or it can be converged to only process the terminal side without the cloud side. Optionally, if the cloud side is involved, the bandwidth required by a single terminal (user) can be calculated, and the data of the terminal used by the current business is calculated to obtain the corresponding cloud demand bandwidth, and then compared with the current cloud actual egress bandwidth to determine whether the bandwidth requirement is met.

[0187] Further, for the case that part of the network parameters meet the normal quality transmission requirement, in one of the embodiments, step 306 can include:

[0188] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and the network transmission delay is less than the maximum delay corresponding to the augmented reality business, a delay adjustment strategy is selected based on the network transmission bandwidth to enhance the network.

[0189] According to the display information, a rendering and encoding strategy linked with the delay adjustment strategy is used to obtain virtual scene data.

[0190] Specifically, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, the server can determine that the bandwidth meets the requirements. If the network transmission delay is greater than the recommended delay and the network transmission delay is less than the maximum delay corresponding to the augmented reality business, the server defaults that the delay does not meet the requirements, and can select a delay adjustment strategy based on the network transmission bandwidth to enhance the network, that is, different processing strategies are selected according to the bandwidth condition. Wherein, the display information can be determined as the target capability information.

[0191] According to the display information, the virtual scene data is obtained by rendering and encoding according to the time delay adjustment strategy. For example, if the bandwidth has sufficient redundancy, the time delay adjustment strategy is determined to first reduce the cloud processing time delay from the service level, and then optimize and reduce the time delay from the network transmission level, wherein high-precision rendering (high-precision rendering can be understood as rendering according to the display information) and shallow compression output are performed. If the bandwidth does not have sufficient redundancy, the time delay is optimized and reduced from the network transmission level, and normal mode rendering and normal encoding output are performed.

[0192] In an exemplary embodiment, the time delay adjustment strategy can include network link selection and network link optimization; according to the display information, the virtual scene data is obtained by rendering and encoding according to the time delay adjustment strategy, including:

[0193] When the network transmission bandwidth is greater than or equal to the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to a second compression ratio; wherein the second compression ratio is less than the first compression ratio;

[0194] When the network transmission bandwidth is less than the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the first compression ratio.

[0195] Specifically, whether the network transmission bandwidth has sufficient redundancy can be determined by the redundancy threshold. When the network transmission bandwidth is greater than or equal to the redundancy threshold, it is determined that the network transmission bandwidth has sufficient redundancy, and then the virtual scene data can be obtained by rendering according to the display information and encoding according to a second compression ratio. When the network transmission bandwidth is less than the redundancy threshold, it is determined that the network transmission bandwidth does not have sufficient redundancy, and then the virtual scene data can be obtained by rendering according to the display information and encoding according to the first compression ratio. The second compression ratio is less than the first compression ratio.

[0196] Taking a server as an example, if the bandwidth has sufficient redundancy, first, cloud high-precision rendering combined with shallow compression with reduced compression rate is used to reduce the cloud processing time delay from the service level, and then network transmission level optimization such as network acceleration / link optimization is used to reduce the time delay. If the bandwidth does not have sufficient redundancy, network transmission acceleration and link optimization are used to reduce the network transmission time delay, and the cloud is rendered and encoded in normal mode. In the embodiment of the application, the network link selection can refer to effectively shortening the actual network transmission time delay by applying network acceleration at the application level; alternatively, the network link optimization can be used to effectively shorten the actual network transmission time delay.

[0197] In one embodiment, the redundancy threshold is a preset multiple of the first transmission bandwidth; and the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0198] Specifically, in the embodiments of the present application, the redundancy threshold can be a preset multiple of the first transmission bandwidth, and the ratio of the first compression ratio to the second compression ratio can be a preset coefficient. The preset multiple and the preset coefficient can be configured as needed, and the preset coefficient can be used to represent the degree of reduction of the compression ratio.

[0199] In one embodiment, the preset multiple is 2 times; and the preset coefficient is the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0200] Specifically, the redundancy threshold can be 2 times the first transmission bandwidth; and the preset coefficient can be the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0201] Taking the network transmission bandwidth b n , the first transmission bandwidth b b , the first compression ratio ct1, and the preset multiple of 2 times as an example, if the network transmission bandwidth b n of the current network is greater than or equal to 2b b , the cloud encoding compression ratio can be reduced to reduce the bandwidth consumption and the delay, the cloud virtual scene processing delay can be reduced first, and then network acceleration and link optimization can be used to reduce the network delay (i.e., network link selection and network link optimization).

[0202] In this way, the degree of reduction of the compression ratio n (i.e., the preset coefficient) can be determined according to the relationship between the current network bandwidth and the bandwidth multiple required for the best display precision of the terminal virtual scene, n = b n / b b , where n is an integer. Then, the compression processing is performed according to the adjusted compression ratio: the original normal compression ratio ct1 is reduced by n times to obtain the second compression ratio cr1 = ct1 / n in the implementation process, so as to reduce the cloud encoding compression complexity and the corresponding delay of the cloud processing.

[0203] If the network transmission bandwidth of the current network cannot satisfy b n <2b b , the transmission delay needs to be reduced by network enhancement (i.e., network link selection and network link optimization), and the following methods can be used:

[0204] First, network delay optimization is performed based on the application layer (for example, network link selection), and then network link optimization is used to further improve the delay capability. For the application layer, a number of acceleration servers in the middle link can be used as a relay, and by selecting the optimal relay acceleration server, a more optimal network transmission link can be found at the application layer, thereby reducing network transmission delay and achieving network acceleration. For network link optimization, network capabilities can be connected, and a network shortest path algorithm or a multi-factor routing algorithm can be used to obtain a more optimal network path, thereby reducing the delay of cloud AR virtual scene transmission.

[0205] The above-mentioned augmented reality processing method effectively reduces the difficulty and delay of cloud processing based on current network bandwidth conditions, based on lossless or shallow compression and other low compression encoding mechanisms, fully utilizes the advantages of large bandwidth transmission, ensures high-quality picture output of cloud AR, and provides more optimized delay experience.

[0206] For the case where all parameters in the network parameters meet the transmission requirements of the basic quality, in one embodiment, the virtual scene data is obtained by rendering and encoding according to the target capability information, including:

[0207] The virtual scene data is obtained by rendering according to the service capability information and encoding according to the first compression ratio.

[0208] Specifically, when the network transmission bandwidth is greater than or equal to the second transmission bandwidth, and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, it can be determined that all parameters in the network parameters meet the transmission requirements of the basic quality, and then the service capability information is determined as the target capability information.

[0209] Taking the case where the network parameters include bandwidth and delay, the display information represents the best display precision of the terminal virtual scene, the service capability information represents the minimum service precision, and the target capability information represents the cloud processing precision as an example, the bandwidth and delay of the current network do not meet the requirements of the best display precision of the terminal virtual scene, and the bandwidth and delay of the current network can simultaneously meet the network transmission requirements of the minimum service precision output, it is determined that the minimum service precision is used as the cloud processing precision, and then the cloud uses the minimum service precision virtual scene rendering (rendering according to the service capability information) and the normal encoding strategy (encoding according to the first compression ratio), and does not perform corresponding network enhancement processing to obtain the virtual scene data.

[0210] In the above, in the case of obtaining virtual scene data, the server can perform virtual scene result delivery: after completing cloud AR virtual scene rendering and flexible encoding compression in the cloud, the corresponding cloud AR virtual scene processing result is delivered to the cloud AR terminal side. The terminal receives the cloud AR virtual scene processing result, and adjusts the brightness according to the real scene lighting conditions, completes the cloud AR virtual-real fusion output, and performs corresponding interactive operations based on the virtual-real fusion result.

[0211] The above-mentioned augmented reality processing method takes the real scene acquisition precision as the benchmark and target, determines the best virtual scene output result (i.e., the best virtual scene display precision of the terminal) that the terminal can achieve according to the terminal processing capability; and taking the best virtual scene output of the terminal as the target, the cloud end obtains the best virtual scene result of the user experience according to the network transmission performance and the augmented attribute combined with flexible virtual scene rendering and adaptive coding; the terminal side obtains the best virtual-real fusion result optimized and guaranteed by the cloud AR terminal through obtaining the virtual scene result, based on the real scene for real-time brightness adjustment, and realizes the cloud AR virtual-real high-precision seamless connection by combining the capabilities of the cloud and network ends.

[0212] In one exemplary embodiment, as shown in FIG. 5, an augmented reality processing method is provided, which is applied to the terminal in FIG. 1 as an example for illustration, including the following steps 502 to 504. Among them:

[0213] Step 502, in the case of obtaining real scene data, sending a service request to the cloud AR cloud end, and sending the obtained display information to the cloud AR cloud end; the service request is used to instruct the cloud AR cloud end to determine the virtual scene data corresponding to the service request according to the display information and the network transmission condition; wherein the display information represents the display capability of the terminal for the virtual scene data;

[0214] Specifically, the service request can be an AR service request; exemplary, the terminal can initiate a cloud AR service request and real scene data acquisition; taking the terminal as a cloud AR terminal as an example, the cloud AR terminal as the request initiator and the final service display presenter of the cloud AR service can start the related service request and synchronously perform the related real scene data acquisition processing.

[0215] Optionally, the terminal can send a service request in the case of obtaining real scene data. For example, the terminal can initiate a service request to the server in the case of obtaining real scene data. Taking the service request as a cloud AR service request as an example, the terminal can initiate a cloud AR service request and real scene acquisition, for example, the terminal starts the related service request and synchronously performs the real scene data acquisition.

[0216] The server receives the service request sent by the terminal and determines that virtual scene processing is needed. For example, in response to receiving the service request, the server can determine the virtual scene data corresponding to the service request according to the display information of the terminal, which can represent the display capability of the terminal for the virtual scene data. For example, the display information of the terminal can be obtained by the terminal in the cloud AR cloud network service cooperation stage.

[0217] For example, the terminal can send the obtained display information to the server, so that the server determines the cloud virtual scene processing based on the display information. The server can obtain the display information of the terminal, for example, the virtual scene best display precision of the terminal. For example, the terminal can obtain the display information and send the obtained display information to the server, so as to achieve the best virtual-real fusion output.

[0218] In some examples, the display information is used to represent the virtual scene best display precision of the terminal, which can be understood as the virtual scene best display precision of the terminal. For example, the virtual scene best display precision of the terminal can refer to the virtual scene output result with the best user experience of the terminal, which is determined based on the real scene collection precision as a benchmark and target and the processing capability of the terminal. Further, the process of determining the virtual scene data by the server according to the display information of the terminal can be understood as that the server targets the virtual scene output with the best user experience of the terminal, performs flexible virtual scene rendering and adaptive coding in the cloud, and obtains the virtual scene result with the best user experience of the cloud, so as to solve the problem of seamless connection of cloud AR virtual-real high precision by means of the cloud network capability and end-to-end optimization. Based on the embodiments of the present application, the best virtual-real fusion result can be finally obtained.

[0219] Step 504: receiving the virtual scene data sent by the cloud AR cloud.

[0220] Specifically, after obtaining the virtual scene data, the server can send the virtual scene data to the terminal. After receiving the virtual scene data sent by the server, the terminal can output the virtual-real fusion result based on the virtual scene data and the collected real scene data.

[0221] In the above-mentioned augmented reality processing method, based on the display capability of the terminal, the network transmission condition is used to enable the cloud to adopt a more flexible virtual scene processing mechanism, so as to achieve high-quality cloud virtual scene processing output and realize end-to-end high-quality user experience and high-quality service output of cloud AR.

[0222] In one of the embodiments, the method can further include:

[0223] Outputting a virtual-real fusion result based on the virtual scene data and the real scene data.

[0224] Specifically, outputting the virtual-real fusion result can refer to enabling the virtual digital capabilities related to the photographed real scene to cooperate with subsequent other virtual image addition and interaction. For example, based on the virtual scene data and the real scene data, the terminal can superimpose the real scene picture and the virtual scene picture to form a reality augmented picture with the real scene picture as the bottom picture and the virtual scene picture as the top picture. It should be understood that the embodiments of the present disclosure are not limited thereto, and in other implementations, the terminal can also superimpose the real scene picture and the virtual scene picture with the virtual scene picture as the bottom picture and the real scene picture as the top picture.

[0225] Further, taking the server as a cloud server as an example, the cloud server will issue the virtual scene result (virtual scene data) after processing to the terminal, the terminal receives the virtual scene result, and performs light and shadow adjustment (such as brightness corresponding adjustment) according to the real scene lighting condition, completes the cloud AR virtual-real fusion output, and performs corresponding interactive operation based on the virtual-real fusion result. For another example, taking the terminal as an AR terminal as an example, cloud AR virtual-real fusion output and interaction can be realized, the AR terminal receives the virtual scene issued by the cloud, and performs corresponding highlight adjustment based on the locally acquired real scene, so as to combine with the real scene collected locally, complete the virtual-real fusion cloud AR service terminal display output, and perform related cloud AR interaction based on the output result.

[0226] In one of the embodiments, the method can further include:

[0227] Determining the display information based on the screen display parameter and the data acquisition parameter for acquiring the real scene data.

[0228] Specifically, the terminal can determine the display information based on the data acquisition parameter for acquiring the real scene data and the screen display parameter, and transmit the display information to the server. Exemplarily, the data acquisition parameter can refer to the related parameter of the real scene acquisition device of the terminal, which is used to represent the real scene acquisition accuracy (referred to as real scene accuracy), for example, the terminal can acquire the real scene accuracy through the real scene acquisition device, including but not limited to resolution, frame rate, etc. Wherein, the real scene acquisition device can include but is not limited to a camera, a light-transmitting lens, etc.

[0229] Further, the screen display parameter can represent the terminal screen display capability (which can be understood as the terminal screen display accuracy), including but not limited to resolution, frame rate, etc. Taking the display information representing the terminal virtual scene best display accuracy as an example, the terminal can obtain the terminal virtual scene best display accuracy by comprehensively considering the real scene acquisition accuracy and the terminal screen display capability, so as to realize the best virtual-real fusion output.

[0230] Exemplarily, the terminal can determine the best display precision of the virtual scene of the terminal based on the real scene collection precision and in combination with the screen display capability of the terminal. For example, the terminal can compare the real scene collection precision with the screen display capability of the terminal to determine the best display precision of the virtual scene of the terminal: ① when the real scene collection precision >= the screen display capability of the terminal, the screen display capability of the terminal is determined as the best display precision of the virtual scene of the terminal; ② when the real scene collection precision < the screen display capability of the terminal, the real scene collection precision is determined as the best display precision of the virtual scene of the terminal. By selecting the real scene collection precision as the best display precision of the virtual scene of the terminal, seamless connection with the corresponding real scene can be achieved, and the best virtual-real fusion output can be realized.

[0231] In one of the embodiments, the data collection parameters include real scene collection resolution, real scene collection frame rate and real scene collection depth of field; the screen display parameters include terminal display resolution, terminal display frame rate and terminal display depth of field.

[0232] The display information includes target resolution, target frame rate and target depth of field; wherein the target resolution is the minimum value of the real scene collection resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene collection frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene collection depth of field and the terminal display depth of field.

[0233] Specifically, the terminal can obtain data collection parameters representing real scene collection precision and screen display parameters representing screen display capability. The data collection parameters can include real scene collection resolution, real scene collection frame rate and real scene collection depth of field; the screen display parameters can include terminal display resolution, terminal display frame rate and terminal display depth of field; and the display information can include target resolution, target frame rate and target depth of field. The target resolution is the minimum value of the real scene collection resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene collection frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene collection depth of field and the terminal display depth of field.

[0234] Exemplarily, the terminal can obtain the relevant resolution, frame rate and depth of field of the current real scene collection device of the terminal as the real scene collection precision, such as the resolution of the camera and the frame rate of the real scene shooting. If the real scene perspective mode is used for acquisition, the real scene collection precision can be defaulted as the real scene precision obtained by the human eye. Further, the terminal can obtain the screen display capability, i.e., obtain the relevant resolution, frame rate and other indicators of the screen display of the terminal.

[0235] In practical applications, taking a terminal as an example of a cloud AR terminal, the real scene acquisition accuracy of the cloud AR terminal is obtained in the cloud AR cloud network side service cooperation stage, including a camera or a light transmission lens of the cloud AR terminal for collecting a real scene, etc. The camera directly acquires its pixel resolution, a shooting corresponding frame rate, a depth of field, etc. The light transmission lens is the accuracy of human observation by default. Assuming that the real scene acquisition resolution of the cloud AR terminal is rl (real scene acquisition resolution), the frame rate thereof is fl (real scene acquisition frame rate), and the depth of field thereof is dl (real scene acquisition depth of field), the terminal screen display accuracy of the cloud AR terminal, including the resolution, the frame rate, and the depth of field of the screen display of the cloud AR terminal, are assumed to be r2 (terminal display resolution), f2 (terminal display frame rate), and d2 (terminal display depth of field), respectively.

[0236] The real scene acquisition accuracy of the cloud AR terminal and the terminal screen display accuracy of the cloud AR terminal are compared to obtain the virtual scene best display accuracy of the cloud AR terminal: r t (target resolution), f t (target frame rate), and d t (target depth of field). The related rules are as follows: the virtual scene best display accuracy of the cloud AR terminal, wherein the resolution r t = Min (rl, r2), the frame rate f t = Min (fl, f2), and the depth of field d t = Min (dl, d2).

[0237] It can be understood that when the real scene data is acquired by using the perspective mode, in one of the embodiments, the display information is determined based on the screen display parameters and the data acquisition parameters of the acquired real scene data, including:

[0238] When the real scene data is obtained by the perspective mode, the screen display parameters are determined as the display information.

[0239] Specifically, the perspective mode can mean that the terminal acquires the real scene data by using a real scene perspective mode. It should be noted that the real scene perspective mode includes but is not limited to a perspective lens mode, for example, the corresponding real-time real scene is obtained by using a light transmission lens.

[0240] In the case where the real scene data is obtained by the terminal through the perspective mode, the terminal can determine the screen display parameters as the display information. Taking a terminal as an example of a cloud AR terminal, if the cloud AR real scene acquisition adopts the perspective lens mode, the real scene accuracy of the cloud AR terminal is greater than the terminal screen display accuracy of the cloud AR terminal by default, that is, in this case, the resolution, the frame rate, and the depth of field of the virtual scene best display accuracy of the cloud AR terminal are equal to r2, f2, and d2, respectively. Further, the server can acquire and analyze the virtual scene best display accuracy of the terminal: the server acquires the virtual scene best display accuracy of the terminal by interfacing with the terminal, and correspondingly acquires the resolution, the frame rate, and the depth of field thereof, and uses the same as a benchmark to determine the cloud virtual scene processing.

[0241] It should be noted that in the augmented reality processing method executed from the terminal, as to the steps implemented from the server, reference can be made to the description in the augmented reality processing method executed from the server in the foregoing, which will not be repeated here.

[0242] In the augmented reality processing method, according to the precision conditions of the corresponding resolution, frame rate, depth of field, etc. of the real scene collected on the terminal side, in combination with the cloud AR terminal display / processing performance, based on the bandwidth, time delay, etc. of the current network transmission, while considering the network acceleration, bandwidth expansion, link optimization, etc. enhancement capabilities, the cloud AR virtual scene processing service quality is determined, the cloud virtual scene is flexibly rendered, the encoding compression ratio, etc. encoding mode is flexibly adjusted, the virtual scene encoding is completed, and the processing result is issued to the terminal side; the terminal side receives the cloud AR virtual scene result, and combines the brightness adjustment of the local virtual scene, and finally realizes the localized output of the cloud AR virtual-real fusion.

[0243] In order to further explain the scheme of the present application, a specific example will be described below, taking the terminal as a cloud AR terminal and the server as a cloud AR cloud as an example, as shown in FIG. 6, the functions possessed by the cloud AR terminal and the cloud AR cloud can include but are not limited to:

[0244] As to the cloud AR terminal, as the request initiator of the cloud AR service and the final service display presenter, it can have the following functions:

[0245] ① Real scene precision acquisition (i.e. real scene precision collection): the terminal acquires the real scene precision that can be obtained by the real scene collection device, which can include resolution, frame rate and depth of field, etc.

[0246] ② Terminal display capability acquisition: the display capability of the terminal is collected, including resolution, frame rate and depth of field, etc.

[0247] ③ Terminal virtual scene best display precision acquisition: the terminal virtual scene best display precision is obtained by comprehensively considering the real scene precision and the terminal display capability, including resolution, frame rate and depth of field, etc.

[0248] When the real scene precision >= terminal display capability: the terminal display capability is selected as the terminal virtual scene best display precision;

[0249] When the real scene precision < terminal display capability: the real scene precision is selected as the terminal virtual scene best display precision, so as to be seamlessly connected with the corresponding real scene and realize the best virtual-real fusion output.

[0250] ④ Initiate cloud AR service request and real scene collection: the cloud AR terminal starts the related service request and synchronously collects and processes the related real scene.

[0251] ⑤Virtual and real output and interaction: The AR terminal will receive the virtual scene data issued by the cloud, and based on the local real scene, it will adjust the corresponding highlights to combine with the real scene collected locally, complete the virtual and real fusion of the cloud AR service terminal display output, and perform related cloud AR interaction based on the output result.

[0252] ⑥Network performance acquisition and analysis: In cooperation with the cloud AR cloud, the current transmission bandwidth, delay and other performance of the corresponding network are acquired.

[0253] Regarding the cloud AR cloud, as the executor of the virtual scene processing of cloud AR, it mainly completes the rendering, compression encoding and other processing of the virtual scene based on the network cooperation ability, and can have the following functions:

[0254] ①Virtual scene best display precision acquisition and analysis: for the terminal side, the terminal virtual scene best display precision is acquired, and the resolution, frame rate and depth are acquired accordingly, which are used as a reference to determine the cloud virtual scene processing.

[0255] ②Network performance acquisition and analysis: the bandwidth, delay and other related performance of the corresponding network transmission are acquired, and the network bandwidth improvement, transmission acceleration and link optimization and other enhancement capabilities are enhanced.

[0256] ③Determine the minimum service precision of virtual scene: according to the minimum requirement of cloud AR business user experience, the minimum service quality of the business is determined, and the minimum resolution, frame rate and depth and other indicators of cloud output are determined.

[0257] ④Determine the cloud processing precision: according to the terminal virtual scene best display precision and the user's better interactive delay experience, the corresponding cloud processing precision is determined according to different combination methods.

[0258] When the network transmission bandwidth and delay of the current network can meet the transmission requirements of the terminal virtual scene best display precision, there is no need to consider calling network bandwidth improvement, network acceleration and link optimization and other capability enhancement, and the terminal virtual scene best display precision requirement is determined as the cloud processing precision.

[0259] When one of the network transmission bandwidth and delay of the current network cannot meet the transmission requirements of the terminal virtual scene best display precision, such as through bandwidth improvement, network acceleration and link optimization and other network transmission enhancement to meet the transmission requirements, the terminal virtual scene best display precision requirement is determined as the cloud processing precision.

[0260] When the network transmission bandwidth and latency of the current network cannot simultaneously meet the transmission requirements of the cloud terminal virtual scene best display precision output, the cloud processing precision can be determined according to different situations: if the bandwidth and latency of the current network do not meet the minimum service precision of the cloud virtual scene, the cloud AR service that guarantees user experience cannot be provided. If the bandwidth and latency of the current network meet the minimum service precision of the cloud virtual scene, the minimum service precision is determined as the cloud processing precision.

[0261] ⑤Determine the network linkage strategy: according to the determined cloud processing precision, considering the instability of network transmission and enhancement, corresponding cloud rendering, coding strategy and network transmission enhancement strategy are formulated.

[0262] In the case that the network transmission bandwidth and latency of the current network can meet the transmission requirements of the terminal virtual scene best display precision, the cloud performs normal cloud processing precision, that is, the terminal virtual scene best display precision for graphic rendering and coding.

[0263] In the case that one of the network transmission bandwidth and latency of the current network cannot meet the transmission requirements of the terminal virtual scene best display precision, the cloud can perform rendering and coding according to the determined cloud processing precision (that is, the terminal virtual scene best display precision) according to different situations:

[0264] When the bandwidth does not meet the requirements, bandwidth expansion is performed by calling the network capability;

[0265] When the latency does not meet the requirements, if the bandwidth is sufficient (that is, the bandwidth has sufficient redundancy), the cloud outputs more than twice the required bandwidth at the terminal virtual scene best display precision, first renders at the highest precision based on the cloud (that is, renders at the terminal virtual scene best display precision), and adopts a shallow compression mode to shorten the compression latency, while reducing network transmission adjustment through network acceleration or link optimization to improve the corresponding transmission latency; if the bandwidth cannot meet the requirements (that is, the bandwidth does not have sufficient redundancy), the cloud outputs more than twice the required bandwidth at the terminal virtual scene best display precision, and reduces network transmission adjustment through network acceleration or link optimization to improve the corresponding transmission latency.

[0266] In the case that the network transmission bandwidth and latency of the current network cannot simultaneously meet the transmission requirements of the cloud terminal virtual scene best display precision output, if the network transmission bandwidth and latency of the current network can simultaneously meet the network transmission requirements of the cloud virtual scene minimum service precision output, the strategy of rendering and normal coding processing based on the current network transmission through the cloud at the minimum service precision is adopted.

[0267] ⑥Cloud AR virtual scene rendering: the determined cloud processing accuracy and business linkage strategy are comprehensively determined, and the rendering processing of the virtual scene is flexibly adjusted according to the adjustable situation of the cloud AR relative application, including two rendering methods of terminal virtual scene best display accuracy output and minimum service accuracy. Among them, the terminal virtual scene best display accuracy output corresponds to the case that all or part of the network transmission bandwidth and delay of the current network meet the transmission requirements of the cloud terminal virtual scene best display accuracy output; the minimum service accuracy corresponds to the case that the network transmission bandwidth and delay of the current network can simultaneously meet the network transmission requirements of the minimum service accuracy output of the cloud virtual scene.

[0268] ⑦Cloud AR virtual scene encoding: in addition to the case that there is redundancy in bandwidth capacity and there is a shortage of delay, a shallow compression method can be used to reduce the cloud processing delay, so as to optimize the overall service delay; in other cases, under the condition that the delay meets certain conditions, the conventional encoding processing can be performed.

[0269] ⑧Virtual scene result delivery: after the cloud AR virtual scene rendering and flexible encoding compression are completed in the cloud, the corresponding cloud AR virtual scene processing result is delivered to the cloud AR terminal side.

[0270] The above-mentioned augmented reality processing method is oriented to the best screen display capability of the cloud AR terminal, and the multi-dimensional enhanced cloud AR virtual scene processing quality of the cloud network side; the terminal virtual scene best display accuracy is determined based on the real scene collection accuracy of the terminal side and in combination with the terminal screen display capability; and based on the terminal virtual scene best display accuracy, in combination with the current bandwidth, delay transmission performance of the network, and in combination with the network acceleration, link optimization and other enhancement capabilities, the cloud virtual scene processing accuracy and network cooperation strategy are obtained, and the cloud virtual scene graphic rendering processing, encoding compression adjustment, network transmission enhancement and the like are completed based on the cloud virtual scene processing accuracy and network cooperation strategy; after the terminal side obtains the delivered virtual scene, the virtual scene brightness processing is completed according to the actual acquired illumination, and finally the high-quality cloud AR virtual-real fusion output is completed, effectively improving the user experience quality.

[0271] Further, the implementation process of the embodiment of the application can include two stages of cloud AR cloud network side business cooperation and cloud AR business operation; wherein the cloud AR cloud network side business cooperation stage is a preparatory stage of the cloud AR business operation stage, and the specific process can be as shown in FIG. 7, and the process of the cloud AR business operation stage can be as shown in FIG. 8.

[0272] As shown in FIG. 7, the cloud AR cloud network side business cooperation stage can include:

[0273] 1) The terminal side acquires the real scene collection accuracy and the terminal screen display capability.

[0274] 1.1 terminal acquires real scene collection accuracy, by obtaining relevant resolution, frame rate, depth of field, etc. of the terminal's current real scene collection device, such as the resolution of the camera and the frame rate of the real scene shooting; if real scene perspective mode is adopted, the accuracy is defaulted as the real scene accuracy acquired by human eyes;

[0275] 1.2 terminal acquires terminal screen display capability, i.e. obtains relevant resolution, frame rate, etc. of the terminal screen display index;

[0276] 2) terminal side real scene collection accuracy and terminal screen display capability comparison, and determination of terminal virtual scene best display accuracy

[0277] 2.1 when real scene collection accuracy >= screen display capability, the terminal screen display capability is determined as the terminal virtual scene best display accuracy;

[0278] 2.2 when real scene collection accuracy < screen display capability, the real scene collection accuracy is determined as the terminal virtual scene best display accuracy.

[0279] 3) cloud simultaneously determines virtual scene minimum service accuracy, acquires and analyzes virtual scene best display accuracy, and obtains network transmission performance of cloud and terminal linkage

[0280] 3.1 determination of virtual scene minimum service accuracy: in combination with the terminal virtual scene best display accuracy and the accuracy range of virtual scene processing of cloud AR business, the virtual scene minimum service accuracy is determined.

[0281] 3.2 acquisition and analysis of virtual scene best display accuracy: the virtual scene best display accuracy sent by the terminal is acquired and decomposed into resolution, frame rate, depth of field, etc., and the required bandwidth requirement is converted according to the compression rate corresponding to the commonly used coding protocol.

[0282] 3.3 cloud and terminal linkage obtain network transmission performance and set delay requirement: through the network connection between the terminal and the cloud, the corresponding network transmission bandwidth and delay between the two are obtained, and the corresponding delay requirement is set according to the user experience characteristics of cloud AR business.

[0283] 4) in combination with the current network transmission performance, whether the corresponding cloud processing requirement is met is judged through the network transmission bandwidth and delay of the current network, so that the cloud determines the corresponding processing accuracy.

[0284] 4.1 the current network meets the terminal virtual scene best display accuracy requirement, and the terminal virtual scene best display accuracy is determined as the cloud processing accuracy.

[0285] 4.2 one of the current network bandwidth and delay does not meet the requirement, but after network enhancement, the terminal virtual scene best display accuracy requirement can be met, and the terminal virtual scene best display accuracy is determined as the cloud processing accuracy, and related industry network capability linkage processing is required.

[0286] 4.3 If neither the current network bandwidth nor the current network latency meets the terminal virtual scene best display accuracy requirement, and the current network can meet the minimum service accuracy requirement, then the minimum service accuracy is determined as the cloud processing accuracy.

[0287] 4.4 If the current network does not meet the minimum service accuracy transmission requirement, it is determined that the cloud AR related service cannot be provided and the process is terminated.

[0288] 5) According to the corresponding determined cloud processing accuracy, the corresponding cloud processing and service linkage strategy is determined according to different situations:

[0289] 5.1 Corresponding to 4.1, no network enhancement is required, and the cloud uses normal terminal virtual scene best display accuracy rendering and normal compression ratio coding.

[0290] 5.2 Corresponding to 4.2, first determine the bandwidth condition, and optimize the related latency according to the different bandwidth conditions

[0291] 5.2.1 If the bandwidth does not meet the requirement (the default latency meets the requirement), the network bandwidth expansion (i.e. network bandwidth improvement) is used to meet the strategy of normal rendering and coding of the cloud.

[0292] 5.2.2 If the bandwidth meets the requirement, the default latency does not meet the requirement, and different processing strategies are selected according to the bandwidth condition.

[0293] 5.2.2.1 If the bandwidth has sufficient redundancy, i.e. meets the transmission of the terminal virtual scene best display accuracy of more than 2 times, first use cloud high-precision rendering (referred to as high-precision rendering) + reduce the compression ratio shallow compression to reduce the cloud processing latency from the service level, and then use network transmission level optimization such as network acceleration / link optimization to reduce the latency.

[0294] 5.2.2.2 If the bandwidth does not have sufficient redundancy, network transmission acceleration, link optimization is used to reduce network transmission latency, and the cloud is normally rendered and normally coded.

[0295] 5.3 Corresponding to 4.3, the minimum service accuracy virtual scene rendering and normal coding strategy is used, and no corresponding network enhancement processing is performed.

[0296] 6, According to the above, the corresponding cloud processing accuracy, cloud processing and service linkage strategy are determined, so as to complete the corresponding cloud AR cloud network service cooperation related process.

[0297] Further, the cloud AR service running process is initiated by the cloud AR terminal, and the process ends when the cloud AR terminal realizes virtual-real fusion output and interaction, which is based on the premise that the virtual scene best display precision is determined on the terminal, the processing precision is determined on the cloud, and the relevant industry network linkage strategy is determined, that is, the cloud AR cloud network end service collaboration process is completed.

[0298] As shown in FIG. 8, the cloud AR service running stage can include:

[0299] 1) The terminal initiates a cloud AR service request and real scene collection: the terminal initiates a cloud AR service request to the cloud, and synchronously obtains the corresponding real-time real scene through a camera, a light-transmitting lens, etc.

[0300] 2) The cloud determines the cloud processing precision and the industry network linkage strategy in the cloud AR cloud network end service collaboration stage, and completes virtual scene rendering.

[0301] 3) The cloud performs encoding and network enhancement processing according to the industry network linkage strategy in the cloud AR cloud network end service collaboration stage.

[0302] 4) The cloud sends the virtual scene result of the completed processing to the terminal.

[0303] 5) The terminal receives the virtual scene result, adjusts the brightness according to the real scene lighting condition, completes the cloud AR virtual-real fusion output, and performs corresponding interaction operation based on the virtual-real fusion result.

[0304] The above-mentioned augmented reality processing method determines the real scene collection precision based on the resolution, frame rate, depth of field, etc. of the terminal real scene collection; at the same time, the terminal screen display precision such as screen display resolution and refresh rate is combined to obtain the best display precision of the virtual scene that can be processed on the terminal side. The cloud determines the cloud processing precision and the industry network linkage strategy based on the target of meeting the best display of the virtual scene that can be processed on the terminal side, the factors such as network transmission bandwidth and time delay, and the enhancement processing capabilities such as network transmission acceleration and link optimization, as well as the minimum service precision of the cloud AR service real scene; then, according to the cloud processing precision and the industry network linkage strategy, the cloud performs graphic rendering and virtual scene encoding mode with adaptive control compression ratio to obtain the virtual scene result output by the cloud; at the same time, according to the industry network linkage strategy, the network transmission enhancement such as network bandwidth expansion, network application acceleration and link optimization is performed for the virtual scene. After the terminal side obtains the corresponding virtual scene result, the brightness of the virtual scene is adjusted according to the real-time light and shadow of the real scene, and finally the corresponding virtual-real fusion output is completed.

[0305] The application aims to improve user cloud AR experience and realize seamless connection between cloud AR virtual scene and real scene. A cloud network end multi-dimensional capability fusion mode is adopted, and the final virtual-real fusion effect is taken as the guide to build a high-quality solution for virtual-real fusion of cloud AR end-to-end. Through cloud network end capability fusion, the cloud AR high-quality cloud virtual scene output is ensured. The cloud virtual scene processing precision is built based on the best display precision of the terminal side, and the network transmission conditions and the appropriate enhancement when insufficient are fused. At the same time, the coding compression ratio optimization in the high-bandwidth environment is adopted to improve the processing effect of the cloud virtual scene. The application can strengthen the network capability supply and improve the service value of cloud AR typical business. By integrating the bandwidth and time delay capabilities of the network into the cloud AR business optimization, the network value can be more efficiently reflected, the network capability supply can be improved, and the possibility of releasing new network service market and new space can be realized. In addition, based on the industry network linkage strategy, the cloud AR business time delay experience can be improved in multiple dimensions. On the one hand, network acceleration enhancement is carried out at the application layer of network transmission, and network link optimization is assisted to improve the transmission time delay. At the same time, the coding compression ratio adjustment based on high bandwidth in the cloud business processing process is fused, and the multiple industry network linkage mechanism for reducing cloud processing time consumption is realized. The end-to-end optimization of cloud AR virtual scene processing and network transmission time delay effectively improves the interactive experience effect of cloud AR business.

[0306] Further, based on the best display precision of the cloud AR terminal virtual scene as the guide, the network current transmission performance and network capability enhancement attribute are combined to determine the cloud virtual scene processing precision. In order to meet the cloud processing precision output, a cloud network end business multi-dimensional cooperative processing mechanism (i.e. industry network linkage strategy) is adopted. Based on flexible rendering, coding processing and network capability enhancement linkage, high-quality output of the cloud AR virtual scene is realized, and the best fusion output with the real scene is achieved.

[0307] The application provides a cloud network end business multi-dimensional coordination and end-to-end cloud AR virtual scene high-quality processing mechanism. The terminal real scene acquisition precision and terminal screen display capability are determined to determine the terminal virtual scene best display precision. The network bandwidth requirement is calculated according to the terminal virtual scene best display precision at the cloud end, and the cloud virtual scene processing precision and cloud network end business cooperative mechanism are determined in combination with the current network transmission performance. Based on the cloud network end business system mechanism, the cloud virtual scene high-quality output is ensured through the cloud flexible coding, network transmission enhancement fusion mode.

[0308] Moreover, the application realizes industry network capability linkage and improves the cloud AR virtual scene cloud processing effect. The business, network and other capabilities are integrated into the virtual scene enhancement processing process. According to the bandwidth redundancy condition, the coding compression ratio is reduced, and the cloud processing time delay is reduced. Through network bandwidth expansion, application layer network acceleration and network link optimization, the network transmission performance is enhanced, and the higher quality output is maximized.

[0309] In addition, the application completes end-to-end capability linkage, and improves the cloud AR virtual-real fusion effect: the terminal determines the optimal display precision of the virtual scene according to the real scene collection precision and the terminal display capability; the cloud end improves the processing effect of the virtual scene by means of service and network capability cooperation, so as to achieve the final optimal display precision output of the virtual scene; the terminal side adjusts the brightness of the virtual scene issued by the cloud end according to the actual light and shadow, and maximizes the high-quality fusion output of the cloud AR real scene and the virtual scene.

[0310] It should be understood that, although the steps in the flowcharts involved in the above-described embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowcharts involved in the above-described embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0311] Based on the same inventive concept, the application also provides an augmented reality processing device for implementing the above-mentioned augmented reality processing method. The implementation scheme of the problem solving provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more augmented reality processing device embodiments provided below can refer to the limitations of the augmented reality processing method in the above text, and will not be repeated here.

[0312] In an exemplary embodiment, as shown in FIG. 9, an augmented reality processing device is provided, which is applied to the virtual scene rendering function of the cloud AR cloud end, and the device comprises:

[0313] The request receiving module 901 is configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case where real scene data is acquired;

[0314] The virtual data determining module 902 is configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information represents the display capability of the terminal for the virtual scene data;

[0315] The data sending module 903 is configured to send the virtual scene data to the terminal.

[0316] In one of the embodiments, the display information is determined based on a terminal-based screen display parameter and a data collection parameter for collecting real scene data by the terminal; the data collection parameter includes a real scene collection resolution, a real scene collection frame rate and a real scene collection depth of field; the screen display parameter includes a terminal display resolution, a terminal display frame rate and a terminal display depth of field; the display information includes a target resolution, a target frame rate and a target depth of field; the target resolution is the minimum value of the real scene collection resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene collection frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene collection depth of field and the terminal display depth of field. In the case that the real scene data is obtained by the terminal in a see-through mode, the display information is the screen display parameter.

[0317] In one of the embodiments, the virtual data determination module 902 includes:

[0318] a parameter acquisition module, configured to acquire, by the virtual scene rendering function, a network parameter for communicating with the terminal by using a current network; the network parameter is a parameter representing a network transmission performance of the current network;

[0319] a target information determination module, configured to determine the display information or the service capability information as the target capability information according to whether the network parameter meets a normal quality transmission requirement required by the display information and whether the network parameter meets a basic quality transmission requirement required by the service capability information; the basic quality transmission requirement is lower than the normal quality transmission requirement; and the service capability information represents a minimum capability supported by the virtual scene rendering function for the augmented reality service;

[0320] a rendering and encoding module, configured to perform rendering and encoding according to the target capability information to obtain the virtual scene data.

[0321] In one of the embodiments, the target information determination module is configured to determine the display information as the target capability information if all the parameters in the network parameter meet the normal quality transmission requirement; determine the display information as the target capability information in the case that it is confirmed that all the parameters in the network parameter after network enhancement can meet the normal quality transmission requirement when part of the parameters in the network parameter meet the normal quality transmission requirement; determine the service capability information as the target capability information if all the parameters in the network parameter do not meet the normal quality transmission requirement and all the parameters in the network parameter meet the basic quality transmission requirement; and not respond to the service request if all or part of the parameters in the network parameter do not meet the basic quality transmission requirement.

[0322] In one of the embodiments, the network parameter includes a network transmission bandwidth and a network transmission time delay; and the display information includes a target resolution, a target frame rate and a target depth of field; wherein:

[0323] When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, it is confirmed that the network transmission bandwidth meets the transmission requirement of normal quality.

[0324] When the network transmission delay is less than or equal to the recommended delay corresponding to the augmented reality service, it is confirmed that the network transmission delay meets the transmission requirement of normal quality.

[0325] The first transmission bandwidth is a quotient of a first product and a first compression ratio; the first product is a product of the target resolution, the target frame rate and the target depth of field; and the first compression ratio represents a compression ratio of normal encoding of the server.

[0326] In one of the embodiments, the rendering and encoding module is configured to, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission delay is less than or equal to the recommended delay, render according to the display information and encode according to the first compression ratio to obtain the virtual scene data.

[0327] In one of the embodiments, the rendering and encoding module is configured to, when the network transmission delay is less than or equal to the recommended delay, if the network transmission bandwidth is less than the first transmission bandwidth, perform network enhancement through bandwidth adjustment, and render according to the display information and encode according to the first compression ratio to obtain the virtual scene data.

[0328] In one of the embodiments, the bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

[0329] In one of the embodiments, the rendering and encoding module is configured to, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and the network transmission delay is less than the maximum delay corresponding to the augmented reality service, select a delay adjustment strategy based on the network transmission bandwidth to perform network enhancement; and render and encode according to the display information using a processing strategy linked with the delay adjustment strategy to obtain the virtual scene data.

[0330] In one of the embodiments, the delay adjustment strategy includes network link selection and network link optimization.

[0331] The rendering and encoding module is further configured to, when the network transmission bandwidth is greater than or equal to a redundancy threshold, render according to the display information and encode according to a second compression ratio to obtain the virtual scene data; and when the network transmission bandwidth is less than the redundancy threshold, render according to the display information and encode according to the first compression ratio to obtain the virtual scene data, wherein the second compression ratio is less than the first compression ratio.

[0332] In one of the embodiments, the redundancy threshold is a first transmission bandwidth multiplied by a preset multiple; and a ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0333] In one of the embodiments, the preset multiple is 2; and the preset coefficient is a ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0334] In one of the embodiments, the network parameters include the network transmission bandwidth and the network transmission delay; the service capability information includes the minimum resolution, the minimum frame rate and the minimum depth of field; and wherein:

[0335] When the network transmission bandwidth is greater than or equal to the second transmission bandwidth, and the network transmission delay is less than or equal to the maximum delay corresponding to the augmented reality service, it is confirmed that all the parameters in the network parameters meet the transmission requirements of the basic quality.

[0336] The second transmission bandwidth is a quotient of a second product and a first compression ratio; the second product is a product of the minimum resolution, the minimum frame rate and the minimum depth of field; and the first compression ratio represents a compression ratio of the server in normal encoding.

[0337] In one of the embodiments, the rendering and encoding module is configured to render according to the service capability information and encode according to the first compression ratio to obtain the virtual scene data.

[0338] In one of the embodiments, as shown in FIG. 10, an augmented reality processing device is provided, which is applied to a terminal, and the device includes:

[0339] The information sending module 1001 is configured to send a service request to the cloud AR cloud end in the case of obtaining the real scene data, and send the obtained display information to the cloud AR cloud end; the service request is used to instruct the cloud AR cloud end to determine the virtual scene data corresponding to the service request according to the display information and the network transmission condition; and the display information represents the display capability of the terminal for the virtual scene data.

[0340] The data receiving module 1002 is configured to receive the virtual scene data sent by the cloud AR cloud end.

[0341] In one of the embodiments, the device further includes:

[0342] The display information determining module is configured to determine the display information based on the screen display parameters and the data acquisition parameters of obtaining the real scene data; the data acquisition parameters include the real scene acquisition resolution, the real scene acquisition frame rate and the real scene acquisition depth of field; the screen display parameters include the terminal display resolution, the terminal display frame rate and the terminal display depth of field; the display information includes the target resolution, the target frame rate and the target depth of field; wherein the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field. When the real scene data is obtained through the perspective mode, the screen display parameters are determined as the display information.

[0343] In one of the embodiments, the apparatus further comprises:

[0344] The virtual-real fusion output module is configured to output a virtual-real fusion result based on the virtual scene data and the real scene data.

[0345] The modules in the augmented reality processing apparatus can be realized by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the modules.

[0346] In one embodiment, a communication device is provided, as shown in FIG. 11, which is a structural schematic diagram of the communication device according to an embodiment of the present application. The communication device can be a server, which can include a receiver 31, a memory 32, a processor 33, at least one communication bus 34, and a transmitter 35. The communication bus 34 is configured to realize the communication connection between the elements. The memory 32 can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory. The memory 32 can store various programs for completing various processing functions and implementing the method steps of the embodiment. In the embodiment, the transmitter 35 can be a radio frequency processing module or a baseband processing module in the server, and the receiver 31 can also be a radio frequency processing module or a baseband processing module in the server. The transmitter 35 and the receiver 31 can be integrated together to form a transceiver. The transmitter 35 and the receiver 31 can be coupled to the processor 33, and can realize the receiving or transmitting action under the indication or control of the processor 33.

[0347] In the embodiment, the receiver 31 is configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case where the terminal acquires real scene data.

[0348] The processor 33 is configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions. The display information represents the display capability of the terminal for the virtual scene data.

[0349] The transmitter 35 is configured to send the virtual scene data to the terminal.

[0350] In one embodiment, the display information is determined based on a terminal-based screen display parameter and a data acquisition parameter for acquiring real scene data by the terminal; the data acquisition parameter comprises a real scene acquisition resolution, a real scene acquisition frame rate and a real scene acquisition depth of field; the screen display parameter comprises a terminal display resolution, a terminal display frame rate and a terminal display depth of field; the display information comprises a target resolution, a target frame rate and a target depth of field; wherein the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field. In the case that the real scene data is obtained by the terminal in a see-through mode, the display information is the screen display parameter.

[0351] In one embodiment, the processor 33 is specifically configured to acquire, by the virtual scene rendering function, a network parameter for communicating with the terminal by using a current network; the network parameter is a parameter representing a network transmission performance of the current network; and the display information or the service capability information is determined as the target capability information according to whether the network parameter meets a normal quality transmission requirement required by the display information and whether the network parameter meets a basic quality transmission requirement required by the service capability information; wherein the basic quality transmission requirement is lower than the normal quality transmission requirement; the service capability information represents a minimum capability supported by the virtual scene rendering function for the augmented reality service; and the virtual scene data is obtained by rendering and encoding according to the target capability information.

[0352] In one embodiment, the processor 33 is specifically configured to determine the display information as the target capability information if all the parameters in the network parameter meet the normal quality transmission requirement; determine the display information as the target capability information if part of the parameters in the network parameter meet the normal quality transmission requirement and if all the parameters in the network parameter after network enhancement meet the normal quality transmission requirement; determine the service capability information as the target capability information if all the parameters in the network parameter do not meet the normal quality transmission requirement and if all the parameters in the network parameter meet the basic quality transmission requirement; and not respond to the service request if all or part of the parameters in the network parameter do not meet the basic quality transmission requirement.

[0353] In one embodiment, the network parameter comprises a network transmission bandwidth and a network transmission time delay; the display information comprises a target resolution, a target frame rate and a target depth of field; and the processor 33 is specifically configured to confirm that the network transmission bandwidth meets the normal quality transmission requirement if the network transmission bandwidth is greater than or equal to a first transmission bandwidth; and confirm that the network transmission time delay meets the normal quality transmission requirement if the network transmission time delay is less than or equal to a recommended time delay corresponding to the augmented reality service; wherein the first transmission bandwidth is a quotient value of a first product and a first compression ratio; the first product is a product of the target resolution, the target frame rate and the target depth of field; and the first compression ratio represents a compression ratio for normal encoding by the server.

[0354] In an embodiment, the processor 33 is specifically configured to, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission delay is less than or equal to the recommended delay, render according to the display information and encode according to the first compression ratio to obtain the virtual scene data.

[0355] In an embodiment, the processor 33 is specifically configured to, when the network transmission delay is less than or equal to the recommended delay, if the network transmission bandwidth is less than the first transmission bandwidth, perform network enhancement through bandwidth adjustment, render according to the display information and encode according to the first compression ratio to obtain the virtual scene data.

[0356] In an embodiment, the bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

[0357] In an embodiment, the processor 33 is specifically configured to, when the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and the network transmission delay is less than the maximum delay corresponding to the augmented reality service, select a delay adjustment strategy based on the network transmission bandwidth to perform network enhancement; render and encode according to the display information using a processing strategy linked with the delay adjustment strategy to obtain the virtual scene data.

[0358] In an embodiment, the delay adjustment strategy includes network link selection and network link optimization; the processor 33 is specifically configured to, when the network transmission bandwidth is greater than or equal to the redundancy threshold, render according to the display information and encode according to the second compression ratio to obtain the virtual scene data; and when the network transmission bandwidth is less than the redundancy threshold, render according to the display information and encode according to the first compression ratio to obtain the virtual scene data; wherein the second compression ratio is less than the first compression ratio.

[0359] In an embodiment, the redundancy threshold is a preset multiple of the first transmission bandwidth; and the ratio of the first compression ratio to the second compression ratio is a preset coefficient.

[0360] In an embodiment, the preset multiple is 2 times; and the preset coefficient is the ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

[0361] In an embodiment, the network parameters comprise a network transmission bandwidth and a network transmission delay; the service capability information comprises a minimum resolution, a minimum frame rate and a minimum depth of field; and the processor 33 is specifically configured to: when the network transmission bandwidth is greater than or equal to a second transmission bandwidth and the network transmission delay is less than or equal to a maximum delay corresponding to the augmented reality service, confirm that all the parameters in the network parameters meet the basic quality transmission requirement; the second transmission bandwidth is a quotient of a second product and a first compression ratio; the second product is a product of the minimum resolution, the minimum frame rate and the minimum depth of field; and the first compression ratio represents a compression ratio of normal encoding of the server.

[0362] In an embodiment, the processor 33 is specifically configured to render according to the service capability information and encode according to the first compression ratio to obtain the virtual scene data.

[0363] In an embodiment, a communication device is provided, which can be a terminal device (terminal for short); referring to FIG. 12, FIG. 12 is a structural schematic diagram of a terminal device provided in an embodiment of the present application. The terminal device 700 shown in FIG. 12 includes at least one processor 701, a memory 702, at least one network interface 704 and a user interface 703. The various components in the terminal device 700 are coupled together through a bus system 705. It can be understood that the bus system 705 is used to realize the connection communication between the components. The bus system 705 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, various buses are marked as the bus system 705 in FIG. 12. In addition, a transceiver 706 is also included in the embodiment of the present application, which can be multiple elements, that is, includes a transmitter and a receiver, and provides a unit for communicating with various other devices on a transmission medium.

[0364] The user interface 703 can include a display, a keyboard or a clicking device (for example, a mouse, a trackball, a touchpad or a touch screen, etc.).

[0365] It is to be understood that the memory 702 in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM). The memory 702 of the system and method described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0366] In some embodiments, the memory 702 stores the following elements, executable modules or data structures, or a subset of them, or an extended set of them: an operating system 7021 and an application program 7022.

[0367] Among them, the operating system 7021 contains various system programs, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application program 7022 contains various application programs, such as media player (Media Player), browser (Browser), etc., for implementing various application services. The program for implementing the method of the embodiments of the present application can be contained in the application program 7022.

[0368] In the embodiments of the present application, the program or instruction stored in the memory 702 is invoked, specifically, the program or instruction stored in the application program 7022, wherein the transmitter is configured to send a service request to the cloud AR cloud end in the case of obtaining real scene data, and send the obtained display information to the cloud AR cloud end; the service request is used to instruct the cloud AR cloud end to determine the virtual scene data corresponding to the service request according to the display information and the network transmission condition; wherein the display information represents the display capability of the terminal for the virtual scene data; and the receiver is configured to receive the virtual scene data sent by the cloud AR cloud end.

[0369] Part or all of the methods disclosed in the embodiments of the present application can also be applied to the processor 701, or implemented by the processor 701, or implemented by the processor 701 in cooperation with other elements (such as a transceiver). The processor 701 can be an integrated circuit chip with a signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit or the instruction in the form of software in the processor 701. The processor 701 described above can be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. Each method, step and logic block diagram disclosed in the embodiments of the present application can be implemented or executed. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 702, and the processor 701 reads the information in the memory 702 and completes the steps of the above method in combination with the hardware.

[0370] It can be understood that the embodiments described in the embodiments of the application can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units for performing the functions described in the embodiments of the application, or a combination thereof.

[0371] For software implementation, the technologies described in the embodiments of the application can be implemented by modules (for example, processes, functions, etc.) for performing the functions described in the embodiments of the application. The software codes can be stored in the memory and executed by the processor 701. The memory can be implemented in the processor 701 or implemented outside the processor 701.

[0372] In one embodiment, the processor is configured to determine the display information based on data acquisition parameters of the real scene data and screen display parameters. The data acquisition parameters include a real scene acquisition resolution, a real scene acquisition frame rate and a real scene acquisition depth of field; the screen display parameters include a terminal display resolution, a terminal display frame rate and a terminal display depth of field; the display information includes a target resolution, a target frame rate and a target depth of field; the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field.

[0373] In one embodiment, the processor is further configured to determine the screen display parameters as the display information when the real scene data is obtained through the see-through mode.

[0374] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium has stored thereon a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0375] In one embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program is executed by a processor to implement the steps in the above method embodiments.

[0376] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0377] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (Resistive Random Access Memory, ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (Artificial Intelligence, AI) processor, etc., without being limited thereto.

[0378] Any combination of the technical features in the above-described embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, it is to be understood that the application embraces all such possible combinations.

[0379] The above-described embodiments only express several implementation manners of the application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the application. It should be pointed out that, for ordinary skilled persons in the art, some modifications and improvements can be made without departing from the concept of the application, and these all belong to the protection scope of the application. Therefore, the patent protection scope of the application should be subject to the appended claims.

Claims

1. A method for augmented reality processing, applied to a virtual scene rendering function of a cloud AR cloud, the method comprising: receiving a service request sent by a terminal, the service request being sent by the terminal in a case where real scene data is acquired; acquiring display information of the terminal, and determining virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information represents display capability of the terminal for virtual scene data; and sending the virtual scene data to the terminal. 2.The method of claim 1, wherein: the display information is determined based on screen display parameters of the terminal and data acquisition parameters of the terminal for acquiring the real scene data; the data acquisition parameters comprise real scene acquisition resolution, real scene acquisition frame rate and real scene acquisition depth of field; the screen display parameters comprise terminal display resolution, terminal display frame rate and terminal display depth of field; the display information comprises target resolution, target frame rate and target depth of field; the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field; wherein, in a case where the real scene data is obtained by the terminal in a see-through mode, the display information is the screen display parameters; and the determining of the virtual scene data corresponding to the service request according to the display information of the terminal and the network transmission conditions comprises: acquiring, by the virtual scene rendering function, network parameters for communication with the terminal by using a current network; the network parameters are parameters representing network transmission performance of the current network; determining target capability information from the display information or service capability information according to whether the network parameters meet normal quality transmission requirements required by the display information and whether the network parameters meet basic quality transmission requirements required by the service capability information; wherein the basic quality transmission requirements are lower than the normal quality transmission requirements; and the service capability information represents minimum capability of the virtual scene rendering function supporting augmented reality services; and rendering and encoding according to the target capability information to obtain the virtual scene data; and the determining of the target capability information from the display information or the service capability information according to whether the network parameters meet the normal quality transmission requirements required by the display information and whether the network parameters meet the basic quality transmission requirements required by the service capability information comprises: determining the display information as the target capability information if all the network parameters meet the normal quality transmission requirements; and determining the display information as the target capability information in a case where it is confirmed that all the network parameters of the current network after network enhancement can meet the normal quality transmission requirements when part of the network parameters meet the normal quality transmission requirements. ​ ​ ​ ​ ​ ​ ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ 4. The method of claim 3, wherein, ​ ​ ​ If all the network parameters do not meet the normal quality transmission requirement, if all or part of the network parameters do not meet the basic quality transmission requirement, the service request is not responded. The network parameters include network transmission bandwidth and network transmission delay; the display information includes target resolution, target frame rate and target depth of field; wherein:

5. The method of claim 4, wherein, When the network transmission bandwidth is greater than or equal to a first transmission bandwidth, it is determined that the network transmission bandwidth meets the normal quality transmission requirement; When the network transmission delay is less than or equal to a recommended delay corresponding to the augmented reality service, it is determined that the network transmission delay meets the normal quality transmission requirement; The first transmission bandwidth is a quotient of a first product and a first compression ratio; the first product is a product of the target resolution, the target frame rate and the target depth of field; and the first compression ratio represents a compression ratio of normal encoding of the server. The rendering and encoding according to the target capability information to obtain the virtual scene data includes:

6. The method of claim 5, wherein, When the network transmission bandwidth is greater than or equal to the first transmission bandwidth and the network transmission delay is less than or equal to the recommended delay, rendering is performed according to the display information and encoding is performed according to the first compression ratio to obtain the virtual scene data. The rendering and encoding according to the target capability information to obtain the virtual scene data includes:

7. The method of claim 5, wherein, When the network transmission delay is less than or equal to the recommended delay, if the network transmission bandwidth is less than the first transmission bandwidth, network enhancement is performed through bandwidth adjustment, rendering is performed according to the display information and encoding is performed according to the first compression ratio to obtain the virtual scene data. The bandwidth adjustment includes increasing the access bandwidth capacity of the terminal and / or increasing the egress bandwidth capacity of the server.

8. The method of claim 7, wherein, The rendering and encoding according to the target capability information to obtain the virtual scene data includes:

9. The method of claim 5, wherein, When the network transmission bandwidth is greater than or equal to the first transmission bandwidth, if the network transmission delay is greater than the recommended delay and the network transmission delay is less than a maximum delay corresponding to the augmented reality service, a time delay adjustment strategy is selected based on the network transmission bandwidth for network enhancement; The rendering and encoding according to the display information are performed using a processing strategy linked with the time delay adjustment strategy to obtain the virtual scene data. The time delay adjustment strategy includes network link selection and network link optimization.

10. The method of claim 9, wherein, The rendering and encoding according to the display information using the processing strategy linked with the time delay adjustment strategy to obtain the virtual scene data includes: When the network transmission bandwidth is greater than or equal to a redundancy threshold, rendering is performed according to the display information and encoding is performed according to a second compression ratio to obtain the virtual scene data; wherein the second compression ratio is less than the first compression ratio; ​ When the network transmission bandwidth is less than the redundancy threshold, the virtual scene data is obtained by rendering according to the display information and encoding according to the first compression ratio.

11. The method of claim 10, wherein, The redundancy threshold is a preset multiple of the first transmission bandwidth; and a ratio of the first compression ratio to the second compression ratio is a preset coefficient.

12. The method of claim 11, wherein, The preset multiple is 2; and the preset coefficient is a ratio of the network transmission bandwidth to the first transmission bandwidth, and the preset coefficient is an integer.

13. The method of claim 4, wherein, The network parameters include network transmission bandwidth and network transmission delay; the service capability information includes minimum resolution, minimum frame rate and minimum depth of field; wherein: When the network transmission bandwidth is greater than or equal to a second transmission bandwidth, and the network transmission delay is less than or equal to a maximum delay corresponding to the augmented reality service, it is confirmed that all parameters in the network parameters meet the transmission requirement of the basic quality; The second transmission bandwidth is a quotient of a second product and a first compression ratio; the second product is a product of the minimum resolution, the minimum frame rate and the minimum depth of field; and the first compression ratio represents a compression ratio of normal encoding by the server.

14. The method of claim 13, wherein, The virtual scene data is obtained by rendering according to the target capability information and encoding. The virtual scene data is obtained by rendering according to the service capability information and encoding according to the first compression ratio.

15. An augmented reality processing method applied to a terminal, the method comprising: sending a service request to a cloud AR cloud end in a case of obtaining real scene data, and sending obtained display information to the cloud AR cloud end; the service request is used to instruct the cloud AR cloud end to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information represents display capability of the terminal for the virtual scene data; receiving the virtual scene data sent by the cloud AR cloud end.

16. The method of claim 15, wherein, The method further comprises: determining the display information based on screen display parameters and data acquisition parameters of the real scene data; the data acquisition parameters include real scene acquisition resolution, real scene acquisition frame rate and real scene acquisition depth of field; the screen display parameters include terminal display resolution, terminal display frame rate and terminal display depth of field; the display information includes target resolution, target frame rate and target depth of field; the target resolution is the minimum value of the real scene acquisition resolution and the terminal display resolution; the target frame rate is the minimum value of the real scene acquisition frame rate and the terminal display frame rate; and the target depth of field is the minimum value of the real scene acquisition depth of field and the terminal display depth of field; when the real scene data is obtained by a perspective mode, the screen display parameters are determined as the display information.

17. The method of claim 15 or 16, wherein, The method further comprises: outputting a virtual-real fusion result based on the virtual scene data and the real scene data.

18. An augmented reality processing device applied to a virtual scene rendering function of a cloud AR cloud end, the device comprising: a request receiving module configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case of obtaining real scene data; a virtual data determining module configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information indicates display capability of the terminal for virtual scene data; a data sending module configured to send the virtual scene data to the terminal. 19.An augmented reality processing apparatus applied to a terminal, the apparatus comprising: an information sending module configured to send a service request to a cloud AR cloud in a case of acquiring real scene data, and send acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information indicates display capability of the terminal for the virtual scene data; a data receiving module configured to receive the virtual scene data sent by the cloud AR cloud.

20. A communication device comprising: a transmitter, a processor and a receiver; the receiver is configured to receive a service request sent by a terminal, the service request being sent by the terminal in a case of acquiring real scene data; the processor is configured to acquire display information of the terminal, and determine virtual scene data corresponding to the service request according to the display information of the terminal and network transmission conditions; wherein the display information indicates display capability of the terminal for virtual scene data; the transmitter is configured to send the virtual scene data to the terminal.

21. A communication device comprising: a transmitter and a receiver; the transmitter is configured to send a service request to a cloud AR cloud in a case of acquiring real scene data, and send acquired display information to the cloud AR cloud; the service request is used to instruct the cloud AR cloud to determine virtual scene data corresponding to the service request according to the display information and network transmission conditions; wherein the display information indicates display capability of the terminal for the virtual scene data; the receiver is configured to receive the virtual scene data sent by the cloud AR cloud.

22. A computer readable storage medium having stored thereon a computer program, wherein, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 17.

23. A computer program product comprising a computer program, wherein, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 17.

Images