A data rendering method, system and device

Abstract

A data rendering method, system and device for solving the problem that the display screen of a terminal display device has black borders due to time delay. The method includes: the server obtains the action prediction information of the user at a future moment, determines the predicted display resources corresponding to the action prediction information and performs basic rendering, and before the future moment arrives, sends the predicted display resources after basic rendering to the terminal display device. In this way, the terminal display device can use the predicted display resources to refresh the display screen before or at the future moment. As long as the prediction is accurate, the user can see the display screen at the future moment in advance before the future moment or see the display screen at the future moment in real time at the future moment, so as to avoid or alleviate the phenomenon that black borders appear on the display screen of the terminal display device due to the time delay problem, and improve the quality of the display screen in the terminal display device and the user's viewing experience.

Description

[0001] This application claims the priority of the Chinese patent application filed with the Intellectual Property Office of the People's Republic of China on June 28, 2020, with application number 202010600378.0 and invention name “A method and UE for providing auxiliary information”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to a data rendering method, system and device. Background Art

[0003] Virtual reality technology is a new human-computer interaction technology created by combining simulation technology with computer graphics human-computer interface technology, multimedia technology, sensor technology, network technology and other technologies, and using computers and the latest sensor technology. With the development of virtual reality technology, various virtual reality products have also become popular in people's production and life. Virtual reality products, for example, can include virtual reality headsets (i.e. head-mounted display devices), virtual reality glasses, augmented reality glasses, etc. By applying virtual reality products in real life, people can truly experience the various customs and landforms of the world without leaving home.

[0004] When users use virtual reality products, they often need to turn their heads left and right to view the display screen. In order to provide users with the display screen corresponding to the head turn in a timely manner, the virtual reality product can detect the user's head movement and obtain the corresponding display resources from the server to display to the user. However, there is a certain delay between the virtual reality product detecting the head movement and obtaining the display resources and refreshing the display screen. In this case, the display screen refreshed by the virtual reality product at the current moment is actually the screen corresponding to the action at the historical moment. During this period, if the user has a new head turning angle, since the virtual reality product has not obtained the display resources corresponding to the new head turning action, the display area corresponding to the head turning action on the display screen of the virtual reality product is a black area or an area similar to "smear", which is called "black edges". The more serious the black edges are, the worse the quality of the display screen in the virtual reality product is, and the user's head turning experience is relatively worse.

[0005] However, at present there is no solution to the problem of black edges appearing on the display screen of virtual reality products due to latency. Summary of the invention

[0006] The present application provides a data rendering method, system and device for improving the display quality of a display screen of a virtual reality product when a user turns his head.

[0007] In a first aspect, the present application provides a data rendering method, which is applied to a server, and the method includes: the server first receives first information sent by a terminal display device, the first information includes first action prediction information of a user at a future moment, and the first action prediction information is used to indicate the user's action at the future moment; then, the server determines the predicted display resources corresponding to the first action prediction information from the initial display resources, and performs basic rendering on the predicted display resources; in this way, before the future moment arrives, the server sends second information to the terminal display device, and carries the predicted display resources after basic rendering in the second information.

[0008] The above design provides the predicted display resources of the future moment to the terminal display device (such as a virtual reality product) in advance by the server, so that the terminal display device can use the predicted display resources to refresh the display screen before or at the future moment. In this case, as long as the prediction is accurate, the user can see the display screen of the future moment in advance before the future moment or see the display screen of the future moment in real time at the future moment. In this way, even if the user turns his head to the future moment in advance, there will be no black edges or very few black edges on the display screen seen by the user. Therefore, this method can avoid or alleviate the phenomenon of black edges appearing on the display screen of the terminal display device due to delay problems, which helps to improve the quality of the display screen in the terminal display device and the user's head turning experience. Furthermore, since this method predicts the user's action at the future moment in advance, the server can only render and transmit the predicted display resources in the direction corresponding to the action at the future moment, and does not need to render and transmit the display resources in all directions, so this method can also save resource consumption in the server.

[0009] In one possible design, the initial display resources can be divided into multiple data streams. In this case, the server determines the predicted display resources corresponding to the first action prediction information from the initial display resources, including: if the available resources of the server are not less than the first resource threshold, it means that the available resources of the cloud server are sufficient. In this case, the server can use all the data streams corresponding to the first action prediction information as predicted display resources to avoid the black edge phenomenon when the processing capacity of the cloud server is sufficient, and try to improve the user's viewing experience. If the available resources of the server are less than the first resource threshold, it means that the available resources of the cloud server are insufficient. In this case, the server can use part of the data streams corresponding to the first action prediction information as predicted display resources to save the resources of the cloud server as much as possible while alleviating the black edge phenomenon. This design can take into account both the server's resource consumption and the user's viewing experience.

[0010] In a possible design, the first information may also include the motion capture information of the user at the first moment, the first moment refers to the moment when the first motion prediction information is predicted, and the motion capture information is used to indicate the user's action at the first moment. In this case, before the server performs basic rendering on the predicted display resources, it may also determine the current display resources corresponding to the motion capture information from the initial display resources, and then perform basic rendering on the predicted display resources. Correspondingly, before the future moment arrives, the server sends the second information to the terminal display device, including: the server determines the target display resources based on the current display resources and the predicted display resources, and performs basic rendering on the target display resources, and before the future moment arrives, sends the second information to the terminal display device, and the second information includes the target display resources after basic rendering. Through this design, the terminal display device can not only display the display resources corresponding to the predicted future action to the user, but also display the display resources corresponding to the captured current action to the user, thereby helping to make the display screen seen by the user more comprehensive.

[0011] In a possible design, the initial display resource may be divided into multiple data streams. In this case, the server may determine the target display resource in the following manner:

[0012] Method 1: The server can use all data streams corresponding to the current display resource and all or part of the data streams corresponding to the predicted display resource as the target display resource. This method determines the target display resource by increasing the number of data streams, and can directly use the original division method to perform subsequent operations without re-dividing the data streams, which is simpler and more convenient to operate.

[0013] In the second method, the server can re-divide the initial display resources so that the data stream corresponding to the re-divided current display resources includes all or part of the data stream corresponding to the predicted display resources, and then use the re-divided current display resources as the target display resources. This method determines the target display resources by increasing the size of the data stream. Although the data stream needs to be re-divided, the re-divided initial display resources can better meet the current needs.

[0014] In one possible design, the server uses all data streams corresponding to the current display resources and all or part of the data streams corresponding to the predicted display resources as target display resources, including: if the available resources of the server are not less than the second resource threshold, it means that the available resources of the server are sufficient. In this case, the server can use all data streams of the current display resources and all or part of the data streams of the predicted display resources as target display resources to improve the user's viewing experience. If the available resources of the server are less than the second resource threshold, it means that the available resources in the server are insufficient. In this case, the server can first reduce the resolution of the predicted display resources, and then use all data streams of the current display resources and all or part of the data streams of the predicted display resources after the resolution is reduced as target display resources to alleviate the serious resource consumption problem of the cloud server.

[0015] In one possible design, the server receives the first information sent by the terminal display device, including: the server receives the first information corresponding to at least two moments sent by the terminal display device, wherein at least two moments are earlier than the future moment, and the first information corresponding to each of the at least two moments includes the first action prediction information of the user at the future moment predicted at the moment. Correspondingly, the server determines the predicted display resources corresponding to the first action prediction information from the initial display resources, including: the server obtains the target action prediction information of the user at the future moment based on the first action prediction information corresponding to the at least two moments and the weight of each first action prediction information, and then determines the predicted display resources from the initial display resources based on the target action prediction information at the future moment. This design improves the accuracy of action prediction by reporting action prediction information multiple times. The more accurate the action prediction is, the more accurate the predicted display resources are, which helps to alleviate the black edge phenomenon.

[0016] In one possible design, the server obtains the user's target action prediction information at a future moment based on the first action prediction information corresponding to at least two moments, including: the server performs weighted averaging on the first action prediction information corresponding to at least two moments, to obtain the user's target action prediction information at a future moment. Among them, for each of the at least two moments, the greater the time difference between the moment and the future moment, the greater the weight of the first action prediction information corresponding to the moment. In this way, by setting a larger weight for relatively accurate action prediction information close to the future moment and setting a smaller weight for relatively inaccurate action prediction information far from the future moment, it helps to make the calculated target action prediction information more accurate.

[0017] In one possible design, the server performs basic rendering on the predicted display resources, including: if the difference between the first action prediction information corresponding to at least two moments is not greater than the preset difference, it means that the accuracy of the action prediction is good. In this case, the server can perform basic rendering on the predicted display resources to maximize the user's viewing experience when the prediction is accurate. If the difference between the first action prediction information corresponding to at least two moments is greater than the preset difference, it means that the accuracy of the action prediction is not high. In this case, the server can reduce the resolution of the predicted display resources and perform basic rendering on the predicted display resources with the reduced resolution to minimize the resource consumption of the cloud server when the prediction is inaccurate.

[0018] In a possible design, the server determines the predicted display resources corresponding to the first action prediction information from the initial display resources, including: the server first uses the prediction model to determine the second action prediction information of the user at a future moment, and then performs weighted averaging on the first action prediction information and the second action prediction information, and determines the predicted display resources from the initial display resources according to the weighted averaged action prediction information. Among them, the prediction model is trained using the learning data reported by one or more terminal display devices, and the learning data reported by each terminal display device is used to indicate the real action of the user wearing the terminal display device at a future moment. The more learning data the prediction model is trained with, the greater the weight of the second action prediction information. Through this design, the server can also use the prediction model to optimize the predicted display resources of the terminal display device. Compared with the method of determining the predicted display resources only based on the action prediction information reported by the terminal display device, the design determines the factors of the predicted display resources more comprehensively, which helps to reduce the subjectivity of the predicted display resources and alleviate the problem of being unable to solve the black edge phenomenon of the display screen due to inaccurate action prediction information reported by the terminal display device.

[0019] In a possible design, the server can use the first action prediction information for learning and prediction before determining the predicted display resource from the initial display resource based on the first action prediction information and the second action prediction information. This design uses the prediction model and the action prediction information reported by the terminal display device to predict the user's action, so that the accuracy of the predicted display resource can depend on the accuracy of the prediction model and the accuracy of the action prediction information at the same time, rather than just on the prediction model or the action prediction information. Therefore, this method can reduce the probability of inaccurate prediction of the predicted display resource and help alleviate the black edge problem.

[0020] In a possible design, when the first action prediction information is only used for learning, the server can determine the prediction display resource corresponding to the second action prediction information from the initial display resource. The design also supports users to independently select the determining factors of the prediction display resource, such as only determined by the action prediction information, or only determined by the prediction model, or determined by the prediction model and the action prediction information. This method has better selectivity and can better meet the different needs of users.

[0021] In the second aspect, the present application provides a data rendering method, which is applied to a terminal display device, and the method includes: the terminal display device predicts the first action prediction information of the user at a future moment, and sends the first information to the server, the first information including the first action prediction information, and the first action prediction information is used to indicate the user's action at the future moment; then, before the future moment arrives, the terminal display device can receive the second information sent by the server, the second information including the predicted display resources corresponding to the action prediction information after basic rendering; in this way, the terminal display device can use the predicted display resources carried in the second information to refresh the display screen before the future moment arrives or at the future moment.

[0022] In the above design, the server provides the predicted display resources for the future moment to the terminal display device in advance, so that the terminal display device can use the predicted display resources to refresh the display screen before or at the future moment. In this way, even if the user turns his head to the future moment in advance, the display screen seen by the user will have no black borders or very few black borders, which helps to improve the quality of the display screen in the terminal display device and the user's head-turning experience.

[0023] In a possible design, before the terminal display device sends the first information to the server, it can also detect the motion capture information of the user at the first moment, and carry the motion capture information in the first information, so that the server can carry the current display resource corresponding to the motion capture information in the second information and send it to the terminal display device. In this case, the terminal display device can refresh the display screen in the following way:

[0024] Method 1: Before the future moment arrives, the terminal display device can generate a super-perspective display screen based on the current display resources and the predicted display resources, and display the super-perspective display screen. Among them, the viewing angle range of the super-perspective display screen is greater than the preset viewing angle range. This design is that the terminal display device provides a super-perspective display screen to the user in advance (increasing the user's current viewing angle). In this way, as long as the user's head turning angle is within this increased current viewing angle, the user will not see the black edge, and the user's viewing experience is better.

[0025] Method 2: When the future moment arrives, the terminal display device refreshes the display screen within the preset viewing angle range according to the current display resources and the predicted display resources. The design refreshes the display screen at the future moment, that is, the display resources corresponding to the predicted user's future action can be displayed to the user in real time at the future moment. This method not only allows the display screen seen by the user to be updated in real time as the user turns his head, improving the user's head turning experience, but also ensures that no black edges appear on the display screen seen by the user when the predicted user action is accurate.

[0026] In one possible design, the terminal display device predicts the user's first action prediction information at a future moment and sends the first information to the server, including: the terminal display device predicts the user's action prediction information at at least two moments in the future, and sends the first information corresponding to each of the at least two moments to the server. Among them, at least two moments are earlier than the future moment, and the first information corresponding to each moment includes the user's first action prediction information at the future moment predicted at the moment. This design improves the accuracy of action prediction by reporting action prediction information multiple times. The more accurate the action prediction is, the more accurate the predicted display resources are, and the more helpful it is to alleviate the black edge phenomenon.

[0027] In a third aspect, the present application provides a data rendering system, which may include a terminal display device and a server. When executing the data rendering method, the terminal display device can predict the user's first action prediction information at a future moment, and send the first information to the server, wherein the first information includes the first action prediction information, and the first action prediction information is used to indicate the user's action at a future moment. Correspondingly, after receiving the first information, the server can determine the predicted display resources corresponding to the first action prediction information in the first information from the initial display resources, and perform basic rendering on the predicted display resources, and before the future moment arrives, send the second information to the terminal display device, wherein the second information includes the predicted display resources after basic rendering. In this way, after receiving the second information, the terminal display device can use the predicted display resources to refresh the display screen before the future moment arrives or at the future moment.

[0028] In a possible design, before sending the first information to the server, the terminal display device can also detect the motion capture information of the user at the first moment, and carry the motion capture information in the first information, wherein the first moment is the moment when the first motion prediction information is predicted. Correspondingly, after receiving the first information and before sending the second information, the server can also determine the current display resource corresponding to the motion capture information from the initial display resource, determine the target display resource according to the current display resource and the predicted display resource, and perform basic rendering on the target display resource, and carry the target display resource after basic rendering in the second information. Correspondingly, after receiving the second information, the terminal display device can also generate a super-perspective display screen according to the current display resource and the predicted display resource before the future moment arrives, and display the super-perspective display screen, and the viewing angle range of the super-perspective display screen is greater than the preset viewing angle range. Alternatively, after receiving the second information, the terminal display device can also refresh the display screen within the preset viewing angle range according to the current display resource and the predicted display resource when the future moment arrives.

[0029] In a possible design, the terminal display device can also predict the user's action prediction information at a future moment at at least two moments, and send the first information corresponding to each of the at least two moments to the server, wherein at least two moments are earlier than the future moment, and the first information corresponding to each of the at least two moments includes the first action prediction information of the user at the future moment predicted at the moment. In this case, the server can also obtain the user's target action prediction information at the future moment based on the first action prediction information corresponding to the at least two moments and the weight of each first action prediction information, and determine the predicted display resource from the initial display resource based on the target action prediction information at the future moment; for each of the at least two moments, the greater the time difference between the moment and the future moment, the greater the weight of the first action prediction information corresponding to the moment.

[0030] In addition, in other possible designs of the third aspect, the server can also execute any other possible design method in the above-mentioned first aspect, and the terminal display device can also execute any other possible design method in the above-mentioned second aspect, which will not be repeated here one by one.

[0031] In a fourth aspect, the present application further provides a data rendering device, which may include a processor and a communication interface. The communication interface may receive a signal from other communication devices other than the data rendering device and transmit it to the processor or send a signal from the processor to other communication devices other than the data rendering device. The processor may be used to implement the data rendering method as described in any one of the first aspect or any one of the second aspect through a logic circuit or by executing code instructions.

[0032] In a fifth aspect, the present application also provides a data rendering device, which may include a processor, the processor is connected to a memory, the memory can be used to store a computer program, and the processor can execute the computer program stored in the memory so that the data rendering device performs a data rendering method as described in any one of the first aspect or any one of the second aspect.

[0033] In a sixth aspect, the present application further provides a data rendering device, which includes a module / unit for executing any possible design method of the first aspect, or includes a module / unit for executing any possible design method of the second aspect. These modules / units can be implemented by hardware, or can be implemented by hardware executing corresponding software.

[0034] In a seventh aspect, the present application further provides a computer-readable storage medium storing a computer program. When the computer program is executed, the data rendering method as described in any one of the first aspect or any one of the second aspect can be implemented.

[0035] In an eighth aspect, the present application further provides a chip, which may include a processor and an interface, wherein the processor may read instructions through the interface to execute the data rendering method described in any one of the first aspect or any one of the second aspect.

[0036] In a ninth aspect, the present application also provides a computer program product, which may include a computer program or instructions. When the computer program or instructions are executed by a communication device, it can implement a data rendering method as described in any one of the first aspects or any one of the second aspects.

[0037] For the beneficial effects of the third to ninth aspects mentioned above, please refer to the technical effects that can be achieved by the corresponding designs in the first and second aspects mentioned above, and they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1A A possible system architecture diagram applicable to the embodiments of the present application is exemplified;

[0039] Figure 1B Another possible system architecture diagram applicable to the embodiments of the present application is exemplarily shown;

[0040] Figure 2 A schematic diagram of an execution flow of a cloud rendering scenario provided by an embodiment of the present application is exemplified;

[0041] Figure 3 A schematic diagram of a display screen in a cloud rendering process provided by an embodiment of the present application is exemplified;

[0042] Figure 4 The following is a schematic diagram showing an exemplary process flow of the rendering method provided in the first embodiment of the present application;

[0043] Figure 5 A schematic diagram showing an exemplary storage method of an initial display resource in a cloud server;

[0044] Figure 6 A schematic diagram exemplarily shows a method of determining a target display resource by increasing the number of data streams;

[0045] Figure 7 A schematic diagram exemplarily shows a method of determining a target display resource by increasing the size of a data stream;

[0046] Figure 8 A schematic diagram of an execution flow of a serial end-to-cloud rendering method is exemplified;

[0047] Fig. 9 A schematic diagram of an execution flow of an asynchronous end-to-cloud rendering method provided in an embodiment of the present application is exemplified;

[0048] Fig.10 A time series diagram exemplarily showing the asynchronous end-to-cloud rendering method provided in an embodiment of the present application;

[0049] Fig.11 A schematic diagram showing a change in viewing angle during a cloud rendering process provided by an embodiment of the present application is exemplified;

[0050] FIG. 12A to FIG. 12D A schematic diagram showing an interface change of establishing a connection between a terminal device and a display device provided in an embodiment of the present application is exemplified;

[0051] FIG. 13A to FIG. 13C A schematic diagram showing, by way of example, changes in interfaces between another terminal device and a display device for establishing a connection provided by an embodiment of the present application;

[0052] Fig.14 The following is a schematic diagram showing an exemplary process flow of the rendering method provided in the second embodiment of the present application;

[0053] Fig.15 The following is a schematic diagram showing an example of a flow chart of a rendering method according to the third embodiment of the present application;

[0054] Fig.16 The following is a schematic diagram showing an exemplary process flow of the rendering method provided in the fourth embodiment of the present application;

[0055] Fig.17 A schematic diagram showing the structure of a data rendering device provided by an embodiment of the present application is exemplified;

[0056] Fig.18 A schematic diagram showing the structure of another data rendering device provided in an embodiment of the present application is exemplified;

[0057] Fig.19 The following is a schematic diagram showing the structure of another data rendering device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0059] Figure 1A and Figure 1B Schematic diagram of a possible system architecture applicable to the embodiment of the present application. The system architecture includes a server and a terminal display device. The terminal display device has a processing function and a display function. For example, the terminal display device can be Figure 1A The terminal device shown includes a terminal device with a processing function and a display device with a display function, and may also be as follows Figure 1B As shown, the processing function and the display function are encapsulated in one device, and the specific details are not limited. It should be understood that the embodiments of the present application are Figure 1A or Figure 1B The number of servers and terminal display devices in the illustrated system architecture is not limited. For example, one server can be connected to multiple terminal display devices at the same time. In addition to servers and terminal display devices, the system architecture applicable to the embodiments of the present application may also include other devices, such as network devices, core network devices, wireless relay devices, and wireless backhaul devices, etc., which are not limited by the embodiments of the present application. In addition, the server in the embodiments of the present application may integrate all functions into an independent physical device, or distribute the functions on multiple independent physical devices, which are not limited by the embodiments of the present application.

[0060] In the embodiment of the present application, the server and the terminal display device can be connected by wire or by wireless. Figure 1A In the illustrated system architecture, the server and the terminal device, and the terminal device and the display device can be connected by wired means, or the server and the terminal device, and the terminal device and the display device can be connected by wireless means, or the server and the terminal device are connected by wired means, and the terminal device and the display device are connected by wireless means, or the server and the terminal device are connected by wireless means, and the terminal device and the display device are connected by wired means. Currently, the most commonly used connection method is: the server and the terminal device are connected by wireless means, and the terminal device and the display device are connected by wired means or wireless means.

[0061] The following is an introduction to the terms and related technologies involved in the embodiments of the present application.

[0062] The server in the embodiment of the present application may refer to a single server or a server cluster. In a specific application scenario, the server may refer to a cloud server, which may provide display resources to a terminal display device. The display resources may include, for example, pictures, videos, or game scenes, etc. The display resources may be pre-stored inside the cloud server, downloaded from the network by the cloud server, or obtained by interacting with other devices. In addition, the cloud service can also provide a simple, efficient, safe, reliable, and scalable computing service. For example, before providing the display resources to the terminal display device, the cloud server can also perform basic rendering on the display resources. The basic rendering may include rendering a two-dimensional plane display resource into a three-dimensional stereoscopic display resource, adjusting the color tone parameters of the display resource, and stretching the display resource screen.

[0063] exist Figure 1A In the illustrated system architecture, the terminal device refers to a device with certain processing and computing capabilities and can support low latency and high-speed transmission performance, such as a device that provides voice and / or data connectivity to users. For example, it may include a handheld device with a wired connection function or a wireless connection function, or a processing device connected to a wired modem or a wireless modem. The terminal device may include, but is not limited to, user equipment (UE), wireless terminal equipment, mobile terminal equipment, device-to-device communication (D2D) terminal equipment, vehicle to everything (V2X) terminal equipment, machine-to-machine / machine-type communications (M2M / MTC) terminal equipment, etc. For example, it may include a mobile phone (or "cellular" phone), a computer with a mobile terminal device, a portable, pocket-sized, handheld, or computer-built-in mobile device, etc. For example, it may also include a device with certain requirements for power consumption, storage capacity, and computing power. In addition, the terminal device may also include components such as barcodes, radio frequency identification (RFID), sensors, global positioning systems (GPS), laser scanners, etc. These components are used to assist the terminal device in sensing the user's business interaction operations.

[0064] exist Figure 1AIn the illustrated system architecture, the display device may be a wearable device with a display function, such as a virtual reality (VR) head display, VR glasses, an augmented reality (AR) head display, AR glasses, etc. In order to reduce the cost and weight of the display device, at this stage, most of the display devices are directly set to have only a display function. However, considering that in the cloud rendering scenario, the terminal device not only needs to provide interactive functions for the user, but also needs to assist the display device in performing operations such as decoding and rendering, and the pressure on the terminal device is relatively large. Therefore, in one embodiment of the present application, the display device may also have a certain processing capability, as long as it does not exceed the cost and weight requirements of the display device.

[0065] exist Figure 1B In the illustrated system architecture, the terminal display device may also refer to the integration of the functions of the above terminal device into the display device, so that the display device has both processing and display functions. In this case, the terminal display device may be a wearable device with processing and display functions, such as a VR head display (all-in-one), VR glasses (all-in-one), AR head display (all-in-one), AR glasses (all-in-one), etc.

[0066] In the embodiment of the present application, the server and the terminal device, or the server and the terminal display device can communicate through a network protocol, and the network protocol may include, for example, transmission control protocol / internet protocol (TCP / IP), user datagram protocol / internet protocol (UDP / IP), hypertext transfer protocol (HTTP), hypertext transfer protocol over secure socket layer (HTTPS), etc. The network protocol may also include the remote procedure call protocol (RPC) protocol and representational state transfer (REST) ​​protocol used on top of the above protocols, without specific limitation.

[0067] The present application can be specifically applied to cloud rendering scenarios. For ease of description, the following embodiments of the present application refer to the server as a cloud server. That is, the "cloud server" that appears below can be replaced by "server".

[0068] First Figure 1A The cloud rendering scenario is introduced by taking the system architecture shown in the figure as an example. Figure 2 The following is a schematic diagram showing an execution flow of a cloud rendering scenario provided by an embodiment of the present application. Figure 2 As shown, during the cloud rendering process, the display device can periodically or in real time capture the user's movements to obtain motion capture information, and then report the motion capture information to the cloud server via the terminal device. After receiving the motion capture information, the cloud server can first perform logical calculations on the motion capture information to determine the user's field of view (FOV), and then obtain the display resources corresponding to the user's FOV from the initial display resources. After basic rendering of the display resources, the basic rendered display resources are encoded and compressed (to reduce network overhead and improve data transmission efficiency) and sent to the terminal device. When the display device has a certain processing function, after receiving the encoded compressed data packet, the terminal device can first decode to obtain the basic rendered display resources, and then send the display resources to the display device, so that the display device performs head motion rendering on the display resources, and then use the head motion rendering display resources to refresh the current display screen, so as to display the display resources corresponding to the user's FOV to the user. It should be understood that this example is based on the consideration that the display device has a certain processing capability. In order to reduce the burden on the terminal device, it is also possible to only place the decoding on the terminal device side and place the head motion rendering on the display device side. In other examples, if the display device does not have processing capabilities, both decoding and head motion rendering can be performed on the terminal device side.

[0069] However, there may be problems with the display screen displayed by the display device to the user. For example, Figure 3The schematic diagram of the display screen in the cloud rendering process provided by the embodiment of the present application is shown as an example. Under normal circumstances, after receiving the motion capture information of the user, the cloud server will send the display resources of the original FOV (such as the area surrounded by the solid line in Figure a) corresponding to the motion capture information to the display device. At this time, the picture seen by the user within the viewing angle is the display picture corresponding to the original FOV. However, there will be a certain delay between reporting the motion capture information and refreshing the display screen. If the user has a new head turning action within the delay, the range of the user's actual FOV (such as the area surrounded by the dotted line in Figure b) may exceed the range of the original FOV. However, since the cloud server only sends the display resources within the original FOV range to the display device, the display device can only display the display screen within the original FOV range to the user, and the area where the actual FOV exceeds the original FOV (such as the area surrounded by the oblique lines in Figure c) will have a black border (displayed as black or similar to "smear"). In this case, the user can only see the black border at the edge of the viewing angle of the actual FOV and cannot see the horse's hoof and part of the horse's tail, so the quality of the display screen and the user's head turning experience are both poor.

[0070] In order to solve the black edge phenomenon, in a high-low quality rendering method, after receiving the motion capture information each time, the cloud server can not only perform basic rendering on the high-quality local display resources corresponding to the motion capture information, but also perform basic rendering on the low-quality initial display resources, and send them to the display device via the terminal device. In this way, the display device can receive a high-quality local display resource and a low-quality initial display resource each time. If the user has a new head turning angle, the display device can also determine the low-quality display resource corresponding to the exceeded angle from the low-quality initial display resource, and then display the high-quality local display resource and the low-quality display resource corresponding to the new head turning angle to the user at the same time. This method can prevent black edges from appearing within the user's viewing angle, which helps to alleviate the black edge phenomenon. However, if this method is adopted, part of the display screen within the user's viewing angle is clear and part of the display screen is unclear. When the clear display screen transitions to the unclear display screen, the user's viewing experience will inevitably be affected. Moreover, the initial display resource is actually a 360° full-view display resource. This method requires the cloud server to render the 360° full-view display resource, which will consume a lot of cloud server resources even at low quality.

[0071] In view of this, an embodiment of the present application provides a rendering method for alleviating the black edge phenomenon generated in the display screen based on comprehensive consideration of the resource consumption of the cloud server and the user viewing experience.

[0072] The terms "system" and "network" in the embodiments of the present application can be used interchangeably. "At least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0073] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the priority or importance of multiple objects. For example, the first terminal display device and the second terminal display device are only used to distinguish different terminal display devices, and do not indicate the difference in priority or importance of the two terminal display devices. Alternatively, the first action prediction information and the second action prediction information are only used to distinguish the action prediction information at different times, and do not indicate the difference in priority or importance of the two action prediction information.

[0074] The data rendering method in this application is introduced below in conjunction with specific embodiments.

[0075] Embodiment 1

[0076] Figure 4 The following is a schematic diagram of an interaction process corresponding to the rendering method provided in the first embodiment of the present application. The method can be executed by a cloud server and a terminal display device, for example: Figure 1A Cloud servers, terminal devices and display devices in the Figure 1B The cloud server and terminal display device in the Figure 4 As shown, the method includes:

[0077] Step 401: The terminal display device predicts and obtains the user's action prediction information at a future moment.

[0078] In an optional implementation, the terminal display device may first obtain the user's motion capture information at a certain moment or certain moments before the future moment arrives, and then predict the user's motion at the future moment based on the obtained motion capture information to obtain motion prediction information. The motion capture information and the motion prediction information may be, for example, the angle through which the user's head has turned. There may be multiple ways to obtain motion capture information. For example, in one way, the terminal display device has a built-in motion sensor, and the terminal display device obtains motion capture information by calling the motion sensor to collect the user's head motion. In another way, the terminal display device has an external camera module, and the terminal display device obtains motion capture information by calling the camera module to take a picture of the user's head motion and analyzing the captured picture. It is understandable that in addition to determining motion capture information by detecting head movements, motion capture information can also be determined by detecting movements at other locations or using other sensors, such as using a somatosensory device to detect hand movements (the hand is used as an example here, of course, it can also be other locations). In this case, the user can wear the terminal display device and hold the somatosensory device in his hand. The terminal display device is connected to the somatosensory device. When the user wants to change the display screen of the terminal display device, the user can directly hold the somatosensory device and turn left or right or flip it. In this way, the display screen provided by the terminal display device to the user can also change accordingly with the hand movement. In this way, even users who have difficulty moving their heads (for example, due to cervical spine injuries that make the head unable to move) can experience the terminal display device well, which helps to meet the viewing needs of various users.

[0079] For ease of understanding, the following embodiments are introduced by taking the prediction of the user's head movement at a future moment as an example, and the prediction methods include but are not limited to the following:

[0080] (1) Dead reckoning (DR): It is a technology that uses the current position and speed to infer the future position and direction. When predicting the user's head movement at a future moment, the terminal display device can input the user's head position at the current moment (for example, if the terminal display device is a VR glasses, the head position can refer to the angle between the center line of the VR glasses and the baseline. When the center line is to the right of the baseline, the angle is positive, and when the center line is to the left of the baseline, the angle is negative) and the head rotation speed into a preset dead reckoning model. The preset dead reckoning model can first calculate the angle that the user will turn at the future moment relative to the current moment based on the time difference between the future moment and the current moment, and the user's head rotation speed at the current moment, and then determine the user's head position at the future moment in combination with the user's head position at the current moment. Among them, the head rotation speed in this example may include the head rotation direction and the rotation speed value. After determining the user's head position at the future moment, the terminal display device can determine the user's head movement at the future moment based on the head position and the current head position. For example, if the current head position is 5 degrees to the right of the baseline, and the head position at the future moment is 10 degrees to the right of the baseline, then the user's head movement at the future moment will be to turn right 5 degrees.

[0081] (2) Kalman predictor: This is a technology that takes into account the influence of noise and uses the numerical values ​​of other components to obtain the future position and direction. When predicting the head movement of the user at the future moment, the terminal display device can simultaneously input the user's head position and head rotation speed at the current moment, as well as the motion data collected by the motion sensor (such as the Global Positioning System (GPS) sensor, pressure sensor, Hall sensor, etc.) into the Kalman predictor. In this case, the Kalman predictor can calculate the theoretical prediction value of the head position at the future moment based on the information such as the user's head position and head rotation speed at the current moment, and on the other hand, it can also calculate the actual measurement value of the head position at the future moment based on the motion data. Then, according to the pre-set weights corresponding to the theoretical prediction value and the weights corresponding to the actual measurement value, the theoretical prediction value and the actual measurement value can be weighted linearly combined to obtain the user's head position at the future moment, and then the terminal display device determines the user's head movement at the future moment.

[0082] (3) Alpha-beta-gamma predictor: also known as ABG predictor, the logical calculation process of this predictor is highly correlated with the logical calculation process of the Kalman predictor, but it is not exactly the same, or it can be said that this predictor is simpler than the Kalman predictor. Specifically, the ABG predictor contains three configuration parameters, namely alpha, beta and gamma. These three configuration parameters are updated according to the actual data and the predicted data. When predicting head movement, the ABG predictor will continuously obtain the actual position of the user's head and the motion data of the motion sensor, and try to combine the three configuration parameters and these data to continuously estimate the head rotation speed and head rotation acceleration, and then apply these estimated data to the prediction. Since the ABG predictor takes the actual data into account in the estimation process, it can reduce some of the noise or error in the estimated data. In addition, the configuration parameters of the ABG predictor are updated in real time, which can also enhance the timeliness of the estimated data response and the noise reduction capability.

[0083] (4) Model fitting method: It is a technology that uses historical positions to infer future positions. This method pre-sets an initial fitting model, the independent variable of the initial fitting model is the moment, the dependent variable is the head position of the user at the moment, and the initial fitting model includes one or more unknown parameters. Among them, the type of the initial fitting model can be set by a person skilled in the art based on experience, for example, it can be a polynomial model, a binary classification model or a b-spline model, etc., without limitation. When predicting the head position of the user at a future moment, the terminal display device can first input each historical moment and the head position of the user at each historical moment into the initial fitting model to calculate each unknown parameter in the initial fitting model, bring each resolved unknown parameter into the initial fitting model to obtain a fitting model, and then input the future moment into the fitting model to calculate the head position of the user at the future moment, and then the terminal display device determines the head movement of the user at the future moment.

[0084] Step 402: The terminal display device sends first information to the cloud server, where the first information includes prediction information of the user's action at a future moment.

[0085] In an embodiment of the present application, the terminal display device may send action prediction information to the cloud server at a certain moment or multiple moments before the future moment arrives. For example, if N action prediction information is predicted at N (N is an integer greater than or equal to 2) moments, the terminal display device may report the N action prediction information to the cloud server at these N moments, or may report the N action prediction information to the cloud server at a certain moment before the future moment, without limitation. Wherein, in the case of multiple reports of action prediction information, the time interval between the most recent reported moment and the future moment needs to be greater than or equal to the first time interval, and the first time interval may be set by a person skilled in the art based on experience. For example, in an optional implementation, the delay between the time after the user turns his head and the terminal display device actually refreshes the display screen can be determined in advance by experiments, and the first time interval is set to the delay, or in the case of multiple experiments, the first time interval is set to the maximum delay in multiple experiments (or it may also be the average or weighted average of the delays of multiple experiments). In this case, when the time interval between the most recently reported moment and the future moment is exactly equal to the first time interval, the terminal display device can display the display resources of the future moment to the user just at the future moment, so that the user can see the display screen corresponding to the head turning angle in real time, and the user's head turning experience is better. When the time interval between the most recently reported moment and the future moment is greater than the first time interval, the terminal display device refreshes the display screen according to the display resources of the future moment before the future moment. This method is equivalent to the terminal display device providing the user with a super-perspective display screen in advance (which can increase the user's current FOV). In this way, as long as the user's head turning angle is within this increased current FOV, the user will not see the black edge. Obviously, compared with the prior art delayed display and possible black edge solutions, this method can alleviate the black edge phenomenon and can greatly improve the user's head turning experience.

[0086] Step 403: The cloud server determines the predicted display resources corresponding to the action prediction information at the future moment from the initial display resources.

[0087] In an optional implementation, the terminal display device can also report the motion capture information corresponding to the moment to the cloud server at a certain moment before the future moment arrives, wherein the time interval between the certain moment and the future moment also needs to be greater than the first time interval. In this case, the cloud server can receive the motion capture information of the current moment and the motion prediction information of the future moment before the distance to the future moment exceeds the delay length, and can determine the current display resources corresponding to the current moment from the initial display resources based on the motion capture information at the current moment, and determine the predicted display resources corresponding to the future moment from the initial display resources based on the motion prediction information at the future moment, and then determine the target display resources based on the current display resources and the predicted display resources, and send the target display resources to the terminal display device before the future moment arrives, so that the terminal display device can display the target display resources to the user in a timely manner.

[0088] Figure 5 A schematic diagram showing an exemplary storage method of an initial display resource in a cloud server is shown as follows: Figure 5 As shown, the initial display resource is divided into multiple data streams in the cloud server, and each data stream corresponds to its own encoding, for example, Figure 5 In the example, the 360° panoramic display resource with a resolution of 8K is evenly divided into 20 data streams (or grids). The codes of these 20 data streams are 1 to 20, and they are arranged in the form of 4 rows and 5 columns. The resolution of each data stream is 8K, and each data stream occupies one cell. One cell is used to indicate the amount of data in one grid. The data streams in the first row include data stream 1, data stream 2, data stream 3, data stream 4, and data stream 5, occupying five cells; the data streams in the second row include data stream 6, data stream 7, data stream 8, data stream 9, and data stream 10, occupying five cells; the data streams in the third row include data stream 11, data stream 12, data stream 13, data stream 14, and data stream 15, occupying five cells; the data streams in the fourth row include data stream 16, data stream 17, data stream 18, data stream 19, and data stream 20, occupying five cells.

[0089] The following uses an example in which a cloud server determines that the current display resources corresponding to the captured user action include data stream 7, data stream 8, data stream 12, and data stream 13 based on the action capture information to introduce how to determine the target display resource.

[0090] In an optional implementation, if the cloud server determines that all data streams corresponding to the predicted user action (e.g., turn right 5 degrees) are data streams 9, 10, 14, and 15 according to the action prediction information, the cloud server can also determine the predicted display resources corresponding to the action prediction information according to the available resources of the cloud server. When the available resources of the cloud server are not less than the first resource threshold, it means that the available resources of the cloud server are sufficient. In this case, all data streams corresponding to the action prediction information (i.e., data streams 9, 10, 14, and 15) can be used as predicted display resources to avoid the black edge phenomenon when the processing capacity of the cloud server is sufficient, and to maximize the user's viewing experience. When the available resources of the cloud server are less than the first resource threshold, it means that the available resources of the cloud server are insufficient. In this case, some data streams corresponding to the action prediction information (e.g., data streams 9 and 14) can be used as predicted display resources, and the amount of data of some data streams can be reduced as the cloud server is insufficient, so as to save the resources of the cloud server as much as possible while alleviating the black edge phenomenon. This implementation method can take into account both the resource consumption of the cloud server and the viewing experience of the user.

[0091] In the embodiment of the present application, after determining the current display resource and the predicted display resource, the cloud server may determine the target display resource in any of the following ways:

[0092] Figure 6 A schematic diagram exemplarily shows a method of determining a target display resource by increasing the number of data streams, such as Figure 6 As shown, in this example, the cloud server can directly use all data streams corresponding to the current display resource and the data stream corresponding to the predicted display resource as the target display resource. For example, in the case where the predicted display resource is all data streams corresponding to the action prediction information, the target display resource corresponds to Figure 6 The target display resource a1 in includes data stream 7, data stream 8, data stream 9, data stream 10, data stream 12, data stream 13, data stream 14 and data stream 15, which occupy 8 cells in total. In the case where the predicted display resource is a partial data stream corresponding to the action prediction information, the target display resource corresponds to Figure 6 The target display resource a2 in , includes data stream 7, data stream 8, data stream 9, data stream 12, data stream 13 and data stream 14, occupying a total of 6 cells.

[0093] Figure 7 A schematic diagram exemplarily shows a method of determining a target display resource by increasing the size of a data stream, such as Figure 7As shown, in this example, the cloud server can re-divide the initial display resources, and make the re-divided current display resources include the predicted display resources. For example, when the predicted display resources are all data streams corresponding to the action prediction information, the cloud server can first expand the size of data stream 8 so that the expanded data stream 8 includes the original data stream 8, the original data stream 9 and the original data stream 10, and at the same time expand the size of data stream 13 so that the expanded data stream 13 includes the original data stream 13, the original data stream 14 and the original data stream 15. In this way, the re-divided initial display resources correspond to Figure 7 In this case, the target display resource can correspond to Figure 7 The target display resource a3 in the example includes data stream 7, enlarged data stream 8, data stream 12, and enlarged data stream 13, wherein data stream 7 and data stream 12 occupy one cell respectively, and enlarged data stream 8 and enlarged data stream 13 occupy three cells respectively. Alternatively, in the case where the predicted display resource is a partial data stream corresponding to the action prediction information, the cloud server can first enlarge the size of data stream 8 so that the enlarged data stream 8 includes the original data stream 8 and the original data stream 9, and at the same time enlarge the size of data stream 13 so that the enlarged data stream 13 includes the original data stream 13 and the original data stream 14. In this way, the initial display resource obtained by re-dividing corresponds to Figure 7 In this case, the target display resource can correspond to Figure 7 The target display resource a4 in , includes data stream 7, enlarged data stream 8, data stream 12 and enlarged data stream 13, wherein data stream 7 and data stream 12 occupy 1 cell respectively, and enlarged data stream 8 and enlarged data stream 13 occupy 2 cells respectively.

[0094] In an embodiment of the present application, when the predicted rotation angle corresponding to the action prediction information is greater than or equal to the actual rotation angle (for example, the user is predicted to turn his head 10° to the right and the user actually turns his head 5° to the right), the entire data stream corresponding to the action prediction information is used as a predicted display resource to completely cover the black edge in the display screen, thereby solving the black edge phenomenon, and although the partial data stream corresponding to the action prediction information may not be able to completely cover the black edge in the display screen as a predicted display resource, it can also play a role in alleviating the black edge phenomenon, thereby improving the user's viewing experience. When the predicted rotation angle corresponding to the action prediction information is less than the actual rotation angle (for example, the user is predicted to turn his head 10° to the right and the user actually turns his head 15° to the right), whether the entire data stream corresponding to the action prediction information is used as a predicted display resource, or the partial data stream corresponding to the action prediction information is used as a predicted display resource, the black edge in the display screen cannot be completely covered, but this method can alleviate the black edge in the display screen and improve the user's viewing experience. It can be seen from this that in either case, using the entire data stream or the partial data stream corresponding to the action prediction information as a predicted display resource can alleviate the black edge phenomenon.

[0095] Step 404 : the cloud server performs basic rendering on the predicted display resources corresponding to the action prediction information at the future moment.

[0096] In the embodiment of the present application, basic rendering may include but is not limited to the following: converting the dimension of display resources, converting the format of display resources, adding filters, deformation, blurring, smoothing, exposure, texturing and other effects to display resources, and adjusting the color parameters of display resources such as hue, color difference, and color temperature. Generally speaking, basic rendering consumes more resources of the cloud server. However, the embodiment of the present application determines the predicted display resources based on the predicted action at the future moment. Therefore, the predicted display resources are only the part of the display resources corresponding to the predicted rotation direction in the initial display resources. In this way, the cloud server can only perform basic rendering on a part of the initial display resources, without the need to perform basic rendering on the entire low-quality initial display resources. Therefore, the solution in the embodiment of the present application can also save resource consumption of the cloud server.

[0097] In the embodiment of the present application, when the quality of the target display resource is higher, the cloud server will consume more resources for basic rendering of the target display resource. In order to take into account the resource consumption of the cloud server and the viewing experience of the user, the embodiment of the present application can determine the quality of the target display resource in the following way:

[0098] If the target display resource is determined by increasing the size of the data stream, since the expanded data stream will include part of the data stream corresponding to the current display resource, in order to ensure that the user obtains the high-definition display image corresponding to the motion capture information, the target display resource can be of high quality. Figure 7 When the target display resource a3 is used, the cloud server can perform basic rendering on data stream 7, data stream 12, expanded data stream 8, and expanded data stream 13, all of which have a resolution of 8K. Figure 7 When the target display resource a4 is selected, the cloud server can perform basic rendering on data stream 7, data stream 12, enlarged data stream 8, and enlarged data stream 13, all of which have a resolution of 8K. When the resolution is 8K, since the enlarged data stream 8 and the enlarged data stream 13 in the target display resource a3 each occupy 3 cells, and the enlarged data stream 8 and the enlarged data stream 13 in the target display resource a4 each occupy 2 cells, the resources consumed by the cloud server for basic rendering of the target display resource a3 are greater than the resources consumed for basic rendering of the target display resource a4.

[0099] If the target display resource is determined by increasing the number of data streams, the quality of the target display resource can be determined based on the available resources of the cloud server: when the available resources of the cloud server are not less than the second resource threshold, it means that the available resources in the cloud server are sufficient. In this case, in order to improve the user's viewing experience, the cloud server can directly use the high-quality current display resources and the high-quality predicted display resources as the target display resources. For example, if the target display resource corresponds to Figure 6 If the target display resource a1 in the image is 8K, basic rendering can be performed on data streams 7, 8, 12, 13, 9, 10, 14, and 15, all of which have a resolution of 8K. When the available resources of the cloud server are less than the second resource threshold, it means that the available resources in the cloud server are insufficient. In this case, in order to alleviate the problem of serious resource consumption of the cloud server, the cloud server can use high-quality current display resources and low-quality predicted display resources as target display resources. For example, if the target display resource corresponds to Figure 6For the target display resource a1 in the image, basic rendering can be performed on data streams 7, 8, 12, and 13, all of which have a resolution of 8K, and data streams 9, 10, 14, and 15, all of which have a resolution of 4K (or 2K). When the resolution of the data stream is low, the cloud server consumes relatively less resources for basic rendering of the data stream. Therefore, using high-quality current display resources and low-quality predicted display resources as target display resources obviously consumes less cloud resources than using high-quality current display resources and high-quality predicted display resources as target display resources.

[0100] Step 405: The cloud server sends the target display resource after basic rendering to the terminal display device before the future time arrives.

[0101] In the embodiment of the present application, the cloud server can also compress the target display resource after basic rendering, and then send the compressed package to the terminal display device. Since the data volume of the compressed package is significantly smaller than the data volume of the uncompressed target display resource, the method of transmitting the compressed package can save network overhead and improve the efficiency of data transmission, which helps the terminal display device refresh the display screen as soon as possible and reduce the delay of the terminal display device.

[0102] In an optional implementation, the time interval between the moment when the cloud server sends the target display resource after basic rendering and the future moment needs to be greater than or equal to the second time interval, and the second time interval can be set by a person skilled in the art based on experience. For example, in an optional implementation, the time difference between the time when the cloud server sends the target display resource after basic rendering and the time when the terminal display device actually refreshes the display image can be determined in advance through experiments, and the second time interval is set to the time difference, or in the case of multiple experiments, the second time interval is set to the maximum time difference in multiple experiments (or it can also be the average or weighted average of the time differences of multiple experiments). This method helps the terminal display device receive the display resources sent by the cloud server before the future moment arrives, so that the terminal display device can display the display resources corresponding to the action prediction information in a timely manner, which helps to alleviate the black edge phenomenon on the display screen.

[0103] Step 406: The terminal display device uses the target display resource after basic rendering to refresh the display screen.

[0104] In an optional implementation, if the terminal display device receives a compressed package, the terminal display device may first decompress the compressed package to obtain the target display resource after basic rendering, and then perform head motion rendering on the target display resource after basic rendering according to the preset end-cloud rendering method, and finally use the target display resource after head motion rendering to refresh the display screen. Among them, head motion rendering means that since the motion capture and reporting process of the terminal display device, the logical calculation of the cloud server, the basic rendering, the encoding compression process, the data transmission process, and the decoding process of the terminal display device all require time, there will be a certain delay after the user's head turns to the terminal display device before decompressing the target display resource, and the user may have a new head movement during the delay, resulting in the target display resource received by the terminal display device not matching the new head movement. In this case, it is not appropriate to directly display the target display screen corresponding to the previous head movement to the user. In this case, the terminal display device can also first perform head motion rendering operations such as stretching, intercepting, and deforming on the target display resource, and after the target display resource after head motion rendering matches the new head movement of the user during the delay period, the display screen is refreshed using the matching target display resource. For example, if the motion capture information sent previously corresponds to a 5-degree right turn, and the terminal device turns right another 2 degrees before decoding and obtaining the target display resources corresponding to a 5-degree right turn, then if the size of the FOV remains unchanged, the terminal display device should extract the target display resources corresponding to 3 degrees on the right from the target display resources corresponding to a 5-degree right turn to refresh the display screen.

[0105] In the embodiments of the present application, the preset end-cloud rendering method can be set by technicians in this field based on experience. At this stage, serial end-cloud rendering is generally used as the preset end-cloud rendering method. Figure 8 An exemplary execution flow diagram of a serial end-to-cloud rendering method is shown as follows: Figure 8As shown, after capturing the posture and position at the Pth moment to obtain motion capture information and reporting this information to the cloud server through instructions, the terminal display device is always in a waiting state until the display resource at the Pth moment sent by the cloud server is received. Then, the terminal display device starts to collect the new head movement of the user within the delay, determines the relevant parameters of the head motion rendering (such as stretching parameters, interception parameters, deformation parameters, etc.) according to the new head movement, and uses the relevant parameters of the head motion rendering to render the display resource. Obviously, in the serial rendering mode, all operations of the cloud server and the terminal display device are executed in series. This serial execution mode will take a long time, resulting in a long delay from the terminal display device detecting the user's head movement to refreshing the display screen (motion to photon, MTP). Generally speaking, if the time difference from the movement of the user's head to the change of the image seen by both eyes is within 20ms, the user will basically not feel uncomfortable, but if it exceeds 20ms, the user's vision is inconsistent with the motion state perceived by the vestibular system, which does not conform to the actual life scene, so it will cause the user to have a stronger dizziness and cause motion sickness. In order to avoid motion sickness as much as possible, the latency of the terminal display device during cloud rendering needs to be less than 20ms. However, the latency generated by the serial end cloud rendering method is difficult to meet this requirement, which is likely to cause motion sickness in users.

[0106] In order to solve the above problems, Fig. 9 An exemplary execution flow diagram of an asynchronous end-to-cloud rendering method provided in an embodiment of the present application is shown as follows: Fig. 9As shown, the terminal display device first performs motion capture on the posture and position at the Pth moment to obtain the motion capture information at the Pth moment, and predicts the motion prediction information at the future moment, and reports this information to the cloud server through instructions. On the one hand, after receiving this information, the cloud server side will determine the display resources at the Pth moment (for example, including the current display resources at the Pth moment and the predicted display resources at the future moment) through logical calculation, and then perform basic rendering on the display resources at the Pth moment, and then encode and compress the display resources at the Pth moment after basic rendering, and transmit them to the terminal display device through instructions. On the other hand, after reporting this information, the terminal display device no longer waits for the processing result of the cloud server, but rotates, translates, distorts, and other head-motion renderings on the basic rendering display resources of the historical moment (for example, the PM moment, P and M are both positive integers greater than 0, and M is less than P) sent back by the cloud server before, and uses the display resources at the PM moment after head-motion rendering to refresh the display screen. After the terminal display device refreshes the display screen, if the next motion capture moment has been reached, the above process is repeated. If the next motion capture moment has not been reached, the display resources of the historical moment (such as the Pth moment) previously sent by the cloud server are continued to be obtained, and the head motion rendering and refresh display screen operations are performed. It can be seen that in the asynchronous end-to-cloud rendering mode, the terminal display device and the cloud server each perform their own rendering process. At the same time, the terminal display device and the cloud server can respectively render the display resources at different times. In this case, the latency of the terminal display device can actually be determined by the speed of the terminal display device's head motion rendering, and no longer depends on the basic rendering speed of the cloud server, which helps to reduce the latency of the terminal display device.

[0107] based on Fig. 9 The execution flow shown is as follows: Fig.10 A time series diagram exemplarily illustrates the asynchronous end-to-cloud rendering method provided by an embodiment of the present application, such as Fig.10 As shown in the figure, assuming that the sum of the delay of motion capture and reporting, logic calculation, basic rendering and encoding compression, head motion rendering and display resource transmission is called a cycle delay, the display screen of the terminal display device is always refreshed according to the cycle delay, and the display screen refreshed by the terminal display device at any time is the display resource corresponding to the head motion at a historical moment one cycle delay before the moment. Therefore, the difference between the display screen that the new head motion generated by the user within the length of a cycle delay and the display resource corresponding to the head motion at the historical moment determines the size of the black border generated after refreshing the display screen.

[0108] For ease of understanding, Fig.11The following is a schematic diagram showing FOV changes during a cloud rendering process provided by an embodiment of the present application. Fig.11 As shown, based on the original FOV, if the motion capture information sent at the Pth moment corresponds to a 5-degree right turn, the display screen corresponding to the display resources received by the terminal display device is shown in FOV1 in Figure A. However, if the user turns right 2 degrees again before the terminal device refreshes the display screen, then according to the default FOV display range, the display screen that the terminal display device should actually display to the user is shown in FOV2 in Figure A. In this case, if according to the existing scheme, since the terminal display device only has display resources corresponding to the display screen of FOV1, but no display resources corresponding to a 2-degree right turn, the display area corresponding to the 2 degrees on the right side of the refreshed display screen is a black border. According to the scheme in the present application, the network device sends the predicted display resources of a 2-degree right turn corresponding to the action prediction information at the future moment to the terminal display device in advance. In this way, the terminal display device can refresh a super-viewing angle FOV3 based on the current FOV1 and the predicted display resources of a 2-degree right turn, and the display range of FOV3 is larger than the default FOV display range. In this way, since the user turns right 2 degrees and is within the display range of FOV3, no black edges will appear on the display screen seen by the user, thereby providing a better viewing experience for the user.

[0109] In the embodiment of the present application, after the terminal display device receives the target display resource, it can immediately perform the head motion rendering and refresh the display screen. In this case, if the terminal display device refreshes the display screen at a future time, the solution in the present application can display the display resources corresponding to the predicted user's future action to the user in real time in the future. This method can not only update the display screen seen by the user in real time with the user's head turning action to improve the user's head turning experience, but also ensure that no black edges appear on the display screen seen by the user when the predicted user action is accurate. If the terminal display device refreshes the display screen before the future time, the terminal display device actually displays a super-viewing angle FOV screen to the user. In this way, as long as the angle at which the user turns his head is within the display range of FOV3, no black edges will appear on the display screen seen by the user, so that the user's viewing experience is better.

[0110] In an optional implementation, when the future moment is over, the terminal display device can restore the display range of the current FOV to the display range of the default FOV to reduce the resource consumption of the terminal display device. Alternatively, the terminal display device can first determine the service type of the current service used by the user. If it is a weakly interactive service (such as watching a movie or live broadcast), the user may not have any new actions for a long time after the future moment. In this case, the terminal display device can restore the display range of the current FOV to the display range of the default FOV, and increase the display range of the FOV when the next action occurs. If it is a strongly interactive service (such as playing games), new actions may occur soon after the future moment. In this case, the terminal display device may not restore the display range of the current FOV, but wait for the next new action to occur and directly update the display range of the current FOV to the display range of the new FOV.

[0111] In the above-mentioned embodiment of the present application, by providing the predicted display resources of the future moment in advance to the terminal display device (such as a virtual reality product), the terminal display device can use the predicted display resources to refresh the display screen before or at the future moment. In this case, as long as the prediction is accurate, the user can see the display screen of the future moment in advance before the future moment or see the display screen of the future moment in real time at the future moment. In this way, even if the user turns his head to the future moment in advance, there will be no black edges or very few black edges on the display screen seen by the user. Therefore, this method can avoid or alleviate the phenomenon of black edges appearing on the display screen of the terminal display device due to delay problems, which helps to improve the quality of the display screen in the terminal display device and the user's head turning experience. Furthermore, since this method predicts the user's action at the future moment in advance, the server can only render and transmit the predicted display resources in the direction corresponding to the action at the future moment, and does not need to render and transmit the display resources in all directions, so this method can also save resource consumption in the server, reduce the network overhead of display resource transmission, and improve the refresh efficiency of the display screen. It can be seen that the data rendering method in the present application can take into account both the resource consumption of the cloud server and the viewing experience of the user.

[0112] The rendering method in the embodiment of the present application can be applied to any scenario that requires data rendering. For example, when a user calls a cloud resource to watch a movie, the rendering method can be used to render the movie screen. When a user calls a cloud resource to play a game, the method can also be used to render the game screen. When a user calls a cloud resource to broadcast a live broadcast, the method can also be used to render the live broadcast screen. The various scenarios described in this content can be pre-bound to the app of the terminal display device. If the system architecture is Figure 1BThe system architecture shown in the figure, the terminal display device can directly execute after entering the corresponding app Figure 4 However, if the system architecture is Figure 1A The system architecture shown in the figure is executed Figure 4 Before the rendering method shown, the terminal device needs to establish a connection with the display device. For example, when a user uses a game app, FIG. 12A to FIG. 12D The following is a schematic diagram showing an interface change of establishing a connection between a terminal device and a display device provided in an embodiment of the present application. In this example, the terminal device is assumed to be a mobile phone X, and the display device is assumed to be a pair of glasses Y. Fig. 12A As shown, the display screen of the mobile phone X may display icons of various apps. If the glasses Y are pre-bound with the game app, when the user clicks the game app icon 1201 on the display screen, the mobile phone X may display the following Fig. 12B The connection interface shown in the figure shows a one-way arrow pointing from phone X to glasses Y, which means that phone X is trying to automatically establish a connection with glasses Y, and the process of establishing the connection can be invisible to the user. Fig. 12C As shown in the figure, a two-way arrow will be displayed on the connection interface of the mobile phone, with the two ends of the arrow pointing to mobile phone X and glasses Y respectively, which means that mobile phone X and glasses Y are successfully connected. In this case, the mobile phone enters Fig.12D The rendering setting interface shown includes multiple configuration boxes, and the contents of the multiple configuration boxes include but are not limited to the following: whether to turn on motion prediction, when to turn on motion prediction, when to end motion prediction, the frequency of reporting motion prediction information when motion prediction is turned on, whether to perform motion capture, when to start motion capture, when to end motion capture, the frequency of reporting motion capture information when motion capture is performed, etc.

[0113] It should be noted that "the game app is pre-bound to the glasses" is only an optional implementation. In another optional implementation, the game app is not bound to the glasses. In this case, the user can manually establish a connection between the mobile phone X and the glasses Y before (or after) opening the game app. The connection establishment process is visible to the user. In this case, the interface changes on the mobile phone can be as follows: FIG. 13A to FIG. 13C As shown. Fig.13A As shown, a connection function key 1301 (such as a Bluetooth on button) may be provided on the mobile phone X. When the user clicks the connection function key 1301, a list of connectable devices may be displayed on the display interface of the mobile phone X, such as Fig. 13BThe list of connectable devices includes one or more icons (and / or text) of connectable devices. One or more connectable devices are devices with good communication quality with mobile phone X, such as bracelet T that is no more than 5 meters away from mobile phone X and has turned on Bluetooth search, or glasses Y that have established a connection with mobile phone X, or tablet R that is connected to the same wifi as mobile phone X, mobile phone W that is a friend or relative device of mobile phone X, etc. Further, the user can select one or more icons of connectable devices to click, such as clicking the icon of glasses Y. At this time, glasses Y can vibrate or ring to show the user that a connection request has been received. The user can press a button on glasses Y to allow glasses Y to establish a connection with mobile phone X. When the connection is successfully established, the display interface of mobile phone X can display the connection result, which can be Fig. 13C The text form shown is "has been connected with glasses Y", which can also be Fig. 12C The double arrow form shown in the figure, the connection result may also include other information, such as the model of glasses Y, Media Access Control Address (MAC), etc., which is not limited in this application. Further, when the user clicks on the game app, if the mobile phone X detects that it has established a connection with the glasses Y, the mobile phone X can directly display the following information to the user: Fig.12D The display rendering configuration interface is shown.

[0114] The solution in the embodiment of the present application can also be applied in other scenarios. For example, there may be a scenario in the future where a user drives while wearing VR glasses (or AR glasses). In this case, if the VR glasses are currently used in navigation mode, the VR glasses can navigate in combination with the actual road conditions. When it is determined that a right turn is required at the current moment, a dotted line mark to the right can be displayed in the display screen of the VR glasses. The user can turn his head to the right according to the prompt of the dotted line mark, and as the head turns, the virtual image displayed in the VR glasses will also turn accordingly. In this scenario, the user can see both the real road conditions and the virtual image displayed by the VR glasses, and the virtual image can also rotate according to the rotation of the person's head, and there is a high probability that no black edges will appear. When the solution in the present application is applied to this scenario, the user can navigate to the destination according to the virtual image with better display effect, the user experience is better, and the navigation effect is also better.

[0115] In the embodiment of the present application, the more the action prediction information matches the user's actual action at a future moment, the higher the accuracy of the action prediction information, and the predicted display resources determined based on the action prediction information can correspond to fewer black edges. The accuracy of the action prediction information is related to the predicted time of obtaining the action prediction information and the amount of reported action prediction information. The closer the predicted time of obtaining the action prediction information is to a future moment and the more the reported action prediction information is, the higher the accuracy of the action prediction information is. The farther the predicted time of obtaining the action prediction information is from a future moment and the less the reported action prediction information is, the lower the accuracy of the action prediction information is.

[0116] Based on this, the present application can also improve the prediction accuracy by reporting the action prediction information multiple times. Two possible implementation methods of reporting the action prediction information multiple times are introduced from the second embodiment and the third embodiment respectively.

[0117] Embodiment 2

[0118] Fig.14 The flowchart corresponding to the rendering method provided in the second embodiment of the present application is exemplarily shown. The method can be applied to a cloud server and a terminal display device, for example Figure 1A The cloud server, terminal device and display device shown, or Figure 1B The cloud server and terminal display device are shown in FIG. Fig.14 As shown, the method may include:

[0119] Step 1401: The terminal display device displays the motion capture information corresponding to time K and the motion prediction information X corresponding to time K+5 at time K. 11 Sent to the cloud server.

[0120] Step 1402: The terminal display device displays the motion capture information corresponding to the K+1 time and the motion prediction information X corresponding to the K+5 time at the K+1 time. 12 Sent to the cloud server.

[0121] Step 1403: The terminal display device displays the motion capture information corresponding to the K+2 time and the motion prediction information X corresponding to the K+5 time at the K+2 time. 13 Sent to the cloud server.

[0122] In the embodiment of the present application, the action prediction information in the above steps 1401 to 1403 may be obtained by performing action prediction at the corresponding time, for example, the action prediction information X 11 It is obtained by predicting the action at time K+5 at time K. The action prediction information X 12 The action prediction information X is obtained by predicting the action at time K+5 at time K+1. 13It is obtained by predicting the action at time K+5 at time K+2. For each of the times K, K+1, and K+2, after the cloud server receives the action capture information corresponding to that time, it can also determine the current display resource from the initial display resources according to the action capture information at that time, and then perform basic rendering on the current display resource and send it to the terminal display device so that the terminal display device can refresh the display screen.

[0123] Step 1404, the cloud server calculates the target action prediction information corresponding to time K+5 according to the action prediction information X 11 , the action prediction information X 12 and the action prediction information X 13 .

[0124] In an optional implementation manner, the cloud server may use the weighted average of the action prediction information X 11 , the action prediction information X 12 and the action prediction information X 13 as the target action prediction information. Among them, the weights corresponding to these three action prediction information can be set according to the time difference between the time when each is predicted and time K+5. When the time difference between the time when the action prediction information is predicted and time K+5 is smaller, the accuracy of the prediction of this action prediction information is higher, so the weight corresponding to the prediction of this action prediction information is larger. When the time difference between the time when the action prediction information is predicted and time K+5 is larger, the accuracy of the prediction of this action prediction information is lower, so the weight corresponding to the prediction of this action prediction information is smaller. For example, assuming that the weights of the action prediction information X 11 , the action prediction information X 12 and the action prediction information X 13 are w1, w2, and w3 respectively, then because the time difference between time K when the action prediction information X 11 is predicted and time K+5 > the time difference between time K+1 when the action prediction information X 12 is predicted and time K+5 > the time difference between time K+2 when the action prediction information X 13 is predicted and time K+5, so it can be set that w1 < w2 < w3, and the sum of w1, w2, and w3 is 1. In this case, the target action prediction information can be: w1 * the action prediction information X 11 + w2 * the action prediction information X 12 and w3 * the action prediction information X 13 . Adopting this implementation manner, in the case of reporting action prediction information multiple times, setting a larger weight for the relatively accurate action prediction information close to the future time and a smaller weight for the relatively inaccurate action prediction information far from the future time helps to make the calculated target action prediction information more accurate.

[0125] It can be understood that “using the weighted average of multiple action prediction information as the target action prediction information” is only an optional implementation. In other optional implementations, the cloud server can also directly use the action prediction information X as the target action prediction information. 11 , action prediction information X 12 and action prediction information X 13 The average value of is used as the target action prediction information, without specific limitation.

[0126] In step 1405, the cloud server determines the current display resources from the initial display resources based on the motion capture information corresponding to the K+2 moment, determines the predicted display resources from the initial display resources based on the target motion prediction information corresponding to the K+5 moment, determines the target display resources based on the current display resources and the predicted display resources, and performs basic rendering on the target display resources.

[0127] In an optional implementation, if the degree of difference between multiple action prediction information is not greater than the preset degree of difference, it means that the action prediction information obtained each time is basically the same, and the accuracy of the action prediction is good. In this case, the high-quality current display resources and the high-quality predicted display resources can be directly used as the target display resources to maximize the user's viewing experience when the prediction is accurate. If the degree of difference between multiple action prediction information is greater than the preset degree of difference, it means that the action prediction information obtained by the multiple predictions is significantly different, and the accuracy of the action prediction is not high. In this case, the high-quality current display resources and the low-quality predicted display resources can be used as the target display resources to minimize the resource consumption of the cloud server when the prediction is inaccurate. Although low-quality predicted display resources are transmitted in this case, low-quality predicted display resources can also alleviate the black edge phenomenon, but the display screen viewed by the user is not as clear as the high-quality predicted display resources.

[0128] In the embodiment of the present application, "using high-quality current display resources and low-quality predicted display resources as target display resources when the prediction is inaccurate" is only an optional implementation method. In other optional implementation methods, the cloud server may also use high-quality current display resources and part of the low-quality display resources corresponding to the initial display resources in all directions as target display resources, or use high-quality current display resources and the entire low-quality initial display resources as target display resources, without specific limitation.

[0129] In the above implementation, the preset difference degree can be set by those skilled in the art based on experience. For example, in one example, the difference degree of the plurality of action prediction information is not greater than the preset difference degree, which may include one or more of the following:

[0130] The difference between any two pieces of action prediction information is not greater than a first preset difference. For example, when the first preset difference is 0.5 degrees, the difference between the head turning angles predicted at any two moments is not greater than 0.5 degrees.

[0131] The average value of the difference between any two pieces of action prediction information is not greater than a second preset difference;

[0132] The standard deviation of the plurality of action prediction information is not greater than a third preset difference value;

[0133] The variance of the plurality of action prediction information is not greater than a fourth preset difference value.

[0134] Correspondingly, the difference degree of the plurality of action prediction information is greater than the preset difference degree, which may include one or more of the following contents:

[0135] There are two action prediction information whose difference is greater than the first preset difference. For example, when the first preset difference is 0.5 degrees, the action prediction information X 11 To turn right 1 degree, the action prediction information X 12 It is a right turn of 1.6 degrees;

[0136] There is an average value of the difference between two pieces of action prediction information that is greater than a second preset difference;

[0137] The standard deviation of the plurality of action prediction information is greater than a third preset difference value;

[0138] The variance of the plurality of action prediction information is greater than a fourth preset difference value.

[0139] Step 1406: The cloud server sends the target display resource after basic rendering to the terminal display device.

[0140] Step 1407: the terminal display device performs head motion rendering on the target display resource after the basic rendering, and uses the target display resource after the head motion rendering to refresh the display screen.

[0141] In the second embodiment, the terminal display device performs the rendering process in a polling manner, reports the action prediction information at the same time at each moment of each polling cycle, and after each polling cycle ends, the next polling cycle is started. According to the scheme in steps 1401 to 1407, the relevant configuration of reporting the action prediction information in the second embodiment is: starting from the first moment of each polling cycle, the action prediction information is reported at a reporting frequency of once at each moment, and after reporting 3 times in a polling cycle, the corresponding prediction display resource is issued. In this case, if the polling cycle is 5, the terminal display device will report the action prediction information of the last moment in the first 3 moments of each polling cycle, so that the user has a better viewing experience at the last moment of each polling cycle. According to this implementation, if the duration of the polling cycle is set small enough, for example, a polling cycle is 5ms, then because the time difference of 5ms is not sensitive to the user's perception, the user's viewing experience can also be always good.

[0142] In the embodiment of the present application, the configuration related to reporting the motion prediction information and reporting the motion capture information may be set by one or more of the following methods, for example:

[0143] In an optional implementation, the terminal display device can display the following information to the user when the app is started: Fig.12D The configuration interface shown in the figure is used to determine whether to start motion prediction, when to start motion prediction, at what frequency to report motion prediction information, whether to perform motion capture, when to start motion capture, and at what frequency to report motion capture information according to the information input by the user on the configuration interface. Alternatively, the cycle of motion prediction and motion capture can be set according to the user's configuration.

[0144] In another optional implementation, the terminal display device may receive high-level indication information sent by the cloud server, and then determine the above-mentioned configuration contents according to the high-level indication information.

[0145] In another optional implementation, the terminal display device may automatically identify the service type of the app being used by the user and execute:

[0146] If it is determined to be a weakly interactive extended reality (XR) service (such as a video service such as a movie), since users basically do not turn their heads in such services, the terminal display device can directly turn off motion prediction and not perform motion capture, or it can turn on motion prediction and perform motion capture, but set the frequency of reporting motion prediction information and motion capture information to be lower to save cloud server resource consumption;

[0147] If it is determined to be a highly interactive XR service (such as a game service or a live broadcast service), then since users frequently turn their heads in such services, the terminal display device can enable motion prediction and motion capture, and the frequency of motion prediction information and motion capture information can be set to be relatively high. For example, the terminal display device calls the cloud sensor to report the motion capture information at each moment and the motion prediction information after a period of time. In this way, the cloud server can not only provide the display resources of the future moment to the terminal display device in advance to alleviate the black edge phenomenon, but also provide the terminal display device with display resources matching the current moment's motion in a smaller cycle, which helps the terminal display device to refresh the display screen in time and improve the user's viewing experience and head turning experience;

[0148] If the service type to which the app belongs cannot be identified, in one case, the terminal display device can detect the user's head turning in real time through a motion sensor. If the data from the motion sensor does not change, it means that the user does not turn his head. In this case, it can be determined that the service type belongs to a weakly interactive XR service, and the above-mentioned configuration contents are set in a manner corresponding to the weakly interactive XR service. If the data from the motion sensor changes frequently, it means that the user turns his head frequently. In this case, it can be determined that the service type belongs to a strongly interactive XR service, and the above-mentioned configuration contents are set in a manner corresponding to the strongly interactive XR service. In another case, the terminal display device can also directly display the following to the user. Fig.12D The configuration interface shown in the figure is displayed, and then the above configuration contents are determined according to the information input by the user on the configuration interface.

[0149] The solution in the second embodiment can make the display screen seen by the user at the action prediction moment have no black edges, but it may not solve the black edge phenomenon at other times except the action prediction moment. When the polling cycle is large, the user's viewing experience at other times may not be good. The solution in the third embodiment can solve the above problem.

[0150] Embodiment 3

[0151] Fig.15 The flowchart corresponding to the rendering method provided in the third embodiment of the present application is exemplarily shown. The method can be applied to a cloud server and a terminal display device, for example Figure 1A The cloud server, terminal device and display device shown, or Figure 1B The cloud server and terminal display device are shown in FIG. Fig.15 As shown, the method includes:

[0152] Step 1501: At time Q, the terminal display device displays the motion capture information corresponding to time Q and the motion prediction information X corresponding to time Q+5. 21Sent to the cloud server.

[0153] Step 1502: The terminal display device displays the motion capture information corresponding to time Q+1 and the motion prediction information X corresponding to time Q+5 at time Q+1. 22 The action prediction information X corresponding to the Q+6 moment 31 Sent to the cloud server.

[0154] Step 1503: The terminal display device displays the motion capture information corresponding to time Q+2 and the motion prediction information X corresponding to time Q+5 at time Q+2. 23 , the action prediction information X corresponding to the Q+6 time 32 The action prediction information X corresponding to the Q+7 time is sent to the cloud server 41 Sent to the cloud server.

[0155] Step 1504: The cloud server predicts the action X corresponding to the Q+5 time. 21 , action prediction information X 22 and action prediction information X 23 Determine the target action prediction information corresponding to time Q+5.

[0156] Step 1505, the cloud server determines the current display resources from the initial display resources based on the motion capture information corresponding to the Q+2 moment, determines the predicted display resources from the initial display resources based on the target motion prediction information corresponding to the Q+5 moment, determines the target display resources based on the current display resources and the predicted display resources, and performs basic rendering on the target display resources.

[0157] Step 1506: The cloud server performs basic rendering on the target display resource, and sends the target display resource after basic rendering to the terminal display device.

[0158] Step 1507: the terminal display device performs head-motion rendering on the target display resource after the basic rendering, and uses the head-motion rendered target display resource to refresh the display screen.

[0159] In the second embodiment, the cloud server will determine the predicted display resources for the future moments at the Q+2 moment and each moment thereafter according to the action prediction information for the future moments reported at the previous three moments, and send the predicted display resources to the terminal display device in advance. For example, after the Q+2 moment, the cloud server will determine the predicted display resources corresponding to the Q+5 moment according to the three action prediction information corresponding to the Q+5 moment reported at the Q moment, the Q+1 moment, and the Q+2 moment, and send them to the terminal display device in advance. It will also store the action prediction information corresponding to the Q+6 moment reported at the Q+1 moment and the Q+2 moment, and then after the Q+3 moment, it will combine the action prediction information corresponding to the Q+6 moment reported at the Q+3 moment, determine the predicted display resources corresponding to the Q+6 moment, and send them to the terminal display device in advance, and so on. Obviously, in addition to reporting only one action prediction information at the first moment at the start, and only reporting two action prediction information at the second moment at the start, this method will report three action prediction information at each subsequent moment, and a maximum of three action prediction information will be reported. In this way, although more data is reported at each subsequent moment, this method can send the corresponding predicted display data to the terminal display device in advance at each subsequent moment starting from the third moment. Therefore, the black edge phenomenon in the display screen of the terminal display device at moment Q+2 and each subsequent moment can be alleviated. Compared with the solution in Example 2, the user's viewing experience and head turning experience are better under the solution in Example 3.

[0160] The above-mentioned Embodiments 2 and 3 both directly determine the predicted display resources based on the action prediction information reported by the terminal display device. The display screen in this way depends on the accuracy of the action prediction information. If the action prediction information is inaccurate, the corresponding predicted display resources may not match the user's actual action, resulting in the display screen not matching the user's actual head turning situation. Based on this, referring to Embodiment 4, the present application also provides a rendering method, which combines the user action information reported by other terminal display devices to comprehensively determine the target display resources, which helps to make the target display resources more consistent with the user's actual head turning situation.

[0161] Embodiment 4

[0162] Fig.16 The interactive flow diagram corresponding to the rendering method provided in the fourth embodiment of the present application is exemplarily shown. The method can be applied to a cloud server, a first terminal display device, and a second terminal display device. The first terminal display device and the second terminal display device can be connected to the cloud server respectively, and the first terminal display device and the cloud server, or the second terminal display device and the cloud server can be connected according to Figure 1A The system architecture shown can also be connected according to Figure 1B The system architecture shown is connected as shown in the figure. Fig.16 As shown, the method may include:

[0163] Step 1601, one or more first terminal display devices report learning data at the first moment to the cloud server, and the learning data at the first moment reported by each first terminal display device is used to indicate the real action information of the user wearing the first terminal display device at the first moment.

[0164] In the embodiments of the present application, a moment can also be referred to as a frame, which specifically refers to a moment in the app used by the user. For example, when a user is watching a movie on a video app, a moment corresponds to a moment in the duration from the start to the end of the movie, and does not refer to a moment in reality. For ease of understanding, the following embodiments are introduced using watching a movie as an example.

[0165] In an optional implementation, the learning data at the first moment sent by each first terminal display device may include the action prediction information and action prediction error information of the user wearing the first terminal display device at the first moment. When the user uses the first terminal display device, if the first terminal display device detects that the action prediction is turned on and the first moment is one of the predicted moments, the first terminal display device will automatically report the user's action prediction information at the first moment to the cloud server before the first moment. The cloud server can use the action prediction information to determine the target display resource at the first moment to execute the rendering scheme in any of the embodiments 1 to 3, and on the other hand, the user's action prediction information at the first moment can also be stored in the internal storage space. Further, the first terminal display device will also determine whether the user agrees to participate in the VR experience project. If the user agrees to participate, when the first moment arrives, the first terminal display device will also call the motion sensor to obtain the user's action capture information at the first moment, calculate the difference between the action prediction information at the first moment and the action capture information to obtain the action prediction error information at the first moment, and report it to the cloud server. In this case, the cloud server can determine the user's real action information at the first moment based on the previously stored action prediction information of the user at the first moment and the action prediction error information. For example, when the user's action prediction information at the first moment is to turn right 5 degrees, and the user's action prediction error information at the first moment is 2 degrees, it can be determined that the user's actual action information at the first moment is to turn right 3 degrees.

[0166] It can be understood that "the learning data includes the action prediction information and action prediction error information of the user at the first moment" is only an optional implementation manner. In another optional implementation manner, when it is determined that the user agrees to participate in the VR experience project, the first terminal display device can also directly report the action capture information of the user at the first moment to the cloud server. In this case, the learning data directly includes the action capture information of the user at the first moment, and the cloud server can directly determine the real action of the user at the first moment based on the action capture information of the user at the first moment. In yet another optional implementation manner, the first terminal display device can also report all the feedback information of the user using the initial display resources as learning data to the cloud server. For example, all the action information of the user watching the entire movie video using the first terminal display device, or all the action information of the user playing games using the first terminal display device, etc.

[0167] In the embodiments of the present application, "whether the user agrees to participate in the VR experience project" can be specifically indicated by high-level configuration information, or can also be configured by the user himself. When the user configures it himself, the first terminal display device can display the configuration interface of the VR experience project to the user when the device is powered on for the first time, or when the user triggers the start of a certain app, or when the user clicks the permission configuration button of the VR experience project. The user can input configuration information according to his own needs on this configuration interface. When the user agrees to participate in the VR experience project, it means that the user allows the first terminal display device to report the VR action information of himself using this device to the cloud server, and agrees that the cloud server improves the accuracy of predicting the user's actions based on the VR action information of other terminal display devices. When the user does not participate in the VR experience project, it means that the user does not allow the first terminal display device to report the VR action information of himself using this device to the cloud server. Correspondingly, the cloud server will not improve the accuracy of predicting the user's actions based on the VR action information of other terminal display devices. By allowing the user to decide whether to report the VR action information to the cloud server himself, the privacy protection awareness of the user can be taken into account during the viewing experience.

[0168] Step 1602, the cloud server trains a prediction model according to the learning data at the first moment reported by each first terminal display device.

[0169] In the embodiments of the present application, the type of the prediction model can be set by those skilled in the art according to experience. For example, it can include but is not limited to the following several types:

[0170] (1) Neural network model: The cloud server can predefine an initial neural network model that contains multiple learnable parameters (also called weights). The initial neural network model includes a multi-layer network structure. After receiving various learning data, the learning data is used for iterative calculations, and the learning data is processed through the multi-layer network structure to calculate the loss value between the output value and the target value. The loss value is back-propagated to the parameters of the initial neural network model, and the initial neural network model is updated according to the preset update rules to obtain a trained neural network model.

[0171] (2) Classification model: It can include supervised learning model, semi-supervised learning model or unsupervised learning model. If the prediction model is a supervised learning model, the cloud server can first label each learning data, and then use one or more of the support vector machine algorithm, linear discriminant algorithm, decision tree algorithm or naive Bayes algorithm to train the labeled learning data to obtain a supervised learning model. If the prediction model is a semi-supervised learning model, the cloud server can first label part of the learning data, and then use the support vector machine algorithm to train the labeled learning data and unlabeled learning data to obtain a semi-supervised learning model. If the prediction model is an unsupervised learning model, the cloud server can directly use the k-clustering algorithm or principal component analysis algorithm to train the unlabeled learning data to obtain an unsupervised learning model.

[0172] (3) Fitting model: The type of fitting model can be a polynomial model, a linear regression model, or a b-spline model. The cloud server can predefine an initial fitting model containing multiple unknown parameters. The independent variable of the initial fitting model is the time, and the dependent variable is the real action information corresponding to the time. After receiving each learning data, the cloud server can substitute the same time in each learning data and the real action information corresponding to the same time into the initial fitting model to solve each unknown parameter. In this way, the prediction model can be obtained by substituting the solved unknown parameters into the initial fitting model.

[0173] In an embodiment of the present application, when the number of first terminal display devices is larger, the amount of learning data is larger, and the prediction model obtained by training with a large amount of learning data can comprehensively consider the characteristics of each user, so that the accuracy of the prediction model is relatively higher. However, if the amount of learning data is very large, it may take a long time to train the prediction model, resulting in low training efficiency. Based on this, in an optional implementation, if the total amount of data of each learning data is not greater than the preset data amount threshold, the cloud server can directly use each learning data to train the prediction model. If the total amount of data of each learning data is greater than the preset data amount threshold, the cloud server can first divide each learning data into multiple sub-learning data, and use multiple threads to train multiple sub-learning data to obtain multiple sub-prediction models, and then aggregate multiple sub-prediction models to obtain a prediction model. This method can improve the efficiency of model training.

[0174] In an embodiment of the present application, the prediction model can also be updated in real time. For example, when a new first terminal display device joins the VR experience project, or when the old first terminal display device has new learning data, the cloud server can use the learning data reported by the new first terminal display device or the new learning data reported by the old first terminal display device to update the prediction model, or it can also combine these new learning data with the previously acquired learning data to retrain the prediction model, without specific limitation.

[0175] In an optional implementation, different business types can correspond to different prediction models. For example, the R&D personnel can classify the usage scenarios of each app in advance. When each first terminal display device reports the learning data of a certain app, it can also report the corresponding usage scenario to the cloud server. Correspondingly, after the cloud server receives the learning data and application scenarios reported by each first terminal display device, it can first classify the learning data belonging to the same application scenario according to the application scenario, and then use the learning data of each application scenario to train the prediction model corresponding to each application scenario. In another optional implementation, considering that different users may have different head turning habits (for example, some users are used to turning their heads to the lower right when turning their heads right, while some users are used to turning their heads to the upper right), the cloud server can also customize a dedicated prediction model for each user. When constructing a prediction model exclusive to a certain user, the cloud server can obtain the learning data corresponding to the app of various types of services used by the user, and extract the features in these learning data, and establish a prediction model exclusive to the user based on the features. In this case, the user-specific prediction model constructed is suitable for predicting the real action information of the user under various types of services. In another optional implementation, a prediction model can be constructed for each user under each business type. In this case, each user can correspond to multiple models. When predicting the user's actions under a certain business type later, the prediction model corresponding to the user under the business type can be directly used for prediction. This method enables the prediction model to take into account both user characteristics and business characteristics, and the prediction effect of the prediction model is better.

[0176] Step 1603: the second terminal display device sends the action prediction information X of the user wearing the second terminal display device at the first moment to the cloud server at one or more moments before the first moment. 51 .

[0177] Step 1604: The cloud server determines whether the second terminal display device has started the VR experience project:

[0178] If the VR experience project is not enabled on the second terminal display device, execute step 1605;

[0179] When the VR experience project is turned on on the second terminal display device, step 1606 is executed.

[0180] Step 1605: The cloud server uses the action prediction information X 51 Determine the predicted display resource corresponding to the first moment.

[0181] In the embodiment of the present application, when the second terminal display device does not open the VR experience project, the user does not want to use the action prediction information X51 Optimize the prediction model, and there is no need to use the prediction model to optimize the viewing experience. In this case, the action prediction information X 51 Used to directly determine the predicted display resource corresponding to the user at the first moment, the accuracy of the predicted display resource depends on the action prediction information X reported by the second terminal device 51 According to the action prediction information X 51 The specific implementation process of determining the predicted display resources may refer to the first embodiment, which will not be described in detail here.

[0182] Step 1606: The cloud server uses the prediction model to determine the action prediction information X corresponding to the user wearing the second terminal display device at the first moment. 52 :

[0183] If the action prediction information X 51 If it is only used for VR experience projects, execute step 1607;

[0184] If the action prediction information X 51 If it is used for both VR experience project and prediction, execute step 1608.

[0185] In an optional implementation, the action prediction information X 51 Whether it is used only for VR experience projects or for both VR experience projects and prediction can be indicated by the user using the second terminal display device, or can be determined by the preset configuration in the cloud server. Among them, the preset configuration can directly indicate a configuration, in which case the preset configuration is applicable to each second terminal display device. Alternatively, the preset configuration can also indicate the configuration corresponding to each second terminal display device. In this case, the action prediction information X sent by different second terminal display devices 51 It may correspond to different configurations.

[0186] Step 1607: The cloud server calculates the action prediction information X. 52 Determine the predicted display resource corresponding to the first moment.

[0187] In the embodiment of the present application, when the second terminal display device starts the VR experience project, if the action prediction information X 51 Only used for VR experience projects, it means that the user wants to directly use the prediction model to predict the user's actions to optimize the viewing experience, and the second terminal displays the action prediction information reported by the device X 51 It is only used to optimize the prediction model, not to optimize the viewing experience. In this case, the cloud server can directly obtain the user's action prediction information X at the first moment predicted by the prediction model. 52 , the action prediction information X 52The corresponding entire data stream or part of the data stream is determined as the predicted display resource. Obviously, in this way, the accuracy of the predicted display resource depends on the accuracy of the prediction model.

[0188] Step 1608: The cloud server calculates the action prediction information X. 51 and action prediction information X 52 Determine the predicted display resource corresponding to the first moment.

[0189] In the embodiment of the present application, when the second terminal display device starts the VR experience project, if the action prediction information X 51 If it is used for VR experience projects and prediction at the same time, it means that the user wants to integrate the prediction model and the action prediction information reported by the second terminal display device X 51 To predict the user's action. In this case, the cloud server can first predict the user's action based on the action prediction information X 51 and action prediction information X 52 Determine the target action prediction information, and then use the entire data stream or part of the data stream corresponding to the target action prediction information as the prediction display resource corresponding to the first moment. This method uses the prediction model and action prediction information X in combination. 51 To predict the user's actions, the accuracy of the predicted display resources can depend on both the accuracy of the prediction model and the action prediction information X 51 The accuracy of the prediction is that the final prediction display resource is obtained by combining two parts of information, rather than relying solely on the prediction model or action prediction information X 51 ,Therefore, this method can reduce the probability of inaccurate prediction of predicted display resources and help alleviate the black edge problem.

[0190] In an optional implementation, the cloud server can send the action prediction information X 51 and action prediction information X 52 The weighted average of is taken as the target action prediction information. For example, in the action prediction information X 51 Including action prediction information X 511 , action prediction information X 512 and action prediction information X 513 In the case of , the target action prediction information can be: t1*action prediction information X 511 +t2*action prediction information X 512 +t3*action prediction information X 513 +t4*action prediction information X 52 , t1, t2, t3 and t4 are the action prediction information X 511 , action prediction information X 512 and action prediction information X 513 and action prediction information X 52The corresponding weights, and the sum of t1, t2, t3, and t4 is 1. Among them, the values of t1, t2, and t3 can be based on the action prediction information X 511 、the action prediction information X 512 and the action prediction information X 513 and the time difference from the first moment are set. For example, if the time difference between the action prediction information X 511 and the first moment, the time difference between the action prediction information X 512 and the first moment, and the time difference between the action prediction information X 513 and the first moment decrease in sequence, then t1 < t2 < t3. And the value of t4 can be determined according to the data volume of the learning data used when training the prediction model. When the data volume of the learning data is large, it indicates that the accuracy of the prediction model is high. Therefore, t4 can be set to a larger value, such as greater than t3. When the data volume of the learning data is small, the accuracy of the prediction model is low. Therefore, t4 can be set to a smaller value, such as less than t1.

[0191] In the above step 1606 or step 1608, since the user wearing the second terminal display device agrees to the VR experience project, the second terminal display device can also call the motion sensor at the first moment to collect the action capture information of the user at the first moment, and determine the action prediction error information of the user at the first moment according to the action capture information and the action prediction information X 51 Then send the action prediction error information to the cloud server, so that the cloud server can use the action prediction information X 51 of the user at the first moment and the action prediction error information to update the prediction model and improve the prediction effect of the prediction model.

[0192] Step 1609, the cloud server sends the predicted display resources after basic rendering to the second terminal display device before the first moment.

[0193] Step 1610, the second terminal display device performs head motion rendering on the predicted display resources after basic rendering and uses the predicted display resources after head motion rendering to refresh the display screen.

[0194] In the fourth embodiment, the cloud server can establish a prediction model in advance based on the learning data of multiple first terminal display devices, and then use the prediction model to optimize the predicted display resources of the second terminal display device. Compared with the method of determining the predicted display resources only based on the action prediction information reported by the second terminal display device, this method can reduce the subjectivity of the predicted display resources and alleviate the problem of being unable to solve the black edge phenomenon of the display screen due to the inaccurate action prediction information reported by the second terminal display device. Furthermore, on this basis, if the predicted display resources are comprehensively determined using the prediction model and the action prediction information reported by the second terminal display device, the predicted display resources can be made more comprehensive, avoiding the problem of inaccurate predicted display resources caused by only using one party's data to determine the predicted display resources, and effectively solving the black edge phenomenon of the display screen.

[0195] It should be noted that the names of the above-mentioned information are only examples. With the evolution of communication technology, the names of any of the above-mentioned information may change. However, no matter how the names change, as long as their meanings are the same as those of the above-mentioned information in this application, they fall within the scope of protection of this application.

[0196] The above mainly introduces the solution provided by the present application from the perspective of the interaction between various network elements. It can be understood that in order to realize the above functions, the above-mentioned network elements include hardware structures and / or software modules corresponding to the execution of various functions. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in this document, the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0197] According to the above method, Fig.17 A schematic diagram of the structure of a data rendering device provided in an embodiment of the present application is shown in FIG. Fig.17 As shown, the data rendering device can be a terminal display device or a server, or a chip or a circuit, such as a chip or circuit that can be arranged in a terminal display device, or a chip or circuit that can be arranged in a server.

[0198] Furthermore, the data rendering device 1701 may further include a bus system, wherein the processor 1702 , the memory 1704 , and the transceiver 1703 may be connected via the bus system.

[0199] It should be understood that the processor 1702 may be a chip. For example, the processor 1702 may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0200] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1702 or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in the processor 1702 for execution. The software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1704, and the processor 1702 reads the information in the memory 1704 and completes the steps of the above method in conjunction with its hardware.

[0201] It should be noted that the processor 1702 in the embodiment of the present application can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor can be combined and performed. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0202] It can be understood that the memory 1704 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0203] When the data rendering device 1701 corresponds to the terminal display device in the above method, the data rendering device may include a processor 1702, a transceiver 1703 and a memory 1704. The memory 1704 is used to store instructions, and the processor 1702 is used to execute the instructions stored in the memory 1704 to implement the above method. Figures 1A to 16 A related scheme of a terminal display device in any one or more corresponding methods shown in .

[0204] When the data rendering 1701 is the above-mentioned terminal display device, the data rendering 1701 can be used to execute the method executed by the terminal display device in any one of the above-mentioned embodiments 1 to 4.

[0205] The data rendering device 1701 is the above-mentioned terminal display device, and when executing the first embodiment, the processor 1702 predicts the user's action prediction information at the future moment, and the action prediction information is used to indicate the user's action at the future moment. The transceiver 1703 sends the first information to the server, and before the future moment arrives, receives the second information sent by the server, the first information includes the action prediction information, and the second information includes the predicted display resources corresponding to the action prediction information after basic rendering. The processor 1702 also uses the predicted display resources to refresh the display screen before or when the future moment arrives.

[0206] When the data rendering device 1701 corresponds to the server in the above method, the data rendering device may include a processor 1702, a transceiver 1703 and a memory 1704. The memory 1704 is used to store instructions, and the processor 1702 is used to execute the instructions stored in the memory 1704 to implement the above method. Figures 1A to 16 A related scheme of a terminal display device in any one or more corresponding methods shown in .

[0207] When the data rendering device 1701 is the above-mentioned server, the data rendering device 1701 can be used to execute the method executed by the server in any one of the above-mentioned embodiments 1 to 4.

[0208] The data rendering device 1701 is the above-mentioned server, and when executing the first embodiment, the transceiver 1703 receives the first information sent by the terminal display device, and the first information includes the user's action prediction information at the future moment, and the action prediction information is used to indicate the user's action at the future moment. The processor 1702 determines the predicted display resource corresponding to the action prediction information from the initial display resource, and performs basic rendering on the predicted display resource. The transceiver 1703 also sends the second information to the terminal display device before the future moment arrives, and the second information includes the predicted display resource after basic rendering.

[0209] For the concepts, explanations, detailed descriptions and other steps involved in the data rendering 1701 and related to the technical solution provided in the embodiment of the present application, please refer to the description of these contents in the aforementioned method or other embodiments, which will not be repeated here.

[0210] According to the above method, Fig.18 A schematic diagram of the structure of a data rendering device provided in an embodiment of the present application is shown in FIG. Fig.18 As shown, the data rendering device 1801 may include a communication interface 1803, a processor 1802 and a memory 1804. The communication interface 1803 is used to input and / or output information; the processor 1802 is used to execute computer programs or instructions so that the data rendering device 1801 can achieve the above Figures 1A to 16The method of the terminal display device side in the related scheme, or the data rendering device 1801 implements the above Figures 1A to 16 In the embodiment of the present application, the communication interface 1803 can implement the above Fig.17 The solution implemented by the transceiver 1703, the processor 1802 can implement the above Fig.17 The solution implemented by the processor 1702, the memory 1804 can implement the above Fig.17 The solution implemented by the memory 1704 will not be described in detail here.

[0211] Based on the above embodiments and the same concept, Fig.19 A schematic diagram of a data rendering device provided in an embodiment of the present application, such as Fig.19 As shown, the data rendering device 1901 can be a terminal display device or a server, or can be a chip or a circuit, such as a chip or a circuit that can be set in a terminal display device or a server.

[0212] The data rendering device 1901 may correspond to the terminal display device in the above method. Figures 1A to 16 The steps performed by the terminal display device in any one or more of the corresponding methods shown in . The data rendering device 1901 may include a processing unit 1902 and a transceiver unit 1903 .

[0213] When the data rendering device 1901 is the above-mentioned terminal display device, the data rendering device 1901 can be used to execute the method executed by the terminal display device in any one of the above-mentioned embodiments 1 to 4.

[0214] The data rendering device 1901 is the above-mentioned terminal display device, and when executing the first embodiment, the processing unit 1902 predicts the user's action prediction information at the future moment, and the action prediction information is used to indicate the user's action at the future moment. The transceiver unit 1903 sends the first information to the server, and before the future moment arrives, receives the second information sent by the server, the first information includes the action prediction information, and the second information includes the predicted display resources corresponding to the action prediction information after basic rendering. The processing unit 1902 also uses the predicted display resources to refresh the display screen before or when the future moment arrives.

[0215] The data rendering device 1901 may correspond to the server in the above method. Figures 1A to 16 The data rendering device may include a processing unit 1902 and a transceiver unit 1903.

[0216] When the data rendering device 1901 is the above-mentioned server, the data rendering device 1901 can be used to execute the method executed by the server in any one of the above-mentioned embodiments 1 to 4.

[0217] The data rendering device 1901 is the above-mentioned server, and when executing the first embodiment, the transceiver unit 1903 receives the first information sent by the terminal display device, and the first information includes the user's action prediction information at the future moment, and the action prediction information is used to indicate the user's action at the future moment. The processing unit 1902 determines the predicted display resource corresponding to the action prediction information from the initial display resource, and performs basic rendering on the predicted display resource. The transceiver unit 1903 also sends the second information to the terminal display device before the future moment arrives, and the second information includes the predicted display resource after basic rendering.

[0218] For the concepts, explanations, detailed descriptions and other steps involved in the data rendering device 1901 and related to the technical solution provided in the embodiment of the present application, please refer to the description of these contents in the aforementioned method or other embodiments, which will not be repeated here.

[0219] It can be understood that the functions of the various units in the above-mentioned data rendering device 1901 can refer to the implementation of the corresponding method embodiment, and will not be repeated here.

[0220] It should be understood that the division of the units of the above data rendering device 1901 is only a division of logical functions. In actual implementation, all or part of the units can be integrated into one physical entity, or they can be physically separated. Fig.17 The processing unit 1902 may be implemented by the transceiver 1703 of Fig.17 The processor 1702 is implemented.

[0221] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figures 1A to 16 A method according to any one of the embodiments shown.

[0222] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figures 1A to 16 A method according to any one of the embodiments shown.

[0223] According to the method provided in the embodiment of the present application, the present application also provides a system, which includes one or more terminal display devices and one or more network devices as mentioned above.

[0224] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0225] The network devices in the above-mentioned various device embodiments correspond to the network devices or terminal devices in the terminal devices and method embodiments, and the corresponding modules or units perform the corresponding steps. For example, the communication unit (transceiver) performs the steps of receiving or sending in the method embodiment, and other steps except sending and receiving can be performed by the processing unit (processor). The functions of the specific units can refer to the corresponding method embodiments. Among them, the processor can be one or more.

[0226] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0227] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0228] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0229] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0230] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0232] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0233] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data rendering method, characterized in that, the method is applied to a server, and the method includes: receiving first information sent by a terminal display device, where the first information includes action capture information of a user at a first moment and first action prediction information of the user at a future moment, the first moment being the moment when the first action prediction information is predicted, the action capture information being used to indicate the action of the user at the first moment, and the first action prediction information being used to indicate the action of the user at the future moment; determining a current display resource corresponding to the action capture information and a predicted display resource corresponding to the first action prediction information from an initial display resource, determining a target display resource according to the current display resource and the predicted display resource, and performing basic rendering on the target display resource; before the future moment arrives, sending second information to the terminal display device, where the second information includes the target display resource after basic rendering.

2. The method according to claim 1, characterized in that, the initial display resource is divided into multiple data streams; the determining the predicted display resource corresponding to the first action prediction information from the initial display resource includes: when the available resources of the server are not less than a first resource threshold, taking all data streams corresponding to the first action prediction information as the predicted display resource; when the available resources of the server are less than the first resource threshold, taking partial data streams corresponding to the first action prediction information as the predicted display resource.

3. The method according to claim 1, characterized in that, the initial display resource is divided into multiple data streams; the determining the target display resource according to the current display resource and the predicted display resource includes: taking all data streams corresponding to the current display resource and all or partial data streams corresponding to the predicted display resource as the target display resource; or, re-dividing the initial display resource so that the data streams corresponding to the current display resource after re-division include all or partial data streams corresponding to the predicted display resource, and taking the current display resource after re-division as the target display resource.

4. The method according to claim 3, characterized in that, the taking all data streams corresponding to the current display resource and all or partial data streams corresponding to the predicted display resource as the target display resource includes: if the available resources of the server are not less than a second resource threshold, taking all data streams of the current display resource and all or partial data streams of the predicted display resource as the target display resource; if the available resources of the server are less than the second resource threshold, reducing the resolution of the predicted display resource, and taking all data streams of the current display resource and all or partial data streams of the predicted display resource after resolution reduction as the target display resource.

5. The method according to any one of claims 1 to 4, characterized in that, the receiving the first information sent by the terminal display device includes: Receive the first information corresponding to at least two moments sent by the terminal display device; the at least two moments are earlier than the future moment, and the first information corresponding to each of the at least two moments includes the first action prediction information of the user at the future moment predicted at that moment. Determining the predicted display resource corresponding to the first action prediction information from the initial display resources includes: Obtain the target action prediction information of the user at the future moment according to the first action prediction information corresponding to the at least two moments respectively; Determine the predicted display resource from the initial display resources according to the target action prediction information at the future moment.

6. The method according to claim 5, characterized in that Obtaining the target action prediction information of the user at the future moment according to the first action prediction information corresponding to the at least two moments respectively includes: Performing weighted averaging on the first action prediction information corresponding to the at least two moments respectively to obtain the target action prediction information of the user at the future moment; wherein, for each of the at least two moments, when the time difference between the moment and the future moment is greater, the weight of the first action prediction information corresponding to the moment is greater.

7. The method according to claim 5, characterized in that Performing basic rendering on the target display resource includes: If the difference degree between the first action prediction information corresponding to the at least two moments respectively is not greater than the preset difference degree, perform basic rendering on the target display resource; If the difference degree between the first action prediction information corresponding to the at least two moments respectively is greater than the preset difference degree, reduce the resolution of the target display resource, and perform basic rendering on the target display resource after reducing the resolution.

8. The method according to any one of claims 1 to 4, characterized in that Determining the predicted display resource corresponding to the first action prediction information from the initial display resources includes: Use a prediction model to determine the second action prediction information of the user at the future moment; Perform weighted averaging on the first action prediction information and the second action prediction information, and determine the predicted display resource from the initial display resources according to the weighted average action prediction information; wherein, the prediction model is trained using the learning data reported by one or more terminal display devices, and the learning data reported by each terminal display device is used to indicate the real action of the user wearing the terminal display device at the future moment; when the more learning data used to train the prediction model, the greater the weight of the second action prediction information.

9. The method according to claim 8, characterized in that Before determining the predicted display resource from the initial display resources according to the first action prediction information and the second action prediction information, it further includes: Using the first action prediction information for learning and prediction.

10. The method according to claim 9, characterized in that When the first action prediction information is only used for learning, then: Determine the predicted display resource corresponding to the second action prediction information from the initial display resource.

11. A data rendering method, characterized in that, the method is applied to a terminal display device, and the method includes: Detect the action capture information of the user at the first moment, and predict the first action prediction information of the user at a future moment. The first moment is the moment when the first action prediction information is obtained. The action capture information is used to indicate the action of the user at the first moment, and the first action prediction information is used to indicate the action of the user at the future moment; Send the first information to the server, where the first information includes the action capture information and the first action prediction information; Before the future moment arrives, receive the second information sent by the server. The second information includes the target display resource after basic rendering, and the target display resource is determined according to the current display resource corresponding to the action capture information and the predicted display resource corresponding to the first action prediction information; Before the future moment arrives or at the future moment, use the target display resource to refresh the display screen.

12. The method according to claim 11, characterized in that, the using the target display resource to refresh the display screen before the future moment arrives or at the future moment includes: Before the future moment arrives, generate a display screen with a super perspective according to the current display resource and the predicted display resource, and display the display screen with the super perspective; the viewing angle range of the display screen with the super perspective is greater than a preset viewing angle range; or, When the future moment arrives, refresh the display screen within a preset viewing angle range according to the current display resource and the predicted display resource.

13. The method according to claim 11 or 12, characterized in that, the predicting the first action prediction information of the user at a future moment and sending the first information to the server includes: Predict the action prediction information of the user at the future moment at at least two moments respectively; the at least two moments are earlier than the future moment; Send the first information corresponding to each of the at least two moments to the server respectively, and the first information corresponding to each moment includes the first action prediction information of the user at the future moment predicted at that moment.

14. A data rendering system, characterized in that, includes: A terminal display device, configured to detect the action capture information of the user at the first moment, predict the first action prediction information of the user at a future moment, and send the first information to the server. The first information includes the action capture information and the first action prediction information; wherein, the first moment is the moment when the first action prediction information is obtained, the action capture information is used to indicate the action of the user at the first moment, and the first action prediction information is used to indicate the action of the user at the future moment; A server, configured to receive the first information, determine, from initial display resources, a current display resource corresponding to the motion capture information and a predicted display resource corresponding to the first motion prediction information, determine a target display resource based on the current display resource and the predicted display resource, perform basic rendering on the target display resource, and send second information to the terminal display device before the future moment arrives; wherein, the second information includes the target display resource after basic rendering. The terminal display device is further configured to, before the future moment arrives or at the future moment, use the target display resource to refresh the display screen.

15. The data rendering system according to claim 14, wherein, after receiving the second information, the terminal display device is specifically configured to: before the future moment arrives, generate a super-view display screen based on the current display resource and the predicted display resource, and display the super-view display screen, the viewing angle range of the super-view display screen being greater than a preset viewing angle range; or, at the future moment, refresh the display screen within a preset viewing angle range based on the current display resource and the predicted display resource.

16. The data rendering system according to claim 14 or 15, wherein, the terminal display device is further configured to: predict the motion prediction information of the user at the future moment at at least two moments respectively, and send the first information corresponding to each of the at least two moments to the server respectively; wherein, the at least two moments are earlier than the future moment, and the first information corresponding to each of the at least two moments includes the first motion prediction information of the user at the future moment predicted at that moment. The server is further configured to: obtain the target motion prediction information of the user at the future moment according to the first motion prediction information corresponding to the at least two moments respectively, and determine the predicted display resource from the initial display resources according to the target motion prediction information at the future moment.

17. A data rendering device, wherein, it includes a processor and a communication interface. The communication interface is configured to receive signals from other communication devices outside the data rendering device and transmit them to the processor or send signals from the processor to other communication devices outside the data rendering device. The processor is configured to implement the data rendering method according to any one of claims 1 to 10, or any one of claims 11 to 13 through logic circuits or by executing code instructions.

18. A data rendering device, wherein, it includes a processor, the processor is connected to a memory, the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory so that the data rendering device executes the data rendering method according to any one of claims 1 to 10, or any one of claims 11 to 13.

19. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program, and when the computer program is run, it implements the data rendering method described in any one of claims 1 to 10 or any one of claims 11 to 13.

20. A chip, characterized in that it includes a processor and an interface; the processor is configured to read instructions through the interface to execute the data rendering method described in any one of claims 1 to 10 or any one of claims 11 to 13.

21. A computer program product, characterized in that the computer program product includes a computer program or instructions, and when the computer program or instructions are executed by a communication device, it implements the data rendering method described in any one of claims 1 to 10 or any one of claims 11 to 13.

Images