Image compensation method, terminal device, cloud server and storage medium

Abstract

The present invention discloses a picture compensation method, terminal device, cloud server and storage medium, and relates to the field of cloud VR technology. The method includes: the terminal device obtains the user's real-time action command and sends it to the cloud server; the cloud server generates a real-time picture according to the real-time action command; when uplink frame loss is detected, a predicted picture is generated according to the real-time action command and an established behavior habit model; the real-time picture or the predicted picture is determined as the first picture, and sent to the terminal device; the terminal device determines whether the first picture sent by the cloud server is received within a preset response delay; if not, a second picture is obtained according to the real-time action command and the stored behavior habit model; and the second picture is displayed. The present invention solves the problem that the existing technology cannot meet the cloud VR processing requirements of real-time interactive large-scale games, resulting in an impact on user experience, and achieves the effect of eliminating the picture freeze caused by the loss of uplink interactive information or the loss of downlink data transmission.

Description

Technical Field

[0001] The present invention relates to the field of cloud VR technology, and in particular to a picture compensation method, terminal equipment, cloud server and storage medium. Background Art

[0002] At present, the common practice of image compensation in cloud processing is that the cloud does not perform any processing at all, and the terminal uses asynchronous time warp (ATW) to process it. This method has a deviation from the content sent from the cloud, resulting in overall distortion and blurring of the image, which can easily make users feel uncomfortable and affect the user experience. At the same time, the terminal performs ATW processing, which has high requirements on the processing power of the terminal itself, and increases the latency when VR is processed in the cloud.

[0003] Therefore, the existing cloud VR technology's picture compensation solution cannot meet the cloud VR processing requirements of large-scale competitive games that require real-time interaction, such as high traffic consumption and high latency requirements. Problems such as picture freezes and picture discontinuities cannot be solved, which affects the user experience. Summary of the invention

[0004] The main purpose of the present invention is to provide a picture compensation method, a terminal device, a cloud server and a storage medium, aiming to solve the problem that the existing cloud VR picture compensation scheme cannot meet the cloud VR processing requirements of real-time interactive large-scale games, thus affecting the user experience.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a picture compensation method, which is applied to a terminal device and includes:

[0007] Obtaining a user's real-time action instruction, and sending the real-time action instruction to a cloud server;

[0008] Determine whether a first picture sent by the cloud server is received within a preset response delay; wherein the first picture is a real-time picture generated by the cloud server according to the real-time action instruction or a predicted picture obtained according to the real-time action instruction and an established behavior habit model; wherein the behavior habit model is established based on the historical picture data and the correspondence between the historical action instructions obtained by parsing the historical picture data;

[0009] If the first picture is not received within the preset response delay, a second picture is obtained according to the real-time action instruction and the stored behavior habit model; wherein the second picture is a historical picture corresponding to the historical action instruction matching the real-time action instruction;

[0010] The second screen is displayed.

[0011] Optionally, in the above-mentioned picture compensation method, after the step of obtaining the user's real-time action instruction and sending the real-time action instruction to the cloud server, the method further includes:

[0012] Determine whether a downlink buffering condition is met; wherein the downlink buffering condition is that the user is in a stationary state and / or the action corresponding to the real-time action instruction does not belong to a preset large-scale action;

[0013] If the downlink cache condition is met, receiving at least one frame of the third picture sent by the cloud server, and saving the at least one frame of the third picture; wherein the at least one frame of the third picture is a cached picture predicted by the cloud server according to the real-time action instruction and the established behavior habit model;

[0014] After the step of determining whether the first picture sent by the cloud server is received within a preset response delay, the method further includes:

[0015] If the first frame is not received within a preset response delay, the at least one frame of the third frame is displayed.

[0016] Optionally, in the above picture compensation method, before the step of obtaining the second picture according to the real-time action instruction and the stored behavior habit model, the method includes:

[0017] Obtaining an initialization instruction, and sending the initialization instruction to a cloud server;

[0018] Receive the behavior habit model sent by the cloud server and store it.

[0019] Optionally, in the above picture compensation method, the step of obtaining the second picture according to the real-time action instruction and the stored behavior habit model includes:

[0020] Determine the behavior habit model corresponding to the target user according to the real-time action instruction and the behavior habit models of different users, and obtain the stored behavior habit model;

[0021] Determining, in the stored behavior habit model, a historical action instruction that matches the real-time action instruction;

[0022] According to the correspondence between the action instructions and the pictures in the stored behavior habit model, the historical pictures corresponding to the historical action instructions are extracted to obtain the second picture.

[0023] In a second aspect, the present invention provides a picture compensation method, which is applied to a cloud server and includes:

[0024] Receive real-time action instructions sent by terminal devices;

[0025] Generate a real-time picture according to the real-time action instruction;

[0026] When uplink frame loss is detected, a predicted action instruction is obtained according to the real-time action instruction and the established behavior habit model; wherein the behavior habit model is established based on the historical picture data and the corresponding relationship between the historical action instructions obtained by parsing the historical picture data;

[0027] generating a prediction picture according to the prediction action instruction;

[0028] The real-time picture or the predicted picture is determined as a first picture, and the first picture is sent to the terminal device, so that the terminal device displays the first picture.

[0029] Optionally, in the above picture compensation method, before the step of obtaining a predicted action instruction according to the real-time action instruction and the established behavior habit model when uplink frame loss is detected, the method further includes:

[0030] Acquire historical action instructions of different users and corresponding historical screen data; wherein the historical action instructions are action instructions of historical operations of users and / or action instructions of users based on screen guidance operations;

[0031] According to the historical action instructions and the historical screen data, a corresponding relationship between the action instructions and the screen is established to obtain the behavior habit models of different users;

[0032] According to the real-time action instruction, the behavior habit model corresponding to the target user is determined to obtain the established behavior habit model.

[0033] Optionally, in the above picture compensation method, after the step of receiving the real-time action instruction sent by the terminal device, the method further includes:

[0034] Obtaining at least one predicted action instruction according to the real-time action instruction and the established behavior habit model;

[0035] Generate a corresponding cached picture according to the at least one predicted action instruction to obtain at least one third picture;

[0036] The at least one frame of the third picture is sent to the terminal device, so that the terminal device saves the at least one frame of the third picture, and displays the at least one frame of the third picture when the first picture is not received within a preset response delay.

[0037] In a third aspect, the present invention provides a terminal device, comprising a processor and a memory, wherein a picture compensation program is stored in the memory, and when the picture compensation program is executed by the processor, the picture compensation method as described above is implemented.

[0038] In a fourth aspect, the present invention provides a cloud server, comprising a processor and a memory, wherein a picture compensation program is stored in the memory, and when the picture compensation program is executed by the processor, the picture compensation method as described above is implemented.

[0039] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by one or more processors, the image compensation method as described above is implemented.

[0040] The above one or more technical solutions provided by the present invention may have the following advantages or at least achieve the following technical effects:

[0041] The present invention proposes a picture compensation method, terminal device, cloud server and storage medium. The terminal device sends a real-time action instruction to the cloud server, and the cloud server generates a real-time picture accordingly. Or when uplink frame loss is detected, a predicted picture is generated according to the real-time action instruction and the established behavior habit model, and the first picture is determined to be sent to the terminal device. When the terminal device does not receive the first picture within the preset response delay, the historical picture is matched according to the real-time action instruction and the stored behavior habit model to obtain and display the second picture, thereby eliminating the picture freeze or picture discontinuity caused by the loss of uplink interactive information or downlink data transmission, and performing cloud VR picture compensation. The present invention solves the problem of uplink loss of action instructions through the behavior habit model established on the cloud server, and solves the problem of downlink loss of VR pictures through the behavior habit model stored on the terminal device. It can also be applied to various cloud VR games with real-time interaction, large traffic consumption and high latency requirements, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 It is a flowchart of a first embodiment of the image compensation method of the present invention;

[0044] Figure 2A schematic diagram of the hardware structure of a terminal device according to the present invention;

[0045] Figure 3 Another schematic diagram of the flow chart of the first embodiment of the image compensation method of the present invention;

[0046] Figure 4 It is a flowchart diagram of a second embodiment of the image compensation method of the present invention;

[0047] Figure 5 The figure is a schematic diagram of the hardware structure of the cloud server involved in the present invention.

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that in the present invention, the term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or system including the elements. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes, taking "A and / or B" as an example, including scheme A, or scheme B, or schemes that A and B meet at the same time. In the present invention, if there is a description involving "first", "second", etc., the description of "first", "second", etc. is only for descriptive purposes, and cannot be understood as indicating or implying its relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features.

[0051] For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical solutions of various embodiments can be combined with each other, but this is based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0052] Analysis of existing technologies shows that, at present, the common practice of image compensation in cloud processing is that the cloud does not perform any processing at all, and the terminal uses asynchronous time warp (ATW) to process, and generates intermediate frames that adapt to the user's movement angle and orientation through the terminal's own prediction of the user's turning to fill the screen content. Although this method can fill the temporary loss of the picture caused by the long cloud processing delay, there is a deviation from the subsequent processing of the cloud content, resulting in overall distortion and blurring of the picture, which can easily cause users to feel dizzy and other discomfort, affecting the user experience. At the same time, the terminal performs ATW processing, which has high requirements on the processing power of the terminal itself, which is somewhat different from the goal of cloud-based methods to lower the threshold of terminals; moreover, when VR is cloud-based, the audio and video stream delivery mode is generally adopted. In the process of the terminal calling the ATW capability, audio and video decoding, frame decomposition and processing, etc. are also required. The processing process is relatively complicated, which increases the delay process of ATW processing.

[0053] In the existing cloud VR technology, in order to solve the latency problem, the signal quality is generally analyzed, and the nearest cell is selected for data transmission according to the movement trajectory only when the transmission requirements are met. This method of reducing the general network transmission latency from aspects such as network quality and edge cloud node selection still cannot meet the high bandwidth requirements of cloud VR. Therefore, the application scenarios of existing cloud VR are generally some scenarios with low requirements for bandwidth and latency, such as small single-player games / promotion / training / scenery, etc., which cannot meet the cloud VR processing requirements of large-scale battle games that require real-time interaction with high traffic consumption and high latency requirements, and are prone to problems such as screen freeze and discontinuous screen. Even if such games can be processed by cloud VR, the local solution using the terminal for processing cannot reduce the terminal latency on the user side, and the problems of screen freeze and discontinuous screen are still not solved, and thus the user experience problems of cloud VR for large-scale battle games caused by latency fluctuations cannot be fundamentally solved.

[0054] In view of the fact that the existing technology cannot meet the cloud VR processing requirements of real-time interactive large-scale games, resulting in a technical problem that affects the user experience, the present invention provides a picture compensation method, and the overall idea is as follows:

[0055] The terminal device obtains the user's real-time action instructions and sends the real-time action instructions to the cloud server;

[0056] The cloud server receives the real-time action instruction sent by the terminal device; generates a real-time picture according to the real-time action instruction; when uplink frame loss is detected, obtains a predicted action instruction according to the real-time action instruction and the established behavior habit model; generates a predicted picture according to the predicted action instruction; wherein the behavior habit model is established based on the historical picture data and the corresponding relationship between the historical action instructions obtained by parsing the historical picture data; determines the real-time picture or the predicted picture as the first picture, and sends the first picture to the terminal device;

[0057] The terminal device determines whether the first picture sent by the cloud server is received within the preset response delay; if not, obtains the second picture according to the real-time action instruction and the stored behavior habit model; and displays the second picture; wherein the second picture is a historical picture corresponding to the historical action instruction matching the real-time action instruction.

[0058] Through the above technical solution, the effect of eliminating the freeze or discontinuity of the screen caused by the loss of uplink interactive information or downlink data transmission is achieved, and cloud VR screen compensation is performed. The present invention solves the problem of uplink loss of action instructions through the behavioral habit model established on the cloud server, and solves the problem of downlink loss of VR screen through the behavioral habit model stored on the terminal device. It can also be applied to various cloud VR games with real-time interaction, high traffic consumption and high latency requirements, improving the user experience.

[0059] The picture compensation method, terminal device, cloud server and storage medium provided by the present invention are described in detail below through specific embodiments and implementation modes in conjunction with the accompanying drawings.

[0060] Embodiment 1

[0061] Reference Figures 1 to 4 , a first embodiment of the image compensation method of the present invention is proposed, and the image compensation method is applied to a terminal device. The terminal device refers to a device that can achieve network connection, and the terminal device can be a VR glasses, a VR display device, a mobile phone, a computer, a tablet computer and other terminal devices.

[0062] like Figure 2 The figure is a schematic diagram of the hardware structure of the terminal device. The terminal device may include: a processor 1001, such as a CPU (Central Processing Unit), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Those skilled in the art will understand that Figure 2 The hardware structure shown in the figure does not constitute a limitation on the terminal device of the present invention, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0063] Specifically, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 is used to connect to the client and communicate data with the client. The user interface 1003 may include an output unit, such as a display screen, an input unit, such as a keyboard; the network interface 1004 is used to connect to the cloud server and communicate data with the cloud server. The network interface 1004 may include an input / output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface; the memory 1005 is used to store various types of data, which may include, for example, instructions of any application or method in the terminal device, and data related to the application. The memory 1005 may be a high-speed RAM memory or a stable memory, such as a disk memory; optionally, the memory 1005 may also be a storage device independent of the processor 1001, and continue to refer to Figure 2 The memory 1005 may include an operating system, a network communication module, a user interface module, and a picture compensation program; the processor 1001 is used to call the picture compensation program stored in the memory 1005 and perform the following operations:

[0064] Obtain the user's real-time action instructions and send the real-time action instructions to the cloud server;

[0065] Determine whether a first picture sent by the cloud server is received within a preset response delay; wherein the first picture is a real-time picture generated by the cloud server according to the real-time action instruction or a predicted picture obtained according to the real-time action instruction and an established behavior habit model; wherein the behavior habit model is established based on the historical picture data and the corresponding relationship between the historical action instructions obtained by parsing the historical picture data;

[0066] If the first picture is not received within the preset response delay, a second picture is obtained according to the real-time action instruction and the stored behavior habit model; wherein the second picture is a historical picture corresponding to the historical action instruction matching the real-time action instruction;

[0067] Display the second screen.

[0068] Based on the above terminal equipment, the following Figure 1 The flowchart shown in FIG. 1 is used to describe the picture compensation method of this embodiment in detail. The method may include the following steps:

[0069] Step S110: Acquire the user's real-time action instruction, and send the real-time action instruction to the cloud server.

[0070] Specifically, the user performs real-time operations on the terminal device, and the terminal device generates real-time action instructions accordingly, which are then uploaded to the cloud server in real time, such as uploading real-time action instructions to the central control server through a head display. At present, the generally required data transmission delay is no more than 20ms, and the delay from the terminal device to the cloud server is mostly around 5-10ms, which can be ignored, so it is considered to be a real-time upload. This embodiment is explained by taking the cloud server including a central control server and a streaming server that communicate with each other as an example, and performs operations such as some large / extra-large real-time interactive nVn battle cloud VR games that consume a lot of traffic and require low latency.

[0071] After receiving the real-time action command sent by the terminal device, the central control server can generate real-time images according to the refresh rate and frame rate of the game itself. Among them, the refresh rate and frame rate are set values. For example, the refresh rate and frame rate of VR games are generally set to 60 / 60. Converted into time, it takes 16.67ms to generate a frame of the picture, and the time relationship between the picture and the action command is 16.67:1.

[0072] After the central control server receives the real-time action instruction of the current user sent by the terminal device, it can also determine the established behavior habit model of the user based on the user information for use. For example, within the next preset number of frames or preset transmission delay, the delay requirement can be set according to the actual situation. If the central control server does not continue to receive the real-time action instruction, it can be determined that there is uplink frame loss. At this time, the cloud server can predict an action instruction based on the real-time action instruction and the behavior habit model, that is, the predicted action instruction, and then generate the predicted picture according to the refresh rate and frame rate of the game itself.

[0073] It can be understood that if the central control server can directly generate a real-time picture based on the real-time action instruction, then the real-time picture will be directly determined as the first picture, and then sent to the terminal device through the streaming server; if the central control server does not continue to receive the next real-time action instruction, correspondingly, the real-time picture cannot be continued to be generated. At this time, the predicted picture can be determined as the first picture, and then sent to the terminal device through the streaming server.

[0074] The way in which the cloud server generates predicted images based on real-time action instructions and behavioral habit models is an image compensation solution established based on the uplink interactive information transmission from the head display to the central control server.

[0075] Step S130: Determine whether the first picture sent by the cloud server is received within a preset response delay.

[0076] Among them, the first picture is a real-time picture generated by the cloud server according to the real-time action instruction or a predicted picture obtained according to the real-time action instruction and an established behavior habit model; wherein the behavior habit model is established based on historical picture data and the correspondence between historical action instructions obtained by analyzing the historical picture data.

[0077] Specifically, when the cloud server generates the first picture in response to the real-time action instruction, frame loss may occur in the process of sending the first picture to the terminal device. At this time, after the terminal device sends the real-time action instruction, it cannot receive the first picture normally. If the first picture is not received within the next preset number of frames or the preset response delay, that is, the first picture is not received within the preset response delay, it can be determined that there is downlink frame loss at this time.

[0078] Step S150: If the first picture is not received within a preset response delay, a second picture is obtained according to the real-time action instruction and the stored behavior habit model.

[0079] The second picture is a historical picture corresponding to the historical action instruction matching the real-time action instruction.

[0080] When it is determined that the downlink frame is lost, the terminal device can match the relevant historical action instructions according to the real-time action instructions and the stored behavior habit model, and obtain a corresponding historical picture, that is, the second picture. This method belongs to the terminal cache solution. The preset response delay can be set according to the actual situation. For example, in this embodiment, when 15 consecutive frames are lost, the user experience is more obvious, and the preset response delay can be set to the delay corresponding to 2-14 frames, that is, it can be taken within 33.34ms-233.38ms.

[0081] Step S170: displaying the second picture.

[0082] After the terminal device obtains the second picture according to step S150, the second picture can be directly displayed to fill the video stream as the response picture frame corresponding to the real-time action instruction. Optionally, if the terminal device subsequently receives the first picture corresponding to the real-time action instruction, that is, the downlink data arrives late, the terminal device can send the next frame of real-time action instruction and receive the next frame of picture normally without any processing.

[0083] Step S180: If the first picture is received within a preset response delay, the first picture is directly displayed.

[0084] The above process is for the case where the terminal device determines based on step S130 that the first picture sent by the cloud server is not received within the preset response delay, that is, there is a downlink frame loss. On the contrary, when the terminal device determines based on step S130 that the first picture sent by the cloud server is received within the preset response delay, it can be known that the downlink data transmission is normal, and the terminal device can display the first picture normally. The user can continue to obtain the next frame of real-time action instructions for the picture displayed by the terminal device and receive the corresponding next frame of the picture.

[0085] In another embodiment, in combination Figure 3 After step S110, the method further includes:

[0086] Step S121: Determine whether the downlink buffering condition is met.

[0087] The downlink buffering condition is that the user is in a stationary state and / or the action corresponding to the real-time action instruction does not belong to a preset large-scale action.

[0088] Specifically, in addition to the above-mentioned terminal caching solution, there can also be a downlink caching solution. When the user can determine that the user is in a stationary state and / or the user's movement amplitude is small based on the operation of the terminal device, the real-time action instruction can be parsed to obtain the corresponding action type. If the action type does not belong to the large-scale action preset by the terminal device, it can be determined that the movement amplitude corresponding to the real-time action instruction is small. When the above-mentioned downlink caching conditions are met, the downlink caching solution can be used to solve the problem of downlink data frame loss, otherwise, the terminal caching solution can be used to solve the problem of downlink data frame loss. The specific downlink compensation solution can be selected by the user or according to the network situation of the cloud VR. For example, under the current network environment and game settings, a terminal caching solution can be used to reduce the use of downlink network bandwidth and improve the accuracy of picture compensation. When the network bandwidth is greatly improved in the future and the bandwidth is no longer a bottleneck, the downlink caching solution can be used to increase the cached data as much as possible. The ultimate experience is that the user's network flash does not affect the user's game fluency, which greatly improves user perception.

[0089] Step S122: If the downlink buffering condition is met, receiving at least one frame of the third picture sent by the cloud server, and saving the at least one frame of the third picture.

[0090] Among them, the at least one frame of the third picture is a cached picture predicted by the cloud server according to the real-time action instruction and the established behavior habit model. The number of frames of the third picture can also be set according to actual needs. In this embodiment, it is set to 15 frames. The cloud server can predict the next 15 predicted action instructions for the real-time action instruction; then generate corresponding cached pictures based on these 15 predicted action instructions to obtain 15 frames of the third picture; then send the 15 frames of the third picture to the terminal device; the terminal device can save these 15 frames of the third picture.

[0091] Correspondingly, after step S130, the method may further include:

[0092] Step S160: If the first frame is not received within a preset response delay, displaying the at least one frame of the third frame.

[0093] Different from the first embodiment, in this embodiment, steps S150 and S170 are replaced by step S160. That is, after saving at least one frame of the third picture, when the judgment result of executing step S130 is that the first picture is not received within the preset response delay, the at least one stored frame of the third picture can be called and displayed in a frame sequence.

[0094] Step S123: If the downlink cache condition is not met, the step of obtaining the second picture according to the real-time action instruction and the stored behavior habit model is executed.

[0095] If the downlink buffering condition is not met, steps S150 and S170 may be executed, that is, a terminal buffering solution is selected.

[0096] The terminal device selects two downlink cache solutions according to the situation of receiving the first picture, namely, the terminal cache solution that matches the historical picture to obtain the second picture and the downlink cache solution that matches the cached picture to obtain the third picture, and the picture compensation solution established by the downlink data transmission of the rendering server to the head display. The picture compensation method combining uplink information transmission and downlink data transmission can perform picture compensation more comprehensively and accurately while ensuring real-time performance, preventing picture frame loss and affecting user experience.

[0097] Furthermore, before step S150, the method includes:

[0098] Step S101: Obtain an initialization instruction, and send the initialization instruction to a cloud server;

[0099] Step S102: Receive the behavior habit model sent by the cloud server and store it.

[0100] The terminal device does not need to establish a behavioral habit model. The cloud server can establish the behavioral habit model of different users based on historical data and send it to the terminal device for direct storage, which can reduce the burden on the terminal device.

[0101] Furthermore, step S150 may include:

[0102] Step S151: determining the behavior habit model corresponding to the target user according to the real-time action instruction and the behavior habit models of different users, and obtaining the stored behavior habit model;

[0103] Step S152: determining, in the stored behavior habit model, a historical action instruction that matches the real-time action instruction;

[0104] Step S153: according to the correspondence between the action instructions and the pictures in the stored behavior habit model, extract the historical pictures corresponding to the historical action instructions to obtain the second picture.

[0105] Specifically, after the terminal device stores the behavioral habit models of different users, it can match the corresponding behavioral habit model according to the real-time action instructions generated by the user's operation as the stored behavioral habit model of the user; different historical action instructions and their corresponding historical screen data can be stored in the behavioral habit model; in the stored behavioral habit model, the appropriate historical action instructions are searched according to the real-time action instructions to determine the historical action instructions that match the real-time action instructions, and then, according to the correspondence between the action instructions and the screen in the stored behavioral habit model, the historical screen corresponding to the determined historical action instruction is extracted to obtain the second screen.

[0106] The picture compensation method provided in this embodiment sends real-time action instructions to the cloud server through the terminal device, and the cloud server generates a real-time picture accordingly, or generates a predicted picture according to the real-time action instructions and the established behavior habit model when uplink frame loss is detected, and determines it as the first picture to be sent to the terminal device. When the terminal device does not receive the first picture within the preset response delay, it matches the historical picture according to the real-time action instructions and the stored behavior habit model, obtains the second picture and displays it, thereby eliminating the picture freeze or picture discontinuity caused by the loss of uplink interactive information or downlink data transmission, and performing cloud VR picture compensation. The present invention solves the problem of uplink loss of action instructions through the behavior habit model established on the cloud server, and solves the problem of downlink loss of VR pictures through the behavior habit model stored on the terminal device. It can also be applied to various cloud VR games with real-time interaction, large traffic consumption and high latency requirements, thereby improving the user experience.

[0107] Embodiment 2

[0108] Based on the same inventive concept, Figures 3 to 5 , a second embodiment of the image compensation method of the present invention is proposed, and the image compensation method is applied to a cloud server. A cloud server refers to a network device or a network system that can achieve network connection. A cloud server can be a network device such as a server or a cloud platform, or a network system composed of a central control server and a streaming server.

[0109] like Figure 5 , which is a schematic diagram of the hardware structure of a cloud server. The cloud server may include: a processor 2001, such as a CPU, a communication bus 2002, a user interface 2003, a network interface 2004, and a memory 2005. Those skilled in the art will appreciate that Figure 5 The hardware structure shown in the figure does not constitute a limitation on the cloud server of the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0110] Specifically, the communication bus 2002 is used to realize the connection and communication between these components; the user interface 2003 is used to connect to the client and communicate data with the client. The user interface 2003 may include an output unit, such as a display screen, an input unit, such as a keyboard; the network interface 2004 is used to connect to the terminal device and communicate data with the terminal device. The network interface 2004 may include an input / output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface; the memory 2005 is used to store various types of data, which may include, for example, instructions of any application or method in the cloud server, and data related to the application. The memory 2005 may be a high-speed RAM memory or a stable memory, such as a disk memory; optionally, the memory 2005 may also be a storage device independent of the processor 2001, and continue to refer to Figure 5 The memory 2005 may include an operating system, a network communication module, a user interface module, and a picture compensation program; the processor 2001 is used to call the picture compensation program stored in the memory 2005 and perform the following operations:

[0111] Receive real-time action instructions sent by terminal devices;

[0112] Generate real-time images according to real-time action instructions;

[0113] When uplink frame loss is detected, a predicted action instruction is obtained according to the real-time action instruction and the established behavior habit model; wherein the behavior habit model is established based on the historical picture data and the corresponding relationship between the historical action instructions obtained by parsing the historical picture data;

[0114] Generate a prediction picture according to the prediction action instruction;

[0115] The real-time picture or the predicted picture is determined as the first picture, and the first picture is sent to the terminal device so that the terminal device displays the first picture.

[0116] Based on the above cloud server, the following is combined Figure 4 The flowchart shown in FIG. 1 is used to describe the picture compensation method of this embodiment in detail. The method may include the following steps:

[0117] Step S210: receiving a real-time action instruction sent by a terminal device.

[0118] Step S230: Generate a real-time picture according to the real-time action instruction.

[0119] Specifically, after receiving the real-time action instruction, the cloud server generates a corresponding rendering picture, that is, a real-time picture, based on the refresh rate and frame rate of the game.

[0120] Step S250: when uplink frame loss is detected, a predicted action instruction is obtained according to the real-time action instruction and the established behavior habit model.

[0121] The behavior habit model is established based on the historical screen data and the corresponding relationship between the historical action instructions obtained by analyzing the historical screen data.

[0122] In this embodiment, when the uplink frame is lost, the central control server predicts the user's next action based on the real-time action instructions and the established behavior habit model, generates corresponding predicted action instructions, and supplements the lost frame data. The central control server can then send the predicted action instructions to the streaming server, which generates the corresponding voice / image data stream according to the refresh rate and frame rate of the game itself, and then sends the data stream to the terminal device, i.e., the head display, for display and playback.

[0123] Step S270: Generate a prediction screen according to the prediction action instruction.

[0124] Step S290: Determine the real-time picture or the predicted picture as the first picture, and send the first picture to the terminal device so that the terminal device displays the first picture.

[0125] Step S230 is the case where there is no uplink frame loss, and the cloud server generates a real-time picture according to the real-time action instruction. In this case, it can directly jump to step S290 to determine the real-time picture as the first picture, and send the first picture to the terminal device so that the first picture is displayed or the picture compensation is performed based on the downlink compensation scheme in the first embodiment. It will not be repeated here. Step S250 is the case where there is uplink frame loss. In this case, the cloud server generates a predicted picture according to the real-time action instruction and the established behavior habit model, and then executes step S290 to determine the predicted picture as the first picture, and send the first picture to the terminal device so that the first picture is displayed or the picture compensation is performed based on the downlink compensation scheme in the first embodiment. It will not be repeated here.

[0126] Furthermore, before step S250, the method may further include:

[0127] Step S241: determining whether the next real-time action instruction sent by the terminal device is received within a preset transmission delay;

[0128] Step S242: If the next real-time action instruction is not received within the preset transmission delay, it is determined that there is uplink frame loss.

[0129] The preset transmission delay can be set according to the actual situation. The uplink interactive information is transmitted in real time. In order to ensure the real-time nature of data transmission, in actual applications, VR products generally require the head display's real-time action command response transmission delay, that is, the delay from the head display sending the real-time action command to receiving the screen cannot exceed 20ms. Based on this, when the game is carried out normally, after the central control server receives the real-time action command, it does not receive other uplink interactive information, that is, the next real-time action command, within 10ms. At this time, it can be determined that there is uplink frame loss. In this embodiment, for cloud VR games with a refresh rate and frame rate of 60 / 60, a 16.67ms command generates a frame of the picture. In large-scale battle games, if the user's action amplitude is large, when 15 consecutive frames are lost, the user's stuttering experience is more obvious. Therefore, it can be set that when 2-14 frames are lost, that is, when the uplink 33.34ms-233.38ms do not receive the real-time action command, uplink transmission compensation is performed. That is to say, based on the situation of this embodiment, the value range of the preset transmission delay can be set to 33.34ms-233.38ms, and the specific threshold value can be set according to the user's professional level or fine-tuned according to the actual use effect to achieve the best frame supplement effect. For example, when the user's action amplitude is small, the threshold of continuous frame loss can be appropriately increased to reduce the number of frame supplements. When the user's action amplitude is large, it is necessary to appropriately reduce the threshold of continuous frame loss, reduce the amount of data for a single frame supplement, and improve the real-time performance of the data.

[0130] It should be noted that the uplink interaction information that is delayed in reaching the cloud server, i.e., the uplink action instructions, will not take effect in the real-time game, but will only be stored in the cloud server as analysis data of user behavior habits, used to verify the user's behavior habit model and continuously update the model.

[0131] Furthermore, if Figure 4 As shown, before step S250, the method may further include:

[0132] Step S221: Acquire historical action instructions of different users and corresponding historical screen data.

[0133] The historical action instructions are action instructions of historical operations of the user and / or action instructions of operations guided by the user based on the screen.

[0134] Step S222: establishing a correspondence between action instructions and pictures according to the historical action instructions and the historical picture data, and obtaining behavior habit models of different users;

[0135] When there is enough user history data, a single user behavior habit model can be established based on each game. User history data may include: for different types of obstacles in the display screen, the user's position movement habits such as hiding, entering and exiting, and the VR screen corresponding to the habit; for different types of obstacles in the display screen, the user's action habits such as pressing buttons and shooting, and the VR screen corresponding to the habit; for multiple obstacles in the display screen, the user's position movement habits between obstacles and the VR screen corresponding to the habit; the user's game play, such as changing magazines and other game play-related action habits and the VR screen corresponding to the habit. The specific data type can also be set according to actual needs. The user's position movement, button action and other game information can be collected in real time by the cloud server, and can be uploaded to the cloud server in real time through the terminal device, such as uploading to the central control server in real time through the head display. At present, the generally required data transmission delay is no more than 20ms, and the delay from the terminal device to the cloud server is mostly around 5-10ms, which can be ignored, so it is considered to be uploaded in real time. The cloud server can store all operations of all users and the corresponding screens to form sufficient user historical data. Then, based on the historical action instructions and corresponding historical screen data in the user's historical data, a behavioral habit model of different users can be established through a deep learning algorithm.

[0136] When there is not enough historical user data or the user information indicates that the user is a new user, the terminal device can be combined with the display of the guide screen to guide the user to operate, and then the corresponding action instructions can be collected as historical action instructions. For example, when the game starts, the user puts on the headset and the VR experience begins. Then, a user position calibration point mark can be set in the VR scene seen by the user. The mark can be the same as the map starting point of the VR scene. Then, the terminal device performs position calibration, and the user completes the first step, that is, the headset is calibrated and the user's position is recorded as the origin, and the user's position information is uploaded to the cloud server. Then, the user can start to experience the game, and the terminal device synchronously records all the user's game information data. For new users, in the first few games, the corresponding screens of game instructions such as how to hide based on obstacles, how to enter and exit obstacles, how to shoot, how to move between obstacles, and how to change magazines can be sent to the terminal device and displayed. The purpose is to guide user behavior habits and improve the accuracy of user behavior habits. After the user completes the guidance, the user's information data is obtained and sent to the cloud server. The cloud server can then build a behavior habit model of the user based on the historical action instructions and corresponding historical screen data in the user's information data through a deep learning algorithm. In the same way, behavior habit models of multiple different users can be built.

[0137] Step S223: Determine the behavior habit model corresponding to the target user according to the real-time action instruction, and obtain the established behavior habit model.

[0138] Specifically, the real-time action instruction is generated by the user operation, and the user is the target user. The real-time action instruction may contain user information, such as geographic location, gender, age and other user information. After the terminal device sends the real-time action instruction to the cloud server, the cloud server can extract the user information or combine the preset rules to match and select the behavior habit model suitable for the user from the behavior habit models of different users obtained above, and determine the behavior habit model corresponding to the target user as the established behavior habit model. So that when uplink frame loss is detected, the predicted action instruction can be obtained according to the real-time action instruction and the established behavior habit model.

[0139] It should be noted that in subsequent actual operations, the real-time action instructions and real-time images generated can be stored for iterative updating of the behavior habit model.

[0140] Optionally, all user data that meets the selected dimensions can be aggregated according to user information, and a general behavior habit model, such as a general user model, can be established through a deep learning algorithm. This general user model can be used in simulation games, allowing users to select a general user model of a specified dimension as their robot teammate or opponent, thereby playing simulation games and achieving the goal of training their game proficiency.

[0141] Furthermore, after step S222, the method may further include:

[0142] Step S224.1: Obtaining the initialization instruction sent by the terminal device;

[0143] Step S224.2: According to the initialization instruction, the behavior habit models of the different users are sent to the terminal device.

[0144] For each game, during initialization, the complete user data can be directly stored in the headset to establish behavioral habit models of different users. The cloud server can also establish behavioral habit models of different users on the cloud server based on the complete user data and send them to the terminal device for storage, thereby reducing the memory pressure of the terminal device.

[0145] Furthermore, if Figure 3 As shown, after step S210, the method may further include:

[0146] Step S225: obtaining at least one predicted action instruction according to the real-time action instruction and the established behavior habit model;

[0147] Step S226: generating a corresponding cached picture according to the at least one predicted action instruction, and obtaining at least one third picture;

[0148] Step S227: Send the at least one frame of the third picture to the terminal device, so that the terminal device saves the at least one frame of the third picture, and displays the at least one frame of the third picture when the first picture is not received within a preset response delay.

[0149] For more implementation details in the specific implementation of the above method steps, please refer to the description of the specific implementation in Example 1. For the sake of brevity of the description, they will not be repeated here.

[0150] It should be noted that the functions that can be realized in each step of the picture compensation method provided in this embodiment and the corresponding technical effects can refer to the description of the specific implementation methods in each embodiment of the picture compensation method of the present invention. For the sake of brevity of the specification, they will not be repeated here.

[0151] Embodiment 3

[0152] Based on the same inventive concept, Figure 2 Schematic diagram of the hardware structure, this embodiment provides a terminal device, the terminal device may include a processor and a memory, the memory stores a picture compensation program, when the picture compensation program is executed by the processor, all or part of the steps of the first embodiment of the picture compensation method of the present invention are implemented.

[0153] Specifically, terminal devices refer to devices that can achieve network connection. Terminal devices can be VR glasses, VR display devices, mobile phones, computers, tablet computers and other terminal devices.

[0154] It can be understood that the terminal device may also include a communication bus, a user interface and a network interface. The communication bus is used to realize the connection and communication between these components; the user interface is used to connect to the client and communicate data with the client, and the user interface may include an output unit, such as a display screen, and an input unit, such as a keyboard; the network interface is used to connect to the cloud server and communicate data with the cloud server, and the network interface may include an input / output interface, such as a standard wired interface and a wireless interface.

[0155] The memory is used to store various types of data, which may include, for example, instructions of any application or method in the terminal device, and data related to the application. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Optionally, the memory may also be a storage device independent of the processor.

[0156] The processor is used to call the picture compensation program stored in the memory and execute the picture compensation method as described above. The processor can be an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute all or part of the steps of the first embodiment of the picture compensation method as described above.

[0157] Embodiment 4

[0158] Based on the same inventive concept, Figure 5 Schematic diagram of the hardware structure, this embodiment provides a cloud server, the cloud server may include a processor and a memory, the memory stores a picture compensation program, when the picture compensation program is executed by the processor, implements all or part of the steps of the second embodiment of the picture compensation method of the present invention.

[0159] Specifically, a cloud server refers to a network device or network system that can achieve network connection. A cloud server can be a network device such as a server or a cloud platform, or it can be a network system composed of a central control server and a streaming server.

[0160] It can be understood that the cloud server may also include a communication bus, a user interface and a network interface. The communication bus is used to realize the connection and communication between these components; the user interface is used to connect to the client and communicate data with the client, and the user interface may include an output unit, such as a display screen, and an input unit, such as a keyboard; the network interface is used to connect to the terminal device and communicate data with the terminal device, and the network interface may include an input / output interface, such as a standard wired interface and a wireless interface.

[0161] The memory is used to store various types of data, which may include, for example, instructions for any application or method in the cloud server, and data related to the application. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk, and optionally, the memory may also be a storage device independent of the processor.

[0162] The processor is used to call the picture compensation program stored in the memory and execute the picture compensation method as described above. The processor can be an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and is used to execute all or part of the steps of the second embodiment of the picture compensation method as described above.

[0163] Embodiment 5

[0164] Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic memory, a disk, an optical disk, a server, etc., and a computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, all or part of the steps of each embodiment of the picture compensation method of the present invention can be implemented.

[0165] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above embodiments are only optional embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structures or equivalent process changes made by using the contents of the present specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A picture compensation method, characterized in that: The method is applied to a terminal device, comprising: Obtaining a user's real-time action instruction, and sending the real-time action instruction to a cloud server; Determine whether a first picture sent by the cloud server is received within a preset response delay; wherein the first picture is a real-time picture generated by the cloud server according to the real-time action instruction or a predicted picture obtained according to the real-time action instruction and an established behavior habit model when uplink frame loss is detected; wherein the behavior habit model is established based on historical picture data and a correspondence between historical action instructions obtained by parsing the historical picture data; If the first picture is not received within the preset response delay, it is determined that there is downlink frame loss, and a second picture is obtained according to the real-time action instruction and the stored behavior habit model; wherein the second picture is a historical picture corresponding to the historical action instruction matching the real-time action instruction, and the preset response delay is a delay corresponding to a preset frame number range; The second screen is displayed.

2. The image compensation method according to claim 1, characterized in that: After the step of obtaining the user's real-time action instruction and sending the real-time action instruction to the cloud server, the method further includes: Determine whether a downlink buffering condition is met; wherein the downlink buffering condition is that the user is in a stationary state and / or the action corresponding to the real-time action instruction does not belong to a preset large-scale action; If the downlink cache condition is met, receiving at least one frame of the third picture sent by the cloud server, and saving the at least one frame of the third picture; wherein the at least one frame of the third picture is a cached picture predicted by the cloud server according to the real-time action instruction and the established behavior habit model; After the step of determining whether the first picture sent by the cloud server is received within a preset response delay, the method further includes: If the first frame is not received within a preset response delay, the at least one frame of the third frame is displayed.

3. The image compensation method according to claim 1, wherein: Before the step of obtaining the second picture according to the real-time action instruction and the stored behavior habit model, the method includes: Obtaining an initialization instruction, and sending the initialization instruction to a cloud server; Receive the behavior habit model sent by the cloud server and store it.

4. The image compensation method according to claim 3, characterized in that: The step of obtaining the second picture according to the real-time action instruction and the stored behavior habit model comprises: Determine the behavior habit model corresponding to the target user according to the real-time action instruction and the behavior habit models of different users, and obtain the stored behavior habit model; Determining, in the stored behavior habit model, a historical action instruction that matches the real-time action instruction; According to the correspondence between the action instructions and the pictures in the stored behavior habit model, the historical pictures corresponding to the historical action instructions are extracted to obtain the second picture.

5. A picture compensation method, characterized in that: The method is applied to a cloud server and includes: Receive real-time action instructions sent by terminal devices; Generate a real-time picture according to the real-time action instruction; When uplink frame loss is detected, a predicted action instruction is obtained according to the real-time action instruction and the established behavior habit model; wherein the behavior habit model is established based on the historical picture data and the corresponding relationship between the historical action instructions obtained by parsing the historical picture data; generating a prediction picture according to the prediction action instruction; The real-time picture or the predicted picture is determined as the first picture, and the first picture is sent to the terminal device, so that the terminal device displays the first picture or the second picture; when the second picture is not received by the terminal device within the preset response delay and it is determined that there is downlink frame loss, the preset response delay is obtained according to the real-time action instruction and the stored behavior habit model, and the preset response delay is the delay corresponding to the preset frame number range.

6. The image compensation method according to claim 5, characterized in that: Before the step of obtaining a predicted action instruction according to the real-time action instruction and the established behavior habit model when uplink frame loss is detected, the method further comprises: Acquire historical action instructions of different users and corresponding historical screen data; wherein the historical action instructions are action instructions of historical operations of users and / or action instructions of users based on screen guidance operations; According to the historical action instructions and the historical screen data, a corresponding relationship between the action instructions and the screen is established to obtain the behavior habit models of different users; According to the real-time action instruction, the behavior habit model corresponding to the target user is determined to obtain the established behavior habit model.

7. The image compensation method according to claim 5, characterized in that: After the step of receiving the real-time action instruction sent by the terminal device, the method further includes: Obtaining at least one predicted action instruction according to the real-time action instruction and the established behavior habit model; Generate a corresponding cached picture according to the at least one predicted action instruction to obtain at least one third picture; The at least one frame of the third picture is sent to the terminal device, so that the terminal device saves the at least one frame of the third picture, and displays the at least one frame of the third picture when the first picture is not received within a preset response delay.

8. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein a picture compensation program is stored in the memory, and when the picture compensation program is executed by the processor, the picture compensation method according to any one of claims 1 to 4 is implemented.

9. A cloud server, characterized in that: The cloud server includes a processor and a memory, wherein a picture compensation program is stored in the memory, and when the picture compensation program is executed by the processor, the picture compensation method according to any one of claims 5 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by one or more processors, the image compensation method according to any one of claims 1 to 7 is implemented.

Images