Cloud game display method and device based on VR device

Abstract

The invention relates to the technical field of VR, and provides a cloud game display method and equipment based on VR equipment, which are used for improving the fluency of picture display. According to the method, aiming at the condition that a left-eye image and a right-eye image are independently coded and transmitted, when the decoding frame rate of a first-eye image video in a left-eye image video stream and a right-eye image video stream is smaller than a display frame rate, a cached first target image, a current frame of a normally received second-eye image and a binocular difference are used, and a second target image is obtained; the current frame of the first-eye image is predicted respectively, and the current frame of the first-eye image and the current frame of the second-eye image are displayed synchronously, so that the problem of picture jamming when the transmission of the monocular image is abnormal is solved, and the picture display fluency is improved; meanwhile, the field angle corresponding to the left eye image and the field angle corresponding to the right eye image are larger than the field angle of the VR equipment, so that the image black edge problem is reduced, the resolution ratio of the edge area is lower than that of the center area, the coding data volume is effectively reduced, and the transmission efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology and provides a cloud game display method and device based on VR devices. Background Art

[0002] When enjoying an immersive experience of cloud gaming based on VR devices, the cloud server is required to perform binocular rendering of the game screen, encode the game screen, and transmit it to the VR device through the network (such as public network or private network). The VR device then performs binocular stereo display and screen control.

[0003] In the traditional display process, when the cloud server encodes the game screen, it usually combines the left eye image and the right eye image into one image. Figure 1 As shown, the synthesized image is then encoded. However, in order to ensure the clarity of the picture, the resolution of the image is generally greater than 4K. Therefore, the data volume of the merged image is large, the encoding time is long, and the display efficiency is low.

[0004] Related technologies In order to solve the above problems, the cloud server independently encodes the left-eye image and the right-eye image of the game screen and transmits them to the VR device for display. Due to the independent encoding and transmission, the monocular image will experience transmission abnormalities, resulting in insufficient display frame rate of the monocular image and freezes, causing dizziness in the user and a poor user experience. Summary of the invention

[0005] The embodiments of the present application provide a cloud game display method and device based on a VR device, which are used to improve the display efficiency of VR images.

[0006] On the one hand, an embodiment of the present application provides a cloud game display method based on a VR device, which is applied to the VR device and includes:

[0007] Respectively receive the left-eye image video stream and the right-eye image video stream of the target game sent by the cloud server;

[0008] Decoding the left-eye image video stream to obtain the left-eye image and cache it, and decoding the right-eye image video stream to obtain the right-eye image and cache it, wherein the field of view corresponding to any monocular image of the left-eye image and the right-eye image is greater than the field of view of the VR device, and the resolution of the edge area of ​​the left-eye image and the right-eye image is lower than the resolution of the central area;

[0009] When the decoding frame rate of the first eye video stream of the left eye image video stream and the right eye image stream is less than the display frame rate, the current frame of the first eye image is predicted respectively using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image, where N is an integer greater than 1;

[0010] According to the field of view of the VR device itself, the predicted current frame of the first image and the decoded current frame of the second image are displayed in binocular stereo.

[0011] On the other hand, an embodiment of the present application provides a cloud game display method based on a VR device, which is applied to a cloud server, including:

[0012] Rendering a left-eye image and a right-eye image of a target game according to a field of view of a VR device, wherein the field of view corresponding to the left-eye image and the right-eye image is greater than the field of view of the VR device;

[0013] For any monocular image of the left-eye image and the right-eye image, segment the monocular image into a central block and a plurality of edge blocks, and downsample the resolution of the edge block so that the resolution of the edge block is lower than the resolution of the central block;

[0014] Combining the timestamp of the monocular image, encoding the center block and the edge block in parallel to obtain a monocular image video stream;

[0015] The left-eye image video stream and the right-eye image video stream are independently sent to the VR device for binocular stereoscopic display.

[0016] On the other hand, an embodiment of the present application provides a VR device, including a processor, a memory, a display, and a communication interface, wherein the communication interface, the display, the memory, and the processor are connected via a bus;

[0017] The communication interface is used to communicate with the cloud server;

[0018] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0019] Respectively receiving a left-eye image video stream and a right-eye image video stream of a target game sent by the cloud server;

[0020] Decoding the left-eye image video stream to obtain the left-eye image and cache it, and decoding the right-eye image video stream to obtain the right-eye image and cache it, wherein the field of view corresponding to any monocular image of the left-eye image and the right-eye image is greater than the field of view of the VR device, and the resolution of the edge area of ​​the left-eye image and the right-eye image is lower than the resolution of the central area;

[0021] When the decoding frame rate of the first eye video stream of the left eye image video stream and the right eye image stream is less than the display frame rate, the current frame of the first eye image is predicted respectively using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image, where N is an integer greater than 1;

[0022] According to the field of view of the VR device itself, the predicted current frame of the first image and the decoded current frame of the second image are displayed in binocular stereoscopic form through the display.

[0023] On the other hand, an embodiment of the present application provides a cloud server, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0024] The communication interface is used to communicate with the VR device;

[0025] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0026] Rendering a left-eye image and a right-eye image of a target game according to the field of view of the VR device, wherein the field of view corresponding to the left-eye image and the right-eye image is greater than the field of view of the VR device;

[0027] For any monocular image of the left-eye image and the right-eye image, segment the monocular image into a central block and a plurality of edge blocks, and downsample the resolution of the edge block so that the resolution of the edge block is lower than the resolution of the central block;

[0028] Combining the timestamp of the monocular image, encoding the center block and the edge block in parallel to obtain a monocular image video stream;

[0029] The left-eye image video stream and the right-eye image video stream are independently sent to the VR device for binocular stereoscopic display.

[0030] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a cloud game display method based on a VR device provided in an embodiment of the present application.

[0031] The beneficial effects of the cloud game display method and device based on VR device provided by the embodiments of the present application are as follows:

[0032] On the cloud server side, when rendering the left-eye image and the right-eye image of the target game, the field of view corresponding to the left-eye image and the right-eye image is made larger than the field of view of the VR device, thereby reducing the image black edge problem caused by asynchronous time distortion of the VR device. When encoding any monocular image, it is divided into a central block and multiple edge blocks for parallel encoding, thereby improving the encoding transmission efficiency. At the same time, since the resolution of the central block is higher than that of the edge block, the data volume is reduced while ensuring the user's viewing effect, further improving the encoding transmission efficiency, thereby ensuring the smoothness of the display on the VR device side.

[0033] On the VR device side, after the left-eye image video stream and the right-eye image video stream are decoded and cached respectively, in the case where the decoding frame rate of the first-eye video stream is lower than the display frame rate, the binocular difference between the first N consecutive frames of the first-eye image and the current frame of the second-eye image and the previous left-eye image and right-eye image is used to predict the current frame of the first-eye image respectively, and it is displayed synchronously with the current frame of the second-eye image. The prediction error between the first-eye images and between the first-eye image and the second-eye image is reduced through prediction, the accuracy of image prediction is improved, the smoothness of the picture display is ensured, and the user's immersive experience is effectively enhanced.

[0034] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0036] Figure 1 A composite effect diagram of a left-eye image and a right-eye image provided in an embodiment of the present application;

[0037] Figure 2 A communication scenario diagram of a VR device and a cloud server provided in an embodiment of the present application;

[0038] Figure 3 A schematic diagram of independent encoding and transmission of left-eye images and right-eye images provided in an embodiment of the present application;

[0039] Figure 4 A flow chart of a cloud game display method based on a VR device implemented on the cloud server side provided in an embodiment of the present application;

[0040] Figure 5A An image segmentation method provided in an embodiment of the present application;

[0041] Figure 5B Another image segmentation method provided in the embodiment of the present application;

[0042] Figure 6 A flow chart of a cloud game display method based on a VR device implemented on the VR device side provided in an embodiment of the present application;

[0043] Figure 7 A flow chart of a current frame prediction method for a monocular image provided in an embodiment of the present application;

[0044] Figure 8 A schematic diagram of the interaction process between the HMD and the cloud server provided in an embodiment of the present application;

[0045] Fig. 9 A structural diagram of a cloud server provided in an embodiment of the present application;

[0046] Fig.10 A structural diagram of a VR device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0048] With the popularity of VR devices, the immersive experience is becoming more and more popular among users, especially in the field of games. However, since VR devices are mostly worn on the user, in order to reduce the user's sense of heaviness, their space structure is limited. Therefore, small and simple processing chips are usually used. Some large-scale games usually require high-configuration processors to run, such as cloud servers, but cloud servers cannot provide immersive experience.

[0049] In order to experience the fun of large-scale games, the advantages of VR devices and cloud servers are combined. The cloud server is responsible for running cloud games and transmitting the audio and video streams of the games to the VR devices. The VR devices display the game screen and play the game sounds according to the received audio and video streams. At the same time, the position and button information of the two handles are transmitted to the cloud server to control the game operation. Figure 2 As shown, in this way, the computing resources of the cloud server can be fully utilized to experience 3D games, while avoiding storing super-large game entities on VR devices.

[0050] In actual applications, when using VR devices to provide an immersive experience of cloud gaming, the cloud server needs to perform binocular rendering of the game screen, encode the game screen, and transmit it to the VR device through the network, so that the VR device can perform binocular stereoscopic display and screen control. However, when the cloud server encodes and transmits the game screen, in order to improve efficiency, the left eye image and the right eye image are encoded and transmitted independently, such as Figure 3 As shown, when the monocular image transmission is abnormal, the monocular image display frame rate on the VR device side will be insufficient, resulting in a freeze phenomenon, causing the user to feel dizzy and a poor user experience.

[0051] In view of this, the embodiment of the present application provides a cloud game display method based on VR devices. After receiving the left eye image and the right eye image independently encoded and transmitted by the cloud server, the VR device caches the left eye image and the right eye image. In this way, when a monocular image transmission exception occurs, the monocular image of the previous frames and the difference between the other eye image and the previous binocular image can be used to predict the current frame and display it respectively, thereby solving the problem of insufficient display frame rate when the monocular image transmission is abnormal, ensuring the smoothness of image display, and improving the user's VR game experience. In addition, when the timestamps of the left eye image and the right eye image are inconsistent on the VR device side, the synchronous display of the left eye image and the right eye image is ensured by interpolating the delayed image.

[0052] On the other hand, for the solution of independently encoding and transmitting the left-eye image and the right-eye image respectively, the field of view corresponding to the left-eye image and the right-eye image rendered by the cloud server during the rendering stage of the game screen is larger than the field of view of the HMD in the VR device, thereby reducing the image black edge problem caused by the automatic white balance of the VR device; and, in the monocular image encoding stage, the cloud server divides any monocular image into multiple image blocks unequally, and the resolution of the middle image block is higher than that of the edge image block, thereby reducing the size of the entire image data and improving the transmission efficiency. At the same time, each image block is independently encoded in parallel, which further improves the image encoding efficiency and ensures the smoothness of the display on the VR device end.

[0053] See also Figure 4, is a flow chart of a cloud game display method based on a VR device provided in an embodiment of the present application. The process is executed by a cloud server running a target game, and mainly includes the following steps:

[0054] S401: For a target game, render a left-eye image and a right-eye image whose field of view angle is larger than the field of view angle of the VR device.

[0055] Considering that during the communication between the cloud server and the VR device, the VR device may perform an asynchronous time warp (ATW) operation to ensure the smoothness of the game screen display, and the ATW operation may cause black edges on the image, affecting the display effect of the game screen. Therefore, in one example, the cloud server renders the left-eye image and the right-eye image of the target game according to the field of view of the VR device, so that the field of view corresponding to the left-eye image and the right-eye image is larger than the field of view of the VR device.

[0056] For example, the field of view of a VR device is 90°, and the field of view corresponding to the left-eye image and the right-eye image rendered by the cloud server is 100°.

[0057] S402: For any monocular image in the left-eye image and the right-eye image, divide the monocular image into a central block and a plurality of edge blocks, and downsample the resolution of the edge block so that the resolution of the edge block is lower than the resolution of the central block.

[0058] In one example, in order to improve the coding efficiency of the image, the cloud server divides the left-eye image and the right-eye image into multiple image blocks respectively. Each image block corresponds to a number and is arranged in order. In this way, when multiple image blocks are encoded in parallel, the VR device can decode the complete monocular image according to the number.

[0059] like Figure 5A As shown, any monocular image is evenly divided into 3*3 image blocks.

[0060] In one example, considering the visual characteristics of human eyes that they pay more attention to the central area and less attention to the edge area, the cloud server unequally divides any monocular image into a central block and multiple edge blocks, such as Figure 5B As shown, the resolution of each edge block is downsampled, so that the resolution of the edge block is lower than that of the center block, so that the center area is encoded using a high-resolution image and the edge area is encoded using a low-resolution image, thereby reducing the amount of image encoding data without affecting the image viewing effect, and further improving the image transmission efficiency.

[0061] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the number of image blocks, which can be appropriately adjusted according to actual conditions.

[0062] S403: Combining the timestamp of the monocular image, encoding the center block and the edge block in parallel to obtain a monocular image video stream.

[0063] In one example, a process is started for a central block and each of multiple edge blocks in any monocular image, and multiple image blocks are encoded in parallel to obtain a monocular image video stream. During the encoding process, the timestamp of the monocular image is carried in the data packet so that the VR device can decode the image and display the binocular image synchronously.

[0064] S404: Send the left-eye image video stream and the right-eye image video stream independently to the VR device for binocular stereoscopic display.

[0065] In one example, a left-eye image video stream obtained by encoding the left-eye image and a right-eye image video stream obtained by encoding the right-eye image are transmitted to a VR device through two links using the User Datagram Protocol (UDP), and are decoded by the VR device for binocular stereoscopic display.

[0066] In an embodiment of the present application, the left-eye image and the right-eye image are independently encoded and transmitted through two encoders. Compared with combining the left-eye image and the right-eye image into one image for encoding and transmission, this can effectively reduce the load of the encoder, reduce the encoding time, and even reduce the amount of retransmitted data when packet loss occurs due to network failure.

[0067] For the VR device side, the cloud game display method based on the VR device provided in the embodiment of the present application is as follows: Figure 6 As shown in FIG. 1 , the process is specifically executed by a head mounted display (HMD) in a VR device and mainly includes the following steps:

[0068] S60 1: Respectively receive the left-eye image video stream and the right-eye image video stream of the target game sent by the cloud server.

[0069] In one example, the VR device receives the left-eye image video stream and the right-eye image video stream sent by the cloud server through two links respectively.

[0070] S602: Decode the left-eye image video stream to obtain the left-eye image and cache it; and decode the right-eye image video stream to obtain the right-eye image and cache it.

[0071] Among them, the field of view angles corresponding to the decoded left-eye image and right-eye image are greater than the field of view angle of the VR device, and the resolution of the edge areas in the left-eye image and the right-eye image is lower than the resolution of the central area.

[0072] S603: For the first target video stream in the left-eye image video stream and the right-eye video stream, determine whether the decoding frame rate of the first target video stream is less than the display frame rate. If so, execute S604; if so, execute S606.

[0073] In one example, since the left-eye image video stream and the right-eye image video stream are transmitted independently in two links, due to network factors, one of the video streams may be abnormally received, resulting in image freezes on the VR device. Therefore, for the first target video stream in the left-eye image video stream and the right-eye video stream, it is necessary to determine whether the decoding frame rate of the first target video stream is less than the display frame rate. When the decoding frame rate is less than the display frame rate, it indicates that the first target image will freeze and requires special processing. When the decoding frame rate is equal to the display frame rate, it indicates that the first target image will not freeze and no processing is required.

[0074] The first target image video stream may be a left-eye image video stream or a right-eye image video stream. When the first target image video stream is a left-eye image video stream, the second target image video stream is a right-eye image video stream, and when the first target image video stream is a right-eye image video stream, the second target image video stream is a left-eye image video stream.

[0075] S604: using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image, respectively predict the current frame of the first eye image, where N is an integer greater than 1.

[0076] In one example, when the decoding frame rate of the first video stream is lower than the display frame rate, in order to avoid display freeze of the first image, the first image needs to be predicted.

[0077] See also Figure 7 , the prediction process of the monocular image provided in the embodiment of the present application mainly includes the following steps:

[0078] S6041: Predict the first current frame of the first image according to the pixel difference between every two adjacent images in the first N frames of the first image.

[0079] In one example, based on the cached first N frames of the first image, the optical flow method is used to implement pixel image prediction, and the process is as follows:

[0080] S6041_1: For each two adjacent frames of images in the first N frames of the first image, calculate the grayscale value change of corresponding pixel points in the two adjacent frames of images.

[0081] In specific implementation, for every two adjacent frames of images, the grayscale values ​​of the corresponding pixels are obtained respectively, and the grayscale values ​​of the pixels in the previous frame are subtracted from the grayscale values ​​of the pixels in the next frame to obtain the grayscale value change of the pixels in the next frame relative to the pixels in the previous frame.

[0082] S6041_2: Obtain the movement speed and direction of each pixel in the first image according to the change in each grayscale value.

[0083] Among them, the gray value change corresponding to each two adjacent frame images is a motion vector, so the movement speed and movement direction of each pixel in the first image can be calculated.

[0084] S6041_3: Predict a first current frame of the first image according to the moving speed and moving direction of each pixel point and the time difference corresponding to the display frame rate.

[0085] The time difference corresponding to the display frame rate, that is, the difference between the timestamps of two adjacent frames of images, combined with the movement speed and direction of each pixel, can be used to predict the current frame of the first image.

[0086] S6042: Subtract the grayscale values ​​of corresponding pixels in the cached previous right-eye image and left-eye image to obtain binocular disparity, and subtract the binocular disparity from the current frame of the second-eye image to predict the second current frame of the first-eye image.

[0087] In one example, assuming that the reception of the second eye image video is normal, the second current frame of the first eye image can be predicted by the decoded current frame of the second eye image and the binocular difference between the left eye image and the right eye image.

[0088] S6043: Obtain a final current frame of the first image according to the first current frame and the second current frame.

[0089] In an embodiment of the present application, the current frame of the first image is predicted by using the cached current frames of the first image and the second image and the binocular difference. Compared with prediction in any other manner, the prediction error between the first images and between the first image and the second image can be reduced, the accuracy of pixel-level prediction can be improved, and the display quality of the predicted image can be ensured.

[0090] S605: Perform binocular stereoscopic display on the predicted current frame of the first eye image and the decoded current frame of the second eye image according to the field of view of the VR device itself.

[0091] In one example, in order to reduce the black edge problem of image display of VR devices, when the cloud server renders the left-eye image and the right-eye image of the target game, the field of view corresponding to the image is larger than the field of view of the VR device. After encoding and transmission, when the VR device performs binocular display, it distorts and color-corrects the left-eye image and the right-eye image according to its own field of view, and then displays them synchronously on the binocular screen, thereby achieving an immersive experience of the target game.

[0092] For example, the field of view of the VR device itself is 90°, and the field of view corresponding to the left-eye image and the right-eye image is 100°. The VR device renders the current frame according to the 90° field of view.

[0093] S606: synchronously displaying the decoded left-eye image and right-eye image.

[0094] When the decoding frame rate of the video stream of the first eye image is equal to the display frame rate, it indicates that both the left eye image video stream and the right eye image video stream are transmitted normally, and the decoded left eye image and right eye image can be distorted and color corrected and then displayed synchronously.

[0095] See also Figure 8 The interaction process between the cloud server and the HMD during the cloud game display based on the VR device provided in the embodiment of the present application mainly includes the following steps:

[0096] S801: The HMD establishes a connection with the cloud server.

[0097] S802: The HMD and the cloud server perform clock synchronization.

[0098] In one example, the HMD sends a clock synchronization request to the cloud server. After the cloud server exchanges clock synchronization information with the HMD multiple times based on the clock synchronization, the HMD calculates the time baseline and the offset of the cloud server relative to the time baseline, thereby completing the clock synchronization.

[0099] S803: The HMD sends its own field of view angle to the cloud server via the UDP protocol.

[0100] S804: The cloud server renders the left-eye image and the right-eye image of the target game according to the field of view of the HMD.

[0101] The field of view angles corresponding to the left-eye image and the right-eye image are greater than the field of view angle of the HMD, and the resolution of the edge areas in the left-eye image and the right-eye image is lower than the resolution of the central area.

[0102] S805: The cloud server encodes the left-eye image into a left-eye image video stream and the right-eye image into a right-eye image video stream by dividing the image into blocks, and transmits them to the HMD through two links.

[0103] S806: The HMD decodes the left-eye image from the left-eye image video stream and caches the left-eye image, and decodes the right-eye image from the right-eye image video stream and caches the right-eye image.

[0104] S807: The HMD determines whether the decoding frame rate of the first video stream is less than the display frame rate. If so, execute S808; if so, execute S810.

[0105] Among them, the first video stream is a left-eye image video stream or a right-eye image video stream.

[0106] S808: The HMD predicts the current frame of the first eye image using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous left eye image and right eye image, respectively, where N is an integer greater than 1.

[0107] Among them, when the first eye image video stream is a left eye image video stream, the second eye image video stream is a right eye image video stream and is transmitted normally; when the first eye image video stream is a right eye image video stream, the second eye image video stream is a left eye image video stream and is transmitted normally.

[0108] S809: The HMD performs binocular stereoscopic display of the predicted current frame of the first eye image and the decoded current frame of the second eye image according to its own field of view.

[0109] S810: synchronously displaying the decoded left-eye image and right-eye image.

[0110] In an embodiment of the present application, when the left-eye image and the right-eye image are independently encoded and transmitted, in the case where the display frame rate of any monocular image is insufficient, the current frame of the image is predicted based on the cached image and the current frame of the other normally received image and the binocular difference, thereby ensuring the display frame rate of the image, solving the problem of stuttering when the monocular image is transmitted abnormally, ensuring the smoothness of the picture display, and helping to enhance the user's immersive experience.

[0111] In one example, during the communication between the VR device and the cloud server, the VR device can also obtain the 6-degree-of-freedom posture, display delay time, etc. of the HMD and the two-hand handles, and send the 6-degree-of-freedom posture, display delay time, etc. to the cloud server. The cloud server predicts and encodes the next frame of binocular image according to the 6-degree-of-freedom posture, display delay time, etc., thereby improving the transmission efficiency of the binocular image and ensuring the smoothness of the display on the VR device end.

[0112] In one example, when the VR device does not receive the first image video stream for a period longer than a preset period, a data retransmission instruction may be sent to the cloud server. The cloud server encodes the first image into an IDR frame and retransmits it to the VR device according to the data retransmission instruction.

[0113] In one example, when the delay of a VR device receiving any monocular image video stream is greater than a preset delay, an encoding instruction may be sent to a cloud server so that the cloud server reduces the encoding bit rate of the monocular image.

[0114] In one example, when a VR device displays a left-eye image and a right-eye image, if the timestamps of the left-eye image and the right-eye image are inconsistent, a time warping operation is performed based on a lagged image, the posture of the lagged image, and the current posture. At the same time, a motion vector of the lagged image is estimated using a motion vector of another eye image and applied to the lagged image, thereby obtaining a binocular image with a consistent timestamp.

[0115] Based on the same technical concept, an embodiment of the present application provides a cloud server that can implement the cloud server side steps in the above-mentioned cloud game display method based on VR device, and can achieve the same technical effect, which will not be repeated here.

[0116] See also Fig. 9 , the cloud server includes a processor 901, a memory 902 and a communication interface 903, wherein the communication interface 903, the memory 902 and the processor 901 are connected via a bus 904;

[0117] The communication interface 903 is used to communicate with the VR device;

[0118] The memory 902 stores a computer program, and the processor 901 performs the following operations according to the computer program:

[0119] Rendering a left-eye image and a right-eye image of a target game according to the field of view of the VR device, wherein the field of view corresponding to the left-eye image and the right-eye image is greater than the field of view of the VR device;

[0120] For any monocular image of the left-eye image and the right-eye image, segment the monocular image into a central block and a plurality of edge blocks, and downsample the resolution of the edge block so that the resolution of the edge block is lower than the resolution of the central block;

[0121] Combining the timestamp of the monocular image, encoding the center block and the edge block in parallel to obtain a monocular image video stream;

[0122] The left-eye image video stream and the right-eye image video stream are independently sent to the VR device for binocular stereoscopic display.

[0123] Based on the same technical concept, an embodiment of the present application provides a VR device, specifically an HMD, which can implement the VR device side steps in the above-mentioned cloud game display method based on VR device, and can achieve the same technical effect, which will not be repeated here.

[0124] See also Fig.10 , the VR device includes a processor 1001, a memory 1002, a display 1003 and a communication interface 1004, wherein the communication interface 1004, the display 1003, the memory 1002 and the processor 1001 are connected via a bus 1005;

[0125] The communication interface 1004 is used to communicate with the cloud server;

[0126] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0127] Respectively receiving a left-eye image video stream and a right-eye image video stream of a target game sent by the cloud server;

[0128] Decoding the left-eye image video stream to obtain the left-eye image and cache it, and decoding the right-eye image video stream to obtain the right-eye image and cache it, wherein the field of view corresponding to any monocular image of the left-eye image and the right-eye image is greater than the field of view of the VR device, and the resolution of the edge area of ​​the left-eye image and the right-eye image is lower than the resolution of the central area;

[0129] When the decoding frame rate of the first eye video stream of the left eye image video stream and the right eye image stream is less than the display frame rate, the current frame of the first eye image is predicted respectively using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image, where N is an integer greater than 1;

[0130] According to the field of view of the VR device itself, the predicted current frame of the first image and the decoded current frame of the second image are displayed in binocular stereoscopic form through the display 1003.

[0131] Optionally, the processor 1001 uses the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image to respectively predict the current frame of the first eye image, and the specific operation is:

[0132] Predicting a first current frame of the first image according to a pixel difference between every two adjacent frames of the first N frames of the first image;

[0133] Subtracting the grayscale values ​​of corresponding pixels in the cached last frame right-eye image and left-eye image to obtain binocular disparity, and subtracting the binocular disparity from the current frame of the second eye image to predict the second current frame of the first eye image;

[0134] A final current frame of the first image is obtained according to the first current frame and the second current frame.

[0135] Optionally, the processor 1001 predicts the first current frame of the first image according to the pixel difference between every two adjacent images in the first N frames of the first image, and the specific operation is:

[0136] For each two adjacent frames of the first N frames of the first image, calculate the grayscale value change of corresponding pixel points in the two adjacent frames of the image;

[0137] According to the change amount of each gray value, the movement speed and movement direction of each pixel in the first image are obtained;

[0138] The first current frame of the first image is predicted according to the movement speed and movement direction of each pixel point and the time difference corresponding to the display frame rate.

[0139] Optionally, the processor 1001 further executes:

[0140] When the decoding frame rates of the left-eye image video stream and the right-eye image stream are equal to the display frame rates, the decoded left-eye image and right-eye image are synchronously displayed through the display 1003 .

[0141] It should be noted that Fig.10 This is just an example, and provides the necessary hardware for a VR device to execute the steps of a cloud game display method based on a VR device provided in an embodiment of the present application. If not shown, the VR device may also include a two-hand handle, an IMU, a speaker, a pickup, a power supply, etc.

[0142] In the game terminal of the present application, the memory in the cloud server and the VR device can be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory can also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory is any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, and the memory can be a combination of the above memories, but is not limited thereto. The processor may include one or more central processing units (CPU) or a general-purpose processor, a graphics processor (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.

[0143] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete a cloud game display method based on a VR device in the aforementioned embodiment.

[0144] An embodiment of the present application also provides a computer program product for storing a computer program, which is used to execute a cloud game display method based on a VR device in the aforementioned embodiment.

[0145] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0146] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cloud game display method based on VR equipment, characterized in that: Applied to VR devices, including: Respectively receive the left-eye image video stream and the right-eye image video stream of the target game sent by the cloud server; Decoding the left-eye image video stream to obtain the left-eye image and cache it, and decoding the right-eye image video stream to obtain the right-eye image and cache it, wherein the field of view corresponding to any monocular image of the left-eye image and the right-eye image is greater than the field of view of the VR device, and the resolution of the edge area of ​​the left-eye image and the right-eye image is lower than the resolution of the central area; When the decoding frame rate of the first eye video stream of the left eye image video stream and the right eye image stream is less than the display frame rate, the current frame of the first eye image is predicted respectively using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image, where N is an integer greater than 1; According to the field of view of the VR device itself, the predicted current frame of the first image and the decoded current frame of the second image are displayed in binocular stereo.

2. The method according to claim 1, characterized in that The method uses the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image to respectively predict the current frame of the first eye image, including: Predicting a first current frame of the first image according to a pixel difference between every two adjacent frames of the first N frames of the first image; Subtracting the grayscale values ​​of corresponding pixels in the cached last frame right-eye image and left-eye image to obtain binocular disparity, and subtracting the binocular disparity from the current frame of the second eye image to predict the second current frame of the first eye image; A final current frame of the first image is obtained according to the first current frame and the second current frame.

3. The method according to claim 2, characterized in that The predicting the first current frame of the first image according to the pixel difference between every two adjacent frames of the first N frames of the first image comprises: For each two adjacent frames of the first N frames of the first image, calculate the grayscale value change of corresponding pixel points in the two adjacent frames of the image; According to the change amount of each gray value, the movement speed and movement direction of each pixel in the first image are obtained; The first current frame of the first image is predicted according to the movement speed and movement direction of each pixel point and the time difference corresponding to the display frame rate.

4. The method according to claim 1, characterized in that The method further comprises: When the decoding frame rates of the left-eye image video stream and the right-eye image stream are equal to the display frame rates, the decoded left-eye image and right-eye image are displayed synchronously.

5. A cloud game display method based on VR equipment, characterized in that: Applied to cloud servers, including: Rendering a left-eye image and a right-eye image of a target game according to a field of view of a VR device, wherein the field of view corresponding to the left-eye image and the right-eye image is greater than the field of view of the VR device; For any monocular image of the left-eye image and the right-eye image, segment the monocular image into a central block and a plurality of edge blocks, and downsample the resolution of the edge block so that the resolution of the edge block is lower than the resolution of the central block; Combining the timestamp of the monocular image, encoding the center block and the edge block in parallel to obtain a monocular image video stream; The left-eye image video stream and the right-eye image video stream are independently sent to the VR device for binocular stereoscopic display.

6. A VR device, characterized in that: It includes a processor, a memory, a display and a communication interface, wherein the communication interface, the display, the memory and the processor are connected via a bus; The communication interface is used to communicate with the cloud server; The memory stores a computer program, and the processor performs the following operations according to the computer program: Respectively receiving a left-eye image video stream and a right-eye image video stream of a target game sent by the cloud server; Decoding the left-eye image video stream to obtain the left-eye image and cache it, and decoding the right-eye image video stream to obtain the right-eye image and cache it, wherein the field of view corresponding to any monocular image of the left-eye image and the right-eye image is greater than the field of view of the VR device, and the resolution of the edge area of ​​the left-eye image and the right-eye image is lower than the resolution of the central area; When the decoding frame rate of the first eye video stream of the left eye image video stream and the right eye image stream is less than the display frame rate, the current frame of the first eye image is predicted respectively using the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image, where N is an integer greater than 1; According to the field of view of the VR device itself, the predicted current frame of the first image and the decoded current frame of the second image are displayed in binocular stereoscopic form through the display.

7. The VR device according to claim 6, wherein: The processor uses the first N consecutive frames of the first eye image and the binocular difference between the current frame of the second eye image and the previous frame of the left eye image and the right eye image to predict the current frame of the first eye image respectively. The specific operation is: Predicting a first current frame of the first image according to a pixel difference between every two adjacent frames of the first N frames of the first image; Subtracting the grayscale values ​​of corresponding pixels in the cached last frame right-eye image and left-eye image to obtain binocular disparity, and subtracting the binocular disparity from the current frame of the second eye image to predict the second current frame of the first eye image; A final current frame of the first image is obtained according to the first current frame and the second current frame.

8. The VR device according to claim 7, characterized in that: The processor predicts the first current frame of the first image according to the pixel difference between each two adjacent images in the first N frames of the first image, and the specific operation is: For each two adjacent frames of the first N frames of the first image, calculate the grayscale value change of corresponding pixel points in the two adjacent frames of the image; According to the change amount of each gray value, the movement speed and movement direction of each pixel in the first image are obtained; The first current frame of the first image is predicted according to the movement speed and movement direction of each pixel point and the time difference corresponding to the display frame rate.

9. The VR device according to claim 6, wherein: The processor also executes: When the decoding frame rates of the left-eye image video stream and the right-eye image stream are equal to the display frame rates, the decoded left-eye image and right-eye image are synchronously displayed through the display.

10. A cloud server, characterized in that: It includes a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is used to communicate with the VR device; The memory stores a computer program, and the processor performs the following operations according to the computer program: Rendering a left-eye image and a right-eye image of a target game according to the field of view of the VR device, wherein the field of view corresponding to the left-eye image and the right-eye image is greater than the field of view of the VR device; For any monocular image of the left-eye image and the right-eye image, segment the monocular image into a central block and a plurality of edge blocks, and downsample the resolution of the edge block so that the resolution of the edge block is lower than the resolution of the central block; Combining the timestamp of the monocular image, encoding the center block and the edge block in parallel to obtain a monocular image video stream; The left-eye image video stream and the right-eye image video stream are independently sent to the VR device for binocular stereoscopic display.

Images