A binocular image display method based on motion compensation and virtual reality device

Abstract

The application relates to the field of VR technology and provides a binocular image display method based on motion compensation and a virtual reality device, which are used for improving the fluency of binocular image display. In a VR streaming scene, when the current receiving frame rate of a VR device is less than the preset sending frame rate of a display terminal, in order to solve problems such as picture jitter, trailing, lag and the like, the VR device sends a control instruction for reducing the sending frame rate of binocular images to the display terminal, and after decoding left eye images and right eye images, only motion compensation is performed on monocular images, then the motion compensation result of the monocular images is adjusted by using the parallax of the binocular images, a predicted image of another eye image after compensation is obtained, relative to motion compensation performed on the binocular images respectively, the calculation resources and calculation time of motion compensation are effectively reduced, the display frame rate of the binocular images is improved, the fluency of picture display in the VR streaming scene is ensured, and the VR experience of a user is improved.

Description

Technical Field

[0001] This application relates to the field of virtual reality (VR) technology, and provides a method for displaying binocular images based on motion compensation and a virtual reality device. Background Technology

[0002] VR streaming technology refers to displaying a 2D image from a display terminal (such as a computer, mobile phone, or tablet) onto a VR device in environments such as Wi-Fi, mobile networks, or high-speed data transmission cables, enabling remote operation on the VR device. Specifically, the audio and video data on the display terminal is hard-encoded and sent to the VR device via network protocols. The VR device then decodes and renders the received audio and video data for binocular display on its screen. This technology is particularly widely used in the gaming industry.

[0003] However, on the one hand, the encoding, transmission, decoding, and rendering of audio and video data takes time, which can lead to insufficient refresh rate of the binocular images on the VR device, resulting in problems such as screen jitter and motion blur in the streaming experience. On the other hand, during data transmission, factors such as network fluctuations, network strength, and network signal can affect the display terminal's audio and video data packets. When the network environment is poor, the VR device may experience display stuttering and jitter, affecting the user's VR experience. Summary of the Invention

[0004] This application provides a method for displaying binocular images based on motion compensation and a virtual reality device, which improves the smoothness of binocular image display.

[0005] On one hand, embodiments of this application provide a motion-compensated binocular image display method applied to VR devices, wherein the VR device and a display terminal perform VR streaming communication, including:

[0006] When the current received frame rate of the stereo image is detected to be less than the preset transmission frame rate of the stereo image, a control command is sent to the display terminal. The control command is used to instruct the display terminal to reduce the frame rate of transmitting the stereo image.

[0007] The display terminal receives the binocular image transmitted at a reduced current transmission frame rate, and decodes it to obtain the left-eye image and the right-eye image;

[0008] Motion compensation is performed on either the left-eye image or the right-eye image to generate at least one first prediction image;

[0009] A disparity map is generated based on the left-eye image and the right-eye image;

[0010] For each first predicted image, a second predicted image is generated by combining the disparity map. The second image is the other eye image in the binocular images besides the reference image.

[0011] Before receiving the next frame of the binocular image, at least one of the first predicted image and the corresponding second predicted image are displayed simultaneously.

[0012] On the other hand, embodiments of this application provide a virtual reality device, including a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected via a bus;

[0013] VR streaming communication is performed with the display terminal through the communication interface;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] When the current received frame rate of the stereo image is detected to be less than the preset transmission frame rate of the stereo image, a control command is sent to the display terminal. The control command is used to instruct the display terminal to reduce the frame rate of transmitting the stereo image.

[0016] The display terminal receives the binocular image transmitted at a reduced current transmission frame rate, and decodes it to obtain the left-eye image and the right-eye image;

[0017] Motion compensation is performed on either the left-eye image or the right-eye image to generate at least one first prediction image;

[0018] A disparity map is generated based on the left-eye image and the right-eye image;

[0019] For each first predicted image, a second predicted image is generated by combining the disparity map. The second image is the other eye image in the binocular images besides the reference image.

[0020] Before receiving the next frame of the binocular image, at least one of the first predicted images and the corresponding second predicted image are simultaneously displayed on the display screen.

[0021] Optionally, for each first predicted image, the processor combines the disparity map to generate a second predicted image of the second image, specifically as follows:

[0022] For each first pixel in the first predicted image, obtain the disparity value of the first pixel in the disparity map;

[0023] A second pixel is generated based on the position of the first pixel and the disparity value;

[0024] If the second pixel is within the range of the second image, then the pixel value of the second pixel is obtained from the second image;

[0025] A second predicted image of the second image is generated based on the pixel values ​​of each second pixel.

[0026] Optionally, after obtaining the second predicted image, the processor further performs:

[0027] For each second pixel in the second predicted image, obtain the neighboring points of the second pixel within a preset neighborhood;

[0028] The pixel value of the second pixel is generated based on the pixel values ​​of each neighboring pixel.

[0029] Optionally, the processor generates a disparity map based on the left-eye image and the right-eye image, specifically by:

[0030] The left-eye image and the right-eye image are matched to obtain multiple pixel matching pairs;

[0031] For each pixel matching pair, a disparity value is calculated based on the positions of the two pixels in the pixel matching pair. The disparity value represents the horizontal offset between the two pixels.

[0032] A disparity map is generated based on the disparity values ​​of each pixel pair.

[0033] Optionally, when it is detected that the current received frame rate of the stereo image is equal to the preset transmitted frame rate of the stereo image, the processor further executes:

[0034] The received binocular images are directly decoded, and the decoded left-eye and right-eye images are displayed simultaneously.

[0035] Optionally, the process of establishing streaming communication between the VR device and the display terminal specifically involves the following steps:

[0036] The display terminal is connected to the same local area network;

[0037] Start the streaming software, receive the broadcast message sent by the TCP service of the display terminal, and establish a TCP connection with the display terminal;

[0038] When the display terminal launches the SteamVR application based on user operation, it establishes a streaming communication connection with the SteamVR application;

[0039] The binocular images hard-coded by the SteamVR application are received through the streaming communication connection.

[0040] On the other hand, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer device to perform the steps of a motion-compensated binocular image display method provided in embodiments of this application.

[0041] The beneficial effects of the motion-compensated binocular image display method provided in this application are as follows:

[0042] When a VR device receives binocular images from a display terminal, if the current received frame rate is lower than the preset transmission frame rate, to address issues such as image jitter, ghosting, and stuttering during streaming, the VR device sends a control command to the display terminal to reduce the transmission frame rate of the binocular images. After decoding the left and right eye images, motion compensation is performed only on the monocular image. Then, using the disparity map of the binocular images, the motion compensation result of the monocular image is adjusted to obtain the compensated predicted image for the other eye, which is then displayed in binocular mode. Obtaining the motion-compensated binocular image from the motion compensation result of the monocular image effectively reduces the computational resources and time for motion compensation, thereby increasing the display frame rate of the binocular image, ensuring smooth display of the image in VR streaming scenarios, and ultimately enhancing the user's VR experience.

[0043] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of a VR streaming scenario provided in an embodiment of this application;

[0046] Figure 2 A flowchart illustrating the method for establishing streaming communication between a display terminal and a VR device, as provided in this application embodiment;

[0047] Figure 3 A flowchart illustrating a method for displaying binocular images based on motion compensation, provided in an embodiment of this application;

[0048] Figure 4This is an example of the effect of motion compensation on a monocular image provided in an embodiment of this application;

[0049] Figure 5 A disparity map of a binocular image provided in an embodiment of this application;

[0050] Figure 6 A flowchart illustrating the method for generating a disparity map provided in an embodiment of this application;

[0051] Figure 7 A flowchart for predicting another image based on motion compensation results of a monocular image, provided in an embodiment of this application;

[0052] Figure 8 The diagram shows the effect of a binocular display provided in the embodiments of this application;

[0053] Figure 9 Example diagram of a motion-compensated binocular image display provided for embodiments of this application;

[0054] Figure 10 This is a structural diagram of the HMD of a virtual reality device provided in an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0056] With the widespread adoption of VR devices, immersive experiences have become increasingly popular. VR streaming technology allows two-dimensional images on a display terminal to be wirelessly transmitted to a VR device, which then displays them in binoculars, creating a stunning stereoscopic visual experience. However, due to factors such as encoding / decoding, data transmission, and network connectivity, VR streaming is prone to issues like image jitter, ghosting, and stuttering, negatively impacting the user's immersive experience.

[0057] To address the aforementioned issues, related technologies employ motion compensation strategies on VR devices to improve the display frame rate. In VR streaming scenarios, the display terminal renders separate images for the left and right eyes, then encodes and transmits them separately. The VR device decodes the received left and right eye data separately to obtain the left and right eye images, and then performs motion compensation on each image individually. However, performing motion compensation separately for each binocular image results in wasted computing resources and time, leading to high compensation costs.

[0058] In view of this, this application provides a method for displaying binocular images based on motion compensation. After receiving binocular images, the VR terminal selects one image as a reference image, performs motion compensation on the reference image to obtain a predicted image, and then adjusts the predicted image according to the disparity between each pixel in the reference image and the other image to obtain the motion-compensated predicted image of the other image. Thus, the motion compensation result of the binocular image is obtained through the motion compensation result of the monocular image, improving the display frame rate of the binocular image and effectively solving problems such as image jitter, ghosting, and stuttering in VR streaming. Moreover, compared with the motion compensation of the binocular image, it saves the computation amount and computation time of motion compensation, further improving the display frame rate of the binocular image, thereby effectively improving the user's VR experience.

[0059] like Figure 1 The diagram illustrates an application scenario provided in this application. The VR device includes a head-mounted display (HMD) with a pose acquisition sensor and two controllers. The display terminal has a SteamVR application installed to run VR games. In a streaming scenario, the VR device and the SteamVR application on the display terminal interact, allowing the user to experience immersive 3D games.

[0060] In VR streaming scenarios, streaming communication needs to be established between the display terminal and the VR device first. The communication process is as follows: Figure 2 As shown, it mainly includes the following steps:

[0061] S201: The VR device and the display terminal are connected to the same local area network.

[0062] S202: The display terminal starts the streaming software, and the streaming software calls the TCP service to send a broadcast message.

[0063] S203: The VR device starts the streaming software and receives broadcast messages sent by the TCP service.

[0064] S204: The VR device establishes a TCP connection with the display terminal based on the broadcast message.

[0065] S205: The streaming software on the display terminal responds to user input and launches the SteamVR application.

[0066] S206: The display terminal establishes a streaming communication connection with the VR device through the SteamVR application.

[0067] S207: The SteamVR application on the display terminal acquires binocular images and performs hardware encoding on each image.

[0068] S208: The display terminal sends the hard-coded binocular image to the VR device via a streaming communication connection.

[0069] S209: The VR device decodes the received binocular images, obtains the left-eye image and the right-eye image respectively, and displays them in binocular form.

[0070] During the streaming communication process between VR devices and display terminals, issues such as jitter, ghosting, and stuttering may occur due to factors such as device hardware conditions and network environment. To address this problem, embodiments of this application provide a motion-compensated binocular display method, see [link to relevant documentation]. Figure 3 This method is executed by the VR device, specifically by the HMD within the VR device, and mainly includes the following steps:

[0071] S301: Determine whether the current received frame rate of the stereo image is less than the preset transmitted frame rate of the stereo image. If it is less, execute S302; if it is equal, execute S308.

[0072] In one example, the preset sending frame rate is the number of image frames transmitted per second, such as 90 frames per second, which can be set according to the requirements of smoothness of the picture; the current receiving frame rate is the number of image frames received per second, such as 90 frames per second, which is affected by data encoding transmission and network environment.

[0073] In practical applications, VR devices determine whether motion compensation is needed by comparing the current received frame rate of the binocular images with the preset transmission frame rate of the binocular images sent by the display device. Specifically, motion compensation is required when the current received frame rate is less than the preset transmission frame rate, and no motion compensation is needed when the current received frame rate is equal to the preset transmission frame rate.

[0074] S302: Send a control command to the display terminal to instruct the reduction of the binocular image transmission frame rate.

[0075] When the current received frame rate is less than the preset sent frame rate, it indicates that the current received frame rate of the stereo image cannot meet the frame rate required to continuously provide a smooth display. At this time, the VR device can send a control command to the display terminal. The display terminal reduces the frame rate of the stereo image according to the control command, such as sending the stereo image to the VR device at half the preset sent frame rate. Then, the VR device uses a motion compensation algorithm to improve the display frame rate of the stereo image and ensure the smoothness of the picture.

[0076] For example, if the preset transmission frame rate is 90 frames per second, and the current transmission frame rate is reduced by half to 45 frames per second, after the frame rate is reduced from 90 frames per second to 45 frames per second, the VR device will subsequently receive 45 frames of stereo images per second. Based on the received 45 frames of stereo images, another 45 frames of stereo images are predicted through motion compensation, thereby restoring the display frame rate of the stereo images to 90 frames per second.

[0077] In one example, the reduced current frame rate ensures that the binocular images sent by the display device are fully received by the VR device, meaning there is no packet loss.

[0078] S303: Receives the binocular image sent by the display terminal at a reduced current transmission frame rate, and decodes it to obtain the left eye image and the right eye image.

[0079] In one example, the reduced current transmission frame rate can determine the number of frames to be motion-compensated. For instance, if the current transmission frame rate is half the preset transmission frame rate, one frame of stereo image needs to be motion-compensated every other frame; or, if the current transmission frame rate is two-thirds of the preset transmission frame rate, one frame of stereo image needs to be motion-compensated every two frames. Each frame of stereo image contains one left-eye image and one right-eye image.

[0080] S304: Perform motion compensation on any one of the reference images in the left-eye and right-eye images to generate at least one first prediction image.

[0081] Considering the slight differences between the binocular images acquired by VR devices, and the fixed distance between the binocular cameras forming the binocular images, this application embodiment addresses issues such as image jitter, ghosting, and stuttering during streaming. While reducing computational resource consumption and time consumption, after receiving and decoding the left and right eye images, motion compensation is performed only on the monocular image. The result of the motion compensation is then used as a reference to reconstruct the motion-compensated result of the other eye image.

[0082] Motion compensation refers to the process of using intelligent algorithms to analyze the previous frame between two adjacent frames and automatically generate one or more new frames, thereby shortening the display time between frames, improving the smoothness of the video, and achieving a better visual experience.

[0083] Taking the left-eye image as a reference image for motion compensation as an example, assuming the current transmission frame rate is reduced to half of the preset transmission frame rate, before the VR device receives the next frame of the left-eye image, it generates a first prediction image for the current left-eye image through a motion compensation algorithm. That is, it predicts a left-eye image every other frame, thereby reducing the time and difference between the current left-eye image and the next frame of the left-eye image and improving the smoothness of the picture.

[0084] The effect of motion compensation for monocular images is as follows: Figure 4 As shown, the motion state of an object in the predicted frame is located between the motion state of the current frame and the next frame, making the display of the image more coherent.

[0085] It should be noted that the embodiments of this application do not impose limiting requirements on motion compensation algorithms, such as including but not limited to motion compensation algorithms based on image patches, motion compensation algorithms based on object contours, motion compensation algorithms based on integral images, and motion compensation algorithms based on sparse representations. These algorithms can be processed by specific chips in VR devices, and will not be described in detail here.

[0086] Since VR streaming scenarios require the display of binocular images, after obtaining the predicted image of the monocular image after motion compensation, it is also necessary to generate the predicted image of the other monocular image.

[0087] S305: Generate a disparity map based on the left-eye and right-eye images.

[0088] In one example, after obtaining the first predicted image, a predicted image for the other visual image can be generated based on the disparity map between the two visual images. Here, the disparity map is a two-dimensional image that stores the disparity values ​​between all corresponding pixels in the stereo-corrected visual images. The disparity value refers to the horizontal distance between the projection points (i.e., pixels) of the same object in the two images.

[0089] The disparity map can be a grayscale image or a color image, such as... Figure 5 The image shown is the disparity map generated from a binocular image.

[0090] In one example, the semi-global block matching (SGBM) algorithm from the OpenCV library can be used. The specific disparity map generation process is as follows: Figure 6 As shown, it mainly includes the following steps:

[0091] S3051: Match the left-eye image and the right-eye image to obtain multiple pixel matching pairs.

[0092] Among them, the pixel matching algorithm for calculating disparity value can be a region matching algorithm (such as block matching algorithm, global matching algorithm) or a feature point-based matching algorithm (such as SIFT, SURF).

[0093] S3052: For each pixel matching pair, calculate the horizontal offset disparity value representing the horizontal offset between the two pixels based on the positions of the two pixels in the pixel matching pair.

[0094] For each pixel pair, the disparity value, representing the horizontal offset between the two pixels, is calculated based on the difference between the position coordinates of the pixel in the left-eye image and the position coordinates of the corresponding pixel in the right-eye image.

[0095] S3053: Generate a disparity map based on the disparity values ​​of each pixel pair.

[0096] The disparity values ​​of all pixel pairs are stored in a two-dimensional image to generate a disparity map between the binocular images.

[0097] S306: For each first predicted image, generate a second predicted image of the second image by combining the disparity map.

[0098] After obtaining the disparity map, a second prediction image of the second image can be generated based on the first prediction image after motion compensation. The second image is the other eye image in the binocular image besides the reference image. For example, when the reference image is the left eye image, the second image is the right eye image, and when the reference image is the right eye image, the second image is the left eye image.

[0099] In specific implementation, the process of generating the second predicted image is as follows: Figure 7 As shown, it mainly includes the following steps:

[0100] S3061: For each first pixel in the first predicted image, obtain the disparity value of the first pixel in the disparity map.

[0101] In one example, since the binocular camera is fixed, the disparity map of the binocular images can be used as the disparity map of the first and second predicted images. Therefore, the disparity value between corresponding pixels in the left and right eye images in the disparity map can be used as the disparity value between the first pixel in the first predicted image and the second pixel in the second predicted image.

[0102] S3062: Generate the second pixel based on the position of the first pixel and the disparity value.

[0103] In one example, for each first pixel in the first predicted image, the corresponding second pixel is generated by subtracting the corresponding disparity value in the disparity map from its position coordinates in the first predicted image.

[0104] S3063: Determine whether the second pixel is within the range of the second image. If yes, execute S3064; otherwise, end.

[0105] S3064: Obtain the pixel value of the second pixel from the second image.

[0106] When the second pixel is within the range of the second image, the pixel value at the corresponding position can be directly obtained from the second image.

[0107] S3065: Generate a second predicted image of the second image based on the pixel values ​​of each second pixel.

[0108] After traversing each first pixel, the pixel values ​​of each second pixel can be obtained through disparity repair, generating the second predicted image of the second image.

[0109] Considering that the second pixel may extend beyond the range of the second image and its pixel value cannot be directly obtained, the generated second predicted image may contain holes or missing parts. Therefore, in some embodiments, after obtaining the second predicted image, estimation and filling can be performed through interpolation or pixel compensation to resolve holes or missing parts in the second predicted image. Specifically, for each second pixel in the second predicted image, its neighboring points within a preset neighborhood are obtained, and the pixel value of the second pixel is generated based on the pixel values ​​of each neighboring point, thereby obtaining a complete second predicted image.

[0110] S307: Before receiving the next frame of the stereo image, simultaneously display at least one first predicted image and the corresponding second predicted image.

[0111] After obtaining the first and second predicted images after motion compensation of the binocular images, the VR device can display the first predicted image and the corresponding second predicted image in binocular mode before receiving the next frame of binocular images, thereby ensuring the smoothness of the VR screen and improving the user experience.

[0112] S308: Directly decodes the received binocular images and simultaneously displays the decoded left-eye and right-eye images.

[0113] In one example, when the current received frame rate of the binocular image is equal to the preset transmitted frame rate of the binocular image, it indicates that the VR device can display the image smoothly without motion compensation. At this time, the received binocular image can be directly decoded and the decoded left-eye and right-eye images can be displayed on the left and right eye screens.

[0114] like Figure 8 The image shown is an illustration of the binocular display provided in an embodiment of this application. Through binocular display, users can obtain stereoscopic vision, thereby better experiencing VR games.

[0115] Taking a PC as the display terminal, the overall process of the motion-compensated binocular image display method provided in this application is as follows: Figure 9 As shown, it mainly includes the following steps:

[0116] S901: The PC establishes a streaming communication connection with the HMD.

[0117] S902: The PC acquires the stereo images from the SteamVR application and performs hard encoding.

[0118] S903: The PC sends the hard-coded stereo image to the HMD according to the predicted transmission frame rate.

[0119] S904: The HMD decodes the received binocular images to obtain the left-eye and right-eye images respectively.

[0120] S905: HMD detects whether the current received frame rate of the stereo image is less than the preset transmitted frame rate of the stereo image. If it is less, then execute S906; if it is equal, then execute S914.

[0121] S906: The HMD sends a control command to the PC to reduce the frame rate of the stereo image transmission.

[0122] S907: The PC transmits stereo images at a reduced current transmission frame rate.

[0123] S908: The HMD decodes the received current binocular image to obtain the current left eye image and the current right eye image respectively.

[0124] S909: The HMD generates a disparity map of the binocular images based on the current left-eye image and the current right-eye image.

[0125] S910: HMD performs motion compensation on the current left-eye image to generate the first predicted image.

[0126] S911: The HMD generates a second prediction image based on the first prediction image and the disparity map.

[0127] S912: HMD fills in holes or missing parts in the second predicted image.

[0128] S913: The HMD renders the first and second predicted images onto the left and right eye screens respectively for binocular display, and executes S914 after receiving the next frame of binocular image.

[0129] S914: Directly decodes the received binocular images and renders the decoded left-eye and right-eye images onto the left and right-eye screens respectively for binocular display.

[0130] In the motion-compensated binocular image display method provided in this application embodiment, when the VR device receives binocular images sent by the display terminal, if the current received frame rate is less than the preset sent frame rate, in order to solve problems such as screen jitter, ghosting, and stuttering during the streaming process, the VR device sends a control command to the display terminal to reduce the sent frame rate of the binocular images. After decoding the left-eye and right-eye images, motion compensation is performed only on the monocular image. Then, the motion compensation result of the monocular image is adjusted using the parallax of the binocular images to obtain the predicted image after compensation for the other eye image. However, by performing motion compensation separately on the binocular images, the computational resources and computation time for motion compensation are effectively reduced, thereby improving the display frame rate of the binocular images, ensuring the smooth display of the image in the VR streaming scene, and thus improving the user's VR experience.

[0131] Based on the same technical concept, this application provides a virtual reality device, in which the HMD can implement the steps of the above-mentioned motion-compensated binocular image display method and achieve the same technical effect.

[0132] See Figure 10 The HMD in the virtual reality device includes a processor 1001, a memory 1002, a display screen 1003, and a communication interface 1004. The communication interface 1004, the display screen 1003, the memory 1002, and the processor 1001 are connected via a bus 1005.

[0133] VR streaming communication is performed with the display terminal through the communication interface 1004;

[0134] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0135] When the current received frame rate of the stereo image is detected to be less than the preset transmission frame rate of the stereo image, a control command is sent to the display terminal. The control command is used to instruct the display terminal to reduce the frame rate of transmitting the stereo image.

[0136] The display terminal receives the binocular image transmitted at a reduced current transmission frame rate, and decodes it to obtain the left-eye image and the right-eye image;

[0137] Motion compensation is performed on either the left-eye image or the right-eye image to generate at least one first prediction image;

[0138] A disparity map is generated based on the left-eye image and the right-eye image;

[0139] For each first predicted image, a second predicted image is generated by combining the disparity map. The second image is the other eye image in the binocular images besides the reference image.

[0140] Before receiving the next frame of the binocular image, at least one first predicted image and a corresponding second predicted image are simultaneously displayed on the display screen 1003.

[0141] Optionally, the processor 1001 generates a second predicted image of the second image for each first predicted image by combining the disparity map. Specifically, the operation is as follows:

[0142] For each first pixel in the first predicted image, obtain the disparity value of the first pixel in the disparity map;

[0143] A second pixel is generated based on the position of the first pixel and the disparity value;

[0144] If the second pixel is within the range of the second image, then the pixel value of the second pixel is obtained from the second image;

[0145] A second predicted image of the second image is generated based on the pixel values ​​of each second pixel.

[0146] Optionally, after obtaining the second predicted image, the processor 1001 further performs:

[0147] For each second pixel in the second predicted image, obtain the neighboring points of the second pixel within a preset neighborhood;

[0148] The pixel value of the second pixel is generated based on the pixel values ​​of each neighboring pixel.

[0149] Optionally, the processor 1001 generates a disparity map based on the left-eye image and the right-eye image, specifically by:

[0150] The left-eye image and the right-eye image are matched to obtain multiple pixel matching pairs;

[0151] For each pixel matching pair, a disparity value is calculated based on the positions of the two pixels in the pixel matching pair. The disparity value represents the horizontal offset between the two pixels.

[0152] A disparity map is generated based on the disparity values ​​of each pixel pair.

[0153] Optionally, when it is detected that the current received frame rate of the stereo image is equal to the preset transmitted frame rate of the stereo image, the processor 1001 further executes:

[0154] The received binocular images are directly decoded, and the decoded left-eye and right-eye images are displayed simultaneously.

[0155] Optionally, during the establishment of streaming communication between the virtual reality device and the display terminal, the processor 1001 performs the following specific operations:

[0156] The display terminal is connected to the same local area network;

[0157] Start the streaming software, receive the broadcast message sent by the TCP service of the display terminal, and establish a TCP connection with the display terminal;

[0158] When the display terminal launches the SteamVR application based on user operation, it establishes a streaming communication connection with the SteamVR application;

[0159] The binocular images hard-coded by the SteamVR application are received through the streaming communication connection.

[0160] in, Figure 10 The memory 1002 can be volatile memory, such as random-access memory (RAM); the memory 1002 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or the memory 1002 can be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory can be a combination of the above-described memories; the processor 1001 can include one or more central processing units (CPUs) or digital processing units, etc.

[0161] It should be noted that, Figure 10 This is merely an example illustrating the hardware necessary for an HMD in a virtual reality device to perform the steps of a motion-compensated binocular image display method provided in this application embodiment. Not shown, the HMD may also include hardware from conventional VR devices such as speakers, microphones, function buttons, a power supply, a camera, and an IMU.

[0162] This application also provides a computer-readable storage medium for storing instructions that, when executed, can perform a motion-compensated binocular image display method as described in the foregoing embodiments.

[0163] This application also provides a computer program product for storing a computer program for executing a motion-compensated binocular image display method according to the foregoing embodiments.

[0164] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0165] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for displaying binocular images based on motion compensation, characterized in that, Applied to VR devices, the VR devices communicate with a display terminal via VR streaming, including: When the current received frame rate of the stereo image is detected to be less than the preset transmission frame rate of the stereo image, a control command is sent to the display terminal. The control command is used to instruct the display terminal to reduce the frame rate of transmitting the stereo image. The display terminal receives the binocular image sent at a reduced current transmission frame rate, and decodes it to obtain the left-eye image and the right-eye image; Motion compensation is performed on either the left-eye image or the right-eye image to generate at least one first prediction image; A disparity map is generated based on the left-eye image and the right-eye image; For each first predicted image, a second predicted image is generated by combining the disparity map. The second image is the other eye image in the binocular images besides the reference image. Before receiving the next frame of the binocular image, at least one of the first predicted image and the corresponding second predicted image are displayed simultaneously.

2. The method as described in claim 1, characterized in that, The step of generating a second predicted image for each first predicted image by combining the disparity map includes: For each first pixel in the first predicted image, obtain the disparity value of the first pixel in the disparity map; A second pixel is generated based on the position of the first pixel and the disparity value; If the second pixel is within the range of the second image, then the pixel value of the second pixel is obtained from the second image; A second predicted image of the second image is generated based on the pixel values ​​of each second pixel.

3. The method as described in claim 2, characterized in that, After obtaining the second predicted image, the method further includes: For each second pixel in the second predicted image, obtain the neighboring points of the second pixel within a preset neighborhood; The pixel value of the second pixel is generated based on the pixel values ​​of each neighboring pixel.

4. The method as described in claim 1, characterized in that, The step of generating a disparity map based on the left-eye image and the right-eye image includes: The left-eye image and the right-eye image are matched to obtain multiple pixel matching pairs; For each pixel matching pair, a disparity value is calculated based on the positions of the two pixels in the pixel matching pair. The disparity value represents the horizontal offset between the two pixels. A disparity map is generated based on the disparity values ​​of each pixel pair.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: When the current received frame rate of the stereo image is detected to be equal to the preset transmission frame rate of the stereo image, the received stereo image is directly decoded, and the decoded left-eye image and right-eye image are displayed simultaneously.

6. The method according to any one of claims 1-4, characterized in that, The process of establishing streaming communication between the VR device and the display terminal includes: The display terminal is connected to the same local area network; Start the streaming software, receive the broadcast message sent by the TCP service of the display terminal, and establish a TCP connection with the display terminal; When the display terminal launches the SteamVR application based on user operation, it establishes a streaming communication connection with the SteamVR application; The binocular images hard-coded by the SteamVR application are received through the streaming communication connection.

7. A virtual reality device, characterized in that, It includes a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected via a bus; VR streaming communication is performed with the display terminal through the communication interface; The memory stores a computer program, and the processor performs the following operations according to the computer program: When the current received frame rate of the stereo image is detected to be less than the preset transmission frame rate of the stereo image, a control command is sent to the display terminal. The control command is used to instruct the display terminal to reduce the frame rate of transmitting the stereo image. The display terminal receives the binocular image sent at a reduced current transmission frame rate, and decodes it to obtain the left-eye image and the right-eye image; Motion compensation is performed on either the left-eye image or the right-eye image to generate at least one first prediction image; A disparity map is generated based on the left-eye image and the right-eye image; For each first predicted image, a second predicted image is generated by combining the disparity map. The second image is the other eye image in the binocular images besides the reference image. Before receiving the next frame of the binocular image, at least one of the first predicted images and the corresponding second predicted image are simultaneously displayed on the display screen.

8. The virtual reality device as described in claim 7, characterized in that, For each first predicted image, the processor combines the disparity map to generate a second predicted image of the second image. Specifically, the operation is as follows: For each first pixel in the first predicted image, obtain the disparity value of the first pixel in the disparity map; A second pixel is generated based on the position of the first pixel and the disparity value; If the second pixel is within the range of the second image, then the pixel value of the second pixel is obtained from the second image; A second predicted image of the second image is generated based on the pixel values ​​of each second pixel.

9. The virtual reality device as described in claim 8, characterized in that, After obtaining the second predicted image, the processor also performs: For each second pixel in the second predicted image, obtain the neighboring points of the second pixel within a preset neighborhood; The pixel value of the second pixel is generated based on the pixel values ​​of each neighboring pixel.

10. The virtual reality device as described in claim 7, characterized in that, The processor generates a disparity map based on the left-eye image and the right-eye image, specifically by: The left-eye image and the right-eye image are matched to obtain multiple pixel matching pairs; For each pixel matching pair, a disparity value is calculated based on the positions of the two pixels in the pixel matching pair. The disparity value represents the horizontal offset between the two pixels. A disparity map is generated based on the disparity values ​​of each pixel pair.

Images