Binocular image display method based on motion compensation and virtual reality equipment

Abstract

The invention relates to the technical field of VR, provides a binocular image display method based on motion compensation and virtual reality equipment, and is used for improving the smoothness of binocular image display. In a VR streaming scene, when a current receiving frame rate of VR equipment is smaller than a preset sending frame rate of a display terminal, in order to solve the problems of picture jitter, smear, jamming and the like, the VR equipment sends a control instruction for reducing a binocular image sending frame rate to the display terminal, and after a left eye image and a right eye image are decoded, motion compensation is only carried out on a monocular image, so that the binocular image sending frame rate is reduced. Then the parallax of the binocular image is utilized to adjust the motion compensation result of the monocular image to obtain the predicted image after the compensation of the other monocular image, and compared with the mode of respectively carrying out motion compensation on the binocular image, the calculation resources and the calculation time of the motion compensation are effectively reduced, so that the display frame rate of the binocular image is improved, and the image quality is improved. Fluent display of pictures in the VR streaming scene is ensured, and the VR experience of a user is improved.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and provides a binocular image display method based on motion compensation and a virtual reality device. Background Art

[0002] VR streaming technology refers to displaying the two-dimensional images on the display terminal (such as computers, mobile phones, tablets, etc.) on the VR device under the environment of WIFI, mobile network or high-speed data transmission cable, so as to achieve the effect of remote operation on the VR device. Specifically, the audio and video data on the display terminal are hard-coded and sent to the VR device through the network protocol. The VR device then decodes and renders the received audio and video data to display binocularly on the screen of the VR device. This technology is widely used in the field of games.

[0003] However, on the one hand, the encoding, transmission, decoding and rendering of audio and video data takes a certain amount of time, which will lead to insufficient refresh frequency of binocular images on the VR device side, resulting in problems such as screen jitter and moving object ghosting in the streaming experience. On the other hand, during the data transmission process, affected by factors such as network fluctuations, network strength, and network signals, when the network environment is poor, the audio and video data packets sent by the display terminal will be lost, and the VR device side will experience display freezes, jitters and other problems, affecting the user's VR experience. Summary of the invention

[0004] The embodiments of the present application provide a binocular image display method and a virtual reality device based on motion compensation, which improve the smoothness of binocular image display.

[0005] On the one hand, an embodiment of the present application provides a binocular image display method based on motion compensation, which is applied to a VR device, wherein the VR device performs VR streaming communication with a display terminal, including:

[0006] When it is detected that the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image, a control instruction is sent to the display terminal, wherein the control instruction is used to instruct the display terminal to reduce the frame rate of sending the binocular image;

[0007] Receiving a binocular image sent by the display terminal at a reduced current sending frame rate, and obtaining a left-eye image and a right-eye image after decoding;

[0008] Performing motion compensation on any reference image of the left-eye image and the right-eye image to generate at least one first predicted image;

[0009] generating a disparity map according to the left-eye image and the right-eye image;

[0010] For each first predicted image, generate a second predicted image of a second image in combination with the disparity map, where the second image is another binocular image in the binocular image except the reference image;

[0011] Before receiving the next frame of the binocular image, at least one of the first predicted image and the corresponding second predicted image is displayed simultaneously.

[0012] On the other hand, an embodiment of the present application provides a virtual reality device, including a processor, a memory, a display screen, and a communication interface, wherein the communication interface, the display screen, the memory, and the processor are connected via a bus;

[0013] Perform VR streaming communication with the display terminal through the communication interface;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] When it is detected that the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image, a control instruction is sent to the display terminal, wherein the control instruction is used to instruct the display terminal to reduce the frame rate of sending the binocular image;

[0016] Receiving a binocular image sent by the display terminal at a reduced current sending frame rate, and obtaining a left-eye image and a right-eye image after decoding;

[0017] Performing motion compensation on any reference image of the left-eye image and the right-eye image to generate at least one first predicted image;

[0018] generating a disparity map according to the left-eye image and the right-eye image;

[0019] For each first predicted image, generate a second predicted image of a second image in combination with the disparity map, where the second image is another binocular image in the binocular image except the reference image;

[0020] Before receiving the next frame of the binocular image, at least one of the first predicted image and the corresponding second predicted image is simultaneously displayed on the display screen.

[0021] Optionally, the processor generates a second predicted image of the second image in combination with the disparity map for each first predicted image, and the specific operation is:

[0022] For each first pixel in the first predicted image, obtaining a disparity value of the first pixel in the disparity map;

[0023] generating a second pixel point according to the position of the first pixel point and the disparity value;

[0024] If the second pixel point is within the range of the second image, obtaining a pixel value of the second pixel point from the second image;

[0025] A second predicted image of the second image is generated according to the pixel value of each second pixel point.

[0026] Optionally, after obtaining the second predicted image, the processor further executes:

[0027] For each second pixel in the second predicted image, obtaining neighboring points of the second pixel in a preset neighborhood;

[0028] The pixel value of the second pixel point is generated according to the pixel values ​​of each neighboring point.

[0029] Optionally, the processor generates a disparity map according to the left-eye image and the right-eye image, and the specific operation is:

[0030] Matching the left-eye image and the right-eye image to obtain a plurality of pixel matching pairs;

[0031] For each pixel point matching pair, a disparity value is calculated according to positions of two pixel points in the pixel point matching pair, where the disparity value represents a horizontal offset between the two pixel points;

[0032] A disparity map is generated based on the disparity values ​​of each matching pair of pixels.

[0033] Optionally, when it is detected that the current receiving frame rate of the binocular image is equal to the preset sending frame rate of the binocular image, the processor further executes:

[0034] The received binocular image is directly decoded, and the decoded left-eye image and right-eye image are displayed simultaneously.

[0035] Optionally, the process of establishing streaming communication between the VR device and the display terminal is specifically performed as follows:

[0036] connected to the same local area network as the display terminal;

[0037] Starting the streaming software, receiving the broadcast message sent by the TCP service of the display terminal, and establishing a TCP connection with the display terminal;

[0038] When the display terminal starts the SteamVR application based on the user operation, a streaming communication connection is established with the SteamVR application;

[0039] The binocular image hard-coded by the SteamVR application is received through the streaming communication connection.

[0040] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of a binocular image display method based on motion compensation provided in an embodiment of the present application.

[0041] The beneficial effects of a binocular image display method based on motion compensation provided by an embodiment of the present application are as follows:

[0042] When the VR device receives the binocular image sent by the display terminal, if the current receiving frame rate is lower than the preset sending frame rate, in order to solve the problems of image jitter, ghosting, and freeze in the streaming process, the VR device sends a control instruction to the display terminal to reduce the sending frame rate of the binocular image, and after decoding the left eye image and the right eye image, only the monocular image is motion compensated, and then the disparity map of the binocular image is used to adjust the motion compensation result of the monocular image, and the predicted image after compensation of the other eye image is obtained and binocular display is performed. The binocular image after motion compensation is obtained through the motion compensation result of the monocular image, which effectively reduces the computing resources and computing time of motion compensation, thereby improving the display frame rate of the binocular image, ensuring the smooth display of the picture in the VR streaming scene, and thus improving the user's VR experience.

[0043] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A schematic diagram of a VR streaming scenario provided in an embodiment of the present application;

[0046] Figure 2 A flow chart of a method for establishing streaming communication between a display terminal and a VR device provided in an embodiment of the present application;

[0047] Figure 3 A flow chart of a binocular image display method based on motion compensation provided in an embodiment of the present application;

[0048] Figure 4This is an effect diagram of the monocular image after motion compensation provided in the embodiment of the present application;

[0049] Figure 5 A disparity map of a binocular image provided in an embodiment of the present application;

[0050] Figure 6 A flow chart of a method for generating a disparity map provided in an embodiment of the present application;

[0051] Figure 7 A flowchart for predicting another eye image based on the motion compensation result of a monocular image provided in an embodiment of the present application;

[0052] Figure 8 The effect diagram of binocular display provided by the embodiment of the present application;

[0053] Fig. 9 An example diagram of binocular image display based on motion compensation provided in an embodiment of the present application;

[0054] Fig.10 A structural diagram of the HMD of the virtual reality device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0056] With the popularity of VR devices, immersive experience is deeply loved by users. Through VR streaming technology, the two-dimensional images on the display terminal can be wirelessly transmitted to the VR device, which performs binocular display to present a three-dimensional visual feast. However, affected by codec, data transmission, and network factors, VR streaming experience is prone to problems such as image jitter, ghosting, and freezes, which affect the user's immersive experience.

[0057] In order to solve the above problems, the relevant technology adopts a motion supplement strategy on the VR device side to improve the display frame rate of the picture. In the VR streaming scenario, the display terminal will render the left eye image and the right eye image respectively, and then encode and transmit the left eye image and the right eye image respectively. The VR device will decode the received left eye data and right eye data respectively to obtain the left eye image and the right eye image, and then perform motion compensation on the left eye image and the right eye image respectively. However, performing motion compensation for binocular images separately will result in a waste of computing resources and computing time, and the compensation cost is high.

[0058] In view of this, an embodiment of the present application provides a binocular image display method based on motion compensation. After receiving the binocular image, the VR end selects one image as the reference image, performs motion compensation on the reference image to obtain a predicted image, and then adjusts the predicted image according to the parallax between the reference image and each pixel in the other image to obtain the predicted image after motion compensation of the other image, thereby obtaining the motion compensation result of the binocular image through the motion compensation result of the monocular image, thereby improving the display frame rate of the binocular image, effectively solving the problems of picture jitter, ghosting, and freeze in the VR streaming process, and compared with the motion compensation of the binocular image, it saves the calculation amount and calculation time of the motion compensation, further improves the display frame rate of the binocular image, and thus effectively improves the user's VR experience.

[0059] like Figure 1 As shown, it is a schematic diagram of the application scenario provided by the embodiment of the present application. The VR device includes a head-mounted display (HMD) with a posture acquisition sensor and a two-hand handle. The SteamVR application is installed on the display terminal to run VR games. In the streaming scenario, the VR device interacts with the SteamVR application on the display terminal, allowing the user to experience 3D games immersively.

[0060] In the VR streaming scenario, streaming communication must be established between the display terminal and the VR device. The communication process is as follows: Figure 2 As shown, it mainly includes the following steps:

[0061] S201: The VR device and the display terminal are connected to the same local area network.

[0062] S202: The display terminal starts the streaming software, and the streaming software calls the TCP service to send a broadcast message.

[0063] S203: The VR device starts the streaming software and receives the broadcast message sent by the TCP service.

[0064] S204: The VR device establishes a TCP connection with the display terminal according to the broadcast message.

[0065] S205: The streaming software on the display terminal responds to the user operation and starts the SteamVR application.

[0066] S206: The display terminal establishes a streaming communication connection with the VR device through the SteamVR application.

[0067] S207: The SteamVR application on the display terminal obtains binocular images and performs hard coding respectively.

[0068] S208: The display terminal sends the hard-coded binocular image to the VR device via a streaming communication connection.

[0069] S209: The VR device decodes the received binocular image to obtain a left-eye image and a right-eye image, respectively, and performs binocular display.

[0070] The streaming communication process between the VR device and the display terminal may be affected by the hardware conditions of the device and the network environment, and may cause jitter, ghosting, freeze and other problems. In order to solve this problem, the embodiment of the present application provides a binocular display method based on motion compensation, see Figure 3 The method flow is executed by the VR device, specifically the HMD in the VR device, and mainly includes the following steps:

[0071] S301: Determine whether the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image. If so, execute S302; if so, execute S308.

[0072] In one example, the preset sending frame rate is the number of image frames transmitted per second, such as 90 frames per second, which can be set according to the requirements of picture smoothness; the current receiving frame rate is the number of image frames received per second, such as 90 frames per second, and the frame rate is affected by data encoding transmission and network environment.

[0073] In actual applications, the VR device can determine whether motion compensation is required by comparing the current receiving frame rate of the binocular image with the preset sending frame rate of the binocular image sent by the display device. Specifically, when the current receiving frame rate is less than the preset sending frame rate, motion compensation is required, and when the current receiving frame rate is equal to the preset sending frame rate, motion compensation is not required.

[0074] S302: Sending a control instruction for instructing to reduce the frame rate of sending binocular images to the display terminal.

[0075] When the current receiving frame rate is lower than the preset sending frame rate, it indicates that the current receiving frame rate of the binocular image cannot meet the frame rate required to continuously provide smooth display. At this time, the VR device may send a control instruction to the display terminal, and the display terminal may reduce the frame rate of sending the binocular image according to the control instruction, such as sending the binocular image to the VR device at half of the preset sending frame rate. Then the VR device uses a motion compensation algorithm to increase the display frame rate of the binocular image to ensure the smoothness of the picture.

[0076] For example, the preset sending frame rate is 90 frames / second, and the current sending frame rate after being reduced by half is 45 frames / second. After being reduced from 90 frames / second to 45 frames / second, the VR device will subsequently receive 45 frames of binocular images per second, and based on the received 45 frames of binocular images, another 45 frames of binocular images are predicted through motion compensation, so that the display frame rate of the binocular images is restored to 90 frames / second.

[0077] In one example, the reduced current sending frame rate can ensure that the binocular image sent by the display device is completely received by the VR device, that is, there is no packet loss.

[0078] S303: receiving the binocular image sent by the display terminal at the reduced current sending frame rate, and obtaining a left-eye image and a right-eye image after decoding.

[0079] In one example, the reduced current sending frame rate can determine the number of frames obtained by motion compensation. For example, the current sending frame rate is half of the preset sending frame rate, and one frame of binocular image needs to be motion compensated every other frame; for another example, the current sending frame rate is two-thirds of the preset sending frame rate, and one frame of binocular image needs to be motion compensated every two frames. Each binocular image frame includes a left-eye image and a right-eye image.

[0080] S304: Perform motion compensation on any reference image in the left-eye image and the right-eye image to generate at least one first predicted image.

[0081] Considering that there are small differences in the binocular images obtained by VR devices, and the distance between the binocular cameras that form the binocular images is fixed. In order to solve the problems of image jitter, ghosting, and freezing in the streaming process, while reducing computing resource consumption and time consumption, in the embodiment of the present application, after receiving and decoding the left eye image and the right eye image, only the monocular image is motion compensated, and the result after motion compensation is used as a reference to restore the result after motion compensation of the other eye image.

[0082] Among them, motion compensation refers to analyzing the previous frame image through an intelligent algorithm between two adjacent frames to automatically generate one or more new frames between two adjacent frames, shortening the display time between frames, thereby improving the smoothness of the video and achieving better visual sensory effects.

[0083] Taking the left eye image as the reference image for motion compensation as an example, assuming that the current sending frame rate is reduced to half of the preset sending frame rate, the VR device generates a first predicted image for the current left eye image through a motion compensation algorithm before receiving the next left eye image frame, that is, predicts a left eye image frame every other frame, thereby reducing the time and difference between the current left eye image and the next left eye image frame, and improving the smoothness of the picture.

[0084] Among them, the effect of monocular image motion compensation is as follows Figure 4 As shown, the motion state of the object in the predicted frame is between the motion states of the current frame and the next frame, making the display of the picture more coherent.

[0085] It should be noted that the embodiments of the present application do not impose restrictive requirements on motion compensation algorithms, such as but not limited to image block-based motion compensation algorithms, object contour-based motion compensation algorithms, integral image-based motion compensation algorithms, and sparse representation-based motion compensation algorithms. These algorithms can be processed by specific chips in VR devices and will not be described in detail.

[0086] Since binocular images need to be displayed in VR streaming scenarios, after obtaining the predicted image after motion compensation of the monocular image, it is also necessary to generate a predicted image of the other eye image.

[0087] S305: Generate a disparity map according to the left-eye image and the right-eye image.

[0088] In one example, after obtaining the first predicted image, a predicted image of another image can be generated based on a disparity map between the binocular images. The disparity map refers to a two-dimensional image storing the disparity values ​​between all corresponding pixels in the stereo-corrected binocular images, and the disparity value refers to the horizontal distance between the projection points (i.e., pixels) of the same object in the two images.

[0089] Among them, the disparity map can be a grayscale image or a color image, such as Figure 5 As shown, this is the disparity map effect generated by the binocular image.

[0090] In one example, the semi-global block matching (SGBM) algorithm in the OpenCV library can be used to implement the disparity map generation process. Figure 6 As shown, it mainly includes the following steps:

[0091] S3051: Match the left-eye image and the right-eye image to obtain a plurality of pixel matching pairs.

[0092] The pixel matching algorithm used when calculating the disparity value may adopt a region matching algorithm (such as a block matching algorithm, a global matching algorithm) or a feature point-based matching algorithm (such as SIFT, SURF), etc.

[0093] S3052: For each pixel matching pair, calculate a disparity value representing a horizontal offset between the two pixels according to positions of the two pixels in the pixel matching pair.

[0094] For each pixel matching pair, a disparity value representing the horizontal offset between the two pixels is calculated based on the difference between the position coordinates of the pixel in the left-eye image and the position coordinates of the corresponding pixel in the right-eye image.

[0095] S3053: Generate a disparity map according to the disparity values ​​of each matching pair of pixel points.

[0096] The disparity values ​​of all pixel pairs are saved in a two-dimensional image to generate a disparity map between binocular images.

[0097] S306: For each first predicted image, generate a second predicted image of the second image in combination with the disparity map.

[0098] After obtaining the disparity map, a second predicted image of the second image can be generated based on the first predicted image after motion compensation, wherein the second image is another eye image in the binocular image except the reference image, for example, when the reference image is a left-eye image, the second image is a right-eye image, and when the reference image is a right-eye image, the second image is a left-eye image.

[0099] In specific implementation, the generation process of the second prediction image is as follows: Figure 7 As shown, it mainly includes the following steps:

[0100] S3061: For each first pixel in the first predicted image, obtain a disparity value of the first pixel in the disparity map.

[0101] In one example, since the binocular camera is fixed, the disparity map of the binocular image can be used as the disparity map of the first predicted image and the second predicted image. Therefore, the disparity values ​​between the corresponding pixels of the left eye image and the right eye image in the disparity map can be used as the disparity values ​​between the first pixel in the first predicted image and the second pixel in the second predicted image.

[0102] S3062: Generate a second pixel point according to the position and disparity value of the first pixel point.

[0103] In an example, for each first pixel point in the first predicted image, the corresponding disparity value in the disparity map is subtracted from the position coordinate of each first pixel point in the first predicted image to generate a corresponding second pixel point.

[0104] S3063: Determine whether the second pixel point is within the range of the second image, if so, execute S3064, if not, end.

[0105] S3064: Obtain a pixel value of a second pixel from the second image.

[0106] When the second pixel point is within the range of the second image, the pixel value of the corresponding position can be directly acquired from the second image.

[0107] S3065: Generate a second predicted image of the second image according to the pixel value of each second pixel point.

[0108] After traversing each first pixel point, the pixel value of each second pixel point can be obtained by parallax restoration to generate a second predicted image of the second image.

[0109] Considering that the second pixel point may exceed the range of the second image, the pixel value cannot be directly obtained, so the generated second predicted image may have holes or missing parts. Therefore, in some embodiments, after obtaining the second predicted image, it can be estimated and filled by interpolation or pixel compensation to solve the holes or missing parts in the second predicted image. In specific implementation, for each second pixel point in the second predicted image, the neighboring points of the second pixel point in a preset neighborhood are obtained, and the pixel value of the second pixel point is generated according to the pixel values ​​of each neighboring point, so as to obtain a complete second predicted image.

[0110] S307: Before receiving the next frame of the binocular image, simultaneously display at least one first predicted image and a corresponding second predicted image.

[0111] After obtaining the first predicted image and the second predicted image after binocular image motion compensation, the VR device can perform binocular display of the first predicted image and the corresponding second predicted image after motion compensation before receiving the next frame of binocular image, thereby ensuring the smoothness of the VR picture and improving the user experience.

[0112] S308: directly decode the received binocular image, and simultaneously display the decoded left-eye image and right-eye image.

[0113] In one example, when the current receiving frame rate of the binocular image is equal to the preset sending frame rate of the binocular image, it indicates that the VR device can display the picture smoothly without motion compensation. At this time, the received binocular image can be directly decoded, and the decoded left eye image and right eye image can be displayed on the left and right eye screens.

[0114] like Figure 8 As shown, it is an effect diagram of binocular display provided in an embodiment of the present application. Through binocular display, the user can obtain stereoscopic vision and thus better experience VR games.

[0115] Taking the display terminal as a PC as an example, the overall process of the binocular image display method based on motion compensation provided by the present application is as follows: Fig. 9 As shown, it mainly includes the following steps:

[0116] S901: The PC establishes a streaming communication connection with the HMD.

[0117] S902: The PC obtains the binocular image of the SteamVR application and performs hard coding.

[0118] S903: The PC sends the hard-coded binocular image to the HMD according to the predicted sending frame rate.

[0119] S904: The HMD decodes the received binocular images to obtain a left-eye image and a right-eye image respectively.

[0120] S905: The HMD detects whether the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image. If so, execute S906; if so, execute S914.

[0121] S906: The HMD sends a control instruction to the PC to reduce the frame rate of the binocular image transmission.

[0122] S907: The PC sends the binocular image at the reduced current sending frame rate.

[0123] S908: The HMD decodes the received current binocular image to obtain a current left-eye image and a current right-eye image.

[0124] S909: The HMD generates a disparity map of the binocular image according to the current left-eye image and the current right-eye image.

[0125] S910: The HMD performs motion compensation on the current left-eye image to generate a first predicted image.

[0126] S911: The HMD generates a second predicted image according to the first predicted image and the disparity map.

[0127] S912: The HMD fills the hole or missing part in the second predicted image.

[0128] S913: The HMD renders the first predicted image and the second predicted image onto the left and right eye screens respectively for binocular display, and executes S914 after receiving the next frame of binocular image.

[0129] S914: directly decode the received binocular image, and render the decoded left-eye image and right-eye image onto left-eye and right-eye screens respectively for binocular display.

[0130] In the binocular image display method based on motion compensation provided in the embodiment of the present application, when the VR device receives the binocular image sent by the display terminal, when the current receiving frame rate is less than the preset sending frame rate, in order to solve the problems of picture jitter, ghosting, and freeze in the streaming process, the VR device sends a control instruction to reduce the sending frame rate of the binocular image to the display terminal, and after decoding the left-eye image and the right-eye image, only motion compensation is performed on the monocular image, and then the motion compensation result of the monocular image is adjusted using the parallax of the binocular image to obtain the predicted image after the compensation of the other eye image. However, by performing motion compensation on the binocular images separately, the computing resources and computing time of the motion compensation are effectively reduced, thereby improving the display frame rate of the binocular image, ensuring the smooth display of the picture in the VR streaming scene, and thus improving the user's VR experience.

[0131] Based on the same technical concept, an embodiment of the present application provides a virtual reality device, in which the HMD can implement the steps of the above-mentioned binocular image display method based on motion compensation and achieve the same technical effect.

[0132] See also Fig.10 , the HMD in the virtual reality device includes a processor 1001, a memory 1002, a display screen 1003 and a communication interface 1004, wherein the communication interface 1004, the display screen 1003, the memory 1002 and the processor 1001 are connected via a bus 1005;

[0133] Perform VR streaming communication with the display terminal through the communication interface 1004;

[0134] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0135] When it is detected that the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image, a control instruction is sent to the display terminal, wherein the control instruction is used to instruct the display terminal to reduce the frame rate of sending the binocular image;

[0136] Receiving a binocular image sent by the display terminal at a reduced current sending frame rate, and obtaining a left-eye image and a right-eye image after decoding;

[0137] Performing motion compensation on any reference image of the left-eye image and the right-eye image to generate at least one first predicted image;

[0138] generating a disparity map according to the left-eye image and the right-eye image;

[0139] For each first predicted image, generate a second predicted image of a second image in combination with the disparity map, where the second image is another binocular image in the binocular image except the reference image;

[0140] Before receiving the next frame of the binocular image, at least one first predicted image and a corresponding second predicted image are simultaneously displayed on the display screen 1003 .

[0141] Optionally, the processor 1001 generates a second predicted image of the second image in combination with the disparity map for each first predicted image, and the specific operation is:

[0142] For each first pixel in the first predicted image, obtaining a disparity value of the first pixel in the disparity map;

[0143] generating a second pixel point according to the position of the first pixel point and the disparity value;

[0144] If the second pixel point is within the range of the second image, obtaining a pixel value of the second pixel point from the second image;

[0145] A second predicted image of the second image is generated according to the pixel value of each second pixel point.

[0146] Optionally, after obtaining the second predicted image, the processor 1001 further executes:

[0147] For each second pixel in the second predicted image, obtaining neighboring points of the second pixel in a preset neighborhood;

[0148] The pixel value of the second pixel point is generated according to the pixel values ​​of each neighboring point.

[0149] Optionally, the processor 1001 generates a disparity map according to the left-eye image and the right-eye image, and the specific operation is:

[0150] Matching the left-eye image and the right-eye image to obtain a plurality of pixel matching pairs;

[0151] For each pixel point matching pair, a disparity value is calculated according to positions of two pixel points in the pixel point matching pair, where the disparity value represents a horizontal offset between the two pixel points;

[0152] A disparity map is generated based on the disparity values ​​of each matching pair of pixels.

[0153] Optionally, when it is detected that the current receiving frame rate of the binocular image is equal to the preset sending frame rate of the binocular image, the processor 1001 further executes:

[0154] The received binocular image is directly decoded, and the decoded left-eye image and right-eye image are displayed simultaneously.

[0155] Optionally, during the process of establishing streaming communication between the virtual reality device and the display terminal, the specific operation of the processor 1001 is:

[0156] connected to the same local area network as the display terminal;

[0157] Starting the streaming software, receiving the broadcast message sent by the TCP service of the display terminal, and establishing a TCP connection with the display terminal;

[0158] When the display terminal starts the SteamVR application based on the user operation, a streaming communication connection is established with the SteamVR application;

[0159] The binocular image hard-coded by the SteamVR application is received through the streaming communication connection.

[0160] in, Fig.10 The memory 1002 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 1002 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1002 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 1001 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0161] It should be noted that Fig.10 This is just an example, and provides the necessary hardware for the HMD in the virtual reality device to execute the steps of the binocular image display method based on motion compensation provided in the embodiment of the present application. If not shown, the HMD may also include the hardware of conventional VR devices such as speakers, microphones, function buttons, power supplies, cameras, and IMUs.

[0162] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, a binocular image display method based on motion compensation in the aforementioned embodiment can be completed.

[0163] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute a binocular image display method based on motion compensation in the aforementioned embodiment.

[0164] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0165] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0166] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0168] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A binocular image display method based on motion compensation, characterized in that: Applied to a VR device, the VR device performs VR streaming communication with a display terminal, including: When it is detected that the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image, a control instruction is sent to the display terminal, wherein the control instruction is used to instruct the display terminal to reduce the frame rate of sending the binocular image; Receiving a binocular image sent by the display terminal at a reduced current sending frame rate, and obtaining a left-eye image and a right-eye image after decoding; Performing motion compensation on any reference image of the left-eye image and the right-eye image to generate at least one first predicted image; generating a disparity map according to the left-eye image and the right-eye image; For each first predicted image, generate a second predicted image of a second image in combination with the disparity map, where the second image is another binocular image in the binocular image except the reference image; Before receiving the next frame of the binocular image, at least one of the first predicted image and the corresponding second predicted image is displayed simultaneously.

2. The method according to claim 1, characterized in that The step of generating a second predicted image of the second image by combining the disparity map with each first predicted image includes: For each first pixel in the first predicted image, obtaining a disparity value of the first pixel in the disparity map; generating a second pixel point according to the position of the first pixel point and the disparity value; If the second pixel point is within the range of the second image, obtaining a pixel value of the second pixel point from the second image; A second predicted image of the second image is generated according to the pixel value of each second pixel point.

3. The method according to claim 2, characterized in that After obtaining the second predicted image, the method further includes: For each second pixel in the second predicted image, obtaining neighboring points of the second pixel in a preset neighborhood; The pixel value of the second pixel point is generated according to the pixel values ​​of each neighboring point.

4. The method according to claim 1, characterized in that The generating a disparity map according to the left-eye image and the right-eye image includes: Matching the left-eye image and the right-eye image to obtain a plurality of pixel matching pairs; For each pixel point matching pair, a disparity value is calculated according to positions of two pixel points in the pixel point matching pair, where the disparity value represents a horizontal offset between the two pixel points; A disparity map is generated based on the disparity values ​​of each matching pair of pixels.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: When it is detected that the current receiving frame rate of the binocular image is equal to the preset sending frame rate of the binocular image, the received binocular image is directly decoded, and the decoded left-eye image and right-eye image are displayed simultaneously.

6. The method according to any one of claims 1 to 4, characterized in that The process of establishing streaming communication between the VR device and the display terminal includes: connected to the same local area network as the display terminal; Starting the streaming software, receiving the broadcast message sent by the TCP service of the display terminal, and establishing a TCP connection with the display terminal; When the display terminal starts the SteamVR application based on the user operation, a streaming communication connection is established with the SteamVR application; The binocular image hard-coded by the SteamVR application is received through the streaming communication connection.

7. A virtual reality device, characterized in that: It includes a processor, a memory, a display screen and a communication interface, wherein the communication interface, the display screen, the memory and the processor are connected via a bus; Perform VR streaming communication with the display terminal through the communication interface; The memory stores a computer program, and the processor performs the following operations according to the computer program: When it is detected that the current receiving frame rate of the binocular image is less than the preset sending frame rate of the binocular image, a control instruction is sent to the display terminal, wherein the control instruction is used to instruct the display terminal to reduce the frame rate of sending the binocular image; Receiving a binocular image sent by the display terminal at a reduced current sending frame rate, and obtaining a left-eye image and a right-eye image after decoding; Performing motion compensation on any reference image of the left-eye image and the right-eye image to generate at least one first predicted image; generating a disparity map according to the left-eye image and the right-eye image; For each first predicted image, generate a second predicted image of a second image in combination with the disparity map, where the second image is another binocular image in the binocular image except the reference image; Before receiving the next frame of the binocular image, at least one of the first predicted image and the corresponding second predicted image is simultaneously displayed on the display screen.

8. The virtual reality device according to claim 7, characterized in that: The processor generates a second predicted image of the second image in combination with the disparity map for each first predicted image, and the specific operation is: For each first pixel in the first predicted image, obtaining a disparity value of the first pixel in the disparity map; generating a second pixel point according to the position of the first pixel point and the disparity value; If the second pixel point is within the range of the second image, obtaining a pixel value of the second pixel point from the second image; A second predicted image of the second image is generated according to the pixel value of each second pixel point.

9. The virtual reality device according to claim 8, characterized in that: After obtaining the second predicted image, the processor further executes: For each second pixel in the second predicted image, obtaining neighboring points of the second pixel in a preset neighborhood; The pixel value of the second pixel point is generated according to the pixel values ​​of each neighboring point.

10. The virtual reality device according to claim 7, characterized in that: The processor generates a disparity map according to the left-eye image and the right-eye image, and the specific operation is: Matching the left-eye image and the right-eye image to obtain a plurality of pixel matching pairs; For each pixel point matching pair, a disparity value is calculated according to positions of two pixel points in the pixel point matching pair, where the disparity value represents a horizontal offset between the two pixel points; A disparity map is generated based on the disparity values ​​of each matching pair of pixels.

Images