Image display method and device based on double-channel video streams

Abstract

The invention relates to the technical field of VR (virtual reality), and provides an image display method and device based on double-channel video streams, which are used for improving the fluency of picture display. According to the method, the game terminal independently encodes the I frames in the left-eye image and the right-eye image according to the recoding request sent by the VR equipment aiming at the condition that the independently encoded images are abnormal when the two-way video streams are transmitted, so that the dependence of one-eye image on the other-eye image in the encoding process is relieved, the encoding efficiency is improved, and the user experience is improved. And in the process of waiting for data transmission, the VR equipment respectively predicts according to the latest left eye cache image and the latest right eye cache image, so that the picture jitter phenomenon is reduced. When the binocular images are stably displayed, the game terminal switches to a mode that one image is independently coded and the other image depends on coding according to the coding notice sent by the VR equipment, so that the data volume of coding is reduced, the transmission speed is improved, and the smooth display of the images is ensured.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and provides an image display method and device based on dual-channel video streams. Background Art

[0002] When users experience games through VR streaming, they need to transmit the left and right eye video streams in the SteamVR application from the game terminal (such as a computer, mobile phone, tablet, etc.) to the head-mounted display (HMD) of the VR device for display. At the same time, the VR device interacts with the SteamVR application on the game terminal based on the 6-degree of freedom (Dof) posture.

[0003] When data is encoded and transmitted, encoding, decoding, transmission and rendering take a certain amount of time. When the receiving frame rate of the VR end is lower than the sending frame rate of the game terminal, problems such as screen jitter, ghosting, and freeze may occur. In order to solve this problem, the relevant technology takes into account that the right eye image is not much different from the left eye image, and about 80% of the screen content is the same. Therefore, one video stream of the monocular image can be encoded and transmitted normally, and the video stream of the other monocular image is encoded based on the encoded video stream. In this way, the other monocular image is based on the encoded image, and only the difference part is encoded and transmitted, thereby reducing the overall data volume of the two video streams by reducing the amount of data transmitted by the other image, thereby improving the transmission speed and solving problems such as screen jitter, ghosting, and freeze.

[0004] However, during the data transmission process, due to the dependence of the other image on the encoded image, when an abnormality occurs in the encoded image, it will affect the generation of the other image, causing abnormalities in both video streams, thereby affecting the smoothness of the picture display. Summary of the invention

[0005] The embodiments of the present application provide an image display method and device based on dual-channel video streams, which are used to improve the smoothness of binocular image display.

[0006] On the one hand, an embodiment of the present application provides an image display method based on a dual-channel video stream, which is applied to a VR device, wherein the VR device performs VR streaming communication with a game terminal, including:

[0007] When an abnormality is detected in the first independently encoded image in the binocular image, a re-encoding request is sent to the game terminal, wherein the re-encoding request is used to instruct the game terminal to independently encode the I frames in the left-eye image and the right-eye image, respectively, and transmit them respectively through two links;

[0008] Predicting respectively according to the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and displaying the left-eye predicted image and the right-eye predicted image before receiving the next frame of data;

[0009] receiving the left-eye I-frame image and the right-eye I-frame image respectively sent by the game terminal through two links and decoding them, and displaying the decoded left-eye I-frame image and the right-eye I-frame image;

[0010] After determining that the binocular image display is stable, sending a coding notification to the game terminal, the coding notification is used to instruct the game terminal to independently encode the first image, to dependently encode the difference between the second image and the first image at the same time point, and to transmit them separately through two links, wherein the first image is any one of the left-eye image and the right-eye image, and the second image is the other one of the left-eye image and the right-eye image;

[0011] The first image and the difference part received by the game terminal transmitted through two links are decoded and binocularly displayed.

[0012] On the other hand, an embodiment of the present application provides an image display method based on a dual-channel video stream, which is applied to a game terminal, wherein the game terminal performs streaming communication with a VR device, including:

[0013] After receiving a re-encoding request sent by the VR device when detecting an abnormality in the independently encoded first image, independently encode the I frames in the left-eye image and the right-eye image according to the re-encoding request, and transmit them to the VR device through two links for binocular display;

[0014] After the vR device determines that the binocular image display is stable, receiving a coding notification sent by the vR device;

[0015] According to the coding notification, the first image is independently encoded, and the difference part of the second image relative to the first image at the same time point is dependently encoded, and the VR device is transmitted through two links respectively for binocular display, wherein the first image is any one of the left eye image and the right eye image, and the second image is the other one of the left eye image and the right eye image.

[0016] On the other hand, an embodiment of the present application provides a VR device, including a processor, a memory, a display, and a communication interface, wherein the communication interface, the display, the memory, and the processor are connected via a bus;

[0017] The communication interface is used for streaming communication with the game terminal;

[0018] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0019] When an abnormality is detected in the first independently encoded image in the binocular image, a re-encoding request is sent to the game terminal, wherein the re-encoding request is used to instruct the game terminal to independently encode the I frames in the left-eye image and the right-eye image, respectively, and transmit them respectively through two links;

[0020] Predicting respectively according to the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and displaying the left-eye predicted image and the right-eye predicted image through the display before receiving the next frame of data;

[0021] receiving a left-eye I-frame image and a right-eye I-frame image respectively sent by the game terminal through two links and decoding the images, and displaying the decoded left-eye I-frame image and right-eye I-frame image through the display;

[0022] After determining that the binocular image display is stable, sending a coding notification to the game terminal, the coding notification is used to instruct the game terminal to independently encode the first image, to dependently encode the difference between the second image and the first image at the same time point, and to transmit them separately through two links, wherein the first image is any one of the left-eye image and the right-eye image, and the second image is the other one of the left-eye image and the right-eye image;

[0023] The first image and the difference part received by the game terminal transmitted through two links are decoded, and binocular display is performed through the display.

[0024] On the other hand, an embodiment of the present application provides a game terminal, including a processor, a memory, and a communication interface, wherein the communication interface, the memory, and the processor are connected via a bus;

[0025] The communication interface is used for performing streaming communication with the vR device;

[0026] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0027] After receiving a re-encoding request sent by the VR device when detecting an abnormality in the independently encoded first image, independently encode the I frames in the left-eye image and the right-eye image according to the re-encoding request, and transmit them to the VR device through two links for binocular display;

[0028] After the vR device determines that the binocular image display is stable, receiving a coding notification sent by the vR device;

[0029] According to the coding notification, the first image is independently encoded, and the difference part of the second image relative to the first image at the same time point is dependently encoded, and the VR device is transmitted through two links respectively for binocular display, wherein the first image is any one of the left eye image and the right eye image, and the second image is the other one of the left eye image and the right eye image.

[0030] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of an image display method based on dual-channel video streams provided in an embodiment of the present application.

[0031] The beneficial effects of the image display method based on dual-channel video streams provided by the embodiment of the present application are as follows:

[0032] In response to the abnormality of the independently encoded first image during dual-channel video stream transmission, the game terminal independently encodes the I frames in the left-eye image and the right-eye image according to the re-encoding request sent by the VR device, and transmits them separately through two links, thereby eliminating the dependence of one eye image on the other eye image during the encoding process, saving the waiting time for the independently encoded image, and improving the encoding efficiency. In the process of waiting for data transmission, the VR device predicts according to the latest left-eye cache image and right-eye cache image respectively, thereby reducing the picture jitter phenomenon. When the binocular image display is stable, the game terminal independently encodes the first image according to the encoding notification sent by the VR device, and performs dependent encoding on the difference part of the second image relative to the first image at the same time point, and transmits them separately through two links, thereby reducing the amount of data transmitted by the second image, improving the transmission speed, and ensuring the smooth display of the picture.

[0033] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0035] Figure 1A schematic diagram of a VR streaming scenario provided in an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the interaction between a VR device and a game terminal provided in an embodiment of the present application;

[0037] Figure 3 A schematic diagram of an interface provided by the SDK of the SteamVR application in the game terminal provided in an embodiment of the present application;

[0038] Figure 4 A composite effect diagram of a left-eye image and a right-eye image provided in an embodiment of the present application;

[0039] Figure 5 A schematic diagram of the encoding and decoding process based on dual-channel video streams provided in an embodiment of the present application;

[0040] Figure 6 A flowchart of an image display method based on dual-channel video streams on a VR device side provided in an embodiment of the present application;

[0041] Figure 7 A flowchart of an image display method based on dual-channel video streams on a game terminal side provided in an embodiment of the present application;

[0042] Figure 8 A flow chart of the interaction method between a VR device and a game terminal provided in an embodiment of the present application;

[0043] Fig. 9 A structural diagram of an HMD of a VR device provided in an embodiment of the present application;

[0044] Fig.10 A structural diagram of a game terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0046] VR streaming refers to projecting the game on the game terminal to the VR device for display in a WIFI, mobile network environment, or wired environment. Through VR streaming, the VR device interacts with the SteamVR application of the game terminal, so that large games can be run on the game terminal, and the resources of the game terminal can be fully utilized to experience 3D games, avoiding the storage of large game entities on the VR device, making up for the problem of insufficient HMD performance in VR devices, and solving the problem that games cannot run across platforms due to differences in system platforms.

[0047] At present, there are two streaming connection methods between VR devices and game terminals: wireless streaming connection and wired streaming connection. Some devices can support both streaming connection methods. Among them, wireless streaming connection requires the VR device and the game terminal to have streaming assistants installed, while wired streaming connection also requires a data cable (such as a USB cable or Type C cable) to connect the game terminal and the VR device.

[0048] like Figure 1 , which is a schematic diagram of a VR streaming scenario provided by an embodiment of the present application, wherein the VR device includes an HMD with a posture acquisition sensor and a two-handed controller, and the computer is installed with a SteamVR application that can run 3D games. In the streaming scenario, the VR device and the SteamVR application interact with the game, allowing the user to immersively experience the 3D game in the SteamVR application.

[0049] Taking unlimited streaming connection as an example, the interaction process between VR device and game terminal is as follows: Figure 2 As shown, the SteamVR application on the game terminal runs a 3D game, a sensor for collecting posture data is installed on the VR device, a PC streaming assistant is installed on the game terminal, and a VR streaming assistant is installed on the VR device. After the PC streaming assistant on the game terminal establishes a streaming connection with the VR streaming assistant on the VR device, the game terminal obtains the game screen of the 3D game from the SteamVR application and collects the game sound, and then encodes the game screen and game sound, and sends the audio and video data to the VR device through the streaming assistant. After receiving the audio and video stream, the VR device performs audio and video decoding, renders the game screen of the left and right eyes to the left and right screens of the HMD respectively for playback, and at the same time, inputs the received game sound into the audio device of the HMD (such as a speaker) for playback, so that the user can see the game screen and hear the game sound. During the interaction process, the sensors in the HMD and handle of the VR device will collect 6DOf posture data (including 3D position and 3D rotation angle). The posture data is sent to the game terminal through the streaming assistant as the control data for the user to operate the game screen. The game terminal controls the game after receiving the posture data.

[0050] At present, VR streaming is mainly based on 3D game screens and the SDK of the SteamVR application. The SDK provides a streaming interface between the game terminal and the VR device. Through the streaming interface, the game terminal encodes the game screen in the SteamVR application and sends it to the VR device and receives the posture data sent by the VR device to control the game screen. Among them, the streaming interface provided by the SDK is as follows: Figure 3 As shown, it includes one audio and video output and one control data input. Since it only includes one audio and video output, the left and right eye images are usually combined into one video stream for encoding and transmission. Figure 4 As shown, this is a composite effect diagram of the left eye image and the right eye image.

[0051] As the user experience in VR streaming scenarios becomes higher and higher, one-channel audio and video output is difficult to meet the demand for smooth picture display. Therefore, in order to improve the display efficiency of the picture, the display method based on dual-channel audio and video data transmission has gradually become a research focus.

[0052] At present, in the scheme based on dual-channel video stream encoding and transmission, the left-eye image and the right-eye image are usually encoded and transmitted in two channels, wherein only one image is independently encoded, and then the other image is dependently encoded according to the difference between the binocular images, and then divided into two video streams for transmission, thereby reducing the amount of data transmitted for the other image, improving the transmission speed, and ensuring the smooth display of the picture.

[0053] Taking the right eye image as a reference to the left eye image as an example, the audio and video stream containing the left eye image is the main audio and video stream. Since the generation of the right eye image depends on the left eye image, the audio and video stream containing the right eye image is used as the auxiliary audio and video stream. The right eye image is decoded through the left eye image and binocular parallax, and the left eye image and the right eye image are cached separately, such as Figure 5 shown.

[0054] To a certain extent, although this method can solve the stuttering problem in VR streaming, during the data transmission process, since the other image is dependent on the independently encoded image, when an abnormality occurs in the encoded image, it will affect the generation of the other image, causing abnormalities in both video streams, thereby affecting the smoothness of the picture display.

[0055] In view of this, the embodiment of the present application provides an image display method based on dual-channel video streams. In this method, when an abnormality occurs in an independently encoded image during dual-channel video stream transmission, the VR device sends a re-encoding request to the game terminal. The game terminal independently encodes the I frames in the left-eye image and the right-eye image according to the re-encoding request, and transmits them separately through two links, thereby eliminating the dependence of one image on the other image during the encoding process, reducing the waiting time for the independently encoded image to return to normal, and improving the encoding efficiency; at the same time, the VR device will use the latest left-eye cache image and right-eye cache image for predictive interpolation, so that the picture will not freeze. When the data transmission returns to normal, the VR device sends a coding notification to the game terminal to restore the dependency coding of the second image on the first image, thereby reducing the amount of data transmitted by the second image coding, increasing the transmission speed, ensuring the smoothness of the picture display, and improving the user experience.

[0056] See also Figure 6 , which is a process of an image display method based on dual-channel video streams provided in an embodiment of the present application. The process is executed by a vR device, specifically an HMD in the vR device, and mainly includes the following steps:

[0057] S601: When an abnormality is detected in the independently encoded first image in the binocular image, a re-encoding request is sent to the game terminal, where the re-encoding request is used to instruct the game terminal to independently encode the I frames in the left-eye image and the right-eye image, respectively, and transmit them separately through two links.

[0058] In one example, when a VR device and a game terminal perform streaming communication, in order to reduce the amount of encoded data and increase the transmission speed, only the first image of the left eye image and the right eye image is independently encoded, and then the second image is dependently encoded based on the difference between the second image and the first image at the same time, thereby reducing the amount of data encoded for the second image and increasing the transmission speed. At this time, the transmission status of the first image will inevitably affect the transmission status of the second image. When the first image is transmitted abnormally, the second image will also be transmitted abnormally, and the recovery of the second image must wait until the first image is restored before it can be executed.

[0059] In order to improve the coding efficiency, when there is an abnormality in the independently encoded first image, the VR device sends a re-encoding request to the game terminal to release the coding dependency of the second image on the first image. The game terminal independently encodes the I frames in the left-eye image and the right-eye image, and transmits them to the VR device through two links respectively.

[0060] The reason why the I frame is selected for independent encoding is that the I frame is an intra-frame coded frame, and after being compressed at a ratio of 6:1, no visually perceptible model phenomenon will be generated, and spatial redundant information can be removed, thereby reducing the amount of coded data.

[0061] It should be noted that the first image in the embodiment of the present application can be any one of the left eye image and the right eye image, and the second image is the other one of the left eye image and the right eye image. For example, when the first image is the left eye image, the second image is the right eye image, and when the first image is the right eye image, the second image is the left eye image.

[0062] In one example, when the game terminal independently encodes the I frames in the left-eye image and the right-eye image, it can also carry the time point and posture information of the I frames so that the VR device can ensure the smoothness of the picture through predictive interpolation.

[0063] In actual applications, the game terminal, as the encoding end, will send the video stream at a set encoding frame rate, and the VR device, as the decoding end, will decode the video stream at a certain decoding frame rate, where the decoding frame rate is less than or equal to the encoding frame rate of the game terminal. When the decoding frame rate is less than the encoding frame rate, the VR device makes up the number of frames through motion compensation, and when the decoding frame rate is equal to the encoding frame rate, the decoded image frame is directly displayed. However, regardless of whether the decoding frame rate is less than the encoding frame rate or the decoding frame rate is equal to the encoding frame rate, the VR device will receive the video stream sent by the game terminal within the preset duration. Therefore, it is possible to determine whether there is a data anomaly based on the reception of the video stream.

[0064] In one example, the VR device decodes the video stream of the independently encoded first image according to the current decoding frame rate. If the first image is not obtained after decoding within a preset number of consecutive frames, it is determined that there is an abnormality in the first image. Otherwise, it is determined that the transmission of the first image is normal, and the transmission of the dual-channel video stream can continue according to the encoding method of the second image depending on the first image.

[0065] S602: Predictions are performed based on the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and the left-eye predicted image and the right-eye predicted image are displayed before receiving the next frame of data.

[0066] In one example, after receiving the left eye image and right eye image sent by the game terminal, the VR device will cache them, and cache the posture (i.e., position and rotation posture) corresponding to each frame of the left eye image and the right eye image. When an abnormality occurs, in order to avoid picture jitter and stuttering during the data encoding and transmission process, the VR device will perform predictive interpolation based on the latest left eye cache image and right eye cache image to obtain the left eye predicted image and the right eye predicted image, and display the left eye predicted image and the right eye predicted image before receiving the next frame of data, thereby reducing picture jitter and stuttering.

[0067] It should be noted that the embodiments do not restrict the prediction interpolation method, including but not limited to asynchronous time warp (ATW), asynchronous space warp (ASW) or a combination of the two.

[0068] Taking ATW as an example, for any latest cached image in the left-eye cached image and the right-eye cached image, the prediction interpolation process includes: predicting the current pose according to the pose sequence corresponding to the latest cached image and its previous n frames of cached images, interpolating according to the current pose and the latest cached image, and generating at least one frame of predicted image.

[0069] S603 receives and decodes the left-eye I-frame image and the right-eye I-frame image respectively sent by the game terminal through two links, and displays the decoded left-eye I-frame image and the right-eye I-frame image.

[0070] In one example, when an abnormality occurs in the independently encoded first image, the game terminal independently encodes the I frame in the left eye image and the I frame in the right eye image, obtains the left eye I frame image and the right eye I frame image, and sends the left eye I frame image and the right eye I frame image to the VR device through two links. Since the left eye image and the right eye image are independently encoded, the VR device can perform parallel decoding after receiving the left eye I frame image and the right eye I frame image, thereby increasing the decoding speed, and stops displaying the left eye predicted image and the right eye predicted image, and directly displays the decoded left eye I frame image and the right eye I frame image.

[0071] S604: After determining that the binocular image display is stable, send a coding notification to the game terminal, the coding notification is used to instruct the game terminal to independently encode the first image, perform dependent encoding on the difference between the second image and the first image at the same time point, and transmit them separately through two links.

[0072] Considering that the amount of data independently encoded for the left-eye image and the right-eye image is greater than the amount of data encoded for the second image that depends on the first image, after the binocular image display of the VR device is stable, that is, the encoding transmission of the first image returns to normal, the VR device sends a coding notification to the game terminal to reduce the amount of encoded data. Based on the coding notification, the game terminal resumes the independent encoding of the first image, calculates the difference between the second image and the first image at the same time point, and performs dependent encoding on the difference, thereby reducing the amount of data encoded for the second image and reducing the transmission delay, and then quickly sends the encoded data to the VR device through two links.

[0073] In one example, when the game terminal encodes the difference between the first image and the second image and the first image, it may also carry the time point information of the first image and the second image, and may also carry the posture information of the first image for predictive interpolation.

[0074] S605: decoding the first image and the difference part transmitted by the receiving game terminal through two links, and performing binocular display.

[0075] During normal encoding transmission, since the second image is encoded dependently with the first image, when the VR device decodes the first image and the second image, the second image will be decoded dependently with the first image.

[0076] In specific implementation, first, the VR device independently decodes the received first image to obtain the decoded first image, then decodes the difference between the second image and the first image at the same time point, combines the decoded first image to obtain the decoded second image, and then performs binocular display of the decoded first image and second image.

[0077] It should be noted that when the first image is encoded and transmitted normally, the VR device can normally decode the first image and the second image transmitted in real time by the game terminal. Therefore, there is no need to predict interpolation based on the posture, and rendering and display can be performed directly. In one example, after decoding the first image and the second image, the display frame rate can also be increased by predictive interpolation.

[0078] Corresponding to the VR device side, Figure 7 The implementation process of the game terminal side in the image display method based on dual-channel video streams is shown, which mainly includes the following steps:

[0079] S701: receiving a re-encoding request sent by a VR device when detecting that an abnormality exists in a first independently encoded image.

[0080] S702: I frames in the left-eye image and the right-eye image are independently encoded according to the re-encoding request, and are transmitted to the VR device through two links for binocular display.

[0081] S703: After the VR device determines that the binocular image display is stable, receive a coding notification sent by the VR device.

[0082] S704: Independently encode the first image according to the encoding notification, perform dependent encoding on the difference between the second image and the first image at the same time point, and transmit them to VR devices through two links for binocular display.

[0083] The following describes the interaction process between the VR device and the game terminal in the image display method based on the dual-channel video stream, taking the first image as the left eye image and the second image as the right eye image. Figure 8 , mainly includes the following steps:

[0084] S801: The VR device and the game terminal are connected to the same local area network.

[0085] S802: The VR device and the game terminal open the streaming assistant respectively to establish a streaming connection.

[0086] S803: The game terminal responds to the user operation and starts SteamVR to stream the game.

[0087] Through S801 to S803, the VR device establishes a streaming connection with the game terminal, so that the VR device can use the computing resources of the game terminal to run large games, and the game terminal can use the VR device to provide a 3D game experience.

[0088] S804: The game terminal independently encodes the left eye image, calculates the difference between the right eye image and the left eye image at the same time point, and performs dependent encoding on the difference.

[0089] Since there is parallax between the two eyes and the parallax is small, there will be a small difference between the left eye image and the right eye image at the same time point. By encoding the difference between the second image and the first image, compared with independently encoding the second image, the amount of encoded data can be effectively reduced and the transmission speed can be increased. However, the right eye image transmitted in this way cannot be decoded independently and needs to rely on the left eye image.

[0090] In one example, during the encoding process, the time and posture information of the image can also be carried so as to perform predictive interpolation when transmission abnormalities occur to prevent picture jitter, freeze, and other phenomena.

[0091] S805: The game terminal transmits the first video stream of the encoded left-eye image and the second video stream of the difference part to the VR device through two links.

[0092] S806: The VR device decodes the first video stream and the second video stream to obtain a left-eye image and a difference between the right-eye image and the left-eye image.

[0093] Since the left-eye image is independently encoded, the left-eye image can be directly decoded through the first video stream.

[0094] S807: The VR device generates a right eye image based on the left eye image and the difference portion.

[0095] Since the second video stream contains only a small amount of difference data of the right eye image and cannot be decoded separately, it is necessary to wait until the left eye image is decoded and then generate the right eye image in combination with the difference data.

[0096] S808: The VR device displays the decoded left-eye image and right-eye image in binocular mode.

[0097] S809: The VR device determines whether the left eye image has not been obtained within a preset number of consecutive frames according to the current decoding frame rate. If so, execute S810; if not, return to S804.

[0098] Affected by the network environment, packet loss may occur during the image encoding and transmission process, resulting in the VR device being unable to decode the left-eye image and the right-eye image. Therefore, a detection mechanism is set up on the VR device side to detect whether the transmission link and image are abnormal by detecting whether the left-eye image is obtained within a preset number of consecutive frames.

[0099] S810: The VR device sends a re-encoding request to the game terminal.

[0100] S811: The VR device performs predictive interpolation based on the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and performs binocular display.

[0101] When encoding transmission anomalies occur, the VR device predicts and inserts frames for the left-eye image and the right-eye image respectively based on the cached images and postures, thereby avoiding jitter and freeze in the encoded image. At the same time, using the latest cached images for predictive interpolation can ensure the smoothness of the image without waiting for data, thus improving prediction efficiency.

[0102] S812: The game terminal independently encodes the I frames in the left-eye image and the right-eye image according to the re-encoding request.

[0103] When an encoding transmission anomaly occurs, the game terminal releases the dependency between the left-eye image and the right-eye image based on the re-encoding request, and encodes the left-eye image and the right-eye image independently, thereby reducing the dependency of the right-eye image on the left-eye image. There is no need to wait for the left-eye image to recover before restoring the right-eye image, making the decoded right-eye image smoother.

[0104] S813: The game terminal transmits the encoded first video stream of the left-eye image and the second video stream of the right-eye image to the VR device through two links.

[0105] S814: The VR device decodes the first video stream to obtain a left-eye image, and decodes the second video stream to obtain a right-eye image.

[0106] S815: The VR device performs binocular display of the decoded left-eye image and right-eye image.

[0107] S816: The VR device determines whether the binocular display is stable. If so, execute S817; if not, return to S811.

[0108] S817: The VR device sends a coding notification to reduce the data amount to the game terminal, so that the game terminal restores the coding method in S804.

[0109] After the binocular display is stable, the encoding method shown in S804 is restored to reduce the amount of data encoded for the right-eye image, thereby reducing the transmission delay and increasing the transmission speed.

[0110] In the image display method based on dual-channel video streams provided by the embodiment of the present application, when an abnormality occurs in an independently encoded image, the VR device predicts and inserts frames through the latest cached left-eye image and right-eye image, respectively, so that the image display will not be stuck. At the same time, by sending a re-encoding request to the game terminal to switch the image encoding method and data retransmission, the independent left-eye image and right-eye image are quickly acquired and displayed, improving the smoothness of the picture and ensuring the smoothness between the abnormal display and the normal display image. When the data encoding transmission returns to normal, the method of switching to independent encoding of one eye image and dependent encoding of the other eye image effectively reduces the amount of encoded data, increases the transmission speed, makes the streaming display smoother, and improves the user's streaming experience.

[0111] Based on the same technical concept, an embodiment of the present application provides a VR device, in which the HMD can implement the steps of the above-mentioned image display method based on dual-channel video streams and achieve the same technical effect.

[0112] See also Fig. 9 , the HMD in the VR device includes a processor 901, a memory 902, a display 903 and a communication interface 904, and the communication interface 904, the display 903, the memory 902 and the processor 901 are connected via a bus 905;

[0113] The communication interface 904 is used for streaming communication with the game terminal;

[0114] The memory 902 stores a computer program, and the processor 901 performs the following operations according to the computer program:

[0115] When an abnormality is detected in the first independently encoded image in the binocular image, a re-encoding request is sent to the game terminal, wherein the re-encoding request is used to instruct the game terminal to independently encode the I frames in the left-eye image and the right-eye image, respectively, and transmit them respectively through two links;

[0116] Predicting respectively according to the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and displaying the left-eye predicted image and the right-eye predicted image through the display 903 before receiving the next frame of data;

[0117] receiving a left-eye I-frame image and a right-eye I-frame image respectively sent by the game terminal through two links and decoding them, and displaying the decoded left-eye I-frame image and right-eye I-frame image through the display;

[0118] After determining that the binocular image display is stable, sending a coding notification to the game terminal, the coding notification is used to instruct the game terminal to independently encode the first image, to dependently encode the difference between the second image and the first image at the same time point, and to transmit them separately through two links, wherein the first image is any one of the left-eye image and the right-eye image, and the second image is the other one of the left-eye image and the right-eye image;

[0119] The first image and the difference part received by the game terminal transmitted through two links are decoded, and binocularly displayed through the display 903.

[0120] Optionally, the processor 901 decodes the first image and the difference part received from the game terminal transmitted through two links, and the specific operation is:

[0121] independently decoding the received first image to obtain a decoded first image;

[0122] A decoded second image is obtained according to the decoded first image and the difference between the first image and the second image at the same time point.

[0123] Optionally, the processor 901 detects whether there is an abnormality in the independently encoded first image in the following manner:

[0124] Decoding the video stream of the independently encoded first image according to the current decoding frame rate;

[0125] If the first image is not obtained after decoding within a preset number of consecutive frames, it is determined that there is an abnormality in the first image, wherein the decoding frame rate is less than or equal to the encoding frame rate of the game terminal, and when the decoding frame rate is less than the encoding frame rate, the frame number is supplemented by motion compensation.

[0126] Optionally, the processor 901 performs predictions according to the latest left-eye cache image and right-eye cache image respectively to obtain a left-eye predicted image and a right-eye predicted image, and the specific operations are:

[0127] For any latest cached image of the left-eye cached image and the right-eye cached image, perform the following operations:

[0128] Predicting the current posture according to the posture sequence corresponding to the latest cached image and the previous n frames of cached images;

[0129] Interpolate frames according to the current posture and the latest cached image to generate at least one frame of predicted image.

[0130] It should be noted that Fig. 9 This is just an example, and the necessary hardware for the HMD in the VR device to execute the steps of the image display method based on dual-channel video streams provided in the embodiment of the present application is given. If not shown, the HMD may also include the hardware of conventional VR devices such as speakers, microphones, function buttons, power supplies, cameras, and IMUs.

[0131] Based on the same technical concept, an embodiment of the present application provides a game terminal, which can be a server, or a PC, a game console, a mobile phone or other terminal, and can implement the steps of the above-mentioned image display method based on dual-channel video streams, and can achieve the same technical effect.

[0132] See also Fig.10 , the game terminal includes a processor 1001, a memory 1002 and a communication interface 1003, wherein the communication interface 1003, the memory 1002 and the processor 1001 are connected via a bus 1004;

[0133] The communication interface 1003 is used for streaming communication with the VR device;

[0134] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0135] After receiving a re-encoding request sent by the VR device when detecting an abnormality in the independently encoded first image, independently encode the I frames in the left-eye image and the right-eye image according to the re-encoding request, and transmit them to the VR device through two links for binocular display;

[0136] After the VR device determines that the binocular image display is stable, receiving a coding notification sent by the VR device;

[0137] According to the coding notification, the first image is independently encoded, and the difference part of the second image relative to the first image at the same time point is dependently encoded, and the VR device is transmitted through two links respectively for binocular display, wherein the first image is any one of the left eye image and the right eye image, and the second image is the other one of the left eye image and the right eye image.

[0138] It should be noted that Fig.10 This is just an example, and provides the necessary hardware for the game terminal to execute the steps of the image display method based on dual-channel video streams provided in the embodiment of the present application. If not shown, the game terminal may also include hardware such as speakers, microphones, function buttons, power supplies, cameras, and displays.

[0139] in, Fig. 9 and Fig.10 The memory in the term "memory" may be a volatile memory, such as a random-access memory (RAM); the memory may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor may include one or more central processing units (CPU) or a digital processing unit, etc.

[0140] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, an image display method based on dual-channel video streams in the aforementioned embodiment can be completed.

[0141] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute an image display method based on a dual-channel video stream in the aforementioned embodiment.

[0142] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0143] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0144] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0145] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0146] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image display method based on dual-channel video streams, characterized in that: Applied to a VR device, the VR device performs streaming communication with a game terminal, including: When an abnormality is detected in the first independently encoded image in the binocular image, a re-encoding request is sent to the game terminal, wherein the re-encoding request is used to instruct the game terminal to independently encode the I frames in the left-eye image and the right-eye image, respectively, and transmit them respectively through two links; Predicting respectively according to the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and displaying the left-eye predicted image and the right-eye predicted image before receiving the next frame of data; receiving the left-eye I-frame image and the right-eye I-frame image respectively sent by the game terminal through two links and decoding them, and displaying the decoded left-eye I-frame image and the right-eye I-frame image; After determining that the binocular image display is stable, sending a coding notification to the game terminal, the coding notification is used to instruct the game terminal to independently encode the first image, to dependently encode the difference between the second image and the first image at the same time point, and to transmit them separately through two links, wherein the first image is any one of the left-eye image and the right-eye image, and the second image is the other one of the left-eye image and the right-eye image; The first image and the difference part received by the game terminal transmitted through two links are decoded and binocularly displayed.

2. The method according to claim 1, characterized in that The step of decoding the first image and the difference part transmitted by the game terminal through two links includes: independently decoding the received first image to obtain a decoded first image; A decoded second image is obtained according to the decoded first image and the difference between the first image and the second image at the same time point.

3. The method according to claim 1, characterized in that The first image encoded independently is detected to be abnormal by: Decoding the video stream of the independently encoded first image according to the current decoding frame rate; If the first image is not obtained after decoding within a preset number of consecutive frames, it is determined that there is an abnormality in the first image, wherein the decoding frame rate is less than or equal to the encoding frame rate of the game terminal, and when the decoding frame rate is less than the encoding frame rate, the frame number is supplemented by motion compensation.

4. The method according to claim 1, characterized in that The step of performing prediction according to the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image comprises: For any latest cached image of the left-eye cached image and the right-eye cached image, perform the following operations: Predicting the current posture according to the posture sequence corresponding to the latest cached image and the previous n frames of cached images; Interpolate frames according to the current posture and the latest cached image to generate at least one frame of predicted image.

5. An image display method based on dual-channel video streams, characterized in that: Applied to a game terminal, the game terminal performs streaming communication with a VR device, including: After receiving a re-encoding request sent by the VR device when detecting an abnormality in the independently encoded first image, independently encode the I frames in the left-eye image and the right-eye image according to the re-encoding request, and transmit them to the VR device through two links for binocular display; After the VR device determines that the binocular image display is stable, receiving a coding notification sent by the VR device; According to the coding notification, the first image is independently encoded, and the difference part of the second image relative to the first image at the same time point is dependently encoded, and the VR device is transmitted through two links respectively for binocular display, wherein the first image is any one of the left eye image and the right eye image, and the second image is the other one of the left eye image and the right eye image.

6. A VR device, characterized in that: It includes a processor, a memory, a display and a communication interface, wherein the communication interface, the display, the memory and the processor are connected via a bus; The communication interface is used for streaming communication with the game terminal; The memory stores a computer program, and the processor performs the following operations according to the computer program: When an abnormality is detected in the first independently encoded image in the binocular image, a re-encoding request is sent to the game terminal, wherein the re-encoding request is used to instruct the game terminal to independently encode the I frames in the left-eye image and the right-eye image, respectively, and transmit them respectively through two links; Predicting respectively according to the latest left-eye cache image and right-eye cache image to obtain a left-eye predicted image and a right-eye predicted image, and displaying the left-eye predicted image and the right-eye predicted image through the display before receiving the next frame of data; receiving a left-eye I-frame image and a right-eye I-frame image respectively sent by the game terminal through two links and decoding the images, and displaying the decoded left-eye I-frame image and right-eye I-frame image through the display; After determining that the binocular image display is stable, sending a coding notification to the game terminal, the coding notification is used to instruct the game terminal to independently encode the first image, to dependently encode the difference between the second image and the first image at the same time point, and to transmit them separately through two links, wherein the first image is any one of the left-eye image and the right-eye image, and the second image is the other one of the left-eye image and the right-eye image; The first image and the difference part received by the game terminal transmitted through two links are decoded, and binocular display is performed through the display.

7. The VR device according to claim 6, wherein: The processor decodes the first image and the difference part transmitted by the game terminal through two links, and the specific operation is: independently decoding the received first image to obtain a decoded first image; A decoded second image is obtained according to the decoded first image and the difference between the first image and the second image at the same time point.

8. The VR device according to claim 6, characterized in that: The processor detects whether the independently encoded first image has an abnormality by: Decoding the video stream of the independently encoded first image according to the current decoding frame rate; If the first image is not obtained after decoding within a preset number of consecutive frames, it is determined that there is an abnormality in the first image, wherein the decoding frame rate is less than or equal to the encoding frame rate of the game terminal, and when the decoding frame rate is less than the encoding frame rate, the frame number is supplemented by motion compensation.

9. The VR device according to claim 6, wherein: The processor performs predictions according to the latest left-eye cache image and right-eye cache image respectively to obtain a left-eye prediction image and a right-eye prediction image, and the specific operations are: For any latest cached image of the left-eye cached image and the right-eye cached image, perform the following operations: Predicting the current posture according to the posture sequence corresponding to the latest cached image and the previous n frames of cached images; Interpolate frames according to the current posture and the latest cached image to generate at least one frame of predicted image.

10. A game terminal, characterized in that: It includes a processor, a memory and a communication interface, wherein the communication interface, the memory and the processor are connected via a bus; The communication interface is used for streaming communication with the VR device; The memory stores a computer program, and the processor performs the following operations according to the computer program: After receiving a re-encoding request sent by the VR device when detecting an abnormality in the independently encoded first image, independently encode the I frames in the left-eye image and the right-eye image according to the re-encoding request, and transmit them to the VR device through two links for binocular display; After the VR device determines that the binocular image display is stable, receiving a coding notification sent by the VR device; According to the coding notification, the first image is independently encoded, and the difference part of the second image relative to the first image at the same time point is dependently encoded, and the VR device is transmitted through two links respectively for binocular display, wherein the first image is any one of the left eye image and the right eye image, and the second image is the other one of the left eye image and the right eye image.

Images