Image coding method based on double-channel video stream and game terminal

Abstract

The invention relates to the technical field of VR (Virtual Reality), and provides an image coding method based on double-channel video streams and a game terminal, which are used for improving the transmission speed of the video streams. According to the method, on the premise of ensuring definition, in order to improve the transmission speed of the two video streams, only the I frame and the P frame in the two video streams are coded, so that the coding efficiency is effectively improved, the data volume of coding transmission is reduced, and meanwhile, when inter-frame prediction coding is carried out on the P frame in the two video streams, the coding efficiency is greatly improved. In consideration of the characteristic that the similarity between the left eye image and the right eye image is relatively high, only the reference P frame image is subjected to inter-frame prediction coding, and the target P frame image is coded based on the coding result of the reference P frame image, so that the coding efficiency is greatly improved compared with inter-frame prediction coding of two paths of P frame images. The coding efficiency is further improved, the size of the coded data volume is reduced, the transmission speed is effectively improved, and therefore the requirement for real-time rendering display of the VR equipment side in the streaming scene is met.

Description

Technical Field

[0001] The present application relates to the field of virtual reality (VR) technology, and provides an image encoding method based on dual-channel video streams and a game terminal. Background Art

[0002] VR streaming is to send the game screen running on the game terminal to the VR device for binocular display, so that users can get a 3D game experience. Figure 1 As shown, during the streaming process, the VR device sends the posture and button information of the head-mounted display (HMD) and the two-hand handles to the game terminal in real time. The game terminal uses the received posture and button information to render the game screen, and obtains the rendered binocular image. After the binocular image is encoded, it is transmitted to the VR device through the network (wired / wireless). The HMD decodes the received video and renders it on the binocular screen.

[0003] To ensure clarity, the resolution of the encoded video should be greater than 4K and the bit rate should be greater than 60Mbps. The traditional encoding method is to combine the left eye image and the right eye image into one image for encoding, such as Figure 2 As shown in FIG. 1 , this method is time-consuming and cannot meet the requirements of high-efficiency encoding. Therefore, currently, most methods use two encoders to perform independent video encoding on the left-eye image and the right-eye image, such as Figure 3 shown.

[0004] At present, in order to increase the transmission speed of the encoded dual-channel video stream, the relevant technology usually increases the encoding compression rate of the dual-channel video stream to reduce the amount of transmitted data. This will cause the clarity of the picture to decrease and affect the user's VR gaming experience. Summary of the invention

[0005] The embodiments of the present application provide an image encoding method based on a dual-channel video stream and a game terminal, which are used to reduce the amount of data transmitted by a binocular image.

[0006] On the one hand, an embodiment of the present application provides an image encoding method based on a dual-channel video stream, which is applied to a game terminal, and the game terminal performs streaming communication with a VR device, including:

[0007] Get one left-eye video stream and one right-eye video stream of the SteamVR application, and select one as the reference video stream and the other as the target video stream;

[0008] For each reference P frame image in the reference video stream, calculate a motion vector of the reference P frame image by dividing the image blocks, perform motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image, obtain a reference P frame prediction image, and calculate a first residual between the reference P frame image and the reference P frame prediction image;

[0009] For each target P frame image in the target video stream, determine a target motion vector of the target P frame image according to a reference motion vector of a corresponding reference P frame image in combination with binocular disparity, obtain a target P frame prediction image according to the target motion vector of the target P frame image and a corresponding reference P frame prediction image, and calculate a second residual between the target P frame image and the target P frame prediction image;

[0010] Inter-frame encoding is performed on the binocular image sent to the VR device according to the first residual and the second residual.

[0011] On the other hand, an embodiment of the present application provides a game terminal, including a processor, a memory, a display, and a communication interface, wherein the communication interface, the display, the memory, and the processor are connected via a bus;

[0012] The communication interface is used for streaming communication with the VR device;

[0013] The display is used to display the game screen after being controlled by the VR device;

[0014] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0015] Get one left-eye video stream and one right-eye video stream of the SteamVR application, and select one as the reference video stream and the other as the target video stream;

[0016] For each reference P frame image in the reference video stream, calculate a motion vector of the reference P frame image by dividing the image blocks, perform motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image, obtain a reference P frame prediction image, and calculate a first residual between the reference P frame image and the reference P frame prediction image;

[0017] For each target P frame image in the target video stream, determine a target motion vector of the target P frame image according to a reference motion vector of a corresponding reference P frame image in combination with binocular disparity, obtain a target P frame prediction image according to the target motion vector of the target P frame image and a corresponding reference P frame prediction image, and calculate a second residual between the target P frame image and the target P frame prediction image;

[0018] Inter-frame encoding is performed on the binocular image sent to the VR device according to the first residual and the second residual.

[0019] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute the steps of an image encoding method based on dual-channel video streams provided in an embodiment of the present application.

[0020] The beneficial effects of the image encoding method based on dual-channel video streams provided by the embodiment of the present application are as follows:

[0021] Under the premise of ensuring clarity, in order to improve the transmission speed of the left-eye video stream and the right-eye video stream, when performing inter-frame prediction encoding on the P frames in the two video streams, considering the high similarity between the left-eye image and the right-eye image, only the reference P frame image is inter-frame prediction encoded, and based on the encoding result of the reference P frame image, combined with binocular parallax, the target P frame image is encoded. Compared with performing inter-frame prediction encoding on both P frame images, the encoding efficiency is further improved and the size of the encoded data is reduced, which effectively improves the transmission speed, thereby ensuring the real-time rendering and display requirements of the VR device end in the streaming scene.

[0022] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 A schematic diagram of a VR streaming scenario provided in an embodiment of the present application;

[0025] Figure 2 This is a binocular image synthesis effect diagram provided in the embodiment of the present application;

[0026] Figure 3 Effect diagrams of left-eye images and right-eye images provided in the embodiments of the present application;

[0027] Figure 4 A schematic diagram of an encoding method for I frames, P frames, and B frames provided in an embodiment of the present application;

[0028] Figure 5 The overall architecture diagram of the image encoding method based on dual-channel video streams provided in the embodiment of the present application;

[0029] Figure 6 A flowchart of an image encoding method based on dual-channel video streams provided in an embodiment of the present application;

[0030] Fig. 7A A schematic diagram of a primary image block division method provided in an embodiment of the present application;

[0031] Figure 7B A schematic diagram of a secondary image block division method provided in an embodiment of the present application;

[0032] Figure 8 A motion vector calculation flow chart provided for an embodiment of the present application;

[0033] Fig. 9 A schematic diagram of motion estimation provided in an embodiment of the present application;

[0034] Fig.10 A motion vector estimation effect diagram provided by an embodiment of the present application;

[0035] Fig.11 A structural diagram of a game terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0037] VR streaming refers to projecting the game on the game terminal to the VR device for display in a WIFI, mobile network environment, or wired environment. Through VR streaming, the VR device interacts with the SteamVR application of the game terminal, so that large games can be run on the game terminal, and the resources of the game terminal can be fully utilized to experience 3D games, avoiding the storage of large game entities on the VR device, making up for the problem of insufficient HMD performance in VR devices, and solving the problem that games cannot run across platforms due to differences in system platforms.

[0038] During the streaming process, the left-eye image and the right-eye image can be independently encoded into a dual-channel video stream using two encoders and sent to the VR device. In order to increase the transmission speed, the dual-channel video stream is usually compressed during the encoding process to reduce the data volume. The higher the compression rate, the smaller the amount of data after encoding and the faster the transmission speed.

[0039] Predictive coding is the most basic coding method in video compression. Common predictive coding is divided into inter-frame prediction and intra-frame prediction. Among them, inter-frame predictive coding can eliminate the temporal information redundancy in the video, and intra-frame predictive coding can eliminate the spatial information redundancy in the video.

[0040] There are three types of video frames in a video stream: I-frames, P-frames, and B-frames. Figure 4 As shown, I frames are compressed using intra-frame prediction coding; P frames can freely choose between intra-frame prediction coding and inter-frame prediction coding according to actual needs, which is more flexible than I frame coding. When inter-frame prediction coding is performed on P frames, a matching block of the current block in the current image can be found from the forward image; B frames are compressed using inter-frame prediction coding. Unlike P frames, B frames will search for a matching block of the current block in the current image in the forward image and the backward image, and finally obtain a unique matching block through weighting.

[0041] However, it takes a long time to compress the I frames, P frames and B frames in the dual-channel video streams separately, and the reduction in data volume requires sacrificing clarity.

[0042] In view of this, an embodiment of the present application provides an image encoding method based on a dual-channel video stream. When encoding the dual-channel video stream, the method only performs intra-frame prediction encoding on the I frame, performs inter-frame prediction encoding on the P frame, and no longer encodes the B frame, thereby improving the video encoding efficiency, ensuring the real-time requirements of the VR stream, and effectively reducing the amount of data after encoding. Among them, when performing inter-frame prediction encoding on the P frame in the dual-channel video stream, the high similarity between the left eye image and the right eye image is utilized, and the binocular images are no longer inter-frame predicted. Instead, only one eye image is inter-frame predicted, and the other eye image is encoded with the encoded image as a reference, thereby further improving the video encoding efficiency.

[0043] See also Figure 5, is an overall architecture diagram of the image encoding method based on dual-channel video streams provided by an embodiment of the present application. The reference video stream and the target video stream are divided into two channels for independent encoding and transmission, and only the I-frame images and P-frame images in the two channels of video are encoded, which effectively reduces the amount of encoded data and improves the video encoding efficiency. Among them, the reference video stream can be any one of the left-eye video stream and the right-eye video stream. When the reference video stream is the left-eye video stream, the target video stream is the right-eye video stream. Similarly, when the reference video stream is the right-eye video stream, the target video stream is the left-eye video stream. For the I-frame images in the reference video stream and the target video stream, the traditional intra-frame prediction coding is performed, which will not be described in detail. When the P-frame images in the reference video stream and the target video stream are inter-frame prediction coding, different from the traditional inter-frame prediction coding, this method first calculates the reference motion vector of the reference P-frame image, and uses the predetermined binocular disparity between the left eye and the right eye to calculate the target motion vector of the target P frame, and then uses the calculated reference motion vector to perform motion compensation on the reference P-frame image to obtain the reference P-frame prediction image, and takes the reference P-frame prediction image as a benchmark, combined with the calculated target Motion vector, obtain the target P frame prediction image, because only the reference P frame image is inter-frame prediction coded, there is no need to perform inter-frame prediction coding on the P frame image of the target video stream, thereby further improving the video coding efficiency; finally, the coding result of the first residual between the reference P frame image and the reference P frame prediction image is combined with the intra-frame prediction coding result of the reference I frame image, and transmitted to the VR device through one video stream, and the coding result of the second residual between the target P frame image and the target P frame prediction image is combined with the intra-frame prediction coding result of the target I frame image, and transmitted to the VR device through another video stream. Among them, the data redundancy can be further reduced by calculating the parameters, thereby further reducing the amount of data transmitted by encoding.

[0044] based on Figure 5 The overall architecture shown in FIG. 1 is as follows: the image encoding process based on dual-channel video streams provided in the embodiment of the present application is as follows Figure 6 As shown in the figure, the process is executed by the game terminal that streams and communicates with the VR device, and mainly includes the following steps:

[0045] S601: Obtain a left-eye video stream and a right-eye video stream of the SteamVR application, and select one as a reference video stream and the other as a target video stream.

[0046] In one example, based on the real-time requirements of VR streaming for screen rendering, the game terminal encodes the left-eye video stream and the right-eye video stream that need to be displayed on the VR device in the SteamVR application. Since the encoding process of the two video streams is different, the two video streams are divided into a reference video stream and a target video stream. Among them, when the reference video stream is the left-eye video stream, the target video stream is the right-eye video stream. Similarly, when the reference video stream is the right-eye video stream, the target video stream is the left-eye video stream.

[0047] The reference video stream includes a reference I frame image, a reference P frame image and a reference B frame image, and the target video stream includes a target I frame image, a target P frame image and a target B frame image, wherein intra-frame prediction encoding is performed for each reference I frame image and each target I frame image, respectively, and inter-frame prediction encoding is performed for each reference P frame image and each target P frame image using the method of an embodiment of the present application, while encoding and transmission are no longer performed for each reference B frame image and each target B frame image, thereby reducing the amount of data transmitted from the game terminal to the VR device.

[0048] Among them, the intra-frame prediction coding can adopt the existing method, which will not be described in detail. The inter-frame prediction coding process is described in detail below.

[0049] S602: For each reference P frame image in the reference video stream, calculate a motion vector of the reference P frame image by dividing the image into blocks.

[0050] A macroblock is the basic processing unit of the coding standard. Its size is usually 16*16 pixels. A 16*16 pixel macroblock can be further divided into smaller sub-blocks.

[0051] like Fig. 7A As shown in the figure, it is a macroblock of 16*16 pixels, which contains part of the content of three eagles, but most of the content is the blue sky background. As for the effective content of the three eagles, in order to better process the partial image of the three eagles, the 16*16 macroblock can be divided into multiple sub-blocks, such as 8*16, 16*8, 8*8, 4*8, 8*4, 4*4, etc. Figure 7B shown.

[0052] During the encoding process, for each reference P frame image, a series of image blocks are divided, and each image block is used as an inter-frame prediction unit (Inter Prediction Unit). Each inter-frame prediction unit can be a square image block or a rectangular image block, and the size can be divided according to actual needs.

[0053] After being divided into image blocks, for each image block, a motion vector (MV) is estimated by searching for the most similar reference block in the forward image (ie, the previous reference P frame image).

[0054] S603: For each target P frame image in the target video stream, determine a target motion vector of the target P frame image according to a reference motion vector of a corresponding reference P frame image in combination with binocular parallax.

[0055] In one example, in order to improve video coding efficiency, considering the high similarity between the left-eye image and the right-eye image in the VR video, only the reference P frame image in the reference video stream is inter-frame predictive coding is performed, and the target P frame image is based on the corresponding reference P frame image. By using binocular parallax, the offset of each target image block in the target P frame image relative to the reference image block most similar to the corresponding current image block is calculated to obtain the target motion vector of the target P frame image.

[0056] In specific implementation, the motion estimation process for the reference motion vector of the reference P frame image and the target reference vector of the target P frame image is as follows: Figure 8 As shown, it mainly includes the following steps:

[0057] S602-3_1: For each current image block divided in the reference P frame image, calculate the difference between all pixels in the current image block and each reference image block in the previous reference P frame image, and obtain the similarity between the current image block and each reference image block.

[0058] For example, Fig. 9 As shown, assuming that the resolution of the reference P frame image is N*M pixels, the current image block B p，q The size of the current image block is P*Q pixels. p，q The pixel position of the upper left corner is (p, q), then the current image block B p，q The formula is expressed as:

[0059] B(p, q)={p, p+1,…,p+P-1}*{q, q+1,…,q+Q-1} Formula 1

[0060] Among them, the P and Q values ​​can be 4, 8, 6, 32, etc.

[0061] In the motion estimation algorithm based on block matching, it is necessary to calculate the similarity between the current image block B(p, q) in the current reference P frame image k and each reference image block B(pi, qj) in the previous reference P frame image h. The lower the similarity, the more matched the current image block is with the reference image block.

[0062] Each similarity can be represented by the difference between all pixels in the current image block and a reference image block. Therefore, each similarity can represent the offset when a reference image block is moved to the position of the current image block, that is, the motion vector.

[0063] Optionally, the difference may be a minimum mean square error (MSE) or a minimum mean absolute error (MAD), and the embodiment of the present application does not impose any restrictive requirements thereto.

[0064] S602-3_2: Use the difference of all pixels corresponding to the minimum similarity as the sub-motion vector of the current image block.

[0065] By sorting the calculated similarities, the minimum similarity can be obtained. A reference image block corresponding to the minimum similarity best matches the current image block. Therefore, the difference between all pixels of the two can be used as the sub-motion vector of the current image block.

[0066] An image block contains multiple pixels, and the unique pixels in the image block share a motion vector. Therefore, the current image block B p，q The sub-motion vector formula is expressed as:

[0067] f k (B p，q )=[f(p, q, k), f(p+1, q, k),…, f(p+P-1, q, k),…, f(p+P-1, q+Q-1, k)]

[0068] Wherein, k is the index of the current reference P frame image.

[0069] S602-3_3: Subtract the binocular disparity from the sub-motion vector of the current image block to obtain a sub-motion vector of a target image block in the corresponding target P frame image.

[0070] Since the binocular parallax between the left and right eyes is constant, the binocular parallax between the left eye image and the right eye image is also constant, and the resolutions of the left eye image and the right eye image are consistent. In this way, after obtaining the sub-motion vector of the current image block in the reference P frame image, when calculating the sub-motion vector of the target image block in the target P frame image, the current image block in the reference P frame image can be used as a reference, and the binocular parallax between the current image block and a target image block corresponding to the current image block in the target P frame image can be subtracted from the sub-motion vector of the current image block, so as to obtain the sub-motion vector of the target image block in the target P frame image.

[0071] S602-3_4: Obtain a reference motion vector of a reference P frame image according to the sub-motion vectors of each current image block, and obtain a target motion vector of a corresponding target P frame image according to the sub-motion vectors of each target image block.

[0072] In the block matching-based motion estimation algorithm, after splicing the sub-motion vectors of each current image block in the reference P frame image, the reference motion vector of the entire reference P frame image can be obtained. Similarly, after splicing the sub-motion vectors of each target image block in the target P frame image, the reference motion vector of the entire target P frame image can be obtained.

[0073] Among them, in the reference P frame image and the target P frame image, for the moving target area in the image, the denser the divided image blocks are, the denser the calculated motion vectors are. Fig.10 As shown, the density of motion vectors in the three eagles area in the image is higher than the density of motion vectors in the sky background area.

[0074] In the embodiment of the present application, when inter-frame prediction coding is performed on a P frame in a video stream, a sub-motion vector of each target image block can be obtained based on the binocular disparity between a target image block in a target P frame image and a corresponding current image block in a reference P frame image. ， This reduces the motion estimation process for the target P frame image. After the sub-motion vectors of each image block are encoded and transmitted to the VR end, the VR end searches for the corresponding image block in the reference P frame image through the sub-motion vector of the target image block. After processing, the reconstructed block of the target image block can be obtained, and the decoding of the target P frame image is completed. By using the sub-motion vector instead of the pixel value of the image block to be transmitted, the amount of data that needs to be encoded and transmitted is effectively reduced.

[0075] S604: Perform motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image to obtain a reference P frame prediction image.

[0076] In one example, after obtaining the reference motion vector of the reference P frame image, the motion state of the moving target in the reference P frame image can be estimated, thereby performing motion compensation on the reference P frame image. Specifically, for each current image block of the reference P frame image, all pixels in the current image block are displaced according to the sub-motion vector of the current image block to obtain the reference P frame prediction image.

[0077] S605: Obtain a target P frame prediction image according to a target motion vector of the target P frame image and a corresponding reference P frame prediction image.

[0078] Different from the motion compensation process of the reference P frame image, since the sub-motion vector of each target image block in the target P frame image is determined based on the reference P frame image, the reference P frame image should also be used as a reference when performing motion compensation on the target P frame image. In specific implementation, for each target image block in the target P frame image, according to the sub-motion vector of the target image block, the similar block in the reference P frame prediction image is displaced to obtain the target P frame prediction image, wherein the similar block is the image block after the current image block is displaced.

[0079] When performing motion compensation on the reference P frame image and the target P frame image in the embodiment of the present application, the motion compensation of the target P frame image is performed based on the motion compensation result of the reference P frame image. Compared with performing traditional motion compensation on both binocular images, the prediction error can be reduced, thereby reducing the residual after encoding.

[0080] S606: Calculate a first residual between the reference P frame image and the reference P frame prediction image.

[0081] S607: Calculate a second residual between the target P frame image and the target P frame prediction image.

[0082] In one example, the first residual is obtained by calculating the difference between the pixel values ​​in the reference P frame image and the reference P frame prediction image. Similarly, the second residual is obtained by calculating the difference between the pixel values ​​in the target P frame image and the target P frame prediction image, so that data redundancy can be reduced by encoding the residual, further reducing the amount of data transmitted by encoding.

[0083] S608: Perform inter-frame coding on the binocular image sent to the VR device according to the first residual and the second residual.

[0084] In one example, each reference P frame prediction image is entropy encoded according to the first residual to obtain a first eye video stream, and each target P frame prediction image is entropy encoded according to the second residual to obtain a second target video stream, and then the first target video stream and the second target video stream are sent to a VR device independently in two ways for displaying binocular images.

[0085] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the entropy coding method, including but not limited to Huffman coding and context-adaptive binary arithmetic coding (CABAC).

[0086] In the image encoding method based on dual-channel video streams provided in the embodiment of the present application, under the premise of ensuring clarity, in order to improve the transmission speed of independent encoding of left-eye images and right-eye images, on the one hand, only I frames and P frames in the two video streams are encoded, and B frames are not encoded, which effectively reduces the amount of data transmitted by encoding and improves the encoding efficiency. On the other hand, when inter-frame prediction encoding is performed on the P frames in the two video streams, considering the high similarity between the left-eye image and the right-eye image, only the reference P frame image is inter-frame prediction encoded, and based on the encoding result of the reference P frame image, the target P frame image is encoded. Compared with inter-frame prediction encoding of both P frame images, while improving the encoding efficiency, the size of the encoded data is further reduced, and the transmission speed is effectively improved, thereby ensuring the real-time rendering and display requirements of the VR device end in the streaming scene.

[0087] Based on the same technical concept, an embodiment of the present application provides a game terminal that can perform streaming communication with a VR device, and can implement the steps of the above-mentioned image encoding method based on dual-channel video streams, and can achieve the same technical effect.

[0088] See also Fig.11 , the game terminal includes a processor 1101, a memory 1102, a display 1103 and a communication interface 1104, wherein the communication interface 1104, the display 1103, the memory 1102 and the processor 1101 are connected via a bus 1105;

[0089] The communication interface 1104 is used for streaming communication with the VR device;

[0090] The display 1103 is used to display the game screen controlled by the VR device;

[0091] The memory 1102 stores a computer program, and the processor 1101 performs the following operations according to the computer program:

[0092] Get one left-eye video stream and one right-eye video stream of the SteamVR application, and select one as the reference video stream and the other as the target video stream;

[0093] For each reference P frame image in the reference video stream, calculate a motion vector of the reference P frame image by dividing the image blocks, perform motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image, obtain a reference P frame prediction image, and calculate a first residual between the reference P frame image and the reference P frame prediction image;

[0094] For each target P frame image in the target video stream, determine a target motion vector of the target P frame image according to a reference motion vector of a corresponding reference P frame image in combination with binocular disparity, obtain a target P frame prediction image according to the target motion vector of the target P frame image and a corresponding reference P frame prediction image, and calculate a second residual between the target P frame image and the target P frame prediction image;

[0095] Inter-frame encoding is performed on the binocular image sent to the VR device according to the first residual and the second residual.

[0096] Optionally, after acquiring the left-eye video stream and the right-eye video stream, the processor 1101 further executes:

[0097] Performing intra-frame prediction coding on each reference I frame image in the reference video stream, and not coding each reference B frame image in the reference video stream; and

[0098] Intra-frame prediction coding is performed on each target I-frame image in the target video stream, and coding is not performed on each target B-frame image in the target video stream.

[0099] Optionally, the processor 1101 calculates the motion vector of the reference P frame image by dividing the image blocks, and determines the target motion vector of the target P frame image according to the reference motion vector of the corresponding reference P frame image in combination with binocular parallax, and the specific operation is:

[0100] For each current image block divided in the reference P frame image, calculate the difference between all pixels in the current image block and each reference image block of the previous reference P frame image, obtain the similarity between the current image block and each reference image block, and use the difference of all pixels corresponding to the minimum similarity as the sub-motion vector of the current image block;

[0101] Subtracting the binocular disparity from the sub-motion vector of the current image block to obtain a sub-motion vector of a target image block in the corresponding target P frame image;

[0102] According to the sub-motion vectors of each current image block, the reference motion vector of the reference P frame image is obtained, and according to the sub-motion vectors of each target image block, the target motion vector of the corresponding target P frame image is obtained.

[0103] Optionally, the processor 1101 performs motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image to obtain a reference P frame prediction image, and obtains a target P frame prediction image according to the target motion vector of the target P frame image and the corresponding reference P frame prediction image, and the specific operations are:

[0104] For each current image block of the reference P frame image, all pixels in the current image block are displaced according to the sub-motion vector of the current image block to obtain the reference P frame prediction image;

[0105] For each target image block in the target P frame image, a similar block in the reference P frame prediction image is displaced according to a sub-motion vector of the target image block to obtain the target P frame prediction image, wherein the similar block is an image block after the current image block is displaced.

[0106] Optionally, the processor 1101 performs inter-frame coding on the binocular image sent to the VR device according to the first residual and the second residual, and the specific operation is:

[0107] According to the first residual, entropy encoding is performed on each reference P frame prediction image to obtain a first video stream; and

[0108] According to the second residual, entropy encoding is performed on each target P frame prediction image to obtain a second target video stream;

[0109] The first target video stream and the second target video stream are sent to the VR device independently in two paths for displaying binocular images.

[0110] in, Fig.11 The memory 1102 in the embodiment may be a volatile memory, such as a random-access memory (RAM); the memory 1102 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1102 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be a combination of the above memories; the processor 1101 may include one or more central processing units (CPU) or a digital processing unit, etc.

[0111] It should be noted that Fig.11 This is just an example, and provides the necessary hardware for the game terminal to execute the steps of the image encoding method based on dual-channel video streams provided in the embodiment of the present application. If not shown, the game terminal may also include the hardware of conventional game devices such as speakers, microphones, function keys, power supplies, cameras, etc.

[0112] An embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, an image encoding method based on dual-channel video streams in the aforementioned embodiment can be completed.

[0113] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute an image encoding method based on a dual-channel video stream in the aforementioned embodiment.

[0114] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0115] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image encoding method based on dual-channel video streams, characterized in that: Applied to a game terminal, the game terminal VR device performs streaming communication, including: Get one left-eye video stream and one right-eye video stream of the SteamVR application, and select one as the reference video stream and the other as the target video stream; For each reference P frame image in the reference video stream, calculate a motion vector of the reference P frame image by dividing the image blocks, perform motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image to obtain a reference P frame prediction image, and calculate a first residual between the reference P frame image and the reference P frame prediction image; For each target P frame image in the target video stream, determine a target motion vector of the target P frame image according to a reference motion vector of a corresponding reference P frame image in combination with binocular disparity, obtain a target P frame prediction image according to the target motion vector of the target P frame image and a corresponding reference P frame prediction image, and calculate a second residual between the target P frame image and the target P frame prediction image; Inter-frame encoding is performed on the binocular image sent to the VR device according to the first residual and the second residual.

2. The method according to claim 1, characterized in that After acquiring the left-eye video stream and the right-eye video stream, the method further includes: Performing intra-frame prediction coding on each reference I frame image in the reference video stream, and not coding each reference B frame image in the reference video stream; and Intra-frame prediction coding is performed on each target I-frame image in the target video stream, and coding is not performed on each target B-frame image in the target video stream.

3. The method according to claim 1, characterized in that Calculating the motion vector of the reference P frame image by dividing the image blocks, and determining the target motion vector of the target P frame image according to the reference motion vector of the corresponding reference P frame image in combination with binocular parallax, including: For each current image block divided in the reference P frame image, calculate the difference between all pixels in the current image block and each reference image block of the previous reference P frame image, obtain the similarity between the current image block and each reference image block, and use the difference of all pixels corresponding to the minimum similarity as the sub-motion vector of the current image block; Subtracting the binocular disparity from the sub-motion vector of the current image block to obtain a sub-motion vector of a target image block in the corresponding target P frame image; According to the sub-motion vectors of each current image block, the reference motion vector of the reference P frame image is obtained, and according to the sub-motion vectors of each target image block, the target motion vector of the corresponding target P frame image is obtained.

4. The method according to claim 3, characterized in that The step of performing motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image to obtain a reference P frame prediction image, and obtaining a target P frame prediction image according to the target motion vector of the target P frame image and the corresponding reference P frame prediction image, comprises: For each current image block of the reference P frame image, all pixels in the current image block are displaced according to the sub-motion vector of the current image block to obtain the reference P frame prediction image; For each target image block in the target P frame image, a similar block in the reference P frame prediction image is displaced according to a sub-motion vector of the target image block to obtain the target P frame prediction image, wherein the similar block is an image block after the current image block is displaced.

5. The method according to claim 1, characterized in that The inter-frame encoding of the binocular image sent to the VR device according to the first residual and the second residual includes: According to the first residual, entropy encoding is performed on each reference P frame prediction image to obtain a first video stream; and According to the second residual, entropy encoding is performed on each target P frame prediction image to obtain a second target video stream; The first target video stream and the second target video stream are sent to the VR device independently in two paths for displaying binocular images.

6. A game terminal, characterized in that: It includes a processor, a memory, a display and a communication interface, wherein the communication interface, the display, the memory and the processor are connected via a bus; The communication interface is used for streaming communication with the VR device; The display is used to display the game screen after being controlled by the VR device; The memory stores a computer program, and the processor performs the following operations according to the computer program: Get one left-eye video stream and one right-eye video stream of the SteamVR application, and select one as the reference video stream and the other as the target video stream; For each reference P frame image in the reference video stream, calculate a motion vector of the reference P frame image by dividing the image blocks, perform motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image to obtain a reference P frame prediction image, and calculate a first residual between the reference P frame image and the reference P frame prediction image; For each target P frame image in the target video stream, determine a target motion vector of the target P frame image according to a reference motion vector of a corresponding reference P frame image in combination with binocular disparity, obtain a target P frame prediction image according to the target motion vector of the target P frame image and a corresponding reference P frame prediction image, and calculate a second residual between the target P frame image and the target P frame prediction image; Inter-frame encoding is performed on the binocular image sent to the VR device according to the first residual and the second residual.

7. The game terminal according to claim 6, characterized in that: After acquiring the left-eye video stream and the right-eye video stream, the processor further executes: Performing intra-frame prediction coding on each reference I frame image in the reference video stream, and not coding each reference B frame image in the reference video stream; and Intra-frame prediction coding is performed on each target I-frame image in the target video stream, and coding is not performed on each target B-frame image in the target video stream.

8. The game terminal according to claim 6, characterized in that: The processor calculates the motion vector of the reference P frame image by dividing the image blocks, and determines the target motion vector of the target P frame image according to the reference motion vector of the corresponding reference P frame image in combination with binocular parallax, and the specific operation is as follows: For each current image block divided in the reference P frame image, calculate the difference between all pixels in the current image block and each reference image block of the previous reference P frame image, obtain the similarity between the current image block and each reference image block, and use the difference of all pixels corresponding to the minimum similarity as the sub-motion vector of the current image block; Subtracting the binocular disparity from the sub-motion vector of the current image block to obtain a sub-motion vector of a target image block in the corresponding target P frame image; According to the sub-motion vectors of each current image block, the reference motion vector of the reference P frame image is obtained, and according to the sub-motion vectors of each target image block, the target motion vector of the corresponding target P frame image is obtained.

9. The game terminal according to claim 8, characterized in that: The processor performs motion compensation on the reference P frame image according to the reference motion vector of the reference P frame image to obtain a reference P frame prediction image, and obtains a target P frame prediction image according to the target motion vector of the target P frame image and the corresponding reference P frame prediction image, and the specific operation is: For each current image block of the reference P frame image, all pixels in the current image block are displaced according to the sub-motion vector of the current image block to obtain the reference P frame prediction image; For each target image block in the target P frame image, a similar block in the reference P frame prediction image is displaced according to a sub-motion vector of the target image block to obtain the target P frame prediction image, wherein the similar block is an image block after the current image block is displaced.

10. The game terminal according to claim 6, characterized in that: The processor performs inter-frame coding on the binocular image sent to the VR device according to the first residual and the second residual, and the specific operation is: According to the first residual, entropy encoding is performed on each reference P frame prediction image to obtain a first video stream; and According to the second residual, entropy encoding is performed on each target P frame prediction image to obtain a second target video stream; The first target video stream and the second target video stream are sent to the VR device independently in two paths for displaying binocular images.

Images