Binocular video compression method, device and readable medium based on deep learning

Abstract

The present invention discloses a binocular video compression method, device and readable medium based on deep learning, which relates to the field of video processing, including: constructing and training a binocular video compression model to obtain a trained binocular video compression model, respectively obtaining the original frame of the left viewpoint at the current moment and the original frame of the previous moment and the original frame of the right viewpoint at the current moment and the original frame of the previous moment in the binocular video to be compressed and inputting them into the trained binocular video compression model, reconstructing the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint in an iterative manner, and obtaining a compressed binocular video. The present invention solves the problems of low redundancy utilization and low compression performance between viewpoints of the existing binocular video.

Description

Technical Field

[0001] The present invention relates to the field of video processing, and in particular to a binocular video compression method, device and readable medium based on deep learning. Background Art

[0002] Video compression technology is a key technology in digital video processing. It reduces the size of video files by reducing the redundancy of video data while maintaining the quality of the video as much as possible. Video compression technology is widely used in many fields such as video storage, transmission and playback.

[0003] Recently, methods based on Convolutional Neural Network (CNN) have emerged in the field of video compression due to their powerful feature representation, end-to-end trainable paradigm and excellent performance. A lot of work has focused on designing more efficient networks with better compression performance to promote CNN-based compression models to further improve the performance and effect of video compression. However, most of the current work is done on monocular videos, and is not widely used in binocular videos and multi-view stereo videos. Redundancy between viewpoints can also be considered to further improve compression performance, and there is more information in binocular videos that can be discovered and used for compression.

[0004] Therefore, how to further utilize the redundancy between viewpoints and the redundancy between the same viewpoints to achieve high-quality video compression has become an urgent problem to be solved. Summary of the invention

[0005] The purpose of this application is to propose a binocular video compression method, device and readable medium based on deep learning to address the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a binocular video compression method based on deep learning, comprising the following steps:

[0007] Constructing and training a binocular video compression model to obtain a trained binocular video compression model, wherein the binocular video compression model includes a right view compression unit and a left view compression unit, wherein the right view compression unit includes a first feature extraction module, a first motion branch, a first disparity branch, a first feature fusion module, a first context combination branch, and a first reconstruction module, and the left view compression unit includes a second feature extraction module, a second motion branch, a second disparity branch, a second feature fusion module, a second context combination branch, and a second reconstruction module;

[0008] The original frame of the left viewpoint at the current moment and the original frame of the previous moment in the binocular video to be compressed and the original frame of the right viewpoint at the current moment and the original frame of the previous moment are respectively obtained and input into the trained binocular video compression model. In the first iteration process, the original frame of the right viewpoint at the current moment is input into the first feature extraction module, and the first shallow feature at the current moment and the second shallow feature at the previous moment are respectively extracted. The first shallow feature at the current moment and the second shallow feature at the previous moment are passed through the first motion branch to obtain the first motion prediction feature at the current moment. The original frame of the left viewpoint at the current moment and the original frame of the previous moment are respectively extracted into the first feature extraction module. The initial frame is input into the second feature extraction module, and the third shallow feature of the current moment and the fourth shallow feature of the previous moment are extracted respectively. The third shallow feature of the current moment and the fourth shallow feature of the previous moment are passed through the second motion branch to obtain the second motion prediction feature of the current moment. The first shallow feature of the current moment and the second motion prediction feature of the current moment are passed through the first disparity branch to obtain the first disparity prediction feature of the current moment. The first disparity prediction feature of the current moment and the first motion prediction feature of the current moment are input into the first feature fusion module to obtain the first fusion feature of the current moment. The first fusion feature of the current moment and the first shallow feature of the current moment are input into the first context combination branch to obtain the first deep global feature of the current moment. The first deep global feature of the current moment is input into the first reconstruction module to obtain the first reconstructed frame of the current moment. The first deep global feature of the current moment and the third shallow feature of the current moment are passed through the second disparity branch to obtain the second disparity prediction feature of the current moment. The second disparity prediction feature of the current moment and the second motion prediction feature of the current moment are input into the second feature fusion module to obtain the second fusion feature of the current moment. The second fusion feature of the current moment and the third shallow feature of the current moment are input into the second context Combine the branches to obtain the second deep global feature of the current moment, input the second deep global feature of the current moment into the second reconstruction module, and obtain the second reconstructed frame of the current moment. In the next iteration, take the current moment as the previous moment, replace the second shallow feature of the previous moment of the right viewpoint with the first deep global feature of the previous moment and input it into the first motion branch, replace the fourth shallow feature of the previous moment of the left viewpoint with the second deep global feature of the previous moment and input it into the second motion branch. Iterate and repeat the above process to obtain the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, and reconstruct the compressed binocular video.

[0009] Preferably, the first motion branch includes a first motion estimation module, a first motion compression module and a first motion compensation module, and the first disparity branch includes a first disparity estimation module, a first disparity compression module and a first disparity compensation module; the second motion branch includes a second motion estimation module, a second motion compression module and a second motion compensation module, and the second disparity branch includes a second disparity estimation module, a second disparity compression module and a second disparity compensation module; the first context combining branch includes a first context extraction module, a first context compression module and a first context compensation module, and the second context combining branch includes a second context extraction module, a second context compression module and a second context compensation module.

[0010] Preferably, the calculation process of the first motion estimation module is as follows:

[0011] ;

[0012] in, represents the first motion feature at time t, Represents a splicing operation, represents a convolution operation with a convolution kernel size of 3×3. represents the residual block, represents the first shallow feature at time t, Represents the second shallow feature at time t-1;

[0013] The calculation process of the second motion estimation module is as follows:

[0014] ;

[0015] in, represents the second motion feature at time t, represents the third shallow feature at time t, represents the fourth shallow feature at time t-1;

[0016] The calculation process of the first motion compression module is as follows:

[0017] ;

[0018] in, represents the first motion compression reference feature at time t, Represents the function corresponding to the deep feature extraction module, Indicates the function corresponding to the quantization module, Represents the function corresponding to the reconstruction module;

[0019] The calculation process of the second motion compression module is as follows:

[0020] ;

[0021] in, represents the second motion compression reference feature at time t, represents the second disparity compression reference feature at time t-1, deformconv represents the function corresponding to the deformable convolutional layer, Represents the function corresponding to the feature compression module;

[0022] The calculation process of the first motion compensation module is as follows:

[0023] ;

[0024] in, represents the first motion prediction feature at time t;

[0025] The calculation process of the second motion compensation module is as follows:

[0026] ;

[0027] in, Represents the second motion prediction feature at time t;

[0028] The calculation process of the first disparity estimation module is as follows:

[0029] ;

[0030] in, represents the first disparity feature at time t;

[0031] The calculation process of the second disparity estimation module is as follows:

[0032] ;

[0033] in, represents the second disparity feature at time t, represents the first deep global feature at time t;

[0034] The calculation formula of the first parallax compression module is as follows:

[0035] ;

[0036] in, represents the first disparity compression reference feature at time t, Represents the first disparity compression reference feature at time t-1;

[0037] The calculation formula of the second parallax compression module is as follows:

[0038] ;

[0039] in, Representing the second parallax compression reference feature at time t;

[0040] The calculation process of the first parallax compensation module is as follows:

[0041] ;

[0042] in, represents the first disparity prediction feature at time t;

[0043] The calculation process of the second parallax compensation module is as follows:

[0044] ;

[0045] in, Represents the second disparity prediction feature at time t.

[0046] Preferably, the calculation process of the first feature extraction module is as follows:

[0047] ;

[0048] in, Represents the original frame of the right view at time t;

[0049] The calculation process of the second feature extraction module is as follows:

[0050] ;

[0051] in, Represents the original frame of the left view at time t;

[0052] The first feature extraction module and the second feature extraction module adopt a feature extraction structure, and the function corresponding to the feature extraction structure is as follows:

[0053] ;

[0054] in, represents the input features of the feature extraction structure, represents the output features of the feature extraction structure;

[0055] ;

[0056] in, represents the ReLU activation function, represents the input features of the residual block, represents the output features of the residual block, Indicates that the convolution kernel size is The function corresponding to the convolutional layer;

[0057] The calculation process of the first feature fusion module is as follows:

[0058] ;

[0059] in, represents the first fusion feature at time t;

[0060] The calculation process of the first context estimation module is as follows:

[0061] ;

[0062] in, represents the first context information at time t;

[0063] The calculation process of the first context compression module is as follows:

[0064] ;

[0065] in, represents the first compression feature at time t;

[0066] The calculation process of the first context compensation module is as follows:

[0067] ;

[0068] in, represents the first deep global feature at time t;

[0069] The calculation process of the first reconstruction module is as follows:

[0070] ;

[0071] in, represents the first reconstructed frame, The convolution kernel is Deconvolution function of

[0072] The calculation process of the second feature fusion module is as follows:

[0073] ;

[0074] in, represents the second fusion feature at time t;

[0075] The calculation process of the second context estimation module is as follows:

[0076] ;

[0077] in, Represents the second context information at time t;

[0078] The calculation process of the second context compression module is as follows:

[0079] ;

[0080] in, represents the second compression feature at time t;

[0081] The calculation process of the second context compensation module is as follows:

[0082] ;

[0083] in, Represents the second deep global feature at time t;

[0084] The calculation process of the second reconstruction module is as follows:

[0085] ;

[0086] in, represents the second reconstructed frame.

[0087] Preferably, the training process of the binocular video compression model includes the following steps:

[0088] The first motion compression length at time t is calculated according to the first motion feature at time t, the second motion compression reference feature at time t, and the first disparity compression reference feature at time t-1, as shown in the following formula:

[0089] ;

[0090] in, is the super prior encoding, is the first motion compression length at time t;

[0091] The second motion compression length at time t is calculated according to the second motion feature at time t, the first motion compression reference feature at time t, and the second parallax compression reference feature at time t-1, as shown in the following formula:

[0092] ;

[0093] in, is the second motion compression length at time t;

[0094] The first disparity compression length at time t is calculated according to the first motion feature at time t, the first disparity feature at time t, the first motion compression reference feature at time t, the second motion compression reference feature at time t and the first disparity compression reference feature at time t-1, as shown in the following formula:

[0095] ;

[0096] in, represents the first parallax compression length;

[0097] The second disparity compression length at time t is calculated according to the second motion feature at time t, the second disparity feature at time t, the first motion compression reference feature at time t, the second motion compression reference feature at time t and the second disparity compression reference feature at time t-1, as shown in the following formula:

[0098] ;

[0099] in, represents the second parallax compression length;

[0100] The first context compression length at time t is calculated according to the first context information at time t, as shown in the following formula:

[0101] ;

[0102] in, is the first context compression length at time t;

[0103] The second context compression length at time t is calculated according to the second context information at time t, as shown in the following formula:

[0104] ;

[0105] in, is the second context compression length at time t;

[0106] The loss function is constructed using the following formula:

[0107] ;

[0108] in, represents the loss function, represents the weight, represents the mean square error loss function;

[0109] The binocular video compression model is trained based on the loss function to obtain a trained binocular video compression model.

[0110] Preferably, the deep feature extraction module includes a convolution layer with a convolution kernel size of 3×3 and two feature extraction structures connected in sequence, as shown in the following formula:

[0111] ;

[0112] in, represents the input features of the deep feature extraction module, Represents the output features of the deep feature extraction module;

[0113] The calculation process of the function corresponding to the reconstruction module is as follows:

[0114] ;

[0115] in, represents the input features of the reconstruction module, represents the output features of the reconstruction module, The convolution kernel is The deconvolution function, Represents the function corresponding to the feature reconstruction module;

[0116] The calculation process of the feature reconstruction module is as follows:

[0117] ;

[0118] in, represents the input features of the feature reconstruction module, Represents the output features of the feature reconstruction module;

[0119] The calculation process of the feature compression module is as follows:

[0120] ;

[0121] in, represents the input features of the feature compression module, Represents the output features of the feature compression module;

[0122] The calculation process of the quantization module is as follows:

[0123] ;

[0124] in, represents the input features of the quantization module, represents the output features of the quantization module, Indicates the range of uniform distribution.

[0125] In a second aspect, the present invention provides a binocular video compression device based on deep learning, comprising:

[0126] A model building module is configured to build and train a binocular video compression model to obtain a trained binocular video compression model, wherein the binocular video compression model includes a right view compression unit and a left view compression unit, wherein the right view compression unit includes a first feature extraction module, a first motion branch, a first disparity branch, a first feature fusion module, a first context combination branch, and a first reconstruction module, and the left view compression unit includes a second feature extraction module, a second motion branch, a second disparity branch, a second feature fusion module, a second context combination branch, and a second reconstruction module;

[0127] The compression module is configured to respectively obtain the original frame of the left viewpoint at the current moment and the original frame at the previous moment in the binocular video to be compressed and the original frame of the right viewpoint at the current moment and the original frame at the previous moment and input them into the trained binocular video compression model. In the first iteration process, the original frame of the right viewpoint at the current moment is input into the first feature extraction module to extract the first shallow feature at the current moment and the second shallow feature at the previous moment respectively. The first shallow feature at the current moment and the second shallow feature at the previous moment are passed through the first motion branch to obtain the first motion prediction feature at the current moment. The original frame of the left viewpoint at the current moment and the original frame of the previous moment are input into the second feature extraction module, and the third shallow feature of the current moment and the fourth shallow feature of the previous moment are extracted respectively. The third shallow feature of the current moment and the fourth shallow feature of the previous moment are passed through the second motion branch to obtain the second motion prediction feature of the current moment. The first shallow feature of the current moment and the second motion prediction feature of the current moment are passed through the first disparity branch to obtain the first disparity prediction feature of the current moment. The first disparity prediction feature of the current moment and the first motion prediction feature of the current moment are input into the first feature fusion module to obtain the first fusion feature of the current moment. The first fusion feature of the current moment and the first shallow feature of the current moment are input into the first context combination branch to obtain the first deep global feature of the current moment. The first deep global feature of the current moment is input into the first reconstruction module to obtain the first reconstructed frame of the current moment; the first deep global feature of the current moment and the third shallow feature of the current moment are passed through the second disparity branch to obtain the second disparity prediction feature of the current moment. The second disparity prediction feature of the current moment and the second motion prediction feature of the current moment are input into the second feature fusion module to obtain the second fusion feature of the current moment. The second fusion feature of the current moment and the third shallow feature of the current moment are input into the second upper The following text combines the branches to obtain the second deep global feature of the current moment, and the second deep global feature of the current moment is input into the second reconstruction module to obtain the second reconstructed frame of the current moment. In the next iteration, the current moment is taken as the previous moment, and the second shallow feature of the previous moment of the right viewpoint is replaced by the first deep global feature of the previous moment and input into the first motion branch. The fourth shallow feature of the previous moment of the left viewpoint is replaced by the second deep global feature of the previous moment and input into the second motion branch. The above process is iteratively repeated to obtain the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, and the compressed binocular video is reconstructed.

[0128] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0129] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0130] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0131] Compared with the prior art, the present invention has the following beneficial effects:

[0132] (1) The binocular video compression method based on deep learning proposed in the present invention adopts a method of alternating the first motion branch of the right viewpoint and the second motion branch of the left viewpoint, and alternating the first disparity branch of the right viewpoint and the second disparity branch of the left viewpoint, and alternately iterates to obtain a reconstructed frame sequence of the left viewpoint and a reconstructed frame sequence of the right viewpoint, thereby realizing binocular video compression.

[0133] (2) The binocular video compression method based on deep learning proposed in the present invention designs a first parallax compression module and a second parallax compression module, which improves the compression performance of the video by compressing the redundancy between viewpoints.

[0134] (3) The binocular video compression method based on deep learning proposed in the present invention designs a first motion compression module and a second motion compression module, which improves the compression performance of the video by compressing the temporal redundancy between the same viewpoints. BRIEF DESCRIPTION OF THE DRAWINGS

[0135] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0136] Figure 1 A schematic diagram of a process flow of a binocular video compression method based on deep learning according to an embodiment of the present application;

[0137] Figure 2 A schematic diagram of a binocular video compression model of a binocular video compression method based on deep learning according to an embodiment of the present application;

[0138] Figure 3 A schematic diagram of a binocular video compression device based on deep learning according to an embodiment of the present application;

[0139] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0140] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0141] Figure 1 A binocular video compression method based on deep learning provided by an embodiment of the present application is shown, characterized in that it includes the following steps:

[0142] S1, construct and train a binocular video compression model to obtain a trained binocular video compression model, wherein the binocular video compression model includes a right viewpoint compression unit and a left viewpoint compression unit, wherein the right viewpoint compression unit includes a first feature extraction module, a first motion branch, a first disparity branch, a first feature fusion module, a first context combination branch and a first reconstruction module, and the left viewpoint compression unit includes a second feature extraction module, a second motion branch, a second disparity branch, a second feature fusion module, a second context combination branch and a second reconstruction module.

[0143] In a specific embodiment, the first motion branch includes a first motion estimation module, a first motion compression module and a first motion compensation module, and the first disparity branch includes a first disparity estimation module, a first disparity compression module and a first disparity compensation module; the second motion branch includes a second motion estimation module, a second motion compression module and a second motion compensation module, and the second disparity branch includes a second disparity estimation module, a second disparity compression module and a second disparity compensation module; the first context combining branch includes a first context extraction module, a first context compression module and a first context compensation module, and the second context combining branch includes a second context extraction module, a second context compression module and a second context compensation module.

[0144] In a specific embodiment, the training process of the binocular video compression model includes the following steps:

[0145] The first motion compression length at time t is calculated according to the first motion feature at time t, the second motion compression reference feature at time t, and the first disparity compression reference feature at time t-1, as shown in the following formula:

[0146] ;

[0147] in, is the super prior encoding, is the first motion compression length at time t;

[0148] The second motion compression length at time t is calculated according to the second motion feature at time t, the first motion compression reference feature at time t, and the second parallax compression reference feature at time t-1, as shown in the following formula:

[0149] ;

[0150] in, is the second motion compression length at time t;

[0151] The first disparity compression length at time t is calculated according to the first motion feature at time t, the first disparity feature at time t, the first motion compression reference feature at time t, the second motion compression reference feature at time t and the first disparity compression reference feature at time t-1, as shown in the following formula:

[0152] ;

[0153] in, represents the first parallax compression length;

[0154] The second disparity compression length at time t is calculated according to the second motion feature at time t, the second disparity feature at time t, the first motion compression reference feature at time t, the second motion compression reference feature at time t and the second disparity compression reference feature at time t-1, as shown in the following formula:

[0155] ;

[0156] in, represents the second parallax compression length;

[0157] The first context compression length at time t is calculated according to the first context information at time t, as shown in the following formula:

[0158] ;

[0159] in, is the first context compression length at time t;

[0160] The second context compression length at time t is calculated according to the second context information at time t, as shown in the following formula:

[0161] ;

[0162] in, is the second context compression length at time t;

[0163] The loss function is constructed using the following formula:

[0164] ;

[0165] in, represents the loss function, represents the weight, represents the mean square error loss function;

[0166] The binocular video compression model is trained based on the loss function to obtain a trained binocular video compression model.

[0167] Specifically, refer to Figure 2(Some connections are omitted), an embodiment of the present application proposes a binocular video compression model for compressing the original frames of the left viewpoint and the original frames of the right viewpoint. The first feature extraction module and the second feature extraction module have the same structure and both adopt a feature extraction structure, which adopts a downsampling convolution layer with a channel of 64 and 3 res blocks; taking the first motion branch and the first disparity branch as examples, the first motion estimation module and the first disparity estimation module both use the first shallow feature of the current moment for splicing, and then extract the first motion feature and the first disparity feature through the convolution layer with a channel of 64 and the res block; the first motion compression module again extracts the depth features of the first motion feature, the second motion compression reference feature and the second disparity compression reference feature, and completes the motion information compression through the deformable convolution layer to obtain the first motion compression reference feature; the first disparity compression module again extracts the depth features of the first disparity feature, the first motion compression reference feature, the second motion compression reference feature and the first disparity compression reference feature at the previous moment, splices the depth features of the first motion compression reference feature, the second motion compression reference feature and the first disparity compression reference feature at the previous moment, and passes them together through the deformable convolution layer Complete the parallax information compression and obtain the first parallax compression reference feature at the current moment; the first motion compensation module inputs the first motion compression reference feature after the convolution layer and the first motion feature at the previous moment into the deformable convolution layer to complete the compensation and obtain the first motion prediction feature. The first parallax compensation module inputs the first parallax compression reference feature after the convolution layer and the second motion feature into the deformable convolution layer to complete the compensation and obtain the first parallax prediction feature; the first feature fusion module fuses the compensation results from the first motion compensation module and the first parallax compensation module to obtain the first fusion feature; the first context estimation module extracts the context feature from the first fusion feature and the first shallow feature from the first feature fusion module; the first context compression module extracts the deep feature of the first fusion feature again and quantizes it; the first context compensation module compensates the first compressed feature after quantization and compression from the first context compression module to the first shallow feature, and obtains the first deep global feature through compensation; the first reconstruction module obtains the first reconstructed frame by upsampling the first deep global feature. The process of the second motion branch and the second parallax branch can refer to the process of the first motion branch and the first parallax branch above to obtain the second reconstructed frame. The original frame of the left view and the original frame of the right view are reconstructed in an iterative manner to obtain a reconstructed frame of the left view and a reconstructed frame of the right view.

[0168] In the training process of the binocular video compression model, the compressed feature bit values ​​are calculated by entropy coding, including the first motion compression length, the second motion compression length, the first disparity compression length, the second disparity compression length, the first context compression length and the second context compression length. The above six compressed feature bit values ​​are combined into the loss function of the binocular video compression model, and the parameters of the binocular video compression model are optimized to obtain a trained binocular video compression model.

[0169] S2, respectively obtain the original frame of the left viewpoint at the current moment and the original frame at the previous moment in the binocular video to be compressed, and the original frame of the right viewpoint at the current moment and the original frame at the previous moment, and input them into the trained binocular video compression model. In the first iteration, the original frame of the right viewpoint at the current moment is input into the first feature extraction module, and the first shallow feature at the current moment and the second shallow feature at the previous moment are respectively extracted. The first shallow feature at the current moment and the second shallow feature at the previous moment are passed through the first motion branch to obtain the first motion prediction feature at the current moment, and the original frame of the left viewpoint at the current moment and the second shallow feature at the previous moment are respectively extracted. The original frame is input into the second feature extraction module, and the third shallow feature at the current moment and the fourth shallow feature at the previous moment are extracted respectively. The third shallow feature at the current moment and the fourth shallow feature at the previous moment are passed through the second motion branch to obtain the second motion prediction feature at the current moment. The first shallow feature at the current moment and the second motion prediction feature at the current moment are passed through the first disparity branch to obtain the first disparity prediction feature at the current moment. The first disparity prediction feature at the current moment and the first motion prediction feature at the current moment are input into the first feature fusion module to obtain the first fusion feature at the current moment. The first fusion feature at the current moment and the first shallow feature at the current moment are input into the first context combination branch to obtain the first deep global feature at the current moment. The first deep global feature at the current moment is input into the first reconstruction module to obtain the first reconstructed frame at the current moment. The first deep global feature at the current moment and the third shallow feature at the current moment are passed through the second disparity branch to obtain the second disparity prediction feature at the current moment. The second disparity prediction feature at the current moment and the second motion prediction feature at the current moment are input into the second feature fusion module to obtain the second fusion feature at the current moment. The second fusion feature at the current moment and the third shallow feature at the current moment are input into the second up and down The text combines branches to obtain the second deep global feature of the current moment, and the second deep global feature of the current moment is input into the second reconstruction module to obtain the second reconstructed frame of the current moment. In the next iteration, the current moment is taken as the previous moment, and the second shallow feature of the previous moment of the right viewpoint is replaced by the first deep global feature of the previous moment and input into the first motion branch. The fourth shallow feature of the previous moment of the left viewpoint is replaced by the second deep global feature of the previous moment and input into the second motion branch. The above process is iteratively repeated to obtain the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, and the compressed binocular video is reconstructed.

[0170] In a specific embodiment, the calculation process of the first motion estimation module is as follows:

[0171] ;

[0172] in, represents the first motion feature at time t, Represents a splicing operation, represents a convolution operation with a convolution kernel size of 3×3. represents the residual block, represents the first shallow feature at time t, Represents the second shallow feature at time t-1;

[0173] The calculation process of the second motion estimation module is as follows:

[0174] ;

[0175] in, represents the second motion feature at time t, represents the third shallow feature at time t, represents the fourth shallow feature at time t-1;

[0176] The calculation process of the first motion compression module is as follows:

[0177] ;

[0178] in, represents the first motion compression reference feature at time t, Represents the function corresponding to the deep feature extraction module, Indicates the function corresponding to the quantization module, Represents the function corresponding to the reconstruction module;

[0179] The calculation process of the second motion compression module is as follows:

[0180] ;

[0181] in, represents the second motion compression reference feature at time t, represents the second disparity compression reference feature at time t-1, deformconv represents the function corresponding to the deformable convolutional layer, Represents the function corresponding to the feature compression module;

[0182] The calculation process of the first motion compensation module is as follows:

[0183] ;

[0184] in, represents the first motion prediction feature at time t;

[0185] The calculation process of the second motion compensation module is as follows:

[0186] ;

[0187] in, Represents the second motion prediction feature at time t;

[0188] The calculation process of the first disparity estimation module is as follows:

[0189] ;

[0190] in, represents the first disparity feature at time t;

[0191] The calculation process of the second disparity estimation module is as follows:

[0192] ;

[0193] in, represents the second disparity feature at time t, represents the first deep global feature at time t;

[0194] The calculation formula of the first parallax compression module is as follows:

[0195] ;

[0196] in, represents the first disparity compression reference feature at time t, Represents the first disparity compression reference feature at time t-1;

[0197] The calculation formula of the second parallax compression module is as follows:

[0198] ;

[0199] in, Representing the second parallax compression reference feature at time t;

[0200] The calculation process of the first parallax compensation module is as follows:

[0201] ;

[0202] in, represents the first disparity prediction feature at time t;

[0203] The calculation process of the second parallax compensation module is as follows:

[0204] ;

[0205] in, Represents the second disparity prediction feature at time t.

[0206] In a specific embodiment, the calculation process of the first feature extraction module is as follows:

[0207] ;

[0208] in, Represents the original frame of the right view at time t;

[0209] The calculation process of the second feature extraction module is as follows:

[0210] ;

[0211] in, Represents the original frame of the left view at time t;

[0212] The first feature extraction module and the second feature extraction module adopt a feature extraction structure, and the function corresponding to the feature extraction structure is as follows:

[0213] ;

[0214] in, represents the input features of the feature extraction structure, represents the output features of the feature extraction structure;

[0215] ;

[0216] in, represents the ReLU activation function, represents the input features of the residual block, represents the output features of the residual block, Indicates that the convolution kernel size is The function corresponding to the convolutional layer;

[0217] The calculation process of the first feature fusion module is as follows:

[0218] ;

[0219] in, represents the first fusion feature at time t;

[0220] The calculation process of the first context estimation module is as follows:

[0221] ;

[0222] in, represents the first context information at time t;

[0223] The calculation process of the first context compression module is as follows:

[0224] ;

[0225] in, represents the first compression feature at time t;

[0226] The calculation process of the first context compensation module is as follows:

[0227] ;

[0228] in, represents the first deep global feature at time t;

[0229] The calculation process of the first reconstruction module is as follows:

[0230] ;

[0231] in, represents the first reconstructed frame, The convolution kernel is Deconvolution function of

[0232] The calculation process of the second feature fusion module is as follows:

[0233] ;

[0234] in, represents the second fusion feature at time t;

[0235] The calculation process of the second context estimation module is as follows:

[0236] ;

[0237] in, Represents the second context information at time t;

[0238] The calculation process of the second context compression module is as follows:

[0239] ;

[0240] in, represents the second compression feature at time t;

[0241] The calculation process of the second context compensation module is as follows:

[0242] ;

[0243] in, Represents the second deep global feature at time t;

[0244] The calculation process of the second reconstruction module is as follows:

[0245] ;

[0246] in, represents the second reconstructed frame.

[0247] In a specific embodiment, the deep feature extraction module includes a convolution layer with a convolution kernel size of 3×3 and two feature extraction structures connected in sequence, as shown in the following formula:

[0248] ;

[0249] in, represents the input features of the deep feature extraction module, Represents the output features of the deep feature extraction module;

[0250] The calculation process of the function corresponding to the reconstruction module is as follows:

[0251] ;

[0252] in, represents the input features of the reconstruction module, represents the output features of the reconstruction module, The convolution kernel is The deconvolution function, Represents the function corresponding to the feature reconstruction module;

[0253] The calculation process of the feature reconstruction module is as follows:

[0254] ;

[0255] in, represents the input features of the feature reconstruction module, Represents the output features of the feature reconstruction module;

[0256] The calculation process of the feature compression module is as follows:

[0257] ;

[0258] in, represents the input features of the feature compression module, Represents the output features of the feature compression module;

[0259] The calculation process of the quantization module is as follows:

[0260] ;

[0261] in, represents the input features of the quantization module, represents the output features of the quantization module, Indicates the range of uniform distribution.

[0262] Specifically, the trained binocular video compression model is deployed, and the original images of the left viewpoint and the original images of the right viewpoint at different times are input into the trained binocular video compression model to obtain the first reconstructed frame and the second reconstructed frame. The first reconstructed frames and the second reconstructed frames at all times constitute the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, respectively. Therefore, the compressed binocular video can be constructed by the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint.

[0263] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a binocular video compression device based on deep learning, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0264] The embodiment of the present application provides a binocular video compression device based on deep learning, including:

[0265] A model building module 1 is configured to build and train a binocular video compression model to obtain a trained binocular video compression model, wherein the binocular video compression model includes a right view compression unit and a left view compression unit, wherein the right view compression unit includes a first feature extraction module, a first motion branch, a first disparity branch, a first feature fusion module, a first context combination branch, and a first reconstruction module, and the left view compression unit includes a second feature extraction module, a second motion branch, a second disparity branch, a second feature fusion module, a second context combination branch, and a second reconstruction module;

[0266] The compression module 2 is configured to respectively obtain the original frame of the left viewpoint at the current moment and the original frame at the previous moment in the binocular video to be compressed and the original frame of the right viewpoint at the current moment and the original frame at the previous moment and input them into the trained binocular video compression model. In the first iteration process, the original frame of the right viewpoint at the current moment is input into the first feature extraction module to extract the first shallow feature at the current moment and the second shallow feature at the previous moment respectively. The first shallow feature at the current moment and the second shallow feature at the previous moment are passed through the first motion branch to obtain the first motion prediction feature at the current moment. The original frame of the left viewpoint at the current moment and the original frame of the previous moment are input into the second feature extraction module, and the third shallow feature of the current moment and the fourth shallow feature of the previous moment are extracted respectively. The third shallow feature of the current moment and the fourth shallow feature of the previous moment are passed through the second motion branch to obtain the second motion prediction feature of the current moment. The first shallow feature of the current moment and the second motion prediction feature of the current moment are passed through the first disparity branch to obtain the first disparity prediction feature of the current moment. The first disparity prediction feature of the current moment and the first motion prediction feature of the current moment are input into the first feature fusion module to obtain the first fusion feature of the current moment. The first fusion feature of the current moment and the first shallow feature of the current moment are input into the first context combination branch to obtain the first deep global feature of the current moment. The first deep global feature of the current moment is input into the first reconstruction module to obtain the first reconstructed frame of the current moment; the first deep global feature of the current moment and the third shallow feature of the current moment are passed through the second disparity branch to obtain the second disparity prediction feature of the current moment. The second disparity prediction feature of the current moment and the second motion prediction feature of the current moment are input into the second feature fusion module to obtain the second fusion feature of the current moment. The second fusion feature of the current moment and the third shallow feature of the current moment are input into the second upper The following text combines the branches to obtain the second deep global feature of the current moment, and the second deep global feature of the current moment is input into the second reconstruction module to obtain the second reconstructed frame of the current moment. In the next iteration, the current moment is taken as the previous moment, and the second shallow feature of the previous moment of the right viewpoint is replaced by the first deep global feature of the previous moment and input into the first motion branch. The fourth shallow feature of the previous moment of the left viewpoint is replaced by the second deep global feature of the previous moment and input into the second motion branch. The above process is iteratively repeated to obtain the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, and the compressed binocular video is reconstructed.

[0267] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0268] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0269] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0270] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 401 executes the computer execution instructions, the above method is implemented.

[0271] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 401, the above method is implemented.

[0272] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0273] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0274] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0275] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 401 to perform some steps of the methods of various embodiments of the present application.

[0276] It should be understood that the processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 401 may be any conventional processor 401, etc. The steps of the method disclosed in the invention may be directly embodied in the hardware processor 401 for execution, or may be executed by a combination of hardware and software modules in the processor 401.

[0277] The memory 402 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0278] The bus 403 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 403 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 403 in the drawings of the present application is not limited to only one bus 403 or one type of bus 403.

[0279] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0280] An exemplary storage medium is coupled to the processor 401, so that the processor 401 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 401. The processor 401 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 401 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0281] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0282] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A binocular video compression method based on deep learning, characterized in that: The following steps are involved: Constructing and training a binocular video compression model to obtain a trained binocular video compression model, wherein the binocular video compression model includes a right view compression unit and a left view compression unit, wherein the right view compression unit includes a first feature extraction module, a first motion branch, a first disparity branch, a first feature fusion module, a first context combination branch, and a first reconstruction module, and the left view compression unit includes a second feature extraction module, a second motion branch, a second disparity branch, a second feature fusion module, a second context combination branch, and a second reconstruction module; The original frame of the left viewpoint at the current moment and the original frame at the previous moment in the binocular video to be compressed and the original frame of the right viewpoint at the current moment and the original frame at the previous moment are respectively obtained and input into the trained binocular video compression model. In the first iteration process, the original frame of the right viewpoint at the current moment is input into the first feature extraction module, and the first shallow feature at the current moment and the second shallow feature at the previous moment are respectively extracted. The first shallow feature at the current moment and the second shallow feature at the previous moment are passed through the first motion branch to obtain the first motion prediction feature at the current moment. The original frame of the left viewpoint at the current moment and the original frame at the previous moment are input into The second feature extraction module extracts the third shallow feature at the current moment and the fourth shallow feature at the previous moment respectively. The third shallow feature at the current moment and the fourth shallow feature at the previous moment are passed through the second motion branch to obtain the second motion prediction feature at the current moment. The first shallow feature at the current moment and the second motion prediction feature at the current moment are passed through the first disparity branch to obtain the first disparity prediction feature at the current moment. The first disparity prediction feature at the current moment and the first motion prediction feature at the current moment are input into the first feature fusion module to obtain the first fusion feature at the current moment. The first fusion feature at the current moment and the first shallow feature at the current moment are input into the first context combination branch to obtain the first deep global feature at the current moment. The first deep global feature at the current moment is input into the first reconstruction module to obtain the first reconstructed frame at the current moment. The first deep global feature at the current moment and the third shallow feature at the current moment are passed through the second disparity branch to obtain the second disparity prediction feature at the current moment. The second disparity prediction feature at the current moment and the second motion prediction feature at the current moment are input into the second feature fusion module to obtain the second fusion feature at the current moment. The second fusion feature at the current moment and the third shallow feature at the current moment are input into the first context combination branch to obtain the first deep global feature at the current moment. The second context is combined with branches to obtain the second deep global feature of the current moment, and the second deep global feature of the current moment is input into the second reconstruction module to obtain the second reconstructed frame of the current moment. In the next iteration, the current moment is taken as the previous moment, and the second shallow feature of the previous moment of the right viewpoint is replaced by the first deep global feature of the previous moment and input into the first motion branch, and the fourth shallow feature of the previous moment of the left viewpoint is replaced by the second deep global feature of the previous moment and input into the second motion branch. The above process is iteratively repeated to obtain the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, and the compressed binocular video is reconstructed.

2. The binocular video compression method based on deep learning according to claim 1, characterized in that: The first motion branch includes a first motion estimation module, a first motion compression module and a first motion compensation module, and the first disparity branch includes a first disparity estimation module, a first disparity compression module and a first disparity compensation module; the second motion branch includes a second motion estimation module, a second motion compression module and a second motion compensation module, and the second disparity branch includes a second disparity estimation module, a second disparity compression module and a second disparity compensation module; the first context combination branch includes a first context extraction module, a first context compression module and a first context compensation module, and the second context combination branch includes a second context extraction module, a second context compression module and a second context compensation module.

3. The binocular video compression method based on deep learning according to claim 2, characterized in that: The calculation process of the first motion estimation module is as follows: in, represents the first motion feature at time t, concat represents the concatenation operation, Conv 3×3 Represents a convolution operation with a convolution kernel size of 3×3, res 3×3 represents the residual block, F t r represents the first shallow feature at time t, Represents the second shallow feature at time t-1; The calculation process of the second motion estimation module is as follows: in, represents the second motion feature at time t, F t l represents the third shallow feature at time t, represents the fourth shallow feature at time t-1; The calculation process of the first motion compression module is as follows: in, represents the first motion compression reference feature at time t, f(·) represents the function corresponding to the deep feature extraction module, Q(·) represents the function corresponding to the quantization module, and recon(·) represents the function corresponding to the reconstruction module; The calculation process of the second motion compression module is as follows: in, represents the second motion compression reference feature at time t, represents the second disparity compression reference feature at time t-1, deformconv represents the function corresponding to the deformable convolution layer, and com(·) represents the function corresponding to the feature compression module; The calculation process of the first motion compensation module is as follows: in, represents the first motion prediction feature at time t; The calculation process of the second motion compensation module is as follows: in, Represents the second motion prediction feature at time t; The calculation process of the first disparity estimation module is as follows: in, represents the first disparity feature at time t; The calculation process of the second disparity estimation module is as follows: in, represents the second disparity feature at time t, represents the first deep global feature at time t; The calculation formula of the first parallax compression module is as follows: in, represents the first disparity compression reference feature at time t, Represents the first disparity compression reference feature at time t-1; The calculation formula of the second parallax compression module is as follows: in, Representing the second parallax compression reference feature at time t; The calculation process of the first parallax compensation module is as follows: in, represents the first disparity prediction feature at time t; The calculation process of the second parallax compensation module is as follows: in, Represents the second disparity prediction feature at time t.

4. The binocular video compression method based on deep learning according to claim 3, characterized in that: The calculation process of the first feature extraction module is as follows: in, Represents the original frame of the right view at time t; The calculation process of the second feature extraction module is as follows: in, Represents the original frame of the left view at time t; The first feature extraction module and the second feature extraction module adopt a feature extraction structure, and the function corresponding to the feature extraction structure is as follows: Extractor(X1)=Conv 5×5 (X1)+res 3×3 (res 3×3 (res 3×3 (Conv 5×5 (X1)))); Among them, X1 represents the input feature of the feature extraction structure, and Extractor(X1) represents the output feature of the feature extraction structure; in, represents the ReLU activation function, x1 represents the input feature of the residual block, and res k×k (x1) represents the output feature of the residual block, Conv k×k (·) represents the function corresponding to the convolution layer with a kernel size of k×k; The calculation process of the first feature fusion module is as follows: in, represents the first fusion feature at time t; The calculation process of the first context estimation module is as follows: in, represents the first context information at time t; The calculation process of the first context compression module is as follows: in, represents the first compression feature at time t; The calculation process of the first context compensation module is as follows: in, represents the first deep global feature at time t; The calculation process of the first reconstruction module is as follows: in, represents the first reconstructed frame, deconv 5×5 (·) represents a deconvolution function with a convolution kernel of 5×5; the calculation process of the second feature fusion module is as follows: in, represents the second fusion feature at time t; The calculation process of the second context estimation module is as follows: in, Represents the second context information at time t; The calculation process of the second context compression module is as follows: in, represents the second compression feature at time t; The calculation process of the second context compensation module is as follows: in, Represents the second deep global feature at time t; The calculation process of the second reconstruction module is as follows: in, represents the second reconstructed frame.

5. The binocular video compression method based on deep learning according to claim 4, characterized in that: The training process of the binocular video compression model includes the following steps: The first motion compression length at time t is calculated according to the first motion feature at time t, the second motion compression reference feature at time t, and the first disparity compression reference feature at time t-1, as shown in the following formula: Among them, Entropy is a super priori code, is the first motion compression length at time t; The second motion compression length at time t is calculated according to the second motion feature at time t, the first motion compression reference feature at time t, and the second parallax compression reference feature at time t-1, as shown in the following formula: in, is the second motion compression length at time t; The first disparity compression length at time t is calculated according to the first motion feature at time t, the first disparity feature at time t, the first motion compression reference feature at time t, the second motion compression reference feature at time t and the first disparity compression reference feature at time t-1, as shown in the following formula: in, represents the first parallax compression length; The second disparity compression length at time t is calculated according to the second motion feature at time t, the second disparity feature at time t, the first motion compression reference feature at time t, the second motion compression reference feature at time t and the second disparity compression reference feature at time t-1, as shown in the following formula: in, represents the second parallax compression length; The first context compression length at time t is calculated according to the first context information at time t, as shown in the following formula: in, is the first context compression length at time t; The second context compression length at time t is calculated according to the second context information at time t, as shown in the following formula: in, is the second context compression length at time t; The loss function is constructed using the following formula: Among them, loss represents the loss function, λ represents the weight, and MSE(·) represents the mean square error loss function; The binocular video compression model is trained based on the loss function to obtain a trained binocular video compression model.

6. The binocular video compression method based on deep learning according to claim 5, characterized in that: The deep feature extraction module includes a convolution layer with a convolution kernel size of 3×3 and two feature extraction structures connected in sequence, as shown in the following formula: f(X2)=Extractor(Extractor(Conv 3×3 (X2))); Where X2 represents the input features of the deep feature extraction module, and f(X2) represents the output features of the deep feature extraction module; The calculation process of the function corresponding to the reconstruction module is as follows: recon(X3)=reconstructor(reconstructor(deconv 3×3 (X3))); Among them, X3 represents the input features of the reconstruction module, recon(X3) represents the output features of the reconstruction module, and deconv 3×3 (·) represents the deconvolution function with a convolution kernel of 3×3, and reconstructor(·) represents the function corresponding to the feature reconstruction module; The calculation process of the feature reconstruction module is as follows: reconstractor(X4)=deconv 5×5 (X4+res 3×3 (res 3×3 (res 3×3 (X4)))); Among them, X4 represents the input features of the feature reconstruction module, and reconstractor(X4) represents the output features of the feature reconstruction module; The calculation process of the feature compression module is as follows: com(x2)=deconv 5×5 (deconv 3×3 deconv 3×3 (Q(Conv 5×5 (Conv 3×3 (Conv 3×3 (x2)))))); Among them, x2 represents the input feature of the feature compression module, and com(x2) represents the output feature of the feature compression module; The calculation process of the quantization module is as follows: Q(X5)=X5+U(-0.5,0.5); Among them, X5 represents the input feature of the quantization module, Q(X5) represents the output feature of the quantization module, and U(-0.5,0.5) represents the uniform distribution in the range of (-0.5,0.5).

7. A binocular video compression device based on deep learning, characterized in that: include: A model building module is configured to build and train a binocular video compression model to obtain a trained binocular video compression model, wherein the binocular video compression model includes a right view compression unit and a left view compression unit, wherein the right view compression unit includes a first feature extraction module, a first motion branch, a first disparity branch, a first feature fusion module, a first context combination branch, and a first reconstruction module, and the left view compression unit includes a second feature extraction module, a second motion branch, a second disparity branch, a second feature fusion module, a second context combination branch, and a second reconstruction module; The compression module is configured to respectively obtain the original frame of the left viewpoint at the current moment and the original frame at the previous moment in the binocular video to be compressed and the original frame of the right viewpoint at the current moment and the original frame at the previous moment and input them into the trained binocular video compression model. In the first iteration process, the original frame of the right viewpoint at the current moment is input into the first feature extraction module to extract the first shallow feature at the current moment and the second shallow feature at the previous moment, respectively. The first shallow feature at the current moment and the second shallow feature at the previous moment are passed through the first motion branch to obtain the first motion prediction feature at the current moment. The original frame of the left viewpoint at the current moment and the original frame of the previous moment are respectively extracted from the first shallow feature at the current moment and the second shallow feature at the previous moment. The original frame is input into the second feature extraction module, and the third shallow feature at the current moment and the fourth shallow feature at the previous moment are extracted respectively. The third shallow feature at the current moment and the fourth shallow feature at the previous moment are passed through the second motion branch to obtain the second motion prediction feature at the current moment. The first shallow feature at the current moment and the second motion prediction feature at the current moment are passed through the first disparity branch to obtain the first disparity prediction feature at the current moment. The first disparity prediction feature at the current moment and the first motion prediction feature at the current moment are input into the first feature fusion module to obtain the first fusion feature at the current moment. The first fusion feature at the current moment and the first shallow feature at the current moment are input into the first context combination branch to obtain the first deep global feature at the current moment. The first deep global feature at the current moment is input into the first reconstruction module to obtain the first reconstructed frame at the current moment. The first deep global feature at the current moment and the third shallow feature at the current moment are passed through the second disparity branch to obtain the second disparity prediction feature at the current moment. The second disparity prediction feature at the current moment and the second motion prediction feature at the current moment are input into the second feature fusion module to obtain the second fusion feature at the current moment. The second fusion feature at the current moment and the third shallow feature at the current moment are input into The second context is combined with the branch to obtain the second deep global feature of the current moment, and the second deep global feature of the current moment is input into the second reconstruction module to obtain the second reconstructed frame of the current moment. In the next iteration, the current moment is taken as the previous moment, and the second shallow feature of the previous moment of the right viewpoint is replaced by the first deep global feature of the previous moment and input into the first motion branch, and the fourth shallow feature of the previous moment of the left viewpoint is replaced by the second deep global feature of the previous moment and input into the second motion branch. The above process is iteratively repeated to obtain the reconstructed frame sequence of the left viewpoint and the reconstructed frame sequence of the right viewpoint, and the compressed binocular video is reconstructed.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images