Encoding method, decoding method, code stream, encoder, decoder, and storage medium

Abstract

Disclosed in the embodiments of the present application are an encoding method, a decoding method, a code stream, an encoder, a decoder, and a storage medium. The decoding method comprises: decoding a code stream, and determining a first representation feature of the current frame; determining predicted motion information of the current frame according to a preset reference frame, and performing correction processing on the predicted motion information, so as to determine decoded motion information; determining a decoding feature of the current frame according to the decoded motion information and the first representation feature of the current frame; and determining reconstruction information of the current frame according to the decoding feature of the current frame. In this way, since the encoding and decoding of motion information are canceled, not only can a training process of a network model be simplified, but the encoding and decoding efficiency can also be improved, thereby improving the video compression performance.

Description

Coding and decoding method, code stream, encoder, decoder and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and in particular to a coding and decoding method, a bit stream, an encoder, a decoder, and a storage medium. Background Art

[0002] In the neural video compression technology based on temporal context mining (TCM), a multi-scale method is proposed here to extract multi-scale temporal context information from the propagation features, which is used as a temporal conditional prior in the conditional encoder, conditional decoder and frame generator of the current frame.

[0003] However, TCM technology is not fully considered and requires the transmission of motion information, which results in encoders and decoders that need to transmit motion information, resulting in high hardware costs. In addition, the training process of existing network models is relatively complex, and the exploration cycle for optimal network parameters is long, which in turn reduces compression performance.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, a code stream, an encoder, a decoder, and a storage medium. By eliminating the coding and decoding of motion information, it can not only simplify the training process of the network model, but also improve the coding and decoding efficiency, thereby improving the video compression performance.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, applied to a decoder, the method comprising:

[0008] Decoding the code stream to determine a first representation feature of the current frame;

[0009] Determine the predicted motion information of the current frame according to the preset reference frame, and perform correction processing on the predicted motion information to determine the decoded motion information;

[0010] determining a decoding feature of the current frame based on the decoded motion information and a first representation feature of the current frame;

[0011] Reconstruction information of the current frame is determined according to the decoding characteristics of the current frame.

[0012] In a second aspect, an embodiment of the present application provides an encoding method, applied to an encoder, the method comprising:

[0013] Determine a preset reference frame corresponding to the current frame;

[0014] Determining encoding motion information of the current frame according to a preset reference frame, and determining a first representation feature of the current frame according to the encoding motion information;

[0015] The first representation feature of the current frame is encoded, and the obtained encoded bits are written into the bitstream.

[0016] In a third aspect, an embodiment of the present application provides a code stream, which is generated by bit encoding based on information to be encoded; wherein the information to be encoded includes at least: a first representation feature of the current frame.

[0017] In a fourth aspect, an embodiment of the present application provides an encoder, comprising a first determining unit and an encoding unit; wherein,

[0018] A first determining unit configured to determine a preset reference frame corresponding to the current frame;

[0019] The first determining unit is further configured to determine the coding motion information of the current frame according to the preset reference frame, and determine the first representation feature of the current frame according to the coding motion information;

[0020] The encoding unit is configured to encode the first representation feature of the current frame and write the obtained encoding bits into the bit stream.

[0021] In a fifth aspect, an embodiment of the present application provides an encoder, comprising a first memory and a first processor; wherein,

[0022] a first memory for storing a computer program capable of running on the first processor;

[0023] The first processor is configured to execute the method according to the second aspect when running a computer program.

[0024] In a sixth aspect, an embodiment of the present application provides a decoder, comprising a decoding unit, a correction unit, and a second determination unit; wherein,

[0025] a decoding unit configured to decode the code stream and determine a first representation feature of a current frame;

[0026] a correction unit configured to determine predicted motion information of a current frame based on a preset reference frame, and perform correction processing on the predicted motion information to determine decoded motion information;

[0027] The second determining unit is configured to determine the decoding feature of the current frame according to the decoded motion information and the first representation feature of the current frame; and further configured to determine the reconstruction information of the current frame according to the decoding feature of the current frame.

[0028] In a seventh aspect, an embodiment of the present application provides a decoder, comprising a second memory and a second processor; wherein,

[0029] a second memory for storing a computer program capable of running on the second processor;

[0030] The second processor is configured to execute the method according to the first aspect when running a computer program.

[0031] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed, implements the method described in the first aspect or the method described in the second aspect.

[0032] The embodiments of the present application provide a coding and decoding method, a bitstream, an encoder, a decoder, and a storage medium. At the encoding end, a preset reference frame corresponding to the current frame is first determined; then, based on the preset reference frame, the coded motion information of the current frame is determined, and based on the coded motion information, a first representation feature of the current frame is determined; then, the first representation feature of the current frame is encoded, and the resulting coded bits are written into the bitstream. At the decoding end, the bitstream is first decoded to determine the first representation feature of the current frame; then, based on the preset reference frame, the predicted motion information of the current frame is determined, and the predicted motion information is corrected to determine the decoded motion information; then, based on the decoded motion information and the first representation feature of the current frame, the decoded features of the current frame are determined; finally, based on the decoded features of the current frame, the reconstruction information of the current frame is determined. In this way, only the first representation feature of the current frame needs to be transmitted in the bitstream, and the transmission of motion information is eliminated, thereby saving the motion information encoder and motion information decoder in the video compression framework, so that a multi-stage training strategy is no longer required during the model training process. This not only simplifies the training process of the network model, but also has better rate-distortion performance. At the same time, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, it can also improve the encoding and decoding efficiency, thereby improving the video compression performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of the overall framework of a neural video compression technology using TCM technology;

[0034] Figure 2 is a schematic diagram of a network structure for learning multi-scale temporal context information;

[0035] FIG3 is a schematic diagram of a neural video compression framework using multi-scale temporal context information;

[0036] FIG4 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0037] FIG5 is a schematic diagram of the composition structure of a frame generator provided in an embodiment of the present application;

[0038] FIG6A is a first structural diagram of a first network module according to an embodiment of the present application;

[0039] FIG6B is a second schematic diagram of the composition structure of a first network module provided in an embodiment of the present application;

[0040] FIG7 is a schematic diagram comparing the training strategies of the embodiment of the present application and the related art;

[0041] FIG8 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0042] FIG9 is a schematic diagram of a network architecture of a video codec provided in an embodiment of the present application;

[0043] FIG10 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0044] FIG11 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

[0045] FIG12 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0046] FIG13 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application;

[0047] FIG14 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0050] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0051] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0052] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0053] Neural Video Compression (NVC);

[0054] Low Delay (LD);

[0055] Motion Vector (MV);

[0056] Multi-Scale Structural Similarity (MS-SSIM);

[0057] Bit Percent Pixel (BPP);

[0058] Temporal Context Mining (TCM) for neural video compression;

[0059] Decoded Picture Buffer (DPB);

[0060] Convolutional Neural Networks (CNN);

[0061] Generalized Divisive Normalization (GDN);

[0062] Peak Signal to Noise Ratio (PSNR);

[0063] Rate Distortion Optimization (RDO);

[0064] High Efficiency Video Coding (HEVC);

[0065] Deep Contextual Video Compression (DCVC)

[0066] Video Concourse Transfer (VCT)

[0067] As can be understood, for deep learning-based video compression, the embodiments of this application provide an efficient conditional coding framework that uses temporal context features as conditional inputs to help the codec encode the current frame, thereby fully tapping the potential of conditional coding. Strictly speaking, "condition" refers to anything that can help improve the compression performance of the current frame.

[0068] For many years, traditional video coding standards are mostly based on the residual coding framework. In residual coding, a predicted frame is first generated from the decoded frame, and then the residual between the current frame and the predicted frame is calculated. The residual will be written into the bitstream, and the decoder obtains the reconstructed residual by decoding the bitstream, and finally adds it to the predicted frame to obtain the decoded frame. Residual coding is a simple and efficient way, but its entropy is greater than or equal to the entropy of conditional coding, which is not the best way. Based on this, the embodiment of the present application makes full use of the advantages of deep learning to mine temporal context features to design conditional coding. A neural compression technology based on temporal context mining of conditional coding has been proposed. The technology is referred to as TCM technology and is applied to neural video compression based on the conditional coding framework. This is a multi-scale method that extracts multi-scale temporal context from propagation features, which acts as a temporal condition prior on the conditional encoder, conditional decoder and frame generator of the current frame to be compressed.

[0069] Figure 1 is a schematic diagram of the overall framework of a neural video compression technology using TCM technology. As shown in Figure 1, the framework can include a context encoder 101, a temporal context encoder 102, a context decoder 103, a motion estimation module 104, an MV encoder 105, an MV decoder 106, a temporal context mining module 107, a frame generator 108, and a decoded picture buffer 109. The black and white blocks represent binary code streams.

[0070] In Figure 1, for the current frame to be compressed, x t On the one hand, x t The context encoder 101 and the super-prior entropy model (Entropy Model) are used to perform entropy coding and write into the bitstream; on the other hand, x t Also the previous decoded frame stored in the decoded picture buffer 109 The data are input to the motion estimation module 104 for motion estimation, and the motion information v is output.t ; then v t The MV encoder 105 and the super-prior entropy model are used to entropy encode and write into the bitstream; the bitstream is then decoded by the MV decoder 106 to obtain the decoded motion information. Further, and the previous decoded frame in the decoded picture buffer 109 The propagation characteristics F t-1 Input together into the time context mining module 107, and then multi-scale time context information can be obtained Among them, x t When entropy coding is performed by the context encoder 101, it is also necessary to The entropy coding is performed by the time context encoder 102 and the time prior entropy model, and the coding result is written into the bitstream; then the decoding process is performed by the context decoder 103 to obtain the decoding feature. Finally, the decoded features and temporal context information Input them together into the frame generator 107 to obtain the propagation feature F of the current frame. t , and can also reconstruct the decoded frame (Decoded Frame), using Indicates; and the obtained F t and The decoded images are stored together in the decoded image buffer 109 for reference in subsequent frames.

[0071] Specifically, TCM technology is mainly implemented through the following processes:

[0072] Taking into account the previously decoded frame x t-1 A lot of information is lost because it only contains three channels, from x t-1 Learning the temporal context is not optimal. Therefore, the TCM technique propagates the features F t-1 First, unlike the previous method of directly learning temporal context from the decoded frame x t-1 Extract features, TCM technology is used to obtain x t-1 Before decoding and reconstructing the frame, the feature F is propagated t-1 Stored in the decoded image buffer. Then propagate the feature F t-1 Used to learn the current frame x t Among them, for the first P frame, since there is no propagation feature in the decoded picture buffer, special processing is required. The method adopted is to extract features from the reconstructed I frame for the temporal context learning of the first P frame.

[0073] Furthermore, learning a single-scale context may not be a good representation of spatiotemporal non-uniform motion and texture. For example, in the largest-scale context, some channels focus on texture information and some on color information; in the smallest-scale context, channels primarily focus on regions with large motion. Therefore, a hierarchical approach can be used to learn multi-scale temporal context, as shown in Figure 2.

[0074] FIG2 is a schematic diagram of a network structure for learning multi-scale temporal context information. As shown in FIG2 , the network structure may include: three feature extraction modules (a first feature extraction module 201, a second feature extraction module 202, and a third feature extraction module 203), three warp modules (a first warp module 204, a second warp module 206, and a third warp module 208), two bilinear filtering modules (a first bilinear filtering module 205 and a second bilinear filtering module 207), two upsampling modules (a first upsampling module 209 and a second upsampling module 210), and three context refinement modules (a first context refinement module 211, a second context refinement module 212, and a third context refinement module 213). Among them, each feature extraction module can be composed of a convolution layer and a residual block. The first feature extraction module 201 is composed of a convolution layer Conv(3,64,64,1) and a residual block Residual block(64,64). The second feature extraction module 202 is composed of a convolution layer Conv(3,64,64,2↓) and a residual block Residual block(64,64). The third feature extraction module 203 is composed of a convolution layer Conv(3,64,64,2↓) and a residual block Residual block(64,64). Here, "2↓" means downsampling with a step size of 2. The alignment module uses the related motion vectors of the same scale to align the propagation features F. For example, the first alignment module 204 uses right To perform alignment, the second alignment module 206 uses the MVs downsampled by the first bilinear filtering module 205. To perform alignment, the third alignment module 208 uses the MVs downsampled by the second bilinear filtering module 207. Perform alignment operations. In addition, each upsampling module can be composed of a subpixel layer and a residual block. The first upsampling module 209 is composed of a subpixel layer Subpixel (3, 128, 64, 2↑) and a residual block Residual block (64, 64). The second upsampling module 210 is composed of a subpixel layer Subpixel (3, 64, 64, 2↑) and a residual block Residual block (64, 64). Each context refinement module can be composed of a convolutional layer and a residual block. The first context refinement module 211 is composed of a convolutional layer Conv (3, 128, 64, 1) and a residual block Residual block (64, 64), which is used to learn the features of the zeroth scale. The second context refinement module 212 is composed of a convolutional layer Conv(3,128,64,1) and a residual block Residual block(64,64), which is used to learn the features of the first scale The third context refinement module 213 is composed of a convolutional layer Conv(3,64,64,1) and a residual block Residual block(64,64) for learning the features of the second scale Finally, the residuals of different scales are added to the propagation features of the corresponding scales to generate the final multi-scale temporal context information

[0075] Specifically, in Figure 2, a three-level feature extraction module consisting of a convolutional layer and a residual block is first used to generate multi-scale features from the propagation feature Ft-1 and At the same time, the bilinear filtering module is used to filter the decoded MV (v t ) is downsampled to generate multi-scale MV ( and Then, the propagation features F are aligned using the related motion vectors MV of the same scale. After the alignment operation, an upsampling module (consisting of a sub-pixel layer and a residual block) is used to upsample the smaller-scale MV and F, and then the upsampled features are concatenated with the corresponding larger-scale features in the channel dimension at the same scale (Contact, C). At each layer of the multi-scale structure, a context refinement module (consisting of a convolutional layer and a residual block) is used to learn the residual information of the features at each scale. Finally, the residuals of different scales are added to the propagation features of the corresponding scale to generate the final multi-scale temporal context information.

[0076] In this way, after obtaining the multi-scale temporal context information, the learned multi-scale temporal context information is refilled into the module of the compression scheme, as shown in Figure 3. Figure 3 is a schematic diagram of a neural video compression framework using multi-scale temporal context information. As shown in Figure 3, it can include a context encoder, a context decoder, a frame generator, and a temporal context encoder. For the current frame x t , the learned multi-scale temporal context information Refill the compression scheme and finally get the reconstructed decoded frame It is worth noting that with the refilled temporal context information, the compression ratio of the proposed scheme is greatly improved.

[0077] In the related art, the module ratio distribution of the spatial correlation information extraction part (conditional coding part), the temporal correlation information extraction part (motion estimation, motion information encoding and motion information compensation part) and the spatiotemporal information fusion frame generation part (frame generator) is not fully weighed. This will result in the use of too many network parameters in less important parts, resulting in module redundancy. At the same time, the related art also needs to transmit motion information, which will lead to the following problems: on the one hand, it is necessary to design a special encoder and decoder for transmitting motion information, and transmit the compressed motion information in the channel; on the other hand, in order to achieve a good rate-distortion performance, it is necessary to use a multi-stage training strategy. At different stages, different loss functions need to be used to train different network parameters, which makes the training process very complicated and the optimal training combination exploration cycle is long, thereby reducing the compression performance.

[0078] Based on this, an embodiment of the present application provides an encoding method, which first determines a preset reference frame corresponding to the current frame; then determines the encoding motion information of the current frame based on the preset reference frame, and determines the first representation feature of the current frame based on the encoding motion information; then encodes the first representation feature of the current frame, and writes the obtained encoding bits into the bitstream.

[0079] An embodiment of the present application also provides a decoding method, which first decodes the code stream to determine the first representation feature of the current frame; then determines the predicted motion information of the current frame based on a preset reference frame, and corrects the predicted motion information to determine the decoded motion information; then determines the decoding feature of the current frame based on the decoded motion information and the first representation feature of the current frame; finally, determines the reconstruction information of the current frame based on the decoding feature of the current frame.

[0080] In this way, only the first representation feature of the current frame needs to be transmitted in the bitstream, and the transmission of motion information is eliminated, thereby saving the motion information encoder and motion information decoder in the video compression framework, so that a multi-stage training strategy is no longer required during the model training process. This not only simplifies the training process of the network model, but also has better rate-distortion performance. At the same time, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, it can also improve the encoding and decoding efficiency, thereby improving the video compression performance.

[0081] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0082] In one embodiment of the present application, referring to FIG4 , a schematic flow chart of a decoding method provided by an embodiment of the present application is shown. As shown in FIG4 , the method may include:

[0083] S401: Decode the code stream and determine the first representation feature of the current frame.

[0084] It should be noted that the decoding method of the embodiment of the present application is applied to the decoder. In addition, the decoding method can refer to an optimization method for end-to-end video compression, specifically optimizing the motion information module; more specifically, a forward multi-reference frame neural video compression method that does not require the transmission of motion information. In other words, the embodiment of the present application eliminates the decoder portion of the motion information and can also expand the use of a single reference frame to multiple reference frames.

[0085] It should also be noted that in this embodiment of the present application, the first representation feature of the current frame can also be referred to as the latent representation feature of the current frame. The bitstream no longer transmits motion information; only the latent representation feature of the current frame needs to be transmitted. Thus, the decoder can obtain the latent representation feature of the current frame by decoding the bitstream. Furthermore, the latent representation feature of the current frame can be used to better conditionally decode the current frame.

[0086] S402: Determine predicted motion information of the current frame according to a preset reference frame, and perform correction processing on the predicted motion information to determine decoded motion information.

[0087] It should be noted that, in the embodiment of the present application, the preset reference frame is adjacent to the current frame, wherein the preset reference frame includes at least one reference frame, and the at least one reference frame is a decoded frame n times before the current frame, where n is a positive integer.

[0088] That is, in the embodiment of the present application, the preset reference frame can be one or more frames. If the current frame is Frame t, where t is a positive integer, then the preset reference frame can be Frame t-1, a decoded frame one moment before the current frame, or Frame t-2, a decoded frame two moments before the current frame, or even Frame tn, a decoded frame n moments before the current frame (e.g., an I frame, the first decoded frame P1, etc.), without specific limitation here.

[0089] In some embodiments, determining the predicted motion information of the current frame according to the preset reference frame may include:

[0090] determining a first reference frame;

[0091] Time information is estimated based on the first reference frame to determine predicted motion information of the current frame.

[0092] It should be noted that in this embodiment of the present application, if the preset reference frame is only one frame, such as the first reference frame, then temporal information can be estimated based on the first reference frame to determine the predicted motion information of the current frame. Exemplarily, motion information is estimated based on the first reference frame and the current frame to obtain the predicted motion information of the current frame.

[0093] In some embodiments, determining the predicted motion information of the current frame according to the preset reference frame may include:

[0094] determining a first reference frame and a second reference frame;

[0095] Estimating time information based on the first reference frame and the second reference frame to determine predicted motion information of the current frame;

[0096] The first reference frame is different from the second reference frame.

[0097] It should also be noted that in this embodiment of the present application, if the preset reference frame can be multiple frames, such as a first reference frame and a second reference frame, then temporal information is estimated based on the first reference frame and the second reference frame to determine the predicted motion information of the current frame. For example, motion information is estimated based on the first reference frame and the second reference frame to obtain the predicted motion information of the current frame.

[0098] In a specific embodiment, performing time information estimation based on the first reference frame and the second reference frame to determine the predicted motion information of the current frame may include: performing motion information estimation on the first reference frame and the second reference frame through an optical flow estimation module to obtain the predicted motion information of the current frame.

[0099] As you can understand, for the optical flow estimation module, optical flow estimation is used to determine the motion information of moving objects. Optical flow, as the name suggests, is the flow of light. For example, the human eye perceives a meteor streaking across the night sky. In computer vision, this defines the movement of objects in an image. This movement can be caused by camera movement or object movement. Specifically, it refers to the movement of pixels representing the same object from one frame of a video image to the next frame, which can be represented by a video image (MV).

[0100] In the embodiment of the present application, the optical flow calculation can adopt the most classic Lucas-Kanade algorithm, or can also adopt the optical flow estimation algorithm based on deep learning, such as the FlowNet / FlowNet2 algorithm, etc., which is not specifically limited here. In addition, it should be noted that when performing optical flow calculation, the first reference frame and the second reference frame as adjacent frames need to meet the following prerequisites: (1) the brightness between adjacent frames is constant; (2) the frame acquisition time of adjacent frames is continuous, or the movement of objects between adjacent frames is relatively "small"; (3) spatial consistency is maintained, that is, the pixels of the same sub-pixel have the same movement.

[0101] Simply put, in an embodiment of the present application, the current method of utilizing time information can be to obtain motion information between two reference frames by optical flow estimation. This is only one way of utilizing spatiotemporal information, but it is not the best. A better method of utilizing time information can also be used here instead of using motion information to obtain better rate-distortion performance.

[0102] It can also be understood that, in the process of video compression, in order to fully utilize the temporal correlation between video frames, the current frame can be decoded with reference to the frames that have been decoded. In the implementation of a single reference frame, for the first reference frame, it can be a decoded frame of the current frame at the previous moment, or it can be an I frame in the current video sequence, or it can be a decoded frame of the current frame at any moment. In other words, the first reference frame can be the decoded frame with the closest forward distance (if the current frame is the first P frame P1, then the first reference frame is the I frame; if the current frame is any P frame P other than the first P frame, then the first reference frame is the I frame; if the current frame is any P frame P other than the first P frame, then the first reference frame is the I frame). t , then the reference frame is P t-1 frame; however, this is not particularly limited.

[0103] It is also understandable that in a multiple-reference frame implementation, taking two reference frames (a first reference frame and a second reference frame) as an example, in one possible implementation, the first reference frame is the decoded frame at the previous moment of the current frame; the second reference frame is the decoded frame two moments before the current frame. In other words, if the current frame is t, then the first reference frame can be the decoded frame Frame t-1, and the second reference frame can be the decoded frame Frame t-2.

[0104] In another possible implementation, the first reference frame is a decoded frame immediately before the current frame, and the second reference frame is a decoded I-frame. In other words, the first reference frame and the second reference frame may also be the decoded frame t-1 and the I-frame, respectively. The I-frame is the first image in a video sequence.

[0105] In short, in the embodiment of the present application, considering the temporal correlation between different frames in the video sequence, the current frame can be decoded with reference to multiple decoded frames. In the multi-frame reference, the types of reference frames can be the same or different. For example, for the first reference frame and the second reference frame, a combination of I frame and P frame (for example, decoded P frame) can be used. t-1 frames and I frames), or a combination of pure P frames (e.g. decoded P t-1 Frame and P t-2 It should be noted that among these reference frame combinations, compared to P t-1 Frame and I frame combination, using P t-1 Frame and P t-2 The rate-distortion performance of frame combining is relatively good.

[0106] Furthermore, for the obtained predicted motion information, in order to better apply it to conditional decoding of the current frame, in some embodiments, the predicted motion information is corrected to determine the decoded motion information, which can include: correcting the predicted motion information through a preset correction module to obtain the decoded motion information.

[0107] It should be noted that in the embodiment of the present application, the preset correction module may be specifically referred to as an MV correction module, which is used to perform correction operations on the predicted motion information. The preset correction module includes an autoregressive module and a residual module. There may be at least one autoregressive module and at least one residual module. The specific number of autoregressive modules and residual modules is not limited herein.

[0108] In a specific implementation, if the autoregressive module is located before the residual module, performing correction processing on the predicted motion information by a preset correction module to obtain the decoded motion information may include:

[0109] Performing autoregressive processing on the motion information of each pixel in the predicted motion information and the motion information of adjacent pixels through an autoregressive module to determine a first intermediate feature;

[0110] The first intermediate features are extracted and fused through the residual module to obtain decoded motion information.

[0111] In another specific implementation, if the autoregressive module is located after the residual module, the predicted motion information is corrected by a preset correction module to obtain the decoded motion information, which may further include:

[0112] The predicted motion information is extracted and fused through the residual module to obtain the second intermediate feature;

[0113] The motion information of each pixel in the second intermediate feature and the motion information of adjacent pixels are autoregressively processed through the autoregressive module to obtain decoded motion information.

[0114] That is, in the embodiments of the present application, the autoregressive module can be placed first and the residual module can be placed second; or vice versa, the residual module can be placed first and the autoregressive module can be placed second; this is not specifically limited here. From a performance perspective, placing the autoregressive module first and the residual module second provides relatively better performance. Furthermore, the number of autoregressive and residual modules can be more than one, but from a cost-effectiveness perspective, having one autoregressive and one residual module provides the best cost-effectiveness.

[0115] It should also be noted that, in an embodiment of the present application, the preset correction module is located after the optical flow estimation module at the decoding end, but before the temporal context mining module. The role of the preset correction module is to correct the predicted motion information (MVdec_p) so as to obtain decoded motion information (MVdec) that is more suitable for conditional coding. Here, if the autoregressive module is in front and the residual module is in the back, then the autoregressive module can make full use of the motion information of each pixel in the predicted motion information (MVdec_p) and the motion information of the adjacent pixels to generate the motion information features of the current pixel, and then perform feature extraction and fusion through a residual module to generate decoded motion information (MVdec). Among them, the decoded motion information (MVdec) is more suitable for conditional decoding of the current frame, and can subsequently be used as a temporal condition prior to act on the conditional decoder of the current frame.

[0116] S403: Determine a decoding feature of the current frame according to the decoded motion information and the first representation feature of the current frame.

[0117] It should be noted that after obtaining the decoded motion information, the decoded features of the current frame can be further determined, thereby determining the reconstruction information of the current frame. In some embodiments, determining the decoded features of the current frame based on the decoded motion information and the first representation feature of the current frame may include:

[0118] Acquire a first representation feature of a preset reference frame;

[0119] The first representation feature of the current frame and the first representation feature of the preset reference frame are converted according to the decoded motion information to determine the decoded feature of the current frame.

[0120] It should also be noted that, in an embodiment of the present application, the first representation feature of the preset reference frame (i.e., the potential representation feature of the preset reference frame) can be obtained from the preset decoding buffer. The preset reference frame here generally refers to a forward reference frame. At the decoding end, since the forward reference frame has been decoded and the potential representation feature of the forward reference frame has been stored in the preset decoding buffer, the first representation feature of the preset reference frame can be directly obtained from the preset decoding buffer. In other words, after the current frame is decoded, the first representation feature of the current frame needs to be stored in the preset decoding buffer so that it can be used as a forward reference for subsequent video frames when the next frame is decoded.

[0121] Furthermore, in some embodiments, the method may also include: applying the decoded motion information as a temporal condition prior to a conditional decoder of the current frame, converting the first representation feature of the current frame and the first representation feature of the preset reference frame to obtain a decoded feature of the current frame.

[0122] That is to say, in an embodiment of the present application, after the predicted motion information generates decoded motion information through a preset correction module, the decoded motion information acts as a temporal condition prior on the conditional decoder of the current frame, and converts the first representation feature of the current frame and the first representation feature of the preset reference frame into the decoded feature of the current frame.

[0123] S404: Determine reconstruction information of the current frame according to the decoding characteristics of the current frame.

[0124] It should be noted that, in the embodiment of the present application, the reconstruction information of the current frame may include the decoded reconstruction features of the current frame and the decoded reconstructed frame of the current frame.

[0125] In some embodiments, determining reconstruction information of the current frame based on decoding features of the current frame may include: performing decoding and reconstruction processing on the decoding features of the current frame by a frame generator to obtain decoding and reconstruction features of the current frame and a decoding and reconstructed frame of the current frame.

[0126] It should also be noted that in the embodiment of the present application, the frame generator may include a first network module and a first convolution module. The number of the first network module may be at least one, and the number of the first convolution module may be at least one. The specific number of the first network module and the first convolution module is not limited here.

[0127] For example, Figure 5 shows a schematic diagram of the composition structure of a frame generator provided in an embodiment of the present application. As shown in Figure 5, the frame generator may include a first network module 501 and a first convolution module 502, and the output end of the first network module 501 is connected to the input end of the first convolution module 502.

[0128] In a specific embodiment, as shown in FIG5 , the decoding feature of the current frame is decoded and reconstructed by the frame generator to obtain the reconstructed information of the current frame, which may include:

[0129] The first network module 501 extracts and fuses the decoding features of the current frame to obtain the decoding reconstruction features of the current frame;

[0130] The first convolution module 502 performs a convolution operation on the decoded and reconstructed features of the current frame to obtain a decoded and reconstructed frame of the current frame.

[0131] It should also be noted that in the embodiment of the present application, the first network module 501 includes at least one U-Net module, and the first convolution module 502 includes at least one convolution layer. At least one U-Net module does not use dense connections; alternatively, at least one U-Net module uses dense connections.

[0132] That is, for the internal structure of the first network module, the number of U-Net modules and whether dense connections are used are not limited. Exemplarily, the first network module may include three U-Net modules, but any natural number of U-Net modules may also be used.

[0133] In one possible implementation, FIG6A is a schematic diagram of the composition structure of a first network module using dense connections provided in an embodiment of the present application. As shown in FIG6A , the first network module 501 includes three U-Net modules and uses dense connections to fuse features from different layers, resulting in better performance.

[0134] In another possible implementation, FIG6B is a schematic diagram of the composition structure of a first network module provided in an embodiment of the present application that does not use dense connections. As shown in FIG6B , the first network module 501 includes three U-Net modules and does not use dense connections, which makes the implementation process simpler and faster.

[0135] That is to say, in an embodiment of the present application, for the first network module 501, the at least one U-Net module can extract features from different stages, and densely connect to fuse features from different stages. Specifically, the decoded and reconstructed features after the U-Net module are obtained after sufficient extraction and fusion of the decoded features. These decoded and reconstructed features will be stored in a preset decoding buffer. Finally, these decoded and reconstructed features will be passed through the first convolution module to generate a decoded and reconstructed frame, which will also be stored in the preset decoding buffer. Among them, the decoded and reconstructed features and decoded and reconstructed frames in the preset decoding buffer will be used for compression of subsequent video frames.

[0136] It is also understandable that since the relevant technology requires special training of the motion information encoder and the motion information decoder, a multi-stage training strategy needs to be adopted; and different loss functions need to be used in different stages. However, in the embodiment of the present application, since there is no need to transmit motion information, the motion information encoder and the motion information decoder are eliminated, so there is no need to adopt a multi-stage training strategy. At this time, only one loss function needs to be used to directly perform end-to-end training on all modules of the entire neural video compression network. Therefore, in some embodiments, the method may also include:

[0137] Determine the preset network model;

[0138] The preset network model is trained through a first loss function, and the trained model is determined as a target network model; wherein the target network model includes at least: a conditional encoder, a conditional decoder, an optical flow estimation module, a preset correction module and a frame generator.

[0139] In the embodiment of the present application, the preset network model includes all modules of the entire neural video compression network. After the preset network model is trained using the first loss function, the trained model can be used as the target network model, where the target network model also includes all modules of the entire neural video compression network, specifically the conditional encoder, conditional decoder, optical flow estimation module, preset correction module, and frame generator.

[0140] In the embodiment of the present application, the first loss function can be a rate-distortion loss function, and the number of first loss functions is one. That is, only one rate-distortion loss function is needed to directly perform end-to-end training on all modules of the entire neural video compression network, thereby greatly simplifying the training method of the neural video compression model.

[0141] For example, FIG7 shows a schematic diagram comparing the training strategies of the embodiment of the present application and the related technology. Among them, related technology 1 and related technology 2 both embody a phased training strategy. Related technology 1 is specifically CNN-based Hybird NVC (DCVC, TCM, DMC), where training is performed in five stages, and the loss function (Loss) corresponding to each stage is shown; related technology 2 is specifically Transformer-based NVC (VCT), where training is performed in three stages, and the loss function (Loss) corresponding to each stage is also shown. However, the technical solution of the embodiment of the present application (specifically Simple NVC) is trained end-to-end, and the corresponding first loss function can be expressed as: D rec +λ(r frame +r mv ).

[0142] As shown in Figure 7, the technical solution of the embodiment of the present application only needs to use a single loss function to train all modules during the entire training process, which greatly simplifies the model training process and makes the training process more concise. At the same time, due to the use of a preset correction module and a better frame generator, the embodiment of the present application also has rate-distortion performance comparable to the best methods in the related art.

[0143] For example, the technical solution proposed in the embodiments of this application, based on TCM technology, redesigned the network framework and training method, and tested on HEVC Class B, C, D, E, UVG, and MCL-JCV test datasets in low-delay P frame (LDP) mode. The specific test results are shown below.

[0144] Table 1

[0145] Table 2

[0146] Here, Table 1 shows the test results corresponding to intra period = 12, and Table 2 shows the test results corresponding to intra period = 32. Among them, the test results are presented in bd-rate save format, and the comparison anchor is TCM. Here, negative numbers represent performance gains, and positive numbers represent performance losses. From all frames (All Frame), it can be seen that the test results have a small performance loss in HEVC_B and HEVC_RGB, but there are performance gains in HEVC_C to HEVC_E.

[0147] In summary, in the embodiments of the present application, on the one hand, for the first time, the motion information encoder and motion information decoder in the neural video compression network framework are removed under the CNN-based framework. On the other hand, it is proposed to use the motion information between the t-1 frame and the t-2 frame to adaptively replace the motion information between the t frame and the t-1 frame, and use the MV correction module composed of the autoregressive module and the residual module to correct the motion information. On the other hand, a module ratio distribution method of the spatial correlation information extraction part (conditional coding part), the temporal correlation information extraction part (motion estimation, motion information encoding and motion information compensation part) and the spatiotemporal information fusion frame generation part (frame generator) is proposed. Specifically, it is recommended to strengthen the spatiotemporal information fusion frame generation part and weaken the temporal correlation information extraction part. On the other hand, since the motion information encoder and motion information decoder are removed in the neural video compression network framework, there is no need to transmit motion information in the channel. This can optimize the training method of the neural video compression network. Previously, due to the existence of the motion information encoder and motion information decoder, in order to achieve better rate-distortion performance during network training, a multi-stage training strategy was required. That is, different loss functions were used to train different network modules at different stages. In the embodiment of the present application, since the motion information encoder and motion information decoder are eliminated, a single loss function can be used to train all modules end-to-end, greatly simplifying the training process.

[0148] In this way, under the premise of almost no impact on performance, the use of the technical solution of the embodiment of the present application can, on the one hand, greatly simplify the training method of the neural video compression model. Since the present technical solution does not require special training for the motion information encoder and the motion information decoder, there is no need to adopt a multi-stage training strategy. It is only necessary to use a rate-distortion loss function to directly perform end-to-end training on all modules of the entire neural video compression network. It can be seen that the present technical solution can greatly simplify the training process of the neural video compression network while maintaining similar rate-distortion performance as the original technology. On the other hand, there is no need to transmit motion information. Since the present technical solution cancels the motion information encoder and the motion information decoder in the entire neural video compression framework, it is not necessary to transmit motion information in the channel. At this time, only the potential representation features of the current frame need to be transmitted in the channel.

[0149] Furthermore, in an embodiment of the present application, in order to achieve a better way of utilizing time information, the current way of utilizing time information is to use the optical flow method to obtain the motion information between the previous and next frames. This is only one way of utilizing spatiotemporal information and is not optimal. A better way of utilizing time information can be used instead of using motion information to obtain better rate-distortion performance.

[0150] Furthermore, in an embodiment of the present application, when the preset reference frames include two reference frames, the two reference frames may use the previous frame Frame t-1 and the I frame. However, from a test performance perspective, the performance of using the previous frame Frame t-1 and the immediately preceding frame Frame t-2 is superior to that of using the previous frame Frame t-1 and the I frame, as shown in Table 3.

[0151] Table 3

[0152] Furthermore, in an embodiment of the present application, the three U-Net modules of the frame generator may not use dense connections, but the test performance is not as good as that of using dense connections, as shown in Table 4.

[0153] Table 4

[0154] Furthermore, in the embodiment of the present application, the scheme of referring to a single frame also has good performance, but is not as good as the rate-distortion performance of referring to a double frame; however, the model parameter amount of the reference single frame is less, and the encoding and decoding speed is also faster, as shown in Table 5.

[0155] Table 5

[0156] This embodiment provides a decoding method that first decodes the bitstream to determine the first representation feature of the current frame; then, based on a preset reference frame, determines the predicted motion information of the current frame, and corrects the predicted motion information to determine the decoded motion information; then, based on the decoded motion information and the first representation feature of the current frame, determines the decoded features of the current frame; and finally, based on the decoded features of the current frame, determines the reconstruction information of the current frame. This method eliminates the need to transmit motion information, simplifying the training process of the network model and achieving better rate-distortion performance. Furthermore, since the predicted motion information is corrected before being used to determine the decoded features of the current frame, encoding and decoding efficiency can be improved, thereby enhancing video compression performance.

[0157] In another embodiment of the present application, see Figure 8, which shows a schematic flow chart of an encoding method provided by an embodiment of the present application. As shown in Figure 8, the method may include:

[0158] S801: Determine a preset reference frame corresponding to the current frame.

[0159] It should be noted that the encoding method of the embodiment of the present application is applied to the encoder. In addition, the encoding method can refer to an optimization method for end-to-end video compression, specifically optimizing the motion information module; more specifically, a forward multi-reference frame neural video compression method that does not require the transmission of motion information. In other words, the embodiment of the present application eliminates the encoder portion of the motion information and can also expand the use of a single reference frame to multiple reference frames.

[0160] It should also be noted that in the embodiment of the present application, the preset reference frame is adjacent to the current frame. The preset reference frame may include at least one reference frame, and the at least one reference frame is an encoded frame n times before the current frame, where n is a positive integer.

[0161] That is, in the embodiment of the present application, the preset reference frame can be one or more frames. If the current frame is Frame t, where t is a positive integer, then the preset reference frame can be Frame t-1, the coded frame one moment before the current frame, or Frame t-2, the coded frame two moments before the current frame, or even Frame tn, the coded frame n moments before the current frame (e.g., an I frame, the first coded frame P1, etc.), without specific limitation here.

[0162] S802: Determine coding motion information of the current frame according to a preset reference frame, and determine a first representation feature of the current frame according to the coding motion information.

[0163] It should be noted that, in the embodiment of the present application, the coded motion information of the current frame may be determined based on one reference frame or multiple reference frames.

[0164] In some embodiments, determining the encoding motion information of the current frame according to the preset reference frame may include:

[0165] determining a first reference frame;

[0166] Time information is estimated based on the first reference frame and the current frame to determine coded motion information.

[0167] It should be noted that in this embodiment of the present application, if the preset reference frame is only one frame, such as the first reference frame, then temporal information can be estimated based on the first reference frame to determine the coded motion information of the current frame. Exemplarily, motion information is estimated based on the first reference frame and the current frame to obtain the coded motion information of the current frame.

[0168] In some embodiments, determining the encoding motion information of the current frame according to the preset reference frame may include:

[0169] determining a first reference frame and a second reference frame;

[0170] Estimating time information based on the first reference frame and the second reference frame to determine coded motion information;

[0171] The first reference frame is different from the second reference frame.

[0172] It should also be noted that in this embodiment of the present application, if the preset reference frame can be multiple frames, such as a first reference frame and a second reference frame, then temporal information is estimated based on the first reference frame and the second reference frame to determine the coded motion information of the current frame. Exemplarily, motion information is estimated based on the first reference frame and the second reference frame to obtain the coded motion information of the current frame.

[0173] In a specific embodiment, performing temporal information estimation based on the first reference frame and the second reference frame to determine the coded motion information may include: performing motion information estimation on the first reference frame and the second reference frame by an optical flow estimation module to obtain the coded motion information.

[0174] It's understandable that the optical flow estimation module on the encoder side has the same structure as the one on the decoder side. Optical flow estimation here determines the motion information of moving objects. Specifically, it refers to the motion information of pixels representing the same object in one frame of a video image moving to the next frame, which can be represented by MV.

[0175] In the embodiment of the present application, the optical flow calculation can adopt the most classic Lucas-Kanade algorithm, or can also adopt the optical flow estimation algorithm based on deep learning, such as the FlowNet / FlowNet2 algorithm, etc., which is not specifically limited here. In addition, it should be noted that when performing optical flow calculation, the first reference frame and the second reference frame as adjacent frames need to meet the following prerequisites: (1) the brightness between adjacent frames is constant; (2) the frame acquisition time of adjacent frames is continuous, or the movement of objects between adjacent frames is relatively "small"; (3) spatial consistency is maintained, that is, the pixels of the same sub-pixel have the same movement.

[0176] Simply put, in an embodiment of the present application, the current method of utilizing time information can be to obtain motion information between two reference frames by optical flow estimation. This is only one way of utilizing spatiotemporal information, but it is not the best. A better method of utilizing time information can also be used here instead of using motion information to obtain better rate-distortion performance.

[0177] It can also be understood that, in the process of video compression, in order to fully utilize the temporal correlation between video frames, the current frame can be encoded with reference to frames that have already been coded and decoded. In the implementation of a single reference frame, for the first reference frame, it can be the coded frame of the current frame at the previous moment, or it can be an I frame in the current video sequence, or it can be the coded frame of the current frame at any moment. In other words, the first reference frame can be the coded frame with the closest forward distance (if the current frame is the first P frame P1, then the first reference frame is the I frame; if the current frame is any P frame P other than the first P frame, then the first reference frame is the I frame). t , then the reference frame is P t-1 frame; however, this is not particularly limited.

[0178] It is also understandable that in a multi-reference frame implementation, taking two reference frames (a first reference frame and a second reference frame) as an example, in one possible implementation, the first reference frame is the coded frame at the previous moment of the current frame; the second reference frame is the coded frame two moments before the current frame. In other words, if the current frame is t, then the first reference frame can be the coded frame Frame t-1, and the second reference frame can be the coded frame Frame t-2.

[0179] In another possible implementation, the first reference frame is the coded frame immediately before the current frame, and the second reference frame is the coded I-frame. In other words, the first reference frame and the second reference frame may also be the coded frame t-1 and the I-frame, respectively. The I-frame is the first frame in a video sequence.

[0180] In short, in the embodiment of the present application, considering the temporal correlation between different frames in the video sequence, the current frame can be encoded with reference to multiple frames that have been coded. In the multi-frame reference, the types of the reference frames can be the same or different. For example, for the first reference frame and the second reference frame, a combination of I frame and P frame (for example, the coded P frame) can be used. t-1 frames and I frames), or a combination of pure P frames (e.g., encoded P t-1 Frame and P t-2 It should be noted that among these reference frame combinations, compared to P t-1 Frame and I frame combination, using P t-1 Frame and P t-2 The rate-distortion performance of frame combining is relatively good.

[0181] Furthermore, the obtained coded motion information can be applied to conditional coding of the current frame to determine the first representation feature of the current frame. In some embodiments, determining the first representation feature of the current frame based on the coded motion information can include: converting the coded motion information to obtain the first representation feature of the current frame.

[0182] In a specific embodiment, determining the first representation feature of the current frame according to the encoded motion information may include:

[0183] The coded motion information is used as a temporal prior to act on the conditional encoder of the current frame to obtain the first representation feature of the current frame.

[0184] It should be noted that in the embodiments of the present application, the first representation feature of the current frame can also be referred to as the potential representation feature of the current frame. The feature obtained by transforming the coded motion information of the current frame into another domain is called the potential representation feature of the current frame, and the encoder needs to write it into the bitstream for transmission.

[0185] Specifically, in the process of video sequence compression, since there is no preset reference frame for the first P frame in the video sequence, it is impossible to determine the coded motion information, so special processing is required. First, the neural image compression method based on deep learning is used to compress the first frame image (I frame) in the video sequence, and the subsequent frames (P frames, P frames) are compressed. t ) will first be combined with its previous frame (P t-1 ) Use the optical flow estimation module to obtain the encoded motion information (MV enc ), the encoded motion information acts as a temporal conditional prior on the conditional encoder of the current frame, thereby obtaining the potential representation features of the current frame.

[0186] It should also be noted that, in the embodiment of the present application, the potential representation features of the current frame will also be used as a reference for subsequent frames; therefore, in some embodiments, the method may further include: storing the first representation features of the current frame in a preset decoding buffer.

[0187] In this way, the coding motion information is first determined based on the preset reference frame, and then the coding motion information is used as a temporal prior in conditional coding to obtain the first representation feature of the current frame. It also needs to be stored in the preset decoding buffer for reference use by subsequent frames.

[0188] S803: Encode the first representation feature of the current frame, and write the obtained coded bits into the bitstream.

[0189] It should be noted that in the embodiment of the present application, motion information is no longer transmitted in the bit stream, and only the first representation feature of the current frame needs to be transmitted here; in this way, the decoding end can obtain the first representation feature of the current frame by decoding the bit stream, which is used to determine the decoded motion information of the current frame, so that the motion information encoder and motion information decoder are no longer required in the entire neural video compression framework.

[0190] It should also be noted that in this embodiment of the present application, the encoder also needs to obtain the reconstruction information of the current frame (decoded reconstruction features and decoded reconstructed frame) and store it in a preset decoding buffer for reference by subsequent frames. The specific operation process of the encoder when obtaining the reconstruction information of the current frame is similar to that of the decoder, that is, the encoder also has an MV correction module, a temporal context mining module, and a frame generator, etc., which will not be detailed here.

[0191] It should also be noted that in the embodiments of the present application, since the encoding end no longer needs to transmit motion information, the motion information encoder and motion information decoder are eliminated, and therefore there is no need to adopt a multi-stage training strategy. In this case, only one loss function is needed to directly perform end-to-end training on all modules of the entire neural video compression network. Therefore, in some embodiments, the method may further include:

[0192] Determine the preset network model;

[0193] The preset network model is trained through a first loss function, and the trained model is determined as a target network model; wherein the target network model includes at least: a conditional encoder, a conditional decoder, an optical flow estimation module, a preset correction module and a frame generator.

[0194] In the embodiment of the present application, the preset network model includes all modules of the entire neural video compression network. After the preset network model is trained using the first loss function, the trained model can be used as the target network model, where the target network model also includes all modules of the entire neural video compression network, specifically the conditional encoder, conditional decoder, optical flow estimation module, preset correction module, and frame generator.

[0195] In the embodiment of the present application, the first loss function can be a rate-distortion loss function, and the number of first loss functions is one. That is, only one rate-distortion loss function is needed to directly perform end-to-end training on all modules of the entire neural video compression network, thereby greatly simplifying the training method of the neural video compression model.

[0196] For example, Figure 7 shows a schematic diagram comparing the training strategies of the embodiment of the present application and the related art. As can be seen from Figure 7, the technical solution of the embodiment of the present application only needs to use a single loss function to train all modules during the entire training process, which greatly simplifies the model training process and makes the training process more concise. At the same time, due to the use of a preset correction module and a better frame generator, the embodiment of the present application also has rate-distortion performance comparable to the best methods in the related art.

[0197] Furthermore, an embodiment of the present application also provides a code stream, wherein the code stream is generated by bit encoding based on information to be encoded; wherein the information to be encoded includes at least: a first representation feature of the current frame.

[0198] In other words, after determining the first representation feature of the current frame, the encoder can encode the first representation feature of the current frame and write it into the bitstream. This is then transmitted to the decoder via the bitstream, allowing the decoder to directly obtain the first representation feature of the current frame through decoding. In this case, motion information is no longer required, allowing the decoder to determine the reconstructed information of the current frame.

[0199] In summary, in the embodiments of the present application, on the one hand, for the first time, the motion information encoder and motion information decoder in the neural video compression network framework are removed under the CNN-based framework. On the other hand, it is proposed to use the motion information between the t-1 frame and the t-2 frame to adaptively replace the motion information between the t frame and the t-1 frame, and use the MV correction module composed of the autoregressive module and the residual module to correct the motion information. On the other hand, a module ratio distribution method of the spatial correlation information extraction part (conditional coding part), the temporal correlation information extraction part (motion estimation, motion information encoding and motion information compensation part) and the spatiotemporal information fusion frame generation part (frame generator) is proposed. Specifically, it is recommended to strengthen the spatiotemporal information fusion frame generation part and weaken the temporal correlation information extraction part. On the other hand, since the motion information encoder and motion information decoder are removed in the neural video compression network framework, there is no need to transmit motion information in the channel. This can optimize the training method of the neural video compression network. Previously, due to the existence of the motion information encoder and motion information decoder, in order to achieve better rate-distortion performance during network training, a multi-stage training strategy was required. That is, different loss functions were used to train different network modules at different stages. In the embodiment of the present application, since the motion information encoder and motion information decoder are eliminated, a single loss function can be used to train all modules end-to-end, greatly simplifying the training process.

[0200] Furthermore, in an embodiment of the present application, in order to achieve a better way of utilizing time information, the current way of utilizing time information is to use the optical flow method to obtain the motion information between the previous and next frames. This is only one way of utilizing spatiotemporal information and is not optimal. A better way of utilizing time information can be used instead of using motion information to obtain better rate-distortion performance.

[0201] Furthermore, in an embodiment of the present application, when the preset reference frames include two reference frames, the two reference frames may use the previous frame t-1 and the I frame. However, from a test performance perspective, the performance of using the previous frame t-1 and the immediately preceding frame t-2 is superior to the performance of using the previous frame t-1 and the I frame, as shown in Table 3 above.

[0202] Furthermore, in an embodiment of the present application, the three U-Net modules of the frame generator may not use dense connections, but the test performance is not as good as that of using dense connections, as shown in Table 4 above.

[0203] Furthermore, in the embodiment of the present application, the scheme of referring to a single frame also has good performance, but is not as good as the rate-distortion performance of referring to a double frame; however, the model parameter amount of the reference single frame is less, and the encoding and decoding speed is also faster, as shown in the aforementioned Table 5.

[0204] This embodiment provides an encoding method that first determines a preset reference frame corresponding to the current frame; then, based on the preset reference frame, determines the encoding motion information of the current frame, and based on the encoding motion information, determines a first representation feature of the current frame; then, encodes the first representation feature of the current frame, and writes the resulting encoded bits into the bitstream. This method eliminates the need to transmit motion information, simplifying the network model training process and achieving better rate-distortion performance. Furthermore, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, encoding and decoding efficiency is improved, thereby enhancing video compression performance.

[0205] In another embodiment of the present application, based on the decoding method and encoding method described in the aforementioned embodiment, the embodiment of the present application provides a network architecture of a video codec system including a decoding method and an encoding method. Figure 9 is a schematic diagram of a network architecture of a video codec provided by an embodiment of the present application. As shown in Figure 9, the network architecture includes one or more electronic devices 13 to 1N and a communication network 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication network 01. During implementation, the electronic device can be various types of devices with video codec functions. For example, the electronic device can include a mobile phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensor device, a server, etc., which is not limited by the embodiment of the present application. Among them, the decoder or encoder in the embodiment of the present application can be the above-mentioned electronic device.

[0206] Among them, the electronic device in the embodiment of the present application has a video encoding and decoding function, generally including a video encoder (ie, encoder) and a video decoder (ie, decoder).

[0207] It can be understood that based on the decoding method and encoding method described in the aforementioned embodiments, the embodiments of the present application propose a forward multi-reference frame neural video compression technology based on a convolutional neural network that does not require the transmission of motion information. Based on the original neural video compression method TCM that transmits motion information, this technical solution eliminates the motion information encoder and motion information decoder, replaces them with a better intra-frame encoding method, adds an MV correction module, optimizes the frame generator, and can expand single reference frames to multiple reference frames.

[0208] In the embodiment of the present application, the detailed video sequence compression process is as follows: first, the first frame image (I frame) in the video sequence is compressed using a neural image compression method based on deep learning, and the subsequent frames (P frames, P t ) will first be compared with its previous frame (P t-1 ) Use the optical flow estimation network (i.e. the aforementioned “optical flow estimation module”) to obtain the motion information (MV enc ), the motion information acts as a temporal prior on the conditional encoder of the current frame to obtain the potential representation features of the current frame. The potential representation features of the current frame will be stored in a preset decoding buffer for reference by subsequent frames. Then, at the decoding end, the previous frame (P t-1 ) of the decoded frame image and the previous frame (P t-2 ) decoded frame image, and use the same optical flow estimation network as the encoding end to obtain the predicted motion information (MV dec_p ), the predicted motion information (MV dec_p ) generates the motion information of the decoding end through an MV correction module (MVdec ), this motion information acts as a temporal prior on the conditional decoder of the current frame, converting the latent representation features of the current frame and the forward reference frame (the latent representation features of the forward reference frame are obtained from a preset decoding buffer) into decoded features of the current frame. The decoded features are then passed through a frame generator to generate a decoded reconstructed frame and decoded reconstructed features of the current frame. The generated decoded reconstructed frame and decoded reconstructed features are then stored in a preset decoding buffer for subsequent video frame compression.

[0209] In the embodiment of the present application, the MV correction module is described in detail: the MV correction module is located after the optical flow estimation module at the decoding end and before the temporal context mining module. dec_p Modify to obtain motion information MV that is more suitable for conditional coding dec The MV correction module can be composed of an autoregressive module and a residual module. For example, the autoregressive module is in front and the residual module is in the back. The autoregressive module can make full use of the predicted motion information MV dec_p The motion information of each pixel in the image and the motion information of the adjacent pixels generate the motion information features of the current pixel, and then a residual module is used to extract and fuse the features to generate the motion information of the decoding end (MV dec ), the motion information (MV dec ) is more suitable for conditional encoding of the current frame and can be used as a conditional decoder for the current frame as a temporal conditional prior.

[0210] In an embodiment of the present application, a detailed description of the frame generator module (dense connection is not applicable, and the number of U-Net modules is not limited, for example, the number can be 3, but any natural number of U-Net modules can also be used): the frame generator is at the very end of the entire neural video compression network. The function is to generate a decoded reconstructed frame and a decoded reconstructed feature of the current frame, where the input of the frame generator is the decoded feature of the current frame. The frame generator can be composed of three U-Net modules and one convolution module, and dense connections are used for these three U-Net modules. The specific dense connection process is shown in Figure 6A.

[0211] Here, the three U-Net modules extract features from different stages, and dense connections fuse these features. The decoded and reconstructed features after the U-Net modules are fully extracted and fused. These decoded and reconstructed features are stored in a preset decoding buffer. Finally, these decoded and reconstructed features are passed through a convolution module to generate decoded and reconstructed frames, which are stored in the preset decoding buffer. The decoded and reconstructed features and decoded and reconstructed frames in the preset decoding buffer are used for subsequent video frame compression.

[0212] It should also be noted that the above-mentioned frame generator and reference frame solutions can be implemented in multiple ways.

[0213] For example, the frame generator scheme may include one with dense connections and one without dense connections. In one frame generator scheme (with dense connections), the specific connection process is shown in FIG6A . Using dense connections allows for the fusion of features from different layers, resulting in better performance. In another frame generator scheme (without dense connections), the specific connection process is shown in FIG6B . Without dense connections, the implementation process is simpler and faster.

[0214] Exemplarily, the scheme of referencing double frames may include single-frame multi-frame and multi-frame reference, wherein, in the multi-frame reference, the type may be different. In a scheme of reference frames (single-frame reference), in the process of video compression, in order to make full use of the temporal correlation between video frames, the current frame may be encoded with reference to the decoded frame. In the single-frame reference, the reference frame may be the decoded frame with the closest forward distance (if the current frame is the first frame P frame P1, then the reference frame is the I frame; if the current frame is any P frame P except the first P frame, then the reference frame is the I frame; if the current frame is any P frame P except the first P frame, then the reference frame is the I frame). t , then the reference frame is the Pt-1 frame). In another reference frame scheme (multi-frame reference), in order to further utilize the temporal correlation between different video frames in the video sequence, the current frame can be encoded with reference to multiple decoded frames. In multi-frame reference, the types of reference frames can be different. For example, a pure P frame reference (e.g., P t-1 、P t-2 ), or use a combination of I frames and P frames (for example: I, P t-1 ), no limitation is given here.

[0215] It can be understood that in the embodiments of the present application, on the one hand, for the first time, the motion information encoder and motion information decoder in the neural video compression network framework are removed under the CNN-based framework. On the other hand, it is proposed to use the motion information between the t-1 frame and the t-2 frame to adaptively replace the motion information between the t frame and the t-1 frame, and use the MV correction module composed of the autoregressive module and the residual module to correct the motion information. On the other hand, a module ratio distribution method of the spatial correlation information extraction part (conditional coding part), the temporal correlation information extraction part (motion estimation, motion information encoding and motion information compensation part) and the spatiotemporal information fusion frame generation part (frame generator) is proposed. Specifically, it is recommended to strengthen the spatiotemporal information fusion frame generation part and weaken the temporal correlation information extraction part. On the other hand, since the motion information encoder and motion information decoder are removed in the neural video compression network framework, there is no need to transmit motion information in the channel. This can optimize the training method of the neural video compression network. Previously, due to the existence of the motion information encoder and motion information decoder, in order to achieve better rate-distortion performance during network training, a multi-stage training strategy was required. That is, different loss functions were used to train different network modules at different stages. In the embodiment of the present application, since the motion information encoder and motion information decoder are eliminated, a single loss function can be used to train all modules end-to-end, greatly simplifying the training process.

[0216] In this way, the technical solution proposed in the embodiment of the present application redesigns the network framework and training method based on the TCM technology, and tests on the HEVC Class B, C, D, E, UVG ​​and MCL-JCV test data sets in LDP mode. Under the premise of almost no impact on performance, the use of the technical solution of the embodiment of the present application has the following beneficial effects: (1) This technical solution can greatly simplify the training method of the neural video compression model. Since this technical solution does not require special training for the motion information encoder and motion information decoder, there is no need to adopt a multi-stage training strategy. Only one rate-distortion loss function is needed to directly perform end-to-end training on all modules of the entire neural video compression network. It can be seen that this technical solution can greatly simplify the training process of the neural video compression network while maintaining similar rate-distortion performance as the original technology. (2) This technical solution does not require the transmission of motion information. Since this technical solution eliminates the motion information encoder and motion information decoder in the entire neural video compression framework, it is not necessary to transmit motion information in the channel. At this time, only the potential representation features of the current frame need to be transmitted in the channel.

[0217] For example, a comparison of the training strategies of the present technical solution and the solutions proposed in the related art is shown in FIG7 . It can be seen that the technical solution of the embodiment of the present application only needs to use one loss function to train all modules during the entire training process, which greatly simplifies the model training process. At the same time, due to the use of the MV correction module and the better frame generator, the present technical solution also has rate-distortion performance comparable to the best previous methods. Among them, the test results of HEVC Class B, C, D, E, UVG ​​and MCL-JCV tests in LD mode are shown in Tables 1 and 2 above.

[0218] In the embodiments of the present application, the specific implementation of the aforementioned embodiments is elaborated in detail based on the above embodiments. It can be seen that according to the technical solution of the aforementioned embodiments, since the transmission of motion information is cancelled, not only can the training process of the network model be simplified, but also better rate-distortion performance can be achieved; at the same time, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, the encoding and decoding efficiency can also be improved, thereby improving the video compression performance.

[0219] In another embodiment of the present application, based on the same inventive concept as the above embodiment, see Figure 10, which shows a schematic diagram of the composition structure of an encoder provided by an embodiment of the present application. As shown in Figure 10, the encoder 100 may include: a first determining unit 1001 and an encoding unit 1002; wherein,

[0220] A first determining unit 1001 is configured to determine a preset reference frame corresponding to a current frame;

[0221] The first determining unit 1001 is further configured to determine the coding motion information of the current frame according to the preset reference frame, and determine the first representation feature of the current frame according to the coding motion information;

[0222] The encoding unit 1002 is configured to encode the first representation feature of the current frame and write the obtained encoding bits into the bitstream.

[0223] In some embodiments, the preset reference frame includes at least one reference frame, and the at least one reference frame is an encoded frame at n moments before the current frame, where n is a positive integer.

[0224] In some embodiments, referring to FIG. 10 , the encoder 100 may further include a first estimation unit 1003 configured to determine a first reference frame; and perform temporal information estimation based on the first reference frame and the current frame to determine the encoded motion information.

[0225] In some embodiments, the first estimation unit 1003 is further configured to determine a first reference frame and a second reference frame; and perform time information estimation based on the first reference frame and the second reference frame to determine the encoded motion information; wherein the first reference frame is different from the second reference frame.

[0226] In some embodiments, the first estimation unit 1003 is further configured to perform motion information estimation on the first reference frame and the second reference frame through an optical flow estimation module to obtain encoded motion information.

[0227] In some embodiments, the first reference frame is a frame that has been encoded at a previous time instant before the current frame; and the second reference frame is a frame that has been encoded at two previous time instants before the current frame.

[0228] In some embodiments, the first determining unit 1001 is further configured to perform conversion processing on the encoded motion information to obtain a first representation feature of the current frame.

[0229] In some embodiments, the first determining unit 1001 is further configured to apply the encoded motion information as a temporal condition a priori to a conditional encoder of the current frame to obtain a first representation feature of the current frame.

[0230] In some embodiments, referring to FIG. 10 , the encoder 100 may further include a first storage unit 1004 configured to store the first representation feature of the current frame in a preset decoding buffer.

[0231] In some embodiments, referring to FIG10 , the encoder 100 may further include a first training unit 1005 configured to determine a preset network model; and to train the preset network model through a first loss function, and determine the trained model as a target network model; wherein the target network model includes at least: a conditional encoder, a conditional decoder, an optical flow estimation module, a preset correction module, and a frame generator.

[0232] In some embodiments, the first loss function is a rate-distortion loss function, and the number of the first loss functions is one.

[0233] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0234] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0235] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 100. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method described in any one of the aforementioned embodiments.

[0236] Based on the composition of the encoder 100 and the computer-readable storage medium, refer to Figure 11, which shows a specific hardware structure diagram of the encoder 100 provided in an embodiment of the present application. As shown in Figure 11, the encoder 100 may include: a first communication interface 1101, a first memory 1102 and a first processor 1103; each component is coupled together through a first bus system 1104. It can be understood that the first bus system 1104 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 1104 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 1104 in Figure 11. Among them,

[0237] The first communication interface 1101 is used to receive and send signals when sending and receiving information with other external network elements;

[0238] A first memory 1102 is used to store computer programs that can be run on the first processor 1103;

[0239] The first processor 1103 is configured to, when running the computer program, execute:

[0240] Determine a preset reference frame corresponding to the current frame;

[0241] Determining encoding motion information of the current frame according to a preset reference frame, and determining a first representation feature of the current frame according to the encoding motion information;

[0242] The first representation feature of the current frame is encoded, and the obtained encoded bits are written into the bitstream.

[0243] It is understood that the first memory 1102 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 1102 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0244] The first processor 1103 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 1103. The above-mentioned first processor 1103 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented as a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 1102 , and the first processor 1103 reads the information in the first memory 1102 and completes the steps of the above method in combination with its hardware.

[0245] It is to be understood that these embodiments described in the present application can be implemented with hardware, software, firmware, middleware, microcode or its combination.For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processor (Digital Signal Processing, DSP), digital signal processing equipment (DSP Device, DSPD), programmable logic device (Programmable Logic Device, PLD), field programmable gate array (Field-Programmable Gate Array, FPGA), general-purpose processor, controller, microcontroller, microprocessor, other electronic units for performing functions described in the present application or its combination.For software implementation, the technology described in the present application can be realized by the module (such as process, function etc.) that performs functions described in the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0246] Optionally, as another embodiment, the first processor 1103 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0247] This embodiment provides an encoder, in which, since the transmission of motion information is eliminated, the motion information encoder and motion information decoder in the video compression framework can be saved, so that a multi-stage training strategy is no longer required during the model training process. This not only simplifies the training process of the network model, but also has better rate-distortion performance. At the same time, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, the encoding and decoding efficiency can be improved, thereby improving the video compression performance.

[0248] In another embodiment of the present application, based on the same inventive concept as the above embodiment, refer to FIG12 , which shows a schematic diagram of the structure of a decoder provided by the embodiment of the present application. As shown in FIG12 , the decoder 120 may include: a decoding unit 1201, a correction unit 1202 and a second determination unit 1203; wherein,

[0249] The decoding unit 1201 is configured to decode the code stream and determine a first representation feature of the current frame;

[0250] The correction unit 1202 is configured to determine the predicted motion information of the current frame according to the preset reference frame, and perform correction processing on the predicted motion information to determine the decoded motion information;

[0251] The second determining unit 1203 is configured to determine the decoding feature of the current frame according to the decoded motion information and the first representation feature of the current frame; and further configured to determine the reconstruction information of the current frame according to the decoding feature of the current frame.

[0252] In some embodiments, the reconstruction information of the current frame includes a decoded reconstruction feature of the current frame and a decoded reconstructed frame of the current frame.

[0253] In some embodiments, the second determination unit 1203 is further configured to obtain a first representation feature of a preset reference frame; and convert the first representation feature of the current frame and the first representation feature of the preset reference frame according to the decoded motion information to determine the decoding feature of the current frame.

[0254] In some embodiments, the second determination unit 1203 is further configured to apply the decoded motion information as a temporal condition prior to the conditional decoder of the current frame, convert the first representation feature of the current frame and the first representation feature of the preset reference frame to obtain the decoded feature of the current frame.

[0255] In some embodiments, the preset reference frame includes at least one reference frame, and the at least one reference frame is a decoded frame at n moments before the current frame, where n is a positive integer.

[0256] In some embodiments, referring to FIG. 12 , the decoder 120 may further include a second estimation unit 1204 configured to determine a first reference frame; and perform temporal information estimation based on the first reference frame to determine predicted motion information of the current frame.

[0257] In some embodiments, the second estimation unit 1204 is further configured to determine a first reference frame and a second reference frame; and perform time information estimation based on the first reference frame and the second reference frame to determine predicted motion information of the current frame; wherein the first reference frame is different from the second reference frame.

[0258] In some embodiments, the second estimation unit 1204 is further configured to perform motion information estimation on the first reference frame and the second reference frame through an optical flow estimation module to obtain predicted motion information of the current frame.

[0259] In some embodiments, the first reference frame is a decoded frame of the current frame at a previous time instant; the second reference frame is a decoded frame of the current frame at two previous time instants.

[0260] In some embodiments, the correction unit 1202 is further configured to perform correction processing on the predicted motion information through a preset correction module to obtain decoded motion information.

[0261] In some embodiments, the preset correction module includes an autoregressive module and a residual module; accordingly, the correction unit 1202 is also configured to perform autoregressive processing on the motion information of each pixel point in the predicted motion information and the motion information of the adjacent pixel points through the autoregressive module to determine the first intermediate feature; and perform feature extraction and fusion on the first intermediate feature through the residual module to obtain decoded motion information.

[0262] In some embodiments, the correction unit 1202 is further configured to perform feature extraction and fusion on the predicted motion information through a residual module to obtain a second intermediate feature; and perform autoregressive processing on the motion information of each pixel point in the second intermediate feature and the motion information of the adjacent pixel points through an autoregressive module to obtain decoded motion information.

[0263] In some embodiments, the number of autoregressive modules is at least one, and the number of residual modules is at least one.

[0264] In some embodiments, the decoding unit 1201 is further configured to perform decoding and reconstruction processing on the decoding features of the current frame through a frame generator to obtain the decoding and reconstruction features of the current frame and the decoding and reconstruction frame of the current frame.

[0265] In some embodiments, the frame generator includes a first network module and a first convolution module; accordingly, the decoding unit 1201 is also configured to extract and fuse the decoding features of the current frame through the first network module to obtain the decoding reconstruction features of the current frame; and perform a convolution operation on the decoding reconstruction features of the current frame through the first convolution module to obtain the decoding reconstructed frame of the current frame.

[0266] In some embodiments, the first network module includes at least one U-Net module; wherein: at least one U-Net module does not use dense connections; or, at least one U-Net module uses dense connections.

[0267] In some embodiments, referring to FIG12 , the decoder 120 may further include a second training unit 1205 configured to determine a preset network model; and to train the preset network model through a first loss function, and determine the trained model as a target network model; wherein the target network model includes at least: a conditional encoder, a conditional decoder, an optical flow estimation module, a preset correction module, and a frame generator.

[0268] In some embodiments, the first loss function is a rate-distortion loss function, and the number of the first loss functions is one.

[0269] It is understood that in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular system. Furthermore, the various components in this embodiment can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The aforementioned integrated units can be implemented in the form of hardware or software functional modules.

[0270] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, this embodiment provides a computer-readable storage medium, which is applied to the decoder 120 and stores a computer program. When the computer program is executed by the second processor, it implements any of the methods in the aforementioned embodiments.

[0271] Based on the composition of the decoder 120 and the computer-readable storage medium, refer to Figure 13, which shows a specific hardware structure diagram of the decoder 120 provided in an embodiment of the present application. As shown in Figure 13, the decoder 120 may include: a second communication interface 1301, a second memory 1302 and a second processor 1303; each component is coupled together through a second bus system 1304. It can be understood that the second bus system 1304 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 1304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the second bus system 1304 in Figure 13. Among them,

[0272] The second communication interface 1301 is used to receive and send signals during the process of sending and receiving information between other external network elements;

[0273] The second memory 1302 is used to store computer programs that can be run on the second processor 1303;

[0274] The second processor 1303 is configured to, when running the computer program, execute:

[0275] Decoding the code stream to determine a first representation feature of the current frame;

[0276] Determine the predicted motion information of the current frame according to the preset reference frame, and perform correction processing on the predicted motion information to determine the decoded motion information;

[0277] determining a decoding feature of the current frame based on the decoded motion information and a first representation feature of the current frame;

[0278] Reconstruction information of the current frame is determined according to the decoding characteristics of the current frame.

[0279] Optionally, as another embodiment, the second processor 1303 is further configured to execute any one of the methods described in the foregoing embodiments when running the computer program.

[0280] It can be understood that the hardware functions of the second memory 1302 and the first memory 1102 are similar, and the hardware functions of the second processor 1303 and the first processor 1103 are similar; they will not be described in detail here.

[0281] This embodiment provides a decoder, in which, since the transmission of motion information is eliminated, the motion information decoder in the video compression framework can be saved, so that a multi-stage training strategy is no longer required during the model training process. This not only simplifies the training process of the network model, but also has better rate-distortion performance. At the same time, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, the encoding and decoding efficiency can be improved, thereby improving the video compression performance.

[0282] In yet another embodiment of the present application, referring to FIG14 , a schematic diagram illustrating the structure of a coding and decoding system provided in an embodiment of the present application is shown. As shown in FIG14 , the coding and decoding system 140 may include an encoder 1401 and a decoder 1402. The encoder 1401 may be any of the encoders described in the aforementioned embodiments, and the decoder 1402 may be any of the decoders described in the aforementioned embodiments.

[0283] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0284] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0285] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0286] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0287] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0288] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0289] In an embodiment of the present application, at the encoding end, the preset reference frame corresponding to the current frame is first determined; then, the coded motion information of the current frame is determined based on the preset reference frame, and the first representation feature of the current frame is determined based on the coded motion information; then, the first representation feature of the current frame is encoded, and the obtained coded bits are written into the bitstream. At the decoding end, the bitstream is first decoded to determine the first representation feature of the current frame; then, the predicted motion information of the current frame is determined based on the preset reference frame, and the predicted motion information is corrected to determine the decoded motion information; then, based on the decoded motion information and the first representation feature of the current frame, the decoded features of the current frame are determined; finally, based on the decoded features of the current frame, the reconstruction information of the current frame is determined. In this way, only the first representation feature of the current frame needs to be transmitted in the bitstream, and the transmission of motion information is eliminated, thereby saving the motion information encoder and motion information decoder in the video compression framework, so that the multi-stage training strategy is no longer required during the model training process, which not only simplifies the training process of the network model, but also has better rate-distortion performance; at the same time, since the predicted motion information is corrected before being used to determine the decoding features of the current frame, the encoding and decoding efficiency can be improved, thereby improving the video compression performance.

Claims

1. A decoding method, applied to a decoder, comprising: Decoding the code stream to determine a first representation feature of the current frame; Determining predicted motion information of the current frame according to a preset reference frame, and performing correction processing on the predicted motion information to determine decoded motion information; determining a decoding feature of the current frame according to the decoded motion information and a first representation feature of the current frame; Reconstruction information of the current frame is determined according to the decoding characteristics of the current frame.

2. The method according to claim 1, wherein The reconstruction information of the current frame includes the decoded reconstruction features of the current frame and the decoded reconstructed frame of the current frame.

3. The method according to claim 1, wherein The determining, based on the decoded motion information and the first representation feature of the current frame, the decoding feature of the current frame includes: Acquire a first representation feature of the preset reference frame; The first representation feature of the current frame and the first representation feature of the preset reference frame are converted according to the decoded motion information to determine the decoded feature of the current frame.

4. The method according to claim 3, wherein: The method further comprises: The decoded motion information is used as a temporal condition prior to act on the conditional decoder of the current frame, and the first representation feature of the current frame and the first representation feature of the preset reference frame are converted to obtain the decoded feature of the current frame.

5. The method according to claim 1, wherein The preset reference frame includes at least one reference frame, and the at least one reference frame is a decoded frame of the current frame at n moments before, where n is a positive integer.

6. The method according to claim 5, wherein: The determining the predicted motion information of the current frame according to the preset reference frame includes: determining a first reference frame; Time information is estimated according to the first reference frame to determine predicted motion information of the current frame.

7. The method according to claim 5, wherein: The determining the predicted motion information of the current frame according to the preset reference frame includes: determining a first reference frame and a second reference frame; performing time information estimation based on the first reference frame and the second reference frame to determine predicted motion information of the current frame; The first reference frame is different from the second reference frame.

8. The method according to claim 7, wherein: The estimating time information according to the first reference frame and the second reference frame to determine the predicted motion information of the current frame includes: Motion information of the first reference frame and the second reference frame is estimated by an optical flow estimation module to obtain predicted motion information of the current frame.

9. The method according to claim 7 or 8, wherein The first reference frame is a decoded frame of the current frame at a previous moment; The second reference frame is a decoded frame two moments before the current frame.

10. The method according to claim 1, wherein The correcting the predicted motion information to determine the decoded motion information includes: The predicted motion information is corrected by a preset correction module to obtain the decoded motion information.

11. The method according to claim 10, wherein: The preset correction module includes an autoregressive module and a residual module; The correcting the predicted motion information by a preset correction module to obtain the decoded motion information includes: performing autoregressive processing on the motion information of each pixel and the motion information of adjacent pixels in the predicted motion information by the autoregressive module to determine a first intermediate feature; The residual module performs feature extraction and fusion on the first intermediate features to obtain the decoded motion information.

12. The method according to claim 11, wherein The correction processing of the predicted motion information by a preset correction module to obtain the decoded motion information further includes: Performing feature extraction and fusion on the predicted motion information through the residual module to obtain a second intermediate feature; The autoregressive module performs autoregressive processing on the motion information of each pixel in the second intermediate feature and the motion information of adjacent pixels to obtain the decoded motion information.

13. The method according to claim 11 or 12, wherein: The number of the autoregressive modules is at least one, and the number of the residual modules is at least one.

14. The method according to claim 2, wherein: The determining, based on the decoding characteristics of the current frame, the reconstruction information of the current frame includes: The frame generator performs decoding and reconstruction processing on the decoding features of the current frame to obtain the decoding and reconstruction features of the current frame and the decoding and reconstruction frame of the current frame.

15. The method according to claim 14, wherein The frame generator includes a first network module and a first convolution module; The decoding and reconstruction processing of the decoding features of the current frame by the frame generator to obtain the reconstruction information of the current frame includes: Extracting and fusing the decoding features of the current frame through the first network module to obtain decoding and reconstruction features of the current frame; A convolution operation is performed on the decoded and reconstructed features of the current frame through the first convolution module to obtain a decoded and reconstructed frame of the current frame.

16. The method according to claim 15, wherein The first network module includes at least one U-Net module; wherein: The at least one U-Net module does not use dense connections; or The at least one U-Net module uses dense connections.

17. The method according to claim 1, wherein The method further comprises: Determine the preset network model; The preset network model is trained by a first loss function, and the trained model is determined as a target network model; wherein the target network model includes at least: a conditional encoder, a conditional decoder, an optical flow estimation module, a preset correction module and a frame generator.

18. The method according to claim 17, wherein The first loss function is a rate-distortion loss function, and the number of the first loss functions is one.

19. A coding method, applied to an encoder, comprising: Determine a preset reference frame corresponding to the current frame; Determining encoding motion information of the current frame according to the preset reference frame, and determining a first representation feature of the current frame according to the encoding motion information; The first representation feature of the current frame is encoded, and the obtained encoding bits are written into a bitstream.

20. The method according to claim 19, wherein The preset reference frame includes at least one reference frame, and the at least one reference frame is an encoded frame at n moments before the current frame, where n is a positive integer.

21. The method according to claim 20, wherein The determining the coding motion information of the current frame according to the preset reference frame includes: determining a first reference frame; Time information is estimated based on the first reference frame and the current frame to determine the coded motion information.

22. The method according to claim 20, wherein The determining the coding motion information of the current frame according to the preset reference frame includes: determining a first reference frame and a second reference frame; Estimating time information based on the first reference frame and the second reference frame to determine the coded motion information; The first reference frame is different from the second reference frame.

23. The method according to claim 22, wherein The estimating time information according to the first reference frame and the second reference frame to determine the coded motion information includes: Motion information of the first reference frame and the second reference frame is estimated by an optical flow estimation module to obtain the encoded motion information.

24. The method according to claim 22 or 23, wherein The first reference frame is a coded frame of the current frame at a previous moment; The second reference frame is a coded frame two moments before the current frame.

25. The method according to claim 19, wherein The determining the first representation feature of the current frame according to the encoded motion information includes: The coded motion information is converted to obtain a first representation feature of the current frame.

26. The method according to claim 19, wherein The determining the first representation feature of the current frame according to the encoded motion information includes: The coded motion information is used as a temporal condition prior to act on a conditional encoder of the current frame to obtain a first representation feature of the current frame.

27. The method according to claim 19, wherein The method further comprises: The first representation feature of the current frame is stored in a preset decoding buffer.

28. The method according to claim 19, wherein The method further comprises: Determine the preset network model; The preset network model is trained by a first loss function, and the trained model is determined as the target network model; wherein The target network model includes at least: a conditional encoder, a conditional decoder, an optical flow estimation module, a preset correction module and a frame generator.

29. The method according to claim 28, wherein The first loss function is a rate-distortion loss function, and the number of the first loss functions is one.

30. A code stream, wherein The code stream is generated by bit coding according to information to be coded; wherein the information to be coded includes at least: a first representation feature of the current frame.

31. An encoder, comprising a first determining unit and an encoding unit; wherein: The first determining unit is configured to determine a preset reference frame corresponding to the current frame; The first determining unit is further configured to determine the encoding motion information of the current frame according to the preset reference frame, and determine the first representation feature of the current frame according to the encoding motion information; The encoding unit is configured to encode the first representation feature of the current frame and write the obtained encoding bits into the bitstream.

32. An encoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 19 to 29 when running the computer program.

33. A decoder comprising a decoding unit, a correction unit, and a second determination unit; wherein: The decoding unit is configured to decode the code stream and determine a first representation feature of the current frame; The correction unit is configured to determine the predicted motion information of the current frame according to a preset reference frame, and perform correction processing on the predicted motion information to determine the decoded motion information; The second determining unit is configured to determine the decoding feature of the current frame according to the decoded motion information and the first representation feature of the current frame; The system is further configured to determine reconstruction information of the current frame according to decoding characteristics of the current frame.

34. A decoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 1 to 18 when running the computer program.

35. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, which, when executed, implements the method according to any one of claims 1 to 18 or the method according to any one of claims 19 to 29.

Images