Training method and apparatus

Abstract

Provided in the present application are a training method and apparatus. The method comprises: during training of a compression model, using a training image sequence as a training input of the compression model, wherein the image sequence comprises a first image frame, a plurality of intermediate frames and a second image frame, and the compression model outputs compression information of the training image sequence; first compression information of each intermediate frame is obtained on the basis of a residual feature parameter of the intermediate frame, and a reference frame of the intermediate frame is used for acquiring the residual feature parameter; and second compression information of the second image frame is obtained on the basis of a coding probability of an image feature of the image frame, and a reference frame of the second image frame is used for acquiring the coding probability. In this way, cumulative errors of a compression model can be effectively reduced in the present application by means of adding a second image frame to a training image sequence.

Description

Training methods and equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311030870.9 and application name “Training Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, and in particular to a training method and device. Background Art

[0003] As convolutional neural networks (CNNs) surpass traditional algorithms in computer vision tasks such as image recognition and object detection, a growing number of researchers are exploring deep learning-based image and video compression methods. Some researchers have designed end-to-end deep learning image and video compression algorithms, for example, by simultaneously optimizing modules such as inter-frame prediction, encoding networks, entropy estimation networks, and decoding networks as a whole.

[0004] Summary of the Invention

[0005] This application provides a training method and apparatus that can effectively reduce the cumulative error of a compression model.

[0006] In a first aspect, the present application provides a training method, comprising: a training device inputs a training image sequence into a compression model, and obtains compression information of the training image sequence output by the compression model. Furthermore, the training device obtains a loss value of the compression model based on the compression information corresponding to each image in the training image sequence. Furthermore, the training device updates the weight value of the compression model based on the loss value of the compression model. The training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame. The compression information of the intermediate frame includes first compression information, which is obtained by the compression model based on the residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on the reference frame of the intermediate frame. The compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0007] In this way, during the training process of the present application, when the compression model performs probabilistic prediction on the image features of the second image frame (i.e., the process of obtaining the coding probability of the image features), since the compression model performs probabilistic prediction on the image features of the second image based on the reference frame, the prediction result of the image features of the second image frame is more accurate in view of the strong correlation between the reference frame and the second image frame, and further, when the probabilistic prediction is performed based on the image features, a more accurate probabilistic prediction result is obtained. Moreover, when the entropy coding process estimates the bit rate of the second image frame (i.e., the second compressed information), it is estimated based on the result of the probabilistic prediction. Therefore, the more accurate the result of the probabilistic prediction, the smaller the estimated bit rate. Accordingly, when obtaining the reconstructed image corresponding to the second image frame based on the bit rate, the smaller the bit rate, the higher the quality of the corresponding reconstructed image, and the smaller the inter-frame redundancy.

[0008] During training, the image quality of the second image frame's reference frame (i.e., the reconstructed image of the P frame) affects the predicted image features of the second image frame. This predicted image feature will affect the predicted coding probability of the second image frame, which in turn will affect the bitrate of the second image frame, which in turn will affect the loss of the second image frame. Furthermore, the lower the loss of the second image frame, the lower the loss of the compression model. During the iterative training of the compression model, the training goal is to continuously reduce the loss of the compression model. To achieve this goal, the loss of the second image frame must be continuously reduced. Based on this mutual constraint, to reduce the loss of the second image frame, the image quality of the reference frame of the second image frame must be continuously improved, thereby achieving a stable constraint on the quality of the reconstructed images of the intermediate frames. In other words, during the iterative training process, the image quality of the intermediate frames output by the compression model increases with each iteration. When the loss reaches a minimum, training ends, and the image quality of the intermediate frames reaches the maximum achievable value. In this way, in subsequent applications, the image compressed by the compression model has higher quality and can effectively reduce the cumulative error of the compression model.

[0009] Exemplarily, the second image frame is the last frame (also referred to as the tail frame) of the training image sequence.

[0010] Exemplarily, the second image frame is the CI (Condition I) frame described in the embodiment of the present application.

[0011] Exemplarily, the second image frame can be an I frame or a P frame. The second image frame is similar to the P frame, and both encoding and decoding (i.e., compression) depend on the reference frame. The difference between the second image frame and the P frame is that the compression of the P frame depends on the reference frame to calculate the residual features, that is, its optical flow is used to calculate the residual information. The compression of the CI frame (i.e., the second image frame) depends on the reference frame to calculate the probability distribution of the feature information of the original image, that is, its optical flow is not used to calculate the residual information, but only to calculate the probability distribution of the feature information of the original image. Therefore, there is no cumulative error between the encoded reconstructed image of the CI frame and the decoded reconstructed image.

[0012] Exemplarily, the training device may repeatedly train the compression model based on the training steps to update the weight values ​​of the compression model and obtain a trained compression model.

[0013] Exemplarily, the weight values ​​of the compression model can also be referred to as the parameters of the compression model (or compression parameters), which is not limited in this application. In other words, the compression model is composed of multiple compression parameters or weight values. Once the weight values ​​of the compression model are obtained, the compression model can be constructed based on the weight values ​​of the compression model.

[0014] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0015] In this way, in the embodiment of the present application, by adding a virtual training sequence, the P frame in the original image can simulate the position in the image sequence in the actual test scene, thereby enhancing the model's ability to reconstruct the entire image sequence and reducing the cumulative error during the test process.

[0016] Exemplarily, the inserted frame is the virtual frame described in the embodiment of the present application.

[0017] In one possible implementation, the first image frame is the leading frame, and at least one inserted frame follows the first image frame. By adding a dummy sequence before the P frame, the position of the P frame in the test scene can be effectively simulated, minimizing the potential cumulative error problem that may arise during testing. This problem can then be addressed during training, allowing the trained compression model to effectively reduce cumulative error in actual applications and improve the quality of reconstructed images for each frame.

[0018] Optionally, if the inserted frame duplicates the I frame, the inserted frame may also be inserted before the I frame (ie, the first frame).

[0019] In one possible implementation, the compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of at least one P frame, and the compression information of the second image frame; wherein, at least one inserted frame is used in the compression process of the compression model for compressing the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

[0020] In this way, the inserted frames only participate in the training but not in the loss value acquisition process of the compression model. This can increase the length of the training image sequence without increasing the training complexity, effectively reducing the computational pressure during the training process.

[0021] In one possible implementation, the compression model includes a first neural network model, a second neural network model, and a third neural network model. The first neural network model is used to compress the first image frame, the second neural network model is used to compress at least one intermediate frame, and the third neural network model is used to compress the second image frame.

[0022] In this way, the compression model may include multiple neural network models, the second neural network model can implement residual encoding and decoding of the intermediate frame, and the third neural network model can implement CI encoding and decoding of the second image frame.

[0023] The second neural network model may be the residual coding model in the embodiment of the present application. The third neural network model may be the CI frame coding model in the embodiment of the present application.

[0024] Exemplarily, the weight values ​​of the compression model include weight values ​​of the first neural network model, weight values ​​of the second neural network model, and weight values ​​of the third neural network model.

[0025] In a possible implementation, the compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes a bit rate of the first image frame.

[0026] In a possible implementation, the compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame, and the reconstructed image of the intermediate frame.

[0027] In one possible implementation, the compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame, and the reconstructed image of the second image frame.

[0028] In one possible implementation, obtaining a loss value corresponding to the training image sequence based on compression information corresponding to each image in the training image sequence includes obtaining a distortion rate of the intermediate frame based on a reconstructed image of the intermediate frame and the intermediate frame, and obtaining a loss value of the intermediate frame based on the distortion rate of the intermediate frame, the optical flow bitrate of the intermediate frame, and the first compression information of the intermediate frame.

[0029] In one possible implementation, a loss value of a compression model is obtained based on compression information corresponding to each image in a training image sequence, including: obtaining the distortion rate of the second image frame based on a reconstructed image of the second image frame and the second image frame; and obtaining the loss value of the second image frame based on the distortion rate of the second image frame, the optical flow code rate of the second image frame, and the second compression information of the second image frame.

[0030] In one possible implementation, the loss value of the compression model is obtained based on the compression information corresponding to each image in the training image sequence, including: obtaining the loss value of the compression model based on the bit rate of the first image frame, the loss value of each intermediate frame in at least one intermediate frame, and the loss value of the second image frame.

[0031] In this way, the loss value of the compression model in the embodiment of the present application is obtained based on the bit rate of the first image frame, the loss value of the P frame and the loss value of the second image frame. Accordingly, since the reconstructed image of the P frame is used as a reference frame for the CI frame, if the overall loss value of the compression model needs to be reduced, the loss value of the CI frame needs to be smaller, and the stability of the reconstructed image of the P frame is further reversely constrained, so that during the training process, the loss value of the compression model becomes smaller and smaller, and accordingly, the quality of the reconstructed image of the P frame becomes higher and higher.

[0032] In one possible implementation, the weight value of the compression model is updated based on the loss value of the compression model, including: when the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training number threshold, stopping updating the weight value of the compression model based on the loss value of the compression model.

[0033] In this way, by setting the training conditions, the compression model can stop training when the training conditions are met, and the trained compression model can be obtained.

[0034] In a second aspect, the present application provides a training device, comprising: a first acquisition module, used to input a training image sequence into a compression model, and obtain compression information of the training image sequence output by the compression model; a second acquisition module, used to obtain a loss value of the compression model based on the compression information corresponding to each image in the training image sequence; an update module, used to update the weight value of the compression model based on the loss value of the compression model; wherein, the training image sequence includes a first image frame, multiple intermediate frames and a second image frame; the compression information of the intermediate frame includes first compression information, which is obtained by the compression model based on the residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on the reference frame of the intermediate frame; the compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0035] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0036] In a possible implementation, the first image frame is a first frame, and at least one inserted frame is after the first image frame.

[0037] In one possible implementation, the compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of at least one P frame, and the compression information of the second image frame; wherein, at least one inserted frame is used in the compression process of the compression model for compressing the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

[0038] In one possible implementation, the compression model includes a first neural network model, a second neural network model, and a third neural network model. The first neural network model is used to compress the first image frame, the second neural network model is used to compress at least one intermediate frame, and the third neural network model is used to compress the second image frame.

[0039] In a possible implementation, the compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes a bit rate of the first image frame.

[0040] In a possible implementation, the compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame, and the reconstructed image of the intermediate frame.

[0041] In one possible implementation, the compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame, and the reconstructed image of the second image frame.

[0042] In one possible implementation, the second acquisition module is specifically used to: obtain the distortion rate of the intermediate frame based on the reconstructed image of the intermediate frame and the intermediate frame; obtain the loss value of the intermediate frame based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame and the first compression information of the intermediate frame.

[0043] In one possible implementation, the second acquisition module is specifically used to: obtain the distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; obtain the loss value of the second image frame based on the distortion rate of the second image frame, the optical flow code rate of the second image frame and the second compression information of the second image frame.

[0044] In a possible implementation, the second acquisition module is specifically used to obtain the loss value of the compression model based on the bit rate of the first image frame, the loss value of each intermediate frame in at least one intermediate frame, and the loss value of the second image frame.

[0045] In one possible implementation, the update module is specifically used to: when the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training number threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

[0046] In a third aspect, the present application provides a coding method, which includes: obtaining video data or image data; encoding the video data or image data based on a coding network to obtain a code stream; wherein the coding network is trained based on a training image sequence, and the training image sequence includes a first image frame, multiple intermediate frames, and a second image frame; the compression information corresponding to each image in the training image sequence is used to obtain a loss value of the coding network, and the loss value of the coding network is used to update the weight value of the coding network; the compression information of the intermediate frame includes first compression information, and the first compression information is obtained by the coding network based on the residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on the reference frame of the intermediate frame; the compression information of the second image frame includes second compression information, and the second compression information is obtained by the coding network based on the coding probability of the image features of the second image frame, and the coding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0047] In a fourth aspect, the present application provides a decoding method, comprising: receiving a code stream; decoding the code stream based on a decoding network to obtain reconstructed video data. The decoding network is trained based on a training image sequence, the training image sequence comprising a first image frame, a plurality of intermediate frames, and a second image frame; the compression information corresponding to each image in the training image sequence is used to obtain a loss value of the decoding network, and the loss value of the decoding network is used to update the weight value of the decoding network; the compression information of the intermediate frame comprises first compression information, which is obtained by the decoding network based on residual feature parameters of the intermediate frame, which are obtained based on a reference frame of the intermediate frame; the compression information of the second image frame comprises second compression information, which is obtained by the decoding network based on the coding probability of the image features of the second image frame, which are obtained based on the reference frame of the second image frame.

[0048] In a fifth aspect, the present application provides an encoding device, comprising a memory and a processor, wherein the memory is coupled to the processor. The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device implements the method in the fourth aspect.

[0049] In a sixth aspect, the present application provides a decoding device comprising a memory and a processor, wherein the memory is coupled to the processor. The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device implements the method of the fifth aspect.

[0050] In the seventh aspect, the present application provides an electronic device comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the above-mentioned first aspects.

[0051] In an eighth aspect, the present application provides a computer-readable storage medium comprising a computer program, wherein when the computer program is executed on a computer, the computer executes the method according to any one of the above-mentioned first aspects.

[0052] In a ninth aspect, the present application provides a computer program, which, when executed by a computer, is used to perform any of the methods described in the first aspect.

[0053] In a tenth aspect, the present application also provides a computer program product, which includes computer program code. When the computer program code runs on a computer, it enables the computer to execute the operations and / or processing performed by the electronic device in any of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG1A is a schematic block diagram of an exemplary decoding system 10;

[0055] FIG1B is an illustrative diagram of an example of a video decoding system 40;

[0056] FIG2 is a schematic diagram of a video decoding device 400 provided by an embodiment of the present invention;

[0057] FIG3 is a simplified block diagram of an apparatus 500 provided by an exemplary embodiment;

[0058] Figure 4 is an example diagram of an end-to-end deep learning image encoding and decoding framework;

[0059] FIG5 is an example diagram of an application scenario of an embodiment of the present application;

[0060] FIG6 is a schematic diagram showing the structure of an exemplary residual coding and decoding framework;

[0061] FIG7 is a schematic diagram showing an exemplary processing flow of a residual codec structure;

[0062] FIG8 is a schematic diagram showing an exemplary GOP;

[0063] FIG9 is a schematic diagram showing exemplary PSNR values ​​of a VTM compression method and PSNR changes of an AI residual compression method;

[0064] FIG10 is a schematic diagram illustrating an exemplary encoding and decoding framework of a CI frame;

[0065] FIG11 is a schematic diagram showing an exemplary processing flow of a CI frame encoding and decoding structure;

[0066] FIG12 is a schematic structural diagram of an exemplary compression model;

[0067] FIG13 is a schematic diagram of an exemplary training image sequence;

[0068] FIG14 is a schematic diagram illustrating an exemplary training process;

[0069] FIG15 is a schematic diagram showing exemplary performance test results;

[0070] FIG16 is a schematic diagram showing exemplary performance test results;

[0071] FIG17 is a schematic diagram of an exemplary training image sequence;

[0072] FIG18 is a schematic diagram illustrating an exemplary training process;

[0073] FIG19 is a schematic diagram illustrating an exemplary training process;

[0074] FIG20 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0075] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0077] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0078] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0079] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0080] The following is a description of the terms involved in the embodiments of this application:

[0081] 1. Bit rate: In image compression, the average code length required to encode a unit pixel.

[0082] 2. Rate-Distortion (RD): This metric describes the relationship between bit rate and distortion. It can also be considered a performance indicator for a compression algorithm, taking into account both bit rate and decoded image distortion.

[0083] 3. Peak Signal to Noise Ratio (PSNR) is an objective criterion for evaluating images. The higher the PSNR, the better the image quality.

[0084] 4. Multi-Scale Structural Similarity Index measure (MS-SSIM) is an objective criterion for evaluating images. A higher value indicates better image quality.

[0085] 5. Video Encoder Performance (Bjontegaard delta rate, BD-Rate) is a metric used to compare video encoder performance. It is typically used to compare the encoding results of different video encoders on the same video sequence and measure the encoder's compression performance. A smaller BD-Rate indicates better results. A negative value indicates a lower bitrate compared to the comparison method at the same quality.

[0086] 6. Codec: A device or program that can encode or decode a signal or data stream. It usually includes an encoder and a decoder.

[0087] 7. Encoder: Encodes a signal or digital stream (usually for transmission, storage, or encryption) or obtains an encoded stream.

[0088] 8. Decoder: The operation of restoring the encoded stream into a signal or digital stream.

[0089] 9. A feature map is a set of values ​​obtained by feeding an image or video into a network. It is typically represented as a matrix of dimensions [c, h, w], where C (channel), H (height), and W (width).

[0090] 10. Optical flow represents the movement speed and direction of each pixel in two adjacent frames of images.

[0091] 11. Error Accumulation (or Accumulated Error) refers to the cumulative error that occurs when the results of a recursive formula are integrated (or accumulated) together. In the context of video compression, this refers specifically to the degradation of compression performance over time.

[0092] 12. Distortion refers to PSNR or MS-SSIM in this application, and may refer to other distortion indicators in other embodiments, which is not limited in this application.

[0093] 13. In the YUV space, the image is divided into three channels, Y / U / V. Y-PSNR refers to the PSNR value on the Y channel, U-PSNR refers to the PSNR value on the U channel, and V-PSNR refers to the PSNR value on the V channel.

[0094] 14. Bitrate refers to the amount of data used per unit time in a video file and is a crucial component of image quality control in video encoding. For the same compression quality, the lower the bitrate, the better.

[0095] Since the embodiments of the present application involve the application of neural networks, for ease of understanding, the following first explains the relevant nouns or terms used in the embodiments of the present application:

[0096] 1. Neural Networks

[0097] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0098] Where s = 1, 2, ... n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a nonlinear function such as ReLU. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0099] 2. Multi-layer perception (MLP)

[0100] MLP is a simple deep neural network (DNN) (different layers are fully connected), also known as a multi-layer neural network, which can be understood as a neural network with many hidden layers. The "many" here does not have a special metric. According to the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since there are many DNN layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscript corresponds to the output of the third layer index 2 and the input of the second layer index 4. In summary, the coefficient from the kth neuron in the L-1th layer to the jth neuron in the Lth layer is defined as It's important to note that the input layer has no W parameter. In deep neural networks, more hidden layers allow the network to better capture complex real-world situations. Theoretically, a model with more parameters has higher complexity and greater "capacity," meaning it can handle more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrices for all layers of a trained deep neural network (a weight matrix formed by the vectors W across many layers).

[0101] 3. Convolutional Neural Networks

[0102] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. A deep learning architecture involves multiple levels of learning at different levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network in which individual neurons respond to input images. A CNN consists of a feature extractor consisting of convolutional and pooling layers. The feature extractor can be thought of as a filter, and the convolution process can be thought of as convolving an input image or feature map with a trainable filter.

[0103] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter that extracts specific information from the input image matrix. A convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation, the weight matrix is ​​typically applied horizontally to the input image, pixel by pixel (or two pixels by two pixels, depending on the stride), to extract specific features from the image. The size of the weight matrix should be proportional to the image size. It is important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during the convolution operation, the weight matrix extends across the entire depth of the input image. Therefore, convolution with a single weight matrix produces a convolved output with a single depth dimension. However, in most cases, multiple weight matrices of the same size (row × column) are applied instead. This is known as multiple homogeneous matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. The dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features from the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors in the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by these multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation. In practical applications, the weight values ​​in these weight matrices require extensive training. The weight matrices formed by the trained weight values ​​can be used to extract information from the input image, allowing the convolutional neural network to make accurate predictions. When a convolutional neural network has multiple convolutional layers, the initial convolutional layers often extract more general features, which can also be called low-level features. As the depth of the convolutional neural network increases, the features extracted by subsequent convolutional layers become increasingly complex, such as high-level semantic features. Features with higher semantics are more applicable to the problem being solved.

[0104] Because it's often necessary to reduce the number of trainable parameters, pooling layers are often periodically introduced after convolutional layers. This can be done in a single convolutional layer followed by a pooling layer, or in a multi-layered system followed by one or more pooling layers. In image processing, the sole purpose of a pooling layer is to reduce the spatial size of an image. Pooling layers can include average pooling and / or max pooling operators, which are used to downsample the input image to produce a smaller image. The average pooling operator calculates the average value of pixel values ​​within a specific range, producing the average pooling result. The max pooling operator takes the pixel with the largest value within a specific range as the max pooling result. Furthermore, just as the size of the weight matrix used in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The output image size after processing by a pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the output image represents the average or maximum value of the corresponding subregion of the image input to the pooling layer.

[0105] After being processed by the convolution layer / pooling layer, the convolutional neural network is still not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained based on relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0106] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the backpropagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0107] 4. Recurrent Neural Networks

[0108] Recurrent neural networks (RNNs) are designed to process sequential data. In traditional neural network models, layers are fully connected, from the input layer to the hidden layer to the output layer, while nodes within each layer are disconnected. While these conventional neural networks have solved many difficult problems, they are still inadequate for many others. For example, to predict the next word in a sentence, you generally need to use the previous words, as the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is dependent on the previous output. Specifically, the network memorizes previous information and applies it to the calculation of the current output. This means that nodes within the hidden layer are no longer disconnected but connected, and the input to a hidden layer includes not only the output of the input layer but also the output of the previous hidden layer. In theory, RNNs can process sequence data of any length. Training an RNN is similar to training a traditional CNN or DNN. This approach also uses the backpropagation algorithm, but with one key difference: if the RNN is expanded, its parameters, such as W, are shared; this is not the case with traditional neural networks, as in the example above. Furthermore, when using gradient descent, the output of each step depends not only on the state of the network at the current step but also on the state of the network at several previous steps. This learning algorithm is called backpropagation through time (BPTT).

[0109] Since we already have convolutional neural networks, why do we still need recurrent neural networks? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, such as cats and dogs. However, in the real world, many elements are interconnected, such as the changes in stock prices over time. Or, for example, someone says, "I love traveling, and my favorite place is Yunnan. I must visit it someday." Humans should know to fill in the blank with "Yunnan." This is because humans make inferences based on context, but how can machines do the same? That's where RNNs come in. RNNs aim to give machines the ability to remember, like humans do. Therefore, the output of an RNN depends on both the current input and historical memory.

[0110] 5. Loss Function

[0111] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, before the first update, there is usually an initialization process, which pre-configures the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss.

[0112] 6. Backpropagation algorithm

[0113] Convolutional neural networks can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during training, reducing the reconstruction error loss of the super-resolution model. Specifically, the forward propagation of the input signal to the output generates error loss. This error loss information is then backpropagated to update the parameters of the initial super-resolution model, thereby converging the error loss. The BP algorithm is a backward propagation movement dominated by error loss, aiming to obtain the optimal compression model parameters, such as the weight matrix.

[0114] As convolutional neural networks (CNNs) surpass traditional algorithms in computer vision tasks like image recognition and object detection, a growing number of researchers are exploring deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms. For example, these algorithms simultaneously optimize the encoding network, entropy estimation network, entropy encoding network, entropy decoding network, decoding network, and inter-frame prediction modules as a whole. The encoding and decoding networks, also known as the transform and inverse transform modules, typically consist of convolutional layers and nonlinear transformation units.

[0115] FIG1A is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that may utilize the techniques of the present application. Video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in video decoding system 10 represent, among other things, devices that may be used to perform the techniques according to various examples described herein.

[0116] As shown in FIG. 1A , a decoding system 10 includes a source device 12 for providing encoded image data 21 such as an encoded image to a destination device 14 for decoding the encoded image data 21 .

[0117] The source device 12 includes an encoder 20 , and optionally, may include a video source 16 , a pre-processor (or pre-processing unit) 18 such as an image pre-processor, and a communication interface (or communication unit) 22 .

[0118] Video source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The video source may be any type of memory or storage that stores any of the aforementioned images.

[0119] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0120] The preprocessor 18 is configured to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color grading, or denoising. It will be appreciated that the preprocessor 18 may be an optional component.

[0121] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide encoded image data 21 (which will be further described below with reference to FIG. 2 and the like).

[0122] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0123] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0124] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0125] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0126] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0127] The communication interface 28 corresponds to the communication interface 22 , and can be used, for example, to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0128] Both the communication interface 22 and the communication interface 28 can be configured as a unidirectional communication interface as indicated by the arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 in Figure 1A, or a bidirectional communication interface, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0129] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (which will be further described below with reference to FIG. 3 and the like).

[0130] The post-processor 32 is configured to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image, to obtain post-processed image data 33, such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color grading, cropping, or resampling, or any other processing for generating the decoded image data 31 for display on a display device 34, etc.

[0131] The display device 34 is configured to receive the post-processed image data 33 and display the image to a user or viewer. The display device 34 may be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or monitor. For example, the display screen may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display screen.

[0132] The decoding system 10 further includes a training engine 25 for training the encoder 20 and / or the decoder 30 to achieve conversion between the image domain and the feature domain.

[0133] In the embodiment of the present application, the training data can be stored in a database (not shown), and the training engine 25 trains the encoding / decoding network, i.e., the system 20 30, based on the training data. It should be noted that the embodiment of the present application does not limit the source of the training data. For example, the training data can be obtained from the cloud or other places for model training.

[0134] In the embodiments of the present application, the compression model training can be performed online or offline. For example, the online method can use Huawei Cloud. The offline method can use a server, server cluster, or terminal, etc., which is not limited in this application.

[0135] Although FIG1A shows source device 12 and destination device 14 as separate devices, device embodiments may also include both source device 12 and destination device 14 or the functionality of both source device 12 and destination device 14, that is, both source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality. In these embodiments, source device 12 or the corresponding functionality and destination device 14 or the corresponding functionality may be implemented using the same hardware and / or software or through separate hardware and / or software, or any combination thereof.

[0136] According to the description, the existence and (accurate) division of different units or functions in the source device 12 and / or the destination device 14 shown in FIG. 1A may vary depending on actual devices and applications, which is obvious to those skilled in the art.

[0137] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both, may be implemented using processing circuitry as shown in FIG. 1B , such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, a dedicated video encoding processor, or any combination thereof. The encoder 20 may be implemented using processing circuitry 46 to include the various modules discussed with reference to the encoder 20 in FIG. 2 and / or any other encoder systems or subsystems described herein. The decoder 30 may be implemented using processing circuitry 46 to include the various modules discussed with reference to the decoder 30 in FIG. 3 and / or any other decoder systems or subsystems described herein. The processing circuitry 46 may be used to perform the various operations discussed below. Alternatively, if portions of the technology are implemented in software, the device may store the software's instructions on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the present technology. One of the video encoder 20 and the video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC), as shown in FIG. 1B .

[0138] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook computer or laptop, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0139] In some cases, the video decoding system 10 shown in FIG1A is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data to a memory and / or retrieve and decode data from a memory.

[0140] 1B is an illustrative diagram of an example video decoding system 40. Video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video codec implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory stores 44, and / or a display device 45.

[0141] 1B , imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 are capable of communicating with one another. In various embodiments, video decoding system 40 may include only video encoder 20 or only video decoder 30.

[0142] In some instances, antenna 42 can be used to transmit or receive an encoded bitstream of video data. Additionally, in some instances, display device 45 can be used to present the video data. Processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Video decoding system 40 can also include an optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, a graphics processor, a general-purpose processor, and the like. Furthermore, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory storage 44 can be implemented as cache memory. In other instances, processing circuitry 46 can include memory (e.g., cache memory, etc.) for implementing an image buffer, etc.

[0143] In some examples, video encoder 20 implemented by logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuitry 46 to implement the various modules discussed with reference to FIG. 2 and / or any other encoder systems or subsystems described herein. Logic circuitry may be used to perform the various operations discussed herein.

[0144] In some examples, the video decoder 30 may be implemented by processing circuitry 46 in a similar manner to implement the various modules discussed with reference to the video decoder 30 of FIG. 3 and / or any other decoder systems or subsystems described herein. In some examples, the logic circuitry-implemented video decoder 30 may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the various modules discussed with reference to FIG. 3 and / or any other decoder systems or subsystems described herein.

[0145] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames, indicators, index values, mode selection data, etc., as discussed herein, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0146] It should be understood that for the examples described herein with reference to video encoder 20, video decoder 30 can be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be configured to receive and parse such syntax elements and decode the associated video data accordingly. In some examples, video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the associated video data accordingly.

[0147] For ease of description, embodiments of the present invention are described with reference to the Versatile Video Coding (VVC) reference software or High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC.

[0148] FIG2 is a schematic diagram of a video decoding device 400 provided in an embodiment of the present invention. Video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, video decoding device 400 may be a decoder, such as video decoder 30 in FIG1A , or an encoder, such as video encoder 20 in FIG1A .

[0149] Video decoding device 400 includes: an input port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit, or central processing unit (CPU) 430 for processing data; for example, processor 430 may be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. Video decoding device 400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to input port 410, receiver unit 420, transmitter unit 440, and output port 450 for outputting or transmitting optical or electrical signals.

[0150] The processor 430 is implemented in hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., a decoding module 470 based on a neural network NN). The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functionality of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented by instructions stored in the memory 460 and executed by the processor 430.

[0151] Memory 460 includes one or more disks, tape drives, and solid-state drives and can be used as overflow data storage for storing programs when such programs are selected for execution, and for storing instructions and data read during program execution. Memory 460 can be volatile and / or non-volatile and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0152] FIG3 is a simplified block diagram of an apparatus 500 according to an exemplary embodiment. The apparatus 500 may be used as either or both of the source device 12 and the destination device 14 in FIG1A .

[0153] The processor 502 in the apparatus 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices, now available or developed in the future, capable of manipulating or processing information. While the disclosed implementations may be implemented using a single processor, such as the processor 502 shown, using more than one processor may provide greater speed and efficiency.

[0154] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that are accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and application programs 510, which include at least one program that allows the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1 through N, as well as a video decoding application that performs the methods described herein.

[0155] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with touch-sensitive elements that can be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0156] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. Furthermore, secondary storage may be directly coupled to other components of device 500 or accessed via a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may have a variety of configurations.

[0157] Figure 4 is an example diagram of an application scenario of an embodiment of the present application. As shown in Figure 4, the application scenario can be a terminal, cloud server, or video surveillance service involving image / video acquisition, storage, or transmission, such as terminal photo / video recording, photo album, cloud photo album, video surveillance, etc.

[0158] Encoding: The camera captures images / videos. The artificial intelligence (AI) image / video encoding network extracts features from the images / video to obtain low-redundancy image features. It then compresses these features to produce a bitstream / image file.

[0159] Decoding end: When images / videos need to be output, the AI ​​image / video decoding network decompresses the code stream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain the reconstructed image / video.

[0160] The storage / transmission module stores the compressed code stream / image files for different services (for example, terminal photography, video surveillance, cloud servers, etc.) or transmits them (for example, cloud services, live broadcast technology, etc.).

[0161] Figure 5 is an example diagram of an application scenario of an embodiment of the present application. As shown in Figure 7, the application scenario can be a business involving image / video acquisition, storage or transmission in a terminal or video surveillance, such as a terminal photo album, video surveillance, live broadcast, etc.

[0162] Encoding: The encoding network transforms the image / video into image features with lower redundancy. The entropy estimation network calculates the coding probability of each data point in the image features. Based on the corresponding probabilities of each data point, the entropy coding network performs lossless encoding on the image features to produce a bitstream / image file, further reducing the amount of data transmitted during the image compression process.

[0163] Decoding: The entropy decoding network performs lossless decoding on the bitstream / image file based on the probabilities corresponding to each data point to reconstruct image features. The decoding network performs an inverse transformation on the image features output by entropy decoding, resolving them into an image / video. The saving module saves the bitstream / image file to the corresponding storage location on the terminal. The loading module loads the bitstream / image file from the corresponding storage location on the terminal and inputs it into the entropy decoding network.

[0164] Optionally, the AI ​​encoding unit, the AI ​​decoding unit and the entropy estimation unit may adopt a network processor (NPU), and the entropy encoding network, the entropy decoding network, the saving module and the loading module may adopt a central processing unit (CPU).

[0165] Figure 6 is a schematic diagram of the structure of the residual coding and decoding framework. Referring to Figure 6, the residual coding and decoding framework (also referred to as the residual coding and decoding model, which is not limited in this application) specifically includes but is not limited to: an image optical flow estimation module, an optical flow compression module, a prediction module, and a residual compression module. Optionally, the model may also include an image reconstruction module and a feature extraction module. It should be noted that the connection relationship in Figure 6 is only an illustrative example. The specific interaction relationship between the sub-modules in each module can be referred to the description in Figure 7, which is not limited here.

[0166] Exemplarily, the image optical flow estimation module may include a motion estimation module. The motion estimation module is a neural network model whose input is the current frame and the reference frame, and whose output is the optical flow.

[0167] The optical flow compression module includes but is not limited to an optical flow entropy estimation module, an optical flow encoding module, and an optical flow decoding module. The optical flow compression module is used to compress the optical flow output by the image optical flow estimation module and output the compressed optical flow.

[0168] The prediction module may include a motion compensation module, which is used to predict the current frame based on the reference frame (ie, the reconstructed image of the previous frame) and the compressed optical flow to obtain a predicted image.

[0169] The residual compression module, including but not limited to the residual entropy estimation module, the residual encoding module, and the residual decoding module, compresses the residual between the original image and the predicted image of the current frame and decompresses it to obtain the reconstructed residual. The reconstructed residual and the predicted image are summed to obtain the reconstructed image of the current frame.

[0170] Optionally, the optical flow entropy estimation module and the residual entropy estimation module can be understood as components of the entropy estimation network. The optical flow encoding module, the motion estimation module, and the residual encoding module can be understood as components of the encoding network. The optical flow decoding module and the residual decoding module can be understood as components of the decoding network.

[0171] 6 , FIG7 is a schematic diagram showing an exemplary processing flow of a residual encoding and decoding structure. Referring to FIG7 , the processing flow includes but is not limited to the following steps:

[0172] Step 1. Feature extraction module for the current frame x t and reference frame Perform feature extraction and get x t The image feature f t , The image feature f t-1 .

[0173] Among them, the reference frame The reconstructed image corresponding to the previous frame. In the embodiments of the present application, the reference frame may refer to the previous frame or the reconstructed image corresponding to the previous frame. It should be noted that the "previous frame" involved in the embodiments of the present application may be a previous frame adjacent to the current frame or a non-adjacent frame, and this application does not limit this.

[0174] Step 2. The motion estimation module is based on the current frame x t and the reconstructed image of the reference frame Get optical flow v t .

[0175] The reference frame can also be understood as the previous frame, and the reconstructed image of the reference frame is output from the image reconstruction module.

[0176] Step 3. The optical flow encoding module encodes the optical flow v t Perform optical flow compression (i.e., optical flow encoding) and output optical flow compression features.

[0177] Step 4. The optical flow entropy estimation module makes a probability prediction on the optical flow compression feature, and performs optical flow entropy coding based on the result of the probability prediction to obtain the optical flow code stream. The code rate of the optical flow code stream is

[0178] Step 5. The optical flow decoding module decodes the optical flow code stream according to the optical flow compression feature to obtain the decoded optical flow

[0179] Step 6. The prediction module is based on the image features f of the reconstructed image of the reference frame t-1 and decoded optical flow Make a prediction and get the predicted features of the current frame (also called predicted image)

[0180] Step 7. The residual coding module is based on the prediction features of the current frame Perform residual coding (also called residual compression) on the image feature f of the current frame to obtain residual compression features.

[0181] The residual calculation process may optionally be: subtracting the pixel value of the prediction block from the pixel value of the image block pixel by pixel (pixel by pixel) to obtain a residual block in the pixel domain.

[0182] Step 8. The residual entropy estimation module calculates the probability of the residual compression feature (also called the residual coding probability) based on the residual compression feature. And performs residual entropy coding based on the residual compression probability to obtain the residual code stream. Among them, the code rate of the residual code stream is

[0183] Step 9. The residual decoding module performs residual decoding on the residual code stream based on the residual compression feature to obtain the residual

[0184] Step 10. Image reconstruction module based on predicted features and residuals Reconstruct the image of the current frame to obtain the reconstructed image of the current frame

[0185] It should be noted that the residual coding and decoding structure in the embodiment of the present application (such as FIG. 7 ) is only an illustrative example and is not limited in this application. Moreover, the specific details of each step in FIG. 7 can be referred to the existing technical embodiments and will not be repeated in this application.

[0186] It should be further explained that the process shown in FIG. 7 can be understood as an encoding process or a decoding process, and this application does not limit this.

[0187] For example, during the training of the network in FIG7 (hereinafter referred to as the residual coding model), the Loss module can obtain the loss function of the residual coding model and, based on the loss function, cyclically train the residual coding model. The specific training process is as follows:

[0188] Step 1. A training device (such as Huawei Cloud or a server, etc., not limited in this application) trains the GOP.

[0189] Exemplarily, a plurality of image frames may constitute a GOP (Group of Pictures). Optionally, a GOP may include one or more I frames and one or more P frames.

[0190] FIG8 is a schematic diagram illustrating an exemplary GOP. Referring to FIG8 , the GOP includes, for example, an I1 frame, a P1 frame, a P2 frame, an I2 frame, and a P3 frame. It should be noted that the type, quantity, and order of the image frames in the GOP in FIG8 are merely illustrative examples and are not limited in this application.

[0191] For example, an I-frame can be called a complete frame or an independently decodable frame, meaning that an I-frame can be decoded independently without relying on other frames. Typically, the first frame of a GOP (i.e., the first frame) is an I-frame, and a P-frame can be called an inter-frame predicted frame. The decoding of a P-frame depends on the most recent preceding I-frame or P-frame. For example, a P1 frame's decoding depends on an I1 frame, and a P2 frame's decoding depends on a P1 frame.

[0192] For example, during GOP training, the training device compresses (i.e., encodes) the I-frame using an image compression model to obtain the bitrate and reconstructed image for the I-frame. The image compression model corresponding to the I-frame (also referred to as an image compression neural network, not limited in this application) may include, but is not limited to, an encoding network, a decoding network, and an image reconstruction network. The specific model can refer to any existing technology and is not limited in this application.

[0193] For example, the training device can compress each P frame in the GOP through the residual coding model shown in FIG7 to obtain the reconstructed image corresponding to each P frame. and code rate R i Among them, the bit rate of P frame R i Including residual bit rate and optical flow rate

[0194] Step 2. Training the device based on each P frame and the reconstructed image of each P frame and code rate R i , and the bit rate R0 of the I frame, obtain the loss function loss of the compression model.

[0195] Exemplarily, the training device calculates the loss function of the compression model based on formula (2):

[0196] Among them, R0 is the bit rate of the I frame. Represents the loss function of a single frame (including P-frames and I-frames). D i Indicates the distortion loss value of each frame. The training device can be based on the P frame (i.e., x in Figure 7 t ) and the reconstructed image of the P frame Determine the distortion loss value D corresponding to the P frame i β i Used to adjust the rate-distortion ratio. I-frame is the same and will not be repeated here.

[0197] Step 3. Based on the loss function of the compression model, iteratively train the compression model.

[0198] For example, as described above, for training a neural network, the weights in the compression model can be adjusted based on the loss function, and the above training steps can be repeated until the training cutoff condition is reached. The compression model includes a neural network for training I frames and a neural network for training P frames (e.g., the residual coding model in FIG7 ).

[0199] In the embodiments of the present application, the training cutoff conditions include, but are not limited to: the loss function reaching a loss function threshold, the number of training times reaching a training number threshold, and / or the fluctuation value of the loss function remaining within a certain range. The above thresholds can be set according to actual needs and are not limited by this application.

[0200] For example, the residual coding model shown in Figure 7 can achieve end-to-end video compression with high RD performance. However, due to the limited number of training sets during training (for example, a GOP typically consists of 6 to 10 image frames), it is difficult to fully simulate a real-world test environment (for example, a GOP in a real-world test environment may be up to 100 image frames) when using the residual coding model in Figure 7 for video residual compression. Therefore, using the residual coding model for video compression will result in a gradually increasing cumulative error. The compression quality of traditional VTM compression fluctuates steadily. Figure 9 shows an example of PSNR values ​​for VTM compression and PSNR variations for AI residual compression. As shown in Figure 9 (the horizontal axis is image frames, the vertical axis is PSNR), the dotted line represents the quality of the reconstructed image using VTM, and the solid line represents the quality of the reconstructed image using AI residual compression. The quality of the VTM reconstructed image is very stable, with no significant degradation. However, the quality of the AI ​​residual compression reconstructed image degrades significantly, and the corresponding cumulative error gradually increases. This phenomenon of cumulative error is particularly pronounced for static and simple motion scenes.

[0201] Figure 10 is a schematic diagram of the encoding and decoding framework of the condition I (condition I, CI) frame. Please refer to Figure 10. The CI frame (also known as the adversarial frame, etc., which is not limited in this application) encoding and decoding framework (hereinafter referred to as the CI frame encoding model) includes but is not limited to: an image optical flow estimation module, an optical flow compression module, a prediction module, and a feature compression module. Optionally, the model can also include an image reconstruction module and a feature extraction module. It should be noted that the connection relationship in Figure 10 is only an illustrative example. The specific interaction relationship between the sub-modules in each module can be referred to the description in Figure 11, which is not limited here.

[0202] Exemplarily, the image optical flow estimation module may include a motion estimation module. The motion estimation module is a neural network model whose input is the current frame and the reference frame, and whose output is the optical flow.

[0203] The optical flow compression module includes but is not limited to an optical flow entropy estimation module, an optical flow encoding module, and an optical flow decoding module. The optical flow compression module is used to compress the optical flow output by the image optical flow estimation module and output the compressed optical flow.

[0204] The prediction module may include a motion compensation module, which is used to predict the current frame based on the reference frame (ie, the reconstructed image of the previous frame) and the compressed optical flow to obtain a predicted image or predicted features.

[0205] The feature compression module includes but is not limited to a feature entropy estimation module, a feature encoding module, and a feature decoding module. The feature compression module, on the one hand, compresses the features of the original image of the current frame (ie, image features), and on the other hand, decompresses the features to obtain reconstructed features.

[0206] Optionally, the optical flow entropy estimation module and the feature entropy estimation module can be understood as components of the entropy estimation network. The optical flow encoding module, the motion estimation module, and the feature encoding module can be understood as components of the encoding network. The optical flow decoding module and the feature decoding module can be understood as components of the decoding network.

[0207] 10 , FIG11 is a schematic diagram illustrating a processing flow of a CI frame encoding and decoding structure. Referring to FIG11 , the processing flow includes but is not limited to the following steps:

[0208] Step 1. Feature extraction module for the current frame x t Perform feature extraction and get x t The image feature f t .

[0209] Step 2. The motion estimation module is based on the current frame x t and the reconstructed image of the reference frame Get optical flow v t .

[0210] The reference frame can also be understood as the previous frame, and the reconstructed image of the reference frame is output from the image reconstruction module.

[0211] Step 3. The optical flow encoding module encodes the optical flow v t Perform optical flow compression (i.e., optical flow encoding) and output optical flow compression features.

[0212] Step 4. The optical flow entropy estimation module makes a probability prediction on the optical flow compression feature, and performs optical flow entropy coding based on the result of the probability prediction to obtain the optical flow code stream. The code rate of the optical flow code stream is

[0213] Step 5. The optical flow decoding module decodes the optical flow code stream according to the optical flow compression feature to obtain the decoded optical flow

[0214] Step 6. The prediction module is based on the image features f of the reconstructed image of the reference frame t-1 and decoded optical flow Make a prediction and get the predicted features of the current frame (it may also be a predicted image)

[0215] The image features of the reconstructed image of the reference frame are: The feature extraction module extracts features based on the reconstructed image of the reference frame (i.e., the previous frame) output by the image reconstruction module, and obtains the image features f of the reconstructed image of the reference frame. t-1 .

[0216] Step 7. The feature encoding module encodes the image features f of the current frame t Perform feature encoding (also called feature compression) to obtain image compression features.

[0217] Step 8. The feature entropy estimation module calculates the image compression feature probability (also called image feature coding probability) based on the image compression feature and the prediction feature. It then performs feature entropy coding based on the image compression feature probability to obtain a feature code stream. The code rate of the feature code stream is

[0218] Step 9. The feature decoding module decodes the feature code stream based on the image compression feature to obtain the feature

[0219] Step 10. Image reconstruction module based on features Reconstruct the image of the current frame to obtain the reconstructed image of the current frame

[0220] For example, the structure of the CI frame coding model is similar to that of the residual coding model. The difference is that in steps 7 to 10, the prediction features in the CI frame coding model are It is not used for image reconstruction, but rather serves as a priori condition for probabilistically estimating image features, thereby reducing inter-frame redundancy (i.e., residual error). In other words, the bitrate of a CI frame is derived based on the coding probabilities corresponding to the CI frame's image features. The CI frame's reference frame (e.g., the reconstructed image of the previous P frame) is used to obtain the coding probabilities of the CI frame's image features.

[0221] For example, during the training of the neural network in FIG11 (hereinafter referred to as the CI frame coding model), the Loss module can obtain the loss function of the CI frame coding model and, based on the loss function, cyclically train the residual coding model. The specific training process is as follows:

[0222] Step 1. A training device (such as Huawei Cloud or a server, etc., not limited in this application) trains the GOP.

[0223] For example, during GOP training, the training device compresses (i.e., encodes) the I-frame using an image compression model to obtain the bitrate and reconstructed image for the I-frame. The image compression model corresponding to the I-frame (also referred to as an image compression neural network, not limited in this application) may include, but is not limited to, an encoding network, a decoding network, and an image reconstruction network. The specific model can refer to any existing technology and is not limited in this application.

[0224] For example, the training device can compress each P frame in the GOP through the CI frame coding model shown in FIG11 to obtain the reconstructed image corresponding to each P frame. and code rate R i Among them, the bit rate of P frame R i Including characteristic code rate and optical flow rate

[0225] Step 2. Training the device based on each P frame and the reconstructed image of each P frame and code rate R i , and the bit rate R0 of the I frame, obtain the loss function loss of the compression model.

[0226] Exemplarily, the training device calculates the loss function of the compression model based on formula (2):

[0227] Among them, R0 is the bit rate of the I frame. Represents the loss function of a single P frame. D i Indicates the distortion loss value of each frame. The training device can be based on the P frame (i.e., x in Figure 11 t ) and the reconstructed image of the P frame Determine the distortion loss value D corresponding to the P frame i β i Used to adjust the rate-distortion ratio.

[0228] Optionally, the training device may also obtain the loss value of the compression model based on formula (2), which is not limited in this application.

[0229] Step 3. Based on the loss function of the compression model, iteratively train the compression model.

[0230] For example, as described above, for training a neural network, the weights in the compression model can be adjusted based on the loss function, and the above training steps can be repeated until the training cutoff condition is reached. The compression model includes a neural network for training I frames and a neural network for training P frames (e.g., the CI frame encoding model in FIG11 ).

[0231] In the embodiments of the present application, the training cutoff conditions include, but are not limited to: the loss function reaching a loss function threshold, the number of training times reaching a training number threshold, and / or the fluctuation value of the loss function remaining within a certain range. The above thresholds can be set according to actual needs and are not limited by this application.

[0232] For example, the CI frame coding model shown in Figure 11 can implement an end-to-end video compression framework. Its decoder only uses the image features of the current frame during image reconstruction, resulting in no cumulative error. However, because the CI frame coding model uses image features for probabilistic prediction and derives the reconstructed image, its prediction results may be inaccurate, resulting in weaker RD performance than the residual coding model.

[0233] An embodiment of the present application provides a compression model, as shown in FIG12 , which may also be referred to as a codec model or a codec neural network, etc., which is not limited in this application. The compression model is used to compress training data in training and image sequences in practical scenarios (such as test scenarios). The compression model includes, but is not limited to: a first codec model (also referred to as a first neural network), a second codec model (also referred to as a second neural network), and a third codec model (also referred to as a third neural network).

[0234] In the embodiment of the present application, the first codec model is used to compress (ie, encode and / or decode) I frames, the second codec model is used to compress P frames, and the third codec model is used to compress CI frames.

[0235] The first encoding / decoding model includes, but is not limited to, an encoding network, a decoding network, and an entropy estimation network. The first encoding model compresses the I-frame as input, outputting the bitrate and reconstructed image of the I-frame. The first encoding / decoding model can employ any architecture, which is not limited in this application.

[0236] In an embodiment of the present application, the second codec model may optionally be the residual codec model shown in Figure 6 or Figure 7. The P frame and the reconstructed image of the reference frame serve as input to the second codec model, which compresses the P frame and outputs the bit rate of the P frame and the reconstructed image of the P frame.

[0237] In an embodiment of the present application, the third codec model may optionally be the CI frame codec model shown in Figure 10 or Figure 11. The CI frame and the reconstructed image of the reference frame serve as input to the third codec model, which compresses the CI frame and outputs the bit rate of the CI frame and the reconstructed image of the CI frame.

[0238] Based on the compression model shown in Figure 12, an embodiment of the present application provides a training method that can reduce the loss value of the compression model by adding several frames during the training process, thereby reducing the cumulative error of the trained compression model during use (for example, during the testing process).

[0239] In one possible implementation method, in an embodiment of the present application, a CI frame can be added to compete with the P frame. The code prior condition of the CI frame is the characteristics of the reconstructed image of the P frame. The loss of the compression model during the training process is reduced, and the quality of the P frame is indirectly constrained to be stable, so as to achieve the effect of reducing the cumulative error.

[0240] In another possible implementation, the embodiment of the present application can simulate the GOP length that may be encountered in the test scenario by adding inserted frames (also called virtual P frames), so that the cumulative error phenomenon can be encountered during the training process, so that the cumulative error of the trained compression model in the actual application scenario can be reduced.

[0241] First, the method of adding CI frames for training is described in detail.

[0242] FIG13 is a diagram of a training image sequence. Referring to FIG13 , in this example, the GOP of the training image sequence is set to n. For example, n can be 6 or 7, that is, the GOP includes 6 image frames. This application does not make specific restrictions. Among them, the first frame x0 of the GOP is an I frame, and the other frames x1, x2…, x n-1The GOP is a P frame, that is, the GOP is an IPPPP structure. During the training process, the GOP is defined by an I frame, that is, the first frame of each GOP is an I frame, and each GOP includes only one I frame. The GOP structure in the embodiment of the present application is only an illustrative example to better illustrate the difference between I frames and CI frames. In actual applications, a GOP may also include but is not limited to multiple I frames (such as the GOP structure shown in Figure 8), which is not limited in this application.

[0243] Still referring to Figure 13, in the training image sequence, a CI frame is added to the end of the GOP, that is, the last frame in the training image sequence is a CI frame. Optionally, if the training image sequence includes multiple I frames, at least one of the middle I frames (that is, except the first frame) can also be replaced with a CI frame, that is, the original I frame is trained using the third encoding and decoding model. Optionally, the training image sequence can also include more image frames, for example, multiple GOPs. The CI frame in the GOP in Figure 13 can also be the first I frame of the next GOP. The setting method of the training image sequence can be set according to actual needs, and this application does not limit it. This application only uses the image sequence in Figure 13 as an example for schematic explanation. Optionally, the CI frame can also replace the P frame, and this application does not limit it.

[0244] As shown in Figure 13, in an embodiment of the present application, the first encoding and decoding model in the compression model compresses the I frame and outputs the compression information of the I frame (including but not limited to the bit rate and reconstructed image, etc.). The second encoding and decoding model compresses the P frames one by one, wherein the reconstructed image of the reference frame of each P frame (for example, the I frame or the previous P frame) is used for the compression process of the P frame. The second coding model outputs the compression information corresponding to a single P frame (including but not limited to the bit rate and reconstructed image, etc.). The third coding model compresses the CI frame, wherein the reconstructed image of the reference frame of the CI frame (for example, the previous P frame) is used for the compression process of the CI frame. The third coding model outputs the compression information of the CI frame (including but not limited to the bit rate and reconstructed image, etc.).

[0245] It should be noted that, as described above, the GOP in the embodiment of the present application is an IPPPP structure. In other embodiments, the training sequence may include multiple I frames.

[0246] FIG14 is a schematic diagram of an exemplary training process. Referring to FIG14 , the specific training process is as follows:

[0247] Step 1: Compress the I frame using the first encoding and decoding model, and output the bit rate and reconstructed image of the I frame.

[0248] For example, the I frame (denoted as x0) in the training image sequence is used as the input of the first codec model, and the first codec model compresses the I frame and outputs the bit rate R0 of the I frame and the reconstructed image

[0249] Step 2: Compress n-1 P frames one by one through the second encoding and decoding model, and output the bit rate R of a single P frame. i and reconstructed image

[0250] For example, as shown in FIG14 , the compression process of a P frame x1 (also referred to as x1 frame, hereinafter referred to as x1, other frames are similar and will not be described below) is taken as an example. x1 is used as the input of the second codec model, which compresses x1 and outputs the bit rate R1 of x1 (i.e., the P frame) and the reconstructed image. Among them, the reconstructed image of the I frame The compression process of x1 by the second coding model can be referred to in detail in the compression process of the residual coding model in FIG7 , which will not be described in detail here.

[0251] Still referring to FIG14, x2 is used as the input of the second codec model, the second codec model compresses x2, and outputs the code rate R2 of x2 (ie, P frame) and the reconstructed image Among them, the reconstructed image of frame x1 The compression process of x2 for the second coding model can be referred to in detail in the compression process of the residual coding model in FIG7 , which will not be described in detail here.

[0252] For example, as mentioned above, the bit rate R of each P frame is i Including optical flow bit rate and residual bit rate For example, the bitrate R1 of P frame x1 includes the optical flow bitrate and residual bit rate Other P frames are similar and will not be described one by one here.

[0253] For example, the second encoding and decoding model compresses each P frame according to the above process, and the bit rate R corresponding to each P frame can be obtained. i and reconstructed image

[0254] Step 3. Reconstructed image based on n-1 P frames and P frames Calculate the distortion value loss D corresponding to each P frame i ,i=1,2,…,n-1.

[0255] For example, the training device can be based on the original image of each P frame and the reconstructed image corresponding to the P frame. Calculate the distortion value loss D corresponding to each P frame i The calculation method can refer to existing technologies and is not limited in this application.

[0256] Step 4. Compress the CI frame through the third encoding and decoding model and output the code rate R of the CI frame n and reconstructed image

[0257] For example, as shown in FIG14 , the CI frame is used as the input of the third codec model in the compression model. The third codec model compresses the CI frame and outputs the code rate R of the CI frame. n and reconstructed image The reference frame of the CI frame (eg, the previous P frame) is used for the compression process of the CI frame by the third coding / decoding model. The specific training process can be referred to the relevant description of FIG11 and will not be repeated here.

[0258] For example, the code rate of the CI frame is R n Including optical flow bit rate and feature code rate

[0259] Step 5. CI frame and reconstructed image based on CI frame Calculate the distortion value loss D corresponding to the CI frame n .

[0260] For example, the training device can be based on the original image of the CI frame and the reconstructed image corresponding to the CI Calculate the distortion value loss D corresponding to the CI frame n The calculation method can refer to existing technologies and is not limited in this application.

[0261] Step 6. Get the loss value loss of the compression model.

[0262] Exemplarily, the training device may calculate the loss value corresponding to the compression model in this iteration based on formula (2) or formula (3).

[0263] Step 7. Update the compression model parameters based on the loss value loss.

[0264] For example, the training device may determine whether a training cutoff condition is met based on the loss value of the compression model. In embodiments of the present application, the training cutoff condition includes, but is not limited to: the loss function reaches a loss function threshold, the number of training times reaches a training number threshold, and / or the fluctuation value of the loss function remains within a certain range. The above thresholds can be set according to actual needs and are not limited in this application.

[0265] In one example, if the training deadline is met, the training is stopped and the trained compressed model is saved.

[0266] In another example, as shown in FIG14 , if the training cutoff condition is not met, the weight value of the compression model (ie, the parameters of the compression model) is updated, and the training is iterated again, that is, training steps 1 to 7 are repeated.

[0267] In one possible implementation, in order to make the CI frame have higher quality (ie, the quality of the reconstructed image is higher), when calculating the loss value loss of the CI frame, its corresponding β value can be a multiple of the β value corresponding to other P frames, and the multiple is greater than 1.

[0268] In another possible implementation, the length of the GOP in each iteration may be gradually lengthened, which is not limited in this application.

[0269] In another possible implementation, the training image sequence used in each iteration process may be the same or different, which is not limited in this application.

[0270] In another possible implementation, the trained compression model can be deployed on a target electronic device, which can be a terminal or other device, and is not limited in this application. The compression model can compress (i.e., encode or decode) the image captured by the electronic device. In actual use, the first codec model in the compression model can be used to compress the first I frame of each sequence. The second codec model can be used to compress the P frame in the sequence. The third codec model can be used to compress at least one intermediate CI frame in each sequence. The specific usage method can be set according to actual needs, and is not limited in this application.

[0271] Figure 15 exemplifies the RD performance test results based on the trained compression model shown in Figure 14. As shown in Figure 15, in the test scenario, the compression model is tested on a HEVC video test sequence. The HEVC video test sequence may include multiple test sequences, such as test sequence 1, test sequence 2, or test sequence 3. Optionally, the test sequence may also be class B-D, which is not limited in this application.

[0272] As shown in Figure 15, in the result diagram corresponding to each test sequence, the solid line represents the test results of the compression model in Figure 14 (i.e., in the embodiment of the present application), and the dotted line represents the test results using the traditional codec model. As can be seen from the figure, the RD performance of the compression model in the embodiment of the present application during the test process is significantly better than the RD performance of the traditional codec model. In other words, by adding CI frame training, the RD performance of the trained compression model can be significantly improved.

[0273] Figure 16 exemplifies the test results of the trained compression model shown in Figure 14. As shown in Figure 16, the solid line represents the test results of the compression model in Figure 14 (i.e., the embodiment of the present application), and the dotted line represents the test results using the traditional codec model. As can be seen from the figure, during the test, the compression model in the embodiment of the present application outperformed the traditional compression model in terms of loss, PSNR, and bpp.

[0274] Analysis of the cause: This application replaces the I frame with the CI frame. Since the CI frame predicts the current frame based on the reference frame during the compression process, the CI frame can reduce inter-frame redundancy, so that the code rate of the CI frame is lower than that of the I frame. In addition, during the training process, the reconstructed image of the reference frame of the CI frame (for example, the previous P frame) (wherein the reconstructed image of the previous P frame can also be understood as a reference frame in the compression process of the CI frame) is used in the compression process of the CI frame. Therefore, in order to make the loss of the CI frame smaller and to make the overall loss (i.e., the compression model) smaller, the quality of the reference frame of the CI frame (i.e., the reconstructed image of the previous P frame) needs to be improved. Therefore, in the iterative process, in order to make the loss of the compression model smaller, the smaller the loss of the CI frame, the lower the corresponding code rate, and accordingly, the higher the quality of the reconstructed image of each P frame, thereby achieving a stable constraint on the quality of the reconstructed image of the P frame. In this way, when the trained compression model is put into the test scene, the quality of the reconstructed image corresponding to each compressed image frame is higher than that of the reconstructed image of the traditional compression model, and the loss value is smaller.

[0275] The following is a detailed description of the training method of adding virtual P frames described above. Figure 17 is a schematic diagram of a training image sequence. Referring to Figure 17, in this example, the GOP of the training image sequence is set to n. For example, n can be 6 or 7, that is, the GOP includes 6 image frames. Among them, the first frame x0 of the GOP is an I frame, and the other frames x1, x2..., x n-1 The GOP is a P frame, that is, the GOP is an IPPPP structure. During the training process, the GOP is defined by an I frame, that is, the first frame of each GOP is an I frame, and each GOP includes only one I frame. The GOP structure in the embodiment of the present application is only an illustrative example to better illustrate the difference between I frames and CI frames. In actual applications, a GOP may also include but is not limited to multiple I frames (such as the GOP structure shown in Figure 8), which is not limited in this application.

[0276] Still referring to Figure 17, in the training image sequence, k (k>1) virtual P frames are added to the GOP. In one example, the training device may copy the I frame to obtain k copied virtual P frames, and insert them into any position in the GOP. Optionally, the k virtual P frames are inserted before the n P frames in the original GOP, so that the P frame can be closer to the position in the GOP in the actual test environment. In another example, the training device may copy the P frame to obtain k copied virtual P frames, and insert them into the GOP. In another example, the training device may also copy the I frame and the P frame to obtain k virtual P frames, wherein some of the k virtual P frames are obtained by copying the I frame, and the other part is obtained by copying the P frame, which is not limited in this application.

[0277] As shown in FIG17 , in an embodiment of the present application, the first encoding / decoding model in the compression model compresses the I frame and outputs compression information of the I frame (including but not limited to the bit rate and reconstructed image, etc.). The second encoding / decoding model compresses the P frames (including n-1 P frames and k virtual P frames) one by one, wherein the reconstructed image of the reference frame of each P frame (for example, the I frame or a previous P frame) is used in the compression process of the P frame. The second encoding model outputs compression information corresponding to a single P frame (including but not limited to the bit rate and reconstructed image, etc.).

[0278] FIG18 is a schematic diagram of an exemplary training process. Referring to FIG18 , the specific training process is as follows:

[0279] Step 1: Compress the I frame using the first encoding and decoding model, and output the bit rate and reconstructed image of the I frame.

[0280] For example, the I frame (denoted as x0) in the training image sequence is used as the input of the first codec model, and the first codec model compresses the I frame and outputs the bit rate R0 of the I frame and the reconstructed image

[0281] Step 2: Compress the k virtual P frames one by one through the second encoding and decoding model, and output the code rate R of a single virtual P frame. i ′ and reconstructed image

[0282] For example, as shown in FIG18 , the compression process of a virtual P frame x1′ (also referred to as x1′ frame, hereinafter referred to as x1′, other frames are similar and will not be described below) is taken as an example. x1′ is used as the input of the second codec model, which compresses x1′ and outputs the bit rate R1′ of x1′ (i.e., the virtual P frame) and the reconstructed image. Among them, the reconstructed image of the I frame The compression process of x1′ for the second coding model can be referred to the compression process of the residual coding model in FIG7 , which will not be described in detail here.

[0283] Still referring to FIG18, x2' is used as the input of the second codec model, and the second codec model compresses x2' and outputs the code rate R2' of x2' (ie, the virtual P frame) and the reconstructed image Among them, the reconstructed image of the x1' frame The compression process of x2′ for the second coding model can be referred to the compression process of the residual coding model in FIG7 , which will not be described in detail here.

[0284] For example, as described above, the bit rate R of each virtual P frame is i 'Including optical flow rate and residual bit rate For example, the code rate R1′ of the virtual P frame x1′ includes the optical flow code rate and residual bit rate Other virtual P frames are similar and will not be described one by one here.

[0285] Exemplarily, the second encoding and decoding model compresses each virtual P frame according to the above process, and the bit rate R corresponding to each virtual P frame can be obtained. i ′ and reconstructed image It should be noted that in the subsequent loss calculation process, the bit rate of the virtual P frame does not participate in the loss calculation. However, during the compression process, the compression model still outputs the corresponding bit rate.

[0286] Step 3. Compress n-1 P frames one by one through the second encoding and decoding model, and output the bit rate R of a single P frame. i and reconstructed image

[0287] For example, as shown in FIG18 , the compression process of a P frame x1 (also referred to as x1 frame, hereinafter referred to as x1, other frames are similar and will not be described below) is taken as an example. x1 is used as the input of the second codec model, which compresses x1 and outputs the bit rate R1 of x1 (i.e., the P frame) and the reconstructed image. Among them, the reconstructed image of the I frame The compression process of x1 by the second coding model can be referred to in detail in the compression process of the residual coding model in FIG7 , which will not be described in detail here.

[0288] Still referring to FIG18, x2 is used as the input of the second codec model, and the second codec model compresses x2 and outputs the code rate R2 of x2 (ie, P frame) and the reconstructed image Among them, the reconstructed image of frame x1 The compression process of x2 for the second coding model can be referred to in detail in the compression process of the residual coding model in FIG7 , which will not be described in detail here.

[0289] For example, as mentioned above, the bit rate R of each P frame is i Including optical flow bit rate and residual bit rate For example, the bitrate R1 of P frame x1 includes the optical flow bitrate and residual bit rate Other P frames are similar and will not be described one by one here.

[0290] For example, the second encoding and decoding model compresses each P frame according to the above process, and the bit rate R corresponding to each P frame can be obtained. i and reconstructed image

[0291] Step 4. Reconstructed image based on n-1 P frames and P frames Calculate the distortion value loss D corresponding to each P frame i ,i=1,2,…,n-1.

[0292] For example, the training device can be based on the original image of each P frame and the reconstructed image corresponding to the P frame. Calculate the distortion value loss D corresponding to each P frame i The calculation method can refer to existing technologies and is not limited in this application.

[0293] Step 5. Get the loss value loss of the compression model.

[0294] For example, the training device can calculate the loss value corresponding to the compression model in this iteration based on formula (2). When calculating the loss, the bit rate of the k virtual P frames is not included in the loss calculation. In other words, the training device calculates the loss of the compression model based on the bit rate and distortion values ​​corresponding to the I frame and n-1 P frames.

[0295] Step 6. Update the compression model parameters based on the loss value loss.

[0296] The detailed description can be found above and will not be repeated here.

[0297] In one possible implementation, k can be set based on the length of the image sequence in the test scene. For example, the length of the image sequence in the test scene may be up to 100, while the length of the original GOP in the training scene (i.e., n) is usually 6 (for illustrative purposes only). Accordingly, the value of k can be selected from 1 to 94 (i.e., 100 minus 6).

[0298] In the test scenario, the video HEVC test sequence is tested based on the trained compression model shown in Figure 18 (i.e., including the first encoding model and the second encoding model), wherein the test sequence includes test sequences corresponding to different resolutions. After the compression model in the embodiment of the present application is tested on image sequences corresponding to different resolutions, the corresponding test results show that the compression model in the embodiment of the present application has a smaller BD-rate than the traditional compression model, which can be improved by an average of 2.63%. It can be seen that in the embodiment of the present application, by adding a virtual training sequence, the P frame in the original image can simulate the position in the GOP in the actual test scenario, thereby enhancing the model's ability to reconstruct the entire GOP and reducing the cumulative error during the test.

[0299] FIG19 is a schematic diagram illustrating an exemplary training process. Referring to FIG19 , in an embodiment of the present application, the training of the compression model can be combined with the training methods of FIG14 and FIG18 . That is, CI frames can be added to the GOP, and virtual P frames (also referred to as interpolated frames, not limited in this application) can be added. Specifically, during the training process, the first codec model compresses I frames, the second codec model compresses k virtual P frames and n-1 P frames, and the third codec model compresses CI frames. The loss value of the compression model is obtained, and the compression model parameters are further updated based on the loss value of the compression model. Training is then iterated until the training cutoff condition is met. The specific training process can be referred to the relevant description of FIG14 and FIG18 , and will not be repeated here. Thus, the compression model (including the first codec model, the second codec model, and the third codec model) trained using the training method shown in FIG19 can effectively reduce cumulative error and improve RD performance in actual use. Furthermore, in the embodiments of the present application, the training process can effectively improve various indicators of the compression model without affecting the test complexity in the test scenario.

[0300] FIG20 is a schematic diagram of the structure of an exemplary training device 2000. Referring to FIG20 , it specifically includes: a first acquisition module 2001, a second acquisition module 2002, and an update module 2003. The first acquisition module 2001 is used to input the training image sequence into the compression model and obtain the compression information of the training image sequence output by the compression model. The second acquisition module 2002 is used to obtain the loss value of the compression model based on the compression information corresponding to each image in the training image sequence. The update module 2003 is used to update the weight value of the compression model based on the loss value of the compression model. The training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame. The compression information of the intermediate frame includes first compression information, which is obtained by the compression model based on the residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on the reference frame of the intermediate frame; the compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0301] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0302] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0303] In a possible implementation, the first image frame is a first frame, and at least one inserted frame is after the first image frame.

[0304] In one possible implementation, the compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of at least one P frame, and the compression information of the second image frame; wherein, at least one inserted frame is used in the compression process of the compression model for compressing the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

[0305] In one possible implementation, the compression model includes a first neural network model, a second neural network model, and a third neural network model. The first neural network model is used to compress the first image frame, the second neural network model is used to compress at least one intermediate frame, and the third neural network model is used to compress the second image frame.

[0306] In a possible implementation, the compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes a bit rate of the first image frame.

[0307] In a possible implementation, the compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame, and the reconstructed image of the intermediate frame.

[0308] In one possible implementation, the compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame, and the reconstructed image of the second image frame.

[0309] In one possible implementation, the second acquisition module is specifically used to: obtain the distortion rate of the intermediate frame based on the reconstructed image of the intermediate frame and the intermediate frame; obtain the loss value of the intermediate frame based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame and the first compression information of the intermediate frame.

[0310] In one possible implementation, the second acquisition module is specifically used to: obtain the distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; obtain the loss value of the second image frame based on the distortion rate of the second image frame, the optical flow code rate of the second image frame and the second compression information of the second image frame.

[0311] In a possible implementation, the second acquisition module is specifically used to obtain the loss value of the compression model based on the bit rate of the first image frame, the loss value of each intermediate frame in at least one intermediate frame, and the loss value of the second image frame.

[0312] In one possible implementation, the update module is specifically used to: when the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training number threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

[0313] During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware coding processor, or can be executed by a combination of hardware and software modules in the coding processor. The software module can be located in a mature storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0314] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0315] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0316] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0317] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0318] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0319] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0320] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to make a computer device (personal computer, server, or network equipment, etc.) perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. Or the computer software product can be transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD), a read-only memory (ROM), a random access memory (RAM)), etc.

[0321] The above description is merely a specific implementation of the embodiments of the present application, but the scope of protection of the embodiments of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the embodiments of the present application should be included in the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims

1. A training method, characterized in that: include: Inputting a training image sequence into a compression model, and obtaining compression information of the training image sequence output by the compression model; Obtaining a loss value of the compression model based on compression information corresponding to each image in the training image sequence; Based on the loss value of the compression model, updating the weight value of the compression model; Wherein, the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; The compression information of the intermediate frame includes first compression information, where the first compression information is obtained by the compression model based on residual feature parameters of the intermediate frame, where the residual feature parameters are obtained based on a reference frame of the intermediate frame; The compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on a reference frame of the second image frame.

2. The method according to claim 1, characterized in that The first image frame is an I frame, and the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

3. The method according to claim 2, characterized in that The first image frame is a first frame, and the at least one inserted frame is after the first image frame.

4. The method according to claim 3, characterized in that The compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of the at least one P frame and the compression information of the second image frame; wherein the at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

5. The method according to any one of claims 1 to 4, characterized in that: The compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress the at least one intermediate frame, and the third neural network model is used to compress the second image frame.

6. The method according to claim 5, characterized in that The compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes the bit rate of the first image frame.

7. The method according to claim 6, characterized in that The compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame and the reconstructed image of the intermediate frame.

8. The method according to claim 7, characterized in that The compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame and the reconstructed image of the second image frame.

9. The method according to claim 8, characterized in that The obtaining, based on compression information corresponding to each image in the training image sequence, a loss value corresponding to the training image sequence, comprises: Based on the reconstructed image of the intermediate frame and the intermediate frame, obtaining a distortion rate of the intermediate frame; A loss value of the intermediate frame is acquired based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame, and the first compression information of the intermediate frame.

10. The method according to claim 8 or 9, characterized in that: The obtaining the loss value of the compression model based on the compression information corresponding to each image in the training image sequence includes: acquiring a distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; A loss value of the second image frame is acquired based on the distortion rate of the second image frame, the optical flow code rate of the second image frame, and the second compression information of the second image frame.

11. The method according to any one of claims 8 to 10, characterized in that The obtaining the loss value of the compression model based on the compression information corresponding to each image in the training image sequence includes: A loss value of the compression model is obtained based on the bit rate of the first image frame, the loss value of each intermediate frame in the at least one intermediate frame, and the loss value of the second image frame.

12. The method according to any one of claims 1 to 11, characterized in that The updating of the weight value of the compression model based on the loss value of the compression model includes: When the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training times threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

13. A training device, characterized in that: include: A first acquisition module, used for inputting a training image sequence into the compression model, and acquiring compression information of the training image sequence output by the compression model; A second acquisition module, used to acquire a loss value of the compression model based on compression information corresponding to each image in the training image sequence; An updating module, used for updating a weight value of the compression model based on a loss value of the compression model; Wherein, the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; The compression information of the intermediate frame includes first compression information, where the first compression information is obtained by the compression model based on residual feature parameters of the intermediate frame, where the residual feature parameters are obtained based on a reference frame of the intermediate frame; The compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on a reference frame of the second image frame.

14. The device according to claim 13, characterized in that The first image frame is an I frame, and the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

15. The device according to claim 14, characterized in that The first image frame is a first frame, and the at least one inserted frame is after the first image frame.

16. The device according to claim 15, characterized in that The compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of the at least one P frame and the compression information of the second image frame; wherein the at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

17. The device according to any one of claims 13 to 16, characterized in that The compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress the at least one intermediate frame, and the third neural network model is used to compress the second image frame.

18. The device according to claim 17, characterized in that The compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes the bit rate of the first image frame.

19. The device according to claim 18, characterized in that The compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame and the reconstructed image of the intermediate frame.

20. The device according to claim 19, characterized in that The compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame and the reconstructed image of the second image frame.

21. The device according to claim 20, characterized in that The second acquisition module is specifically used for: Based on the reconstructed image of the intermediate frame and the intermediate frame, obtaining a distortion rate of the intermediate frame; A loss value of the intermediate frame is acquired based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame, and the first compression information of the intermediate frame.

22. The device according to claim 20 or 21, characterized in that The second acquisition module is specifically used for: acquiring a distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; A loss value of the second image frame is acquired based on the distortion rate of the second image frame, the optical flow code rate of the second image frame, and the second compression information of the second image frame.

23. The device according to any one of claims 20 to 22, characterized in that The second acquisition module is specifically used for: A loss value of the compression model is obtained based on the bit rate of the first image frame, the loss value of each intermediate frame in the at least one intermediate frame, and the loss value of the second image frame.

24. The device according to any one of claims 13 to 23, characterized in that The update module is specifically used for: When the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training times threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

25. A coding method, characterized in that: The method comprises: Acquiring video data or image data; Encoding the video data or the image data based on a coding network to obtain a code stream; wherein the coding network is trained based on a training image sequence, and the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; compression information corresponding to each image in the training image sequence is used to obtain a loss value of the coding network, and the loss value of the coding network is used to update a weight value of the coding network; The compression information of the intermediate frame includes first compression information, where the first compression information is obtained by the encoding network based on residual feature parameters of the intermediate frame, where the residual feature parameters are obtained based on a reference frame of the intermediate frame; The compression information of the second image frame includes second compression information, which is obtained by the encoding network based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

26. A decoding method, characterized in that: The method comprises: Receive code stream; The bitstream is decoded based on a decoding network to obtain reconstructed video data; wherein the decoding network is obtained based on the training method described in any one of claims 1 to 12.

27. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 12.

28. A coding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to claim 25.

29. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device performs the method according to claim 26.

30. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 12.

31. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

Images