Training method and device

Abstract

The invention provides a training method and device. The method comprises the steps that in the process of training a compression model, a training image sequence serves as training input of the compression model, the image sequence comprises a first image frame, a plurality of middle frames and a second image frame, and the compression model outputs compression information of the training image sequence. Wherein the first compression information of the middle frame is obtained based on the residual error characteristic parameter of the middle frame, and the reference frame of the middle frame is used for obtaining the residual error characteristic parameter. Moreover, the second compression information of the second image frame is obtained based on the coding probability of the image feature of the image frame, and the reference frame of the second image frame is used for obtaining the coding probability. Therefore, the second image frame is added in the training image sequence, so that the accumulative error of the compression model can be effectively reduced.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method and device. Background Art

[0002] As convolutional neural networks (CNNs) outperform traditional algorithms in computer vision tasks such as image recognition and object detection, more and more researchers are beginning to explore deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms, for example, optimizing modules such as inter-frame prediction, encoding network, entropy estimation network, and decoding network as a whole. Summary of the invention

[0003] The present application provides a training method and device. The training method can effectively reduce the cumulative error of the compression model.

[0004] In a first aspect, the present application provides a training method, comprising: a training device inputs a training image sequence into a compression model, and obtains compression information of the training image sequence output by the compression model. Furthermore, the training device obtains a loss value of the compression model based on the compression information corresponding to each image in the training image sequence. Furthermore, the training device updates a weight value of the compression model based on the loss value of the compression model. Wherein, the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame. The compression information of the intermediate frame includes first compression information, which is obtained by the compression model based on residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on a reference frame of the intermediate frame. The compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0005] In this way, during the training process of the present application, when the compression model performs probability prediction on the image features of the second image frame (i.e., the process of obtaining the coding probability of the image features), since the compression model performs probability prediction on the image features of the second image based on the reference frame, the prediction result of the image features of the second image frame is more accurate in view of the strong correlation between the reference frame and the second image frame, and further, when the probability prediction is performed based on the image features, a more accurate probability prediction result is obtained. Moreover, when the entropy coding process estimates the bit rate of the second image frame (i.e., the second compressed information), it is estimated based on the result of the probability prediction. Therefore, the more accurate the result of the probability prediction, the smaller the estimated bit rate. Accordingly, when obtaining the reconstructed image corresponding to the second image frame based on the bit rate, the smaller the bit rate, the higher the quality of the corresponding reconstructed image, and the smaller the inter-frame redundancy.

[0006] During the training process, the image quality of the reference frame of the second image frame (i.e., the reconstructed image of the P frame) will affect the prediction result of the image feature of the second image frame, and the prediction result of the image feature will affect the prediction result of the coding probability of the second image frame, and the prediction result of the coding probability of the second image frame will affect the code rate of the second image frame, and the code rate of the second image frame will affect the size of the loss (loss value) of the second image frame. Further, the smaller the loss of the second image frame, the smaller the loss of the compression model. In the repeated iteration process of the compression model training, the training goal is to make the loss of the compression model smaller and smaller. To achieve this goal, the loss of the second image frame needs to be smaller and smaller. Based on the above mutual constraint relationship, in order to make the loss of the second image frame smaller and smaller, the image quality of the reference frame of the second image frame will be higher and higher, thereby realizing the constraint on the quality stability of the reconstructed image of the intermediate frame. In other words, in the repeated iterative training process, the image quality of the intermediate frame output by the compression model each time becomes higher and higher. When the loss reaches the minimum value, the training ends, and the image quality of the intermediate frame will reach the maximum value that can be achieved by the training. In this way, in subsequent applications, the image compressed by the compression model has higher quality and can effectively reduce the cumulative error of the compression model.

[0007] Exemplarily, the second image frame is the last frame (also referred to as the tail frame) of the training image sequence.

[0008] Exemplarily, the second image frame is a CI (Condition I) frame described in the embodiment of the present application.

[0009] Exemplarily, the second image frame can be an I frame or a P frame. The second image frame is similar to the P frame, and both encoding and decoding (i.e., compression) depend on the reference frame. The difference between the second image frame and the P frame is that the compression of the P frame depends on the reference frame to calculate the residual features, that is, its optical flow is used to calculate the residual information. The compression of the CI frame (i.e., the second image frame) depends on the reference frame to calculate the probability distribution of the feature information of the original image, that is, its optical flow is not used to calculate the residual information, but only to calculate the probability distribution of the feature information of the original image. Therefore, there is no cumulative error between the encoded reconstructed image of the CI frame and the decoded reconstructed image.

[0010] Exemplarily, the training device may repeatedly train the compression model based on the training steps to update the weight value of the compression model and obtain a trained compression model.

[0011] Exemplarily, the weight value of the compression model can also be called the parameter of the compression model (or compression parameter), which is not limited in this application. In other words, the compression model is composed of multiple compression parameters or weight values. Once the weight value of the compression model is obtained, the compression model can be constructed based on the weight value of the compression model.

[0012] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0013] In this way, in the embodiment of the present application, by adding a virtual training sequence, the P frame in the original image can simulate the position in the image sequence in the actual test scene, thereby enhancing the model's ability to reconstruct the entire image sequence and reducing the cumulative error during the test process.

[0014] Exemplarily, the inserted frame is the virtual frame described in the embodiments of the present application.

[0015] In a possible implementation, the first image frame is the first frame, and at least one inserted frame is after the first image frame. In this way, by adding a virtual sequence before the P frame, the position of the P frame in the test scene can be effectively simulated to meet the problem of cumulative error that may occur during the test as much as possible, so as to train for this problem during the training process, so that the trained compression model can effectively reduce the cumulative error in the actual application process and improve the quality of the reconstructed image of each image frame.

[0016] Optionally, if the inserted frame duplicates the I frame, the inserted frame may also be inserted before the I frame (ie, the first frame).

[0017] In one possible implementation, the compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes compression information of the first image frame, compression information of at least one P frame, and compression information of the second image frame; wherein, at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

[0018] In this way, the inserted frames only participate in the training but not in the loss value acquisition process of the compression model. This can increase the length of the training image sequence without increasing the training complexity, effectively reducing the computational pressure during the training process.

[0019] In one possible implementation, the compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress at least one intermediate frame, and the third neural network model is used to compress the second image frame.

[0020] In this way, the compression model may include multiple neural network models, the second neural network model can realize residual encoding and decoding of the intermediate frame, and the third neural network model can realize CI encoding and decoding of the second image frame.

[0021] The second neural network model may be the residual coding model in the embodiment of the present application. The third neural network model may be the CI frame coding model in the embodiment of the present application.

[0022] Exemplarily, the weight values ​​of the compression model include weight values ​​of the first neural network model, weight values ​​of the second neural network model, and weight values ​​of the third neural network model.

[0023] In a possible implementation, the compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes a bit rate of the first image frame.

[0024] In a possible implementation, the compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame, and the reconstructed image of the intermediate frame.

[0025] In a possible implementation, the compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes second compression information, the optical flow code rate of the second image frame, and the reconstructed image of the second image frame.

[0026] In a possible implementation, based on the compression information corresponding to each image in the training image sequence, obtaining the loss value corresponding to the training image sequence includes: obtaining the distortion rate of the intermediate frame based on the reconstructed image of the intermediate frame and the intermediate frame. Obtaining the loss value of the intermediate frame based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame, and the first compression information of the intermediate frame.

[0027] In one possible implementation, based on the compression information corresponding to each image in the training image sequence, a loss value of the compression model is obtained, including: based on the reconstructed image of the second image frame and the second image frame, obtaining the distortion rate of the second image frame; based on the distortion rate of the second image frame, the optical flow code rate of the second image frame and the second compression information of the second image frame, obtaining the loss value of the second image frame.

[0028] In one possible implementation, a loss value of a compression model is obtained based on compression information corresponding to each image in a training image sequence, including: obtaining the loss value of the compression model based on a bit rate of a first image frame, a loss value of each intermediate frame in at least one intermediate frame, and a loss value of a second image frame.

[0029] In this way, the loss value of the compression model in the embodiment of the present application is obtained based on the bit rate of the first image frame, the loss value of the P frame and the loss value of the second image frame. Correspondingly, since the reconstructed image of the P frame is used as a reference frame of the CI frame, if the overall loss value of the compression model needs to be reduced, the loss value of the CI frame needs to be smaller, and the stability of the reconstructed image of the P frame is further reversely constrained, so that during the training process, the loss value of the compression model becomes smaller and smaller, and accordingly, the quality of the reconstructed image of the P frame becomes higher and higher.

[0030] In one possible implementation, the weight value of the compression model is updated based on the loss value of the compression model, including: when the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training number threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

[0031] In this way, by setting the training conditions, the compression model can stop training when the training conditions are met, and the trained compression model can be obtained.

[0032] In a second aspect, the present application provides a training device, comprising: a first acquisition module, used to input a training image sequence into a compression model, and obtain compression information of the training image sequence output by the compression model; a second acquisition module, used to obtain a loss value of the compression model based on the compression information corresponding to each image in the training image sequence; an update module, used to update the weight value of the compression model based on the loss value of the compression model; wherein the training image sequence comprises a first image frame, multiple intermediate frames, and a second image frame; the compression information of the intermediate frame comprises first compression information, which is obtained by the compression model based on residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on a reference frame of the intermediate frame; the compression information of the second image frame comprises second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0033] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0034] In a possible implementation manner, the first image frame is a first frame, and at least one inserted frame is after the first image frame.

[0035] In one possible implementation, the compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes compression information of the first image frame, compression information of at least one P frame, and compression information of the second image frame; wherein, at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

[0036] In one possible implementation, the compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress at least one intermediate frame, and the third neural network model is used to compress the second image frame.

[0037] In a possible implementation, the compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes a bit rate of the first image frame.

[0038] In a possible implementation, the compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame, and the reconstructed image of the intermediate frame.

[0039] In a possible implementation, the compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes second compression information, the optical flow code rate of the second image frame, and the reconstructed image of the second image frame.

[0040] In a possible implementation, the second acquisition module is specifically used to: acquire the distortion rate of the intermediate frame based on the reconstructed image of the intermediate frame and the intermediate frame; acquire the loss value of the intermediate frame based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame and the first compression information of the intermediate frame.

[0041] In a possible implementation, the second acquisition module is specifically used to: acquire the distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; acquire the loss value of the second image frame based on the distortion rate of the second image frame, the optical flow code rate of the second image frame and the second compression information of the second image frame.

[0042] In a possible implementation, the second acquisition module is specifically used to acquire the loss value of the compression model based on the bit rate of the first image frame, the loss value of each intermediate frame in at least one intermediate frame, and the loss value of the second image frame.

[0043] In one possible implementation, the update module is specifically used to: when the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training times threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

[0044] In a third aspect, the present application provides a coding method, which includes: obtaining video data or image data; encoding the video data or image data based on a coding network to obtain a code stream; wherein the coding network is trained based on a training image sequence, and the training image sequence includes a first image frame, multiple intermediate frames, and a second image frame; the compression information corresponding to each image in the training image sequence is used to obtain a loss value of the coding network, and the loss value of the coding network is used to update the weight value of the coding network; the compression information of the intermediate frame includes first compression information, and the first compression information is obtained by the coding network based on the residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on a reference frame of the intermediate frame; the compression information of the second image frame includes second compression information, and the second compression information is obtained by the coding network based on the coding probability of the image features of the second image frame, and the coding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0045] In a fourth aspect, the present application provides a decoding method, the method comprising: receiving a code stream; decoding the code stream based on a decoding network to obtain reconstructed video data. The decoding network is trained based on a training image sequence, the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; the compression information corresponding to each image in the training image sequence is used to obtain a loss value of the decoding network, and the loss value of the decoding network is used to update a weight value of the decoding network; the compression information of the intermediate frame includes first compression information, the first compression information is obtained by the decoding network based on residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on a reference frame of the intermediate frame; the compression information of the second image frame includes second compression information, the second compression information is obtained by the decoding network based on the coding probability of the image features of the second image frame, and the coding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

[0046] In a fifth aspect, the present application provides a coding device, comprising a memory and a processor, wherein the memory is coupled to the processor. The memory stores program instructions, and when the program instructions are executed by the processor, the coding device implements the method in the fourth aspect.

[0047] In a sixth aspect, the present application provides a decoding device, including a memory and a processor, wherein the memory is coupled to the processor. The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device implements the method of the fifth aspect.

[0048] In the seventh aspect, the present application provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement a method as described in any one of the above-mentioned first aspects.

[0049] In an eighth aspect, the present application provides a computer-readable storage medium, comprising a computer program, wherein when the computer program is executed on a computer, the computer executes any one of the methods described in the first aspect.

[0050] In a ninth aspect, the present application provides a computer program, which, when executed by a computer, is used to execute any one of the methods in the first aspect above.

[0051] In a tenth aspect, the present application also provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer executes the operations and / or processing performed by the electronic device in any of the above method embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1A is a schematic block diagram of an exemplary decoding system 10;

[0053] Figure 1B is an illustrative diagram of an example of a video decoding system 40;

[0054] Figure 2 A schematic diagram of a video decoding device 400 provided in an embodiment of the present invention;

[0055] Figure 3 A simplified block diagram of an apparatus 500 is provided for an exemplary embodiment;

[0056] Figure 4 An example diagram of an end-to-end deep learning image encoding and decoding framework;

[0057] Figure 5 This is an example diagram of an application scenario of an embodiment of the present application;

[0058] Figure 6 is a schematic diagram of the structure of a residual coding and decoding framework shown exemplarily;

[0059] Figure 7 Schematic diagram of the processing flow of the residual codec structure shown as an example;

[0060] Figure 8 is a schematic diagram of an exemplary GOP;

[0061] Fig. 9Schematic diagram of PSNR value of VTM compression method and PSNR change of AI residual compression method;

[0062] Fig.10 Schematic diagram of the encoding and decoding framework of the CI frame shown as an example;

[0063] Fig.11 Schematic diagram of the processing flow of the CI frame encoding and decoding structure shown as an example;

[0064] Fig.12 is a schematic diagram of the structure of a compression model shown as an example;

[0065] Fig.13 is a schematic diagram of an exemplary training image sequence;

[0066] Fig.14 is a schematic diagram of an exemplary training process;

[0067] Fig.15 is a schematic diagram of performance test results shown as an example;

[0068] Fig.16 is a schematic diagram of performance test results shown as an example;

[0069] Fig.17 is a schematic diagram of an exemplary training image sequence;

[0070] Fig.18 is a schematic diagram of an exemplary training process;

[0071] Fig.19 is a schematic diagram of an exemplary training process;

[0072] Fig. 20 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0075] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0076] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0077] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0078] The following are descriptions of terms involved in the embodiments of this application:

[0079] 1. Bit rate: In image compression, the average coding length required to encode a unit pixel.

[0080] 2. Rate-Distortion (RD): It is used to describe the relationship between bit rate and distortion. It can also be understood as an indicator to measure the performance of the compression algorithm, which comprehensively considers the bit rate and the distortion of the decoded image.

[0081] 3. Peak Signal to Noise Ratio (PSNR) is an objective standard for evaluating images. The higher the PSNR, the better the image quality.

[0082] 4. Multi-Scale Structural Similarity Index measure (MS-SSIM) is an objective standard for evaluating images. The higher the value, the better the image quality.

[0083] 5. Video encoder performance (Bjontegaard delta rate, BD-Rate) is an indicator used to compare video encoder performance. It is usually used to compare the encoding effects of different video encoders on the same video sequence and measure the compression performance of the encoder. The smaller the BD-Rate, the better the relative result. A negative value means that the bit rate is saved compared to the comparison method at the same quality.

[0084] 6. Codec refers to a device or program that can encode or decode a signal or a data stream. It usually includes an encoder and a decoder.

[0085] 7. Encoder: An operation that encodes a signal or digital stream (usually for transmission, storage, or encryption) or obtains a coded stream.

[0086] 8. Decoder: The operation of restoring the encoded stream to a signal or digital stream.

[0087] 9. A feature map is a set of values ​​obtained by inputting an image or video into the network, generally represented as a matrix of dimension [c, h, w], where C (channel) represents channel, H (height) represents height, and W (width) represents width.

[0088] 10. Optical flow refers to the movement speed and direction of each pixel in two adjacent frames.

[0089] 11. Error Accumulation or Accumulated error: When the recursive formula is used to integrate (or accumulate) the results of each part of the calculation, the errors are accumulated, and the final sum of errors is called accumulated error. In the video compression scenario, it specifically refers to the compression performance becoming lower and lower over time.

[0090] 12. Distortion: In this application, it refers to PSNR or MS-SSIM. In other embodiments, it may also refer to other distortion indicators, which is not limited in this application.

[0091] 13. The image is divided into three channels, Y / U / V, in the YUV space. Y-PSNR refers to the PSNR value on the Y channel, U-PSNR refers to the PSNR value on the U channel, and V-PSNR refers to the PSNR value on the V channel.

[0092] 14. Bit rate refers to the data flow used per unit time of a video file, and is an important component of image quality control in video encoding. Under the same compression quality, the smaller the rate, the better.

[0093] Since the embodiments of the present application involve the application of a neural network, for ease of understanding, the following first explains the relevant nouns or terms used in the embodiments of the present application:

[0094] 1. Neural Networks

[0095] A neural network (NN) is a machine learning model. A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0096]

[0097] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a nonlinear function such as ReLU. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0098] 2. Multi-layer perception (MLP)

[0099] MLP is a simple deep neural network (DNN) (different layers are fully connected), also known as a multi-layer neural network, which can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Taking coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0100] 3. Convolutional Neural Networks

[0101] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron can respond to the image input into it. A convolutional neural network contains a feature extractor consisting of a convolutional layer and a pooling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve an input image or convolution feature plane (feature map).

[0102] A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution on the input signal. A convolutional layer can include many convolution operators, also known as kernels. In image processing, the convolution operator is essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is ​​usually processed horizontally on the input image one pixel after another (or two pixels after two pixels... depending on the value of the stride) to complete the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, instead of using a single weight matrix, multiple weight matrices of the same size (rows × columns) are applied, that is, multiple isotype matrices. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise points in the image. The multiple weight matrices have the same size (rows × columns), and the size of the feature maps extracted by the multiple weight matrices of the same size is also the same. The extracted multiple feature maps of the same size are then merged to form the output of the convolution operation. The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network can make correct predictions. When the convolutional neural network has multiple convolutional layers, the initial convolutional layer often extracts more general features, which can also be called low-level features; as the depth of the convolutional neural network deepens, the features extracted by the later convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0103] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. It can be a convolution layer followed by a pooling layer, or multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator to sample the input image to obtain an image of smaller size. The average pooling operator can calculate the pixel values ​​in the image within a specific range to produce an average value as the result of average pooling. The maximum pooling operator can take the pixel with the largest value in the range as the result of maximum pooling within a specific range. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer can be smaller than the size of the image input to the pooling layer. Each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0104] After being processed by the convolution layer / pooling layer, the convolution neural network is not sufficient to output the required output information. Because as mentioned above, the convolution layer / pooling layer will only extract features and reduce the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network needs to use the neural network layer to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer may include multiple hidden layers, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of the specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0105] Optionally, after the multiple hidden layers in the neural network layer, an output layer of the entire convolutional neural network is also included. The output layer has a loss function similar to the classification cross entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation will begin to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0106] 4. Recurrent Neural Networks

[0107] Recurrent neural networks (RNN) are used to process sequence data. In the traditional neural network model, from the input layer to the hidden layer and then to the output layer, the layers are fully connected, and the nodes within each layer are disconnected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer disconnected but connected, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN. The same error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded, the parameters, such as W, are shared; this is not the case with the traditional neural network mentioned above. And when using the gradient descent algorithm, the output of each step depends not only on the network at the current step, but also on the state of the network at the previous steps. This learning algorithm is called the Back Propagation Through Time (BPTT).

[0108] Since we already have convolutional neural networks, why do we need recurrent neural networks? The reason is simple. In convolutional neural networks, there is a premise that the elements are independent of each other, and the input and output are also independent, such as cats and dogs. But in the real world, many elements are interconnected, such as the change of stocks over time, or someone said: I like traveling, and my favorite place is Yunnan. I must go there if I have the chance in the future. Fill in the blank here, humans should all know to fill in "Yunnan". Because humans will make inferences based on the content of the context, but how can machines do this? RNN came into being. RNN is designed to give machines the ability to remember like humans. Therefore, the output of RNN needs to rely on the current input information and historical memory information.

[0109] 5. Loss Function

[0110] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0111] 6. Back Propagation Algorithm

[0112] Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the compression model, such as the weight matrix.

[0113] As convolutional neural networks (CNNs) outperform traditional algorithms in computer vision tasks such as image recognition and target detection, more and more researchers are beginning to explore deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms, for example, by optimizing the encoding network, entropy estimation network, entropy encoding network, entropy decoding network, decoding network, inter-frame prediction and other modules as a whole, where the encoding network and decoding network can also be called transformation modules and inverse transformation modules, which are generally composed of convolutional layers and nonlinear transformation units.

[0114] Figure 1A1 is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply a decoding system 10) that can utilize the techniques of the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices that can be used to perform various techniques according to various examples described in the present application.

[0115] like Figure 1A As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21 such as encoded images to a destination device 14 for decoding the encoded image data 21.

[0116] The source device 12 includes an encoder 20 , and may additionally or optionally include a video source 16 , a preprocessor (or a preprocessing unit) 18 such as an image preprocessor, and a communication interface (or a communication unit) 22 .

[0117] The video source 16 may include or may be any type of image capture device for capturing real-world images, etc., and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The video source may be any type of memory or storage that stores any of the above images.

[0118] In order to distinguish the processing performed by the pre-processor (or pre-processing unit) 18 , the image (or image data) 17 may also be referred to as a raw image (or raw image data) 17 .

[0119] The preprocessor 18 is used to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color adjustment, or denoising. It is understood that the preprocessing unit 18 may be an optional component.

[0120] The video encoder (or encoder) 20 is used to receive the pre-processed image data 19 and provide the encoded image data 21 (hereinafter referred to as Figure 2 etc. for further description).

[0121] The communication interface 22 in the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device through the communication channel 13 for storage or direct reconstruction.

[0122] The destination device 14 includes a decoder 30 and, in addition or alternatively, may include a communication interface (or communication unit) 28 , a post-processor (or post-processing unit) 32 , and a display device 34 .

[0123] The communication interface 28 in the destination device 14 is used to receive the encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is a encoded image data storage device, and provide the encoded image data 21 to the decoder 30.

[0124] The communication interface 22 and the communication interface 28 can be used to send or receive encoded image data (or encoded data) 21 through a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any type of combination thereof.

[0125] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission coding or processing for transmission over a communication link or network.

[0126] The communication interface 28 corresponds to the communication interface 22 , for example, and can be used to receive transmission data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21 .

[0127] Both the communication interface 22 and the communication interface 28 can be configured as follows Figure 1A A unidirectional communication interface, or a bidirectional communication interface, indicated by an arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14, and can be used to send and receive messages, etc. to establish a connection, confirm and exchange any other information related to the communication link and / or data transmission such as encoded image data transmission, etc.

[0128] The video decoder (or decoder) 30 is used to receive the encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as Figure 3 etc. for further description).

[0129] The post-processor 32 is used to post-process the decoded image data 31 (also called reconstructed image data) such as the decoded image to obtain the post-processed image data 33 such as the post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color adjustment, cropping or resampling, or any other processing for generating the decoded image data 31 for display by the display device 34 or the like.

[0130] The display device 34 is used to receive the post-processed image data 33 to display the image to a user or viewer, etc. The display device 34 can be or include any type of display for displaying the reconstructed image, such as an integrated or external display screen or display. For example, the display screen can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display screen.

[0131] The decoding system 10 further includes a training engine 25 for training the encoder 20 and / or the decoder 30 to achieve conversion between the image domain and the feature domain.

[0132] In the embodiment of the present application, the training data can be stored in a database (not shown), and the training engine 25 trains the encoding / decoding network, i.e., the system 20 30, based on the training data. It should be noted that the embodiment of the present application does not limit the source of the training data, for example, the training data can be obtained from the cloud or other places for model training.

[0133] In the embodiment of the present application, the training of the compression model can be performed online or offline. For example, the online method can use Huawei Cloud. The offline method can use a server or a server cluster or a terminal, etc., which is not limited in the present application.

[0134] although Figure 1A The source device 12 and the destination device 14 are shown as independent devices, but the device embodiment may also include the source device 12 and the destination device 14 or the functions of the source device 12 and the destination device 14 at the same time, that is, the source device 12 or the corresponding function and the destination device 14 or the corresponding function at the same time. In these embodiments, the source device 12 or the corresponding function and the destination device 14 or the corresponding function can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0135] According to the description, Figure 1A The presence and (exact) division of different units or functions in the source device 12 and / or the destination device 14 shown may vary depending on the actual device and application, which will be obvious to the skilled person.

[0136] The encoder 20 (eg, video encoder 20) or the decoder 30 (eg, video decoder 30) or both may be configured as follows: Figure 1B The processing circuitry shown may be implemented, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. The encoder 20 may be implemented by processing circuitry 46 to include reference Figure 2 The various modules discussed in encoder 20 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented by processing circuit 46 to include reference Figure 3 The processing circuit 46 may be used to perform the various operations discussed below. Alternatively, if part of the technique is implemented in software, the device may store the instructions of the software in a suitable non-transitory computer-readable storage medium and use one or more processors to execute the instructions in hardware to perform the technique of the present invention. One of the video encoder 20 and the video decoder 30 may be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as Figure 1B shown.

[0137] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (e.g., a content service server or a content distribution server), a broadcast receiving device, a broadcast transmitting device, etc., and may not use or use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0138] In some cases, Figure 1AThe video decoding system 10 shown is merely exemplary, and the techniques provided herein may be applicable to video encoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between an encoding device and a decoding device. In other examples, data is retrieved from a local memory, sent over a network, and so on. A video encoding device may encode data and store the data in a memory, and / or a video decoding device may retrieve data from a memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but only encode data to a memory and / or retrieve and decode data from a memory.

[0139] Figure 1B 4 is an illustrative diagram of an example of a video decoding system 40. The video decoding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video codec implemented by a processing circuit 46), an antenna 42, one or more processors 43, one or more memory storages 44, and / or a display device 45.

[0140] like Figure 1B As shown, imaging device 41, antenna 42, processing circuit 46, video encoder 20, video decoder 30, processor 43, memory storage 44 and / or display device 45 can communicate with each other. In different examples, video decoding system 40 can include only video encoder 20 or only video decoder 30.

[0141] In some instances, antenna 42 can be used to transmit or receive a coded bit stream of video data. In addition, in some instances, display device 45 can be used to present video data. Processing circuit 46 can include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. Video decoding system 40 can also include optional processor 43, which can similarly include application-specific integrated circuit (ASIC) logic, graphics processor, general processor, etc. In addition, memory storage 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, memory storage 44 can be implemented by cache memory. In other instances, processing circuit 46 can include memory (e.g., cache, etc.) for implementing image buffer, etc.

[0142] In some examples, video encoder 20 implemented by logic circuits may include an image buffer (e.g., implemented by processing circuit 46 or memory storage 44) and a graphics processing unit (e.g., implemented by processing circuit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented by processing circuit 46 to implement reference Figure 2 The various modules discussed in connection with the encoder and / or any other encoder system or subsystem described herein. Logic circuits may be used to perform the various operations discussed herein.

[0143] In some examples, video decoder 30 may be implemented in a similar manner by processing circuitry 46 to implement reference Figure 3 The various modules discussed in connection with the video decoder 30 of the present invention and / or any other decoder system or subsystem described herein. In some examples, the video decoder 30 implemented by logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented, for example, by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include the video decoder 30 implemented by processing circuitry 46 to implement the video decoder 30 of the present invention. Figure 3 and / or the various modules discussed above for any other decoder system or subsystem described herein.

[0144] In some examples, antenna 42 may be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream may include data related to the encoded video frames discussed herein, indicators, index values, mode selection data, etc., such as data related to the encoded partitions (e.g., transform coefficients or quantized transform coefficients, (as discussed) optional indicators, and / or data defining the encoded partitions). Video decoding system 40 may also include video decoder 30 coupled to antenna 42 and used to decode the encoded bitstream. Display device 45 is used to present the video frames.

[0145] It should be understood that for the examples described with reference to the video encoder 20 in the embodiments of the present application, the video decoder 30 can be used to perform the reverse process. With respect to the signaling syntax elements, the video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, the video encoder 20 can entropy encode the syntax elements into an encoded video bitstream. In such examples, the video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0146] For ease of description, the embodiments of the present invention are described with reference to the Versatile video coding (VVC) reference software or the High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). Those skilled in the art will appreciate that the embodiments of the present invention are not limited to HEVC or VVC.

[0147] Figure 2 Schematic diagram of a video decoding device 400 provided in an embodiment of the present invention. The video decoding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 400 may be a decoder, such as Figure 1A The video decoder 30 in the embodiment may also be an encoder, for example Figure 1A The video encoder 20 in.

[0148] The video decoding device 400 includes: an input port 410 (or input port 410) and a receiving unit (Rx) 420 for receiving data; a processor, a logic unit or a central processing unit (CPU) 430 for processing data; for example, the processor 430 here can be a neural network processor 430; a transmitter unit (Tx) 440 and an output port 450 (or output port 450) for transmitting data; and a memory 460 for storing data. The video decoding device 400 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 410, the receiving unit 420, the transmitting unit 440 and the output port 450 for the output or output of optical or electrical signals.

[0149] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the input port 410, the receiving unit 420, the sending unit 440, the output port 450, and the memory 460. The processor 430 includes a decoding module 470 (e.g., a decoding module 470 based on a neural network NN). The decoding module 470 implements the embodiments disclosed above. For example, the decoding module 470 performs, processes, prepares, or provides various encoding operations. Therefore, the decoding module 470 provides substantial improvements to the functions of the video decoding device 400 and affects the switching of the video decoding device 400 to different states. Alternatively, the decoding module 470 is implemented with instructions stored in the memory 460 and executed by the processor 430.

[0150] The memory 460 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device for storing such programs when such programs are selected for execution, and for storing instructions and data read during program execution. The memory 460 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0151] Figure 3 A simplified block diagram of an apparatus 500 is provided for an exemplary embodiment, and the apparatus 500 may be used as Figure 1A Either or both of the source device 12 and the destination device 14 in .

[0152] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices that are currently available or will be developed in the future and are capable of manipulating or processing information. Although a single processor such as the processor 502 shown in the figure may be used to implement the disclosed implementation, using more than one processor is faster and more efficient.

[0153] In one implementation, the memory 504 in the apparatus 500 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 accessed by the processor 502 via a bus 512. The memory 504 may also include an operating system 508 and an application 510, which includes at least one program that allows the processor 502 to perform the methods described herein. For example, the application 510 may include applications 1 to N, and also include a video decoding application that performs the methods described herein.

[0154] The apparatus 500 may also include one or more output devices, such as a display 518. In one example, the display 518 may be a touch-sensitive display that combines a display with a touch-sensitive element that may be used to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0155] Although bus 512 in device 500 is described herein as a single bus, bus 512 may include multiple buses. In addition, auxiliary storage may be directly coupled to other components of device 500 or accessed through a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Therefore, device 500 may have a variety of configurations.

[0156] Figure 4 This is an example diagram of an application scenario of an embodiment of the present application, such as Figure 4 As shown, the application scenario can be a business involving image / video acquisition, storage or transmission in a terminal, cloud server, or video surveillance, such as terminal photo / video recording, photo album, cloud photo album, video surveillance, etc.

[0157] Encoding end: The camera collects images / videos. The artificial intelligence (AI) image / video encoding network extracts features from the images / videos to obtain image features with low redundancy, and then compresses the image features to obtain a code stream / image file.

[0158] Decoding end: When images / videos need to be output, the AI ​​image / video decoding network decompresses the bitstream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain the reconstructed image / video.

[0159] The storage / transmission module stores the compressed code stream / image files for different services (for example, terminal photography, video surveillance, cloud servers, etc.) or transmits (for example, cloud services, live broadcast technology, etc.).

[0160] Figure 5This is an example diagram of an application scenario of an embodiment of the present application, such as Figure 7 As shown, the application scenario can be a service involving image / video acquisition, storage or transmission in a terminal or video surveillance, such as a terminal photo album, video surveillance, live broadcast, etc.

[0161] Encoding side: The encoding network transforms the image / video into image features with lower redundancy. The entropy estimation network is responsible for calculating the encoding probability of each data in the image features. The entropy coding network performs lossless encoding on the image features according to the probability corresponding to each data to obtain the bitstream / image file, further reducing the amount of data transmission during the image compression process.

[0162] Decoding end: The entropy decoding network performs lossless decoding on the bitstream / image file according to the probability corresponding to each data to obtain the reconstructed image features. The decoding network performs a reverse transformation on the image features output by entropy decoding and parses it into an image / video. The saving module saves the bitstream / image file to the corresponding storage location of the terminal. The loading module loads the bitstream / image file from the corresponding storage location of the terminal and inputs it into the entropy decoding network.

[0163] Optionally, the AI ​​encoding unit, the AI ​​decoding unit and the entropy estimation unit may adopt a network processor (Network Processing Unit, NPU), and the entropy encoding network, the entropy decoding network, the saving module and the loading module may adopt a central processing unit (Central Processing Unit, CPU).

[0164] Figure 6 is a schematic diagram of the structure of the residual coding and decoding framework shown as an example. Figure 6 The residual codec framework (also referred to as the residual codec model, which is not limited in this application) specifically includes but is not limited to: an image optical flow estimation module, an optical flow compression module, a prediction module, and a residual compression module. Optionally, the model may also include an image reconstruction module and a feature extraction module. It should be noted that Figure 6 The connection relationship in the example is only an illustrative example. The specific interaction relationship between the submodules in each module can be referred to Figure 7 The description in is not limited here.

[0165] Exemplarily, the image optical flow estimation module may include a motion estimation module. The motion estimation module is a neural network model, whose input is the current frame and the reference frame, and whose output is the optical flow.

[0166] The optical flow compression module includes but is not limited to an optical flow entropy estimation module, an optical flow encoding module and an optical flow decoding module. The optical flow compression module is used to compress the optical flow output by the image optical flow estimation module and output the compressed optical flow.

[0167] The prediction module may include a motion compensation module, which is used to predict the current frame according to the reference frame (ie, the reconstructed image of the previous frame) and the compressed optical flow to obtain a predicted image.

[0168] The residual compression module includes but is not limited to a residual entropy estimation module, a residual encoding module, and a residual decoding module. The residual compression module compresses the residual between the original image and the predicted image of the current frame on the one hand, and decompresses it to obtain a reconstructed residual on the other hand. The reconstructed residual and the predicted image are summed to obtain the reconstructed image of the current frame.

[0169] Optionally, the optical flow entropy estimation module and the residual entropy estimation module can be understood as components of the entropy estimation network. The optical flow encoding module, the motion estimation module and the residual encoding module can be understood as components of the encoding network. The optical flow decoding module and the residual decoding module can be understood as components of the decoding network.

[0170] Combination Figure 6 , Figure 7 For an exemplary processing flow diagram of the residual codec structure, please refer to Figure 7 , including but not limited to the following steps:

[0171] Step 1. The feature extraction module extracts the current frame x t and reference frame Perform feature extraction and get x t The image feature f t , The image feature f t-1 .

[0172] The reference frame The reference frame is the reconstructed image corresponding to the previous frame. In the embodiment of the present application, the reference frame may refer to the previous frame or the reconstructed image corresponding to the previous frame. It should be noted that the "previous frame" involved in the embodiment of the present application may be the previous frame adjacent to the current frame or a non-adjacent frame, and the present application does not limit this.

[0173] Step 2. The motion estimation module is based on the current frame x t and the reconstructed image of the reference frame Get the optical flow v t .

[0174] The reference frame can also be understood as the previous frame, and the reconstructed image of the reference frame is output from the image reconstruction module.

[0175] Step 3. The optical flow encoding module encodes the optical flow v t Perform optical flow compression (i.e., optical flow encoding) and output optical flow compression features.

[0176] Step 4. The optical flow entropy estimation module performs probability prediction on the optical flow compression features, and performs optical flow entropy coding based on the results of the probability prediction to obtain the optical flow code stream. The code rate of the optical flow code stream is

[0177] Step 5. The optical flow decoding module decodes the optical flow code stream according to the optical flow compression features to obtain the decoded optical flow

[0178] Step 6. The prediction module is based on the image features f of the reconstructed image of the reference frame t-1 and decoded optical flow Make a prediction and get the predicted features of the current frame (also called the predicted image)

[0179] Step 7. The residual coding module predicts the features of the current frame based on Perform residual encoding (also called residual compression) with the image feature f of the current frame to obtain a residual compression feature.

[0180] The residual calculation process may optionally be: subtracting the pixel value of the prediction block from the pixel value of the image block pixel by pixel (pixel by pixel) to obtain a residual block in the pixel domain.

[0181] Step 8. The residual entropy estimation module calculates the probability of residual compression features (also called residual coding probability) based on the residual compression features. And performs residual entropy coding based on the residual compression probability to obtain a residual code stream. Among them, the code rate of the residual code stream is

[0182] Step 9. The residual decoding module performs residual decoding on the residual code stream based on the residual compression feature to obtain the residual

[0183] Step 10. Image reconstruction module based on predicted features and residual Reconstruct the image of the current frame to obtain the reconstructed image of the current frame

[0184] It should be noted that the residual encoding and decoding structure (eg Figure 7 ) is only an illustrative example and is not limited in this application. Figure 7 The specific details of each step in the embodiment can be referred to the existing technical embodiments, and will not be repeated in this application.

[0185] It should be further explained that Figure 7 The process shown in can be understood as an encoding process or a decoding process, and this application does not limit this.

[0186] For example, in Figure 7 During the training of the network in (hereinafter referred to as the residual coding model), the Loss module can obtain the loss function of the residual coding model and cyclically train the residual coding model based on the loss function. The specific training process is as follows:

[0187] Step 1. A training device (such as Huawei Cloud or a server, etc., not limited in this application) trains the GOP.

[0188] Exemplarily, a plurality of image frames may constitute a GOP (Group of Pictures). Optionally, a GOP may include one or more I frames and one or more P frames.

[0189] Figure 8 For a schematic diagram of an exemplary GOP, refer to Figure 8 , exemplarily, the GOP includes I1 frame, P1 frame, P2 frame, I2 frame, and P3 frame. It should be noted that, Figure 8 The type, quantity and order of the image frames in the GOP are only illustrative examples and are not limited in this application.

[0190] Exemplarily, an I frame can be called a complete frame, or an independently decodable frame, that is, an I frame does not need to rely on other frames and can be decoded independently. Usually, the first frame (i.e., the first frame) of a GOP is an I frame, and a P frame can be called an inter-frame prediction frame. The decoding of a P frame depends on the most recent I frame or P frame in front of it. For example, a P1 frame needs to be decoded based on an I1 frame, and a P2 frame needs to be decoded based on a P1 frame.

[0191] Exemplarily, during the training of GOP by the training device, the I frame is compressed (i.e., image encoded) by the image compression model to obtain the bit rate and reconstructed image of the I frame. The image compression model corresponding to the I frame (also referred to as an image compression neural network, which is not limited in this application) may include but is not limited to: an encoding network, a decoding network, and an image reconstruction network. The specific model may refer to any one of the existing technologies, which is not limited in this application.

[0192] For example, the training device can be Figure 7 The residual coding model shown compresses each P frame in the GOP to obtain a reconstructed image corresponding to each P frame. and code rate R i Among them, the bit rate of P frame is R i Including residual bit rate and optical flow rate

[0193] Step 2. Training the device based on each P frame and the reconstructed image of each P frame and code rate R i, and the bit rate R of the I frame 0 , get the loss function loss of the compression model.

[0194] Exemplarily, the training device calculates the loss function loss of the compression model based on formula (2):

[0195]

[0196] Among them, R 0 is the bit rate of the I frame. Represents the loss function of a single frame (including P frames and I frames). D i represents the distortion loss value of each frame. The training device can be based on P frames (i.e. Figure 7 x in t ) and the reconstructed image of the P frame Determine the distortion loss value D corresponding to the P frame i . β i Used to adjust the rate distortion ratio. The I frame is the same and will not be described here.

[0197] Step 3. Based on the loss function of the compression model, iteratively train the compression model.

[0198] For example, as described above, for training the neural network, the weights in the compression model may be adjusted based on the loss function, and the above training steps may be repeated until the training cutoff condition is reached. The compression model includes a neural network for training I frames and a neural network for training P frames (e.g. Figure 7 The residual coding model in .

[0199] In the embodiment of the present application, the training cutoff condition includes but is not limited to: the loss function reaches the loss function threshold, the number of training times reaches the training times threshold, and / or the fluctuation value of the loss function remains within a certain range. The above thresholds can be set according to actual needs and are not limited in this application.

[0200] For example, Figure 7 The residual coding model shown in can achieve end-to-end video compression with high RD performance. However, due to the limited number of training sets in the training process (for example, a GOP usually includes 6 to 10 image frames), the use of Figure 7 When the residual coding model in the video is used for residual compression, it is difficult to fully simulate the real test environment during the training process (for example, the length of GOP in the real test environment may reach 100 image frames). Therefore, the way of using the residual coding model for video compression will gradually increase the cumulative error. The compression quality of the traditional VTM fluctuates steadily. Fig. 9 Schematic diagram showing the PSNR value of the VTM compression method and the PSNR change of the AI ​​residual compression method. Fig. 9 As shown in the figure (the horizontal axis is the image frame, the vertical axis is the PSNR), where the dotted line indicates the quality of the reconstructed image by VTM, and the solid line indicates the quality of the reconstructed image by AI residual compression. The quality of the reconstructed image by VTM is very stable, without obvious degradation, but the quality of the reconstructed image by AI residual compression is obviously gradually reduced, and the corresponding cumulative error will gradually increase. Especially for static and simple motion scenes, the phenomenon of cumulative error is particularly obvious.

[0201] Fig.10 Schematic diagram of the encoding and decoding framework of condition I (condition I, CI) frame is shown as an example. Fig.10 The CI frame (also called adversarial frame, etc., which is not limited in this application) encoding and decoding framework (hereinafter referred to as CI frame encoding model) includes but is not limited to: image optical flow estimation module, optical flow compression module, prediction module, feature compression module. Optionally, the model may also include an image reconstruction module and a feature extraction module. It should be noted that Fig.10 The connection relationship in the example is only an illustrative example. The specific interaction relationship between the submodules in each module can be referred to Fig.11 The description in is not limited here.

[0202] Exemplarily, the image optical flow estimation module may include a motion estimation module. The motion estimation module is a neural network model, whose input is the current frame and the reference frame, and whose output is the optical flow.

[0203] The optical flow compression module includes but is not limited to an optical flow entropy estimation module, an optical flow encoding module and an optical flow decoding module. The optical flow compression module is used to compress the optical flow output by the image optical flow estimation module and output the compressed optical flow.

[0204] The prediction module may include a motion compensation module, which is used to predict the current frame based on the reference frame (ie, the reconstructed image of the previous frame) and the compressed optical flow to obtain a predicted image or a predicted feature.

[0205] The feature compression module includes but is not limited to: a feature entropy estimation module, a feature encoding module and a feature decoding module. The feature compression module, on the one hand, compresses the features of the original image of the current frame (ie, image features), and on the other hand, decompresses to obtain reconstructed features.

[0206] Optionally, the optical flow entropy estimation module and the feature entropy estimation module can be understood as components of the entropy estimation network. The optical flow encoding module, the motion estimation module and the feature encoding module can be understood as components of the encoding network. The optical flow decoding module and the feature decoding module can be understood as components of the decoding network.

[0207] Combination Fig.10 , Fig.11For an exemplary processing flow diagram of the CI frame encoding and decoding structure, please refer to Fig.11 , including but not limited to the following steps:

[0208] Step 1. The feature extraction module extracts the current frame x t Perform feature extraction and get x t The image feature f t .

[0209] Step 2. The motion estimation module is based on the current frame x t and the reconstructed image of the reference frame Get the optical flow v t .

[0210] The reference frame can also be understood as the previous frame, and the reconstructed image of the reference frame is output from the image reconstruction module.

[0211] Step 3. The optical flow encoding module encodes the optical flow v t Perform optical flow compression (i.e., optical flow encoding) and output optical flow compression features.

[0212] Step 4. The optical flow entropy estimation module performs probability prediction on the optical flow compression features, and performs optical flow entropy coding based on the results of the probability prediction to obtain the optical flow code stream. The code rate of the optical flow code stream is

[0213] Step 5. The optical flow decoding module decodes the optical flow code stream according to the optical flow compression features to obtain the decoded optical flow

[0214] Step 6. The prediction module is based on the image features f of the reconstructed image of the reference frame t-1 and decoded optical flow Make a prediction and get the predicted features of the current frame (it may also be a predicted image)

[0215] The image features of the reconstructed image of the reference frame are: The feature extraction module extracts features based on the reconstructed image of the reference frame (i.e., the previous frame) output by the image reconstruction module to obtain the image features f of the reconstructed image of the reference frame. t-1 .

[0216] Step 7. The feature encoding module encodes the image features f of the current frame t Perform feature encoding (also called feature compression) to obtain image compression features.

[0217] Step 8. The feature entropy estimation module calculates the image compression feature probability (also called the image feature coding probability) based on the image compression feature and the prediction feature. It then performs feature entropy coding based on the image compression feature probability to obtain a feature code stream. The code rate of the feature code stream is

[0218] Step 9. The feature decoding module decodes the feature code stream based on the image compression feature to obtain the feature

[0219] Step 10. Image reconstruction module based on features Reconstruct the image of the current frame to obtain the reconstructed image of the current frame

[0220] Exemplarily, the structure of the CI frame coding model is similar to that of the residual coding model, except that in steps 7 to 10, the prediction features in the CI frame coding model are It is not used for image reconstruction, but is used as a priori condition to estimate the probability of image features, thereby reducing inter-frame redundancy (i.e., residual error). In other words, the bit rate of the CI frame is obtained based on the coding probability corresponding to the image features of the CI frame, wherein the reference frame of the CI frame (e.g., the reconstructed image of the previous P frame) is used in the process of obtaining the coding probability of the image features of the CI frame.

[0221] For example, in Fig.11 During the training of the neural network in (hereinafter referred to as CI frame coding model), the Loss module can obtain the loss function of the CI frame coding model and cyclically train the residual coding model based on the loss function. The specific training process is as follows:

[0222] Step 1. A training device (such as Huawei Cloud or a server, etc., not limited in this application) trains the GOP.

[0223] Exemplarily, during the training of GOP by the training device, the I frame is compressed (i.e., image encoded) by the image compression model to obtain the bit rate and reconstructed image of the I frame. The image compression model corresponding to the I frame (also referred to as an image compression neural network, which is not limited in this application) may include but is not limited to: an encoding network, a decoding network, and an image reconstruction network. The specific model may refer to any one of the existing technologies, which is not limited in this application.

[0224] For example, the training device can be Fig.11 The CI frame coding model shown compresses each P frame in the GOP to obtain a reconstructed image corresponding to each P frame. and code rate R i Among them, the bit rate of P frame is R i Including characteristic code rate and optical flow rate

[0225] Step 2. Training the device based on each P frame and the reconstructed image of each P frame and code rate Ri , and the bit rate R of the I frame 0 , get the loss function loss of the compression model.

[0226] Exemplarily, the training device calculates the loss function loss of the compression model based on formula (2):

[0227]

[0228] Among them, R 0 is the bit rate of the I frame. Represents the loss function of a single P frame. D i represents the distortion loss value of each frame. The training device can be based on P frames (i.e. Fig.11 x in t ) and the reconstructed image of the P frame Determine the distortion loss value D corresponding to the P frame i . β i Used to adjust the rate-distortion ratio.

[0229] Optionally, the training device may also obtain the loss value of the compression model based on formula (2), which is not limited in this application.

[0230] Step 3. Based on the loss function of the compression model, iteratively train the compression model.

[0231] For example, as described above, for training the neural network, the weights in the compression model may be adjusted based on the loss function, and the above training steps may be repeated until the training cutoff condition is reached. The compression model includes a neural network for training I frames and a neural network for training P frames (e.g. Fig.11 CI frame coding model in ).

[0232] In the embodiment of the present application, the training cutoff condition includes but is not limited to: the loss function reaches the loss function threshold, the number of training times reaches the training times threshold, and / or the fluctuation value of the loss function remains within a certain range. The above thresholds can be set according to actual needs and are not limited in this application.

[0233] For example, Fig.11 The CI frame coding model shown can realize an end-to-end video compression framework, and its decoding end only uses the image features of the current frame in the image reconstruction process, so there is no accumulated error. However, since the CI frame coding model uses image features for probabilistic prediction and obtains the way to reconstruct the image, its prediction result may be inaccurate, making the RD performance of the CI frame coding model weaker than the RD performance of the residual coding model.

[0234] The present application embodiment provides a compression model, such as Fig.12As shown, the compression model may also be referred to as a codec model or a codec neural network, etc., which is not limited in this application. The compression model is used to compress the training data in training and the image sequence in the practical scenario (such as the test scenario). The compression model includes but is not limited to: a first codec model (also referred to as a first neural network), a second codec model (also referred to as a second neural network), and a third codec model (also referred to as a third neural network).

[0235] In the embodiment of the present application, the first coding model is used to compress (ie, encode and / or decode) I frames, the second coding model is used to compress P frames, and the third coding model is used to compress CI frames.

[0236] The first coding model includes but is not limited to: a coding network, a decoding network, an entropy estimation network, etc. The I frame is used as the input of the first coding model, and the first coding model compresses the I frame and outputs the bit rate and reconstructed image of the I frame. The first coding model can adopt any possible architecture, which is not limited in this application.

[0237] In the embodiment of the present application, the second coding model can be optionally Figure 6 or Figure 7 The residual coding model shown. The P frame and the reconstructed image of the reference frame are used as inputs of the second coding model, and the second coding model compresses the P frame and outputs the bit rate of the P frame and the reconstructed image of the P frame.

[0238] In the embodiment of the present application, the third coding model can be optionally Fig.10 or Fig.11 The CI frame coding and decoding model shown. The CI frame and the reconstructed image of the reference frame are used as inputs of the third coding and decoding model, and the third coding and decoding model compresses the CI frame and outputs the code rate of the CI frame and the reconstructed image of the CI frame.

[0239] based on Fig.12 The compression model shown in the embodiment of the present application provides a training method, which can reduce the loss value of the compression model by adding several frames during the training process, so that the cumulative error of the trained compression model can be reduced during use (such as the testing process).

[0240] In one possible implementation, in an embodiment of the present application, a CI frame can be added to compete with the P frame. Based on the fact that the code prior condition of the CI frame is the characteristic of the reconstructed image of the P frame, the loss of the compression model in the training process is reduced, and the quality stability of the P frame is indirectly constrained to achieve the effect of reducing the cumulative error.

[0241] In another possible implementation, the embodiment of the present application can simulate the GOP length that may be encountered in the test scenario by adding inserted frames (also called virtual P frames), so that the cumulative error phenomenon can be encountered during the training process, so that the cumulative error of the trained compression model in the actual application scenario can be reduced.

[0242] First, the method of adding CI frames for training is described in detail.

[0243] like Fig.13 The following is a schematic diagram of the training image sequence. Fig.13 In this example, the GOP of the training image sequence is set to n. For example, n can be 6 or 7, that is, the GOP includes 6 image frames, which is not specifically limited in this application. 0 For I frame, other frames are x 1 ,x 2 …,x n-1 The structure of the GOP is P frame, that is, the GOP is an IPPPP structure. In the training process, the GOP is defined by an I frame, that is, the first frame of each GOP is an I frame, and each GOP includes only one I frame. The GOP structure in the embodiment of the present application is only an illustrative example to better illustrate the difference between an I frame and a CI frame. In practical applications, a GOP may also include but is not limited to multiple I frames (for example, Figure 8 The GOP structure shown in FIG. 1 is not limited in this application.

[0244] Still refer to Fig.13 In the training image sequence, a CI frame is added to the end of the GOP, that is, the last frame in the training image sequence is a CI frame. Optionally, if the training image sequence includes multiple I frames, at least one I frame in the middle (i.e., except the first frame) can also be replaced with a CI frame, that is, the original I frame is trained using the third codec model. Optionally, the training image sequence can also include more image frames, for example, multiple GOPs. Fig.13 The CI frame in the GOP in the training image sequence can also be the first I frame of the next GOP. The setting method of the training image sequence can be set according to actual needs and is not limited in this application. Fig.13 Optionally, the CI frame may also replace the P frame, which is not limited in this application.

[0245] like Fig.13As shown, in an embodiment of the present application, the first coding model in the compression model compresses the I frame and outputs the compression information of the I frame (including but not limited to the bit rate and reconstructed image, etc.). The second coding model compresses the P frames one by one, wherein the reconstructed image of the reference frame of each P frame (for example, the I frame or the previous P frame) is used in the compression process of the P frame. The second coding model outputs the compression information corresponding to a single P frame (including but not limited to the bit rate and reconstructed image, etc.). The third coding model compresses the CI frame, wherein the reconstructed image of the reference frame of the CI frame (for example, the previous P frame) is used in the compression process of the CI frame. The third coding model outputs the compression information of the CI frame (including but not limited to the bit rate and reconstructed image, etc.).

[0246] It should be noted that, as described above, the GOP in the embodiment of the present application is an IPPPP structure, and in other embodiments, the training sequence may include multiple I frames.

[0247] Fig.14 For an exemplary training flow diagram, please refer to Fig.14 , the specific training process is as follows:

[0248] Step 1. Compress the I frame through the first encoding and decoding model, and output the bit rate and reconstructed image of the I frame.

[0249] For example, the I frame in the training image sequence (denoted as x 0 ) as the input of the first codec model, the first codec model compresses the I frame and outputs the bit rate R of the I frame 0 and reconstruct the image

[0250] Step 2: Compress n-1 P frames one by one through the second encoding and decoding model, and output the bit rate R of a single P frame i and reconstruct the image

[0251] For example, Fig.14 As shown, P frame x 1 (Also called x 1 Frame, hereinafter referred to as x 1 , other frames are similar and will not be described below) as an example. 1 As the input of the second codec model, the second codec model processes x 1 Compress and output x 1 (i.e. P frame) bit rate R 1 and reconstruct the image Among them, the reconstructed image of I frame For the second codec model pair x 1 The specific compression process can be found in Figure 7The compression process of the residual encoding and decoding model in will not be repeated here.

[0252] Still refer to Fig.14 , x 2 As the input of the second codec model, the second codec model processes x 2 Compress and output x 2 (i.e. P frame) bit rate R 2 and reconstruct the image Among them, x 1 Reconstructed image of frame For the second codec model pair x 2 The specific compression process can be found in Figure 7 The compression process of the residual encoding and decoding model in will not be repeated here.

[0253] Exemplarily, as described above, the bit rate R of each P frame i Including optical flow bit rate and residual bit rate For example, P frame x 1 The code rate R 1 Including optical flow bit rate and residual bit rate Other P frames are similar and will not be described one by one here.

[0254] Exemplarily, the second encoding and decoding model compresses each P frame according to the above process, and the bit rate R corresponding to each P frame can be obtained. i and reconstruct the image

[0255] Step 3. Reconstructed image based on n-1 P frames and P frames Calculate the distortion value loss D corresponding to each P frame i ,i=1,2,…,n-1.

[0256] For example, the training device may be based on the original image of each P frame and the reconstructed image corresponding to the P frame. Calculate the distortion value loss D corresponding to each P frame i The calculation method can refer to the existing technology and is not limited in this application.

[0257] Step 4: Compress the CI frame through the third encoding and decoding model and output the code rate R of the CI frame n and reconstruct the image

[0258] For example, Fig.14 As shown, the CI frame is used as the input of the third codec model in the compression model. The third codec model compresses the CI frame and outputs the code rate R of the CI frame. n and reconstruct the image The reference frame of the CI frame (for example, the previous P frame) is used for the compression process of the CI frame by the third coding model. Fig.11 The relevant description will not be repeated here.

[0259] For example, the code rate R of the CI frame n Including optical flow bit rate and feature rate

[0260] Step 5. CI frame and reconstructed image based on CI frame Calculate the distortion value loss D corresponding to the CI frame n .

[0261] For example, the training device may be based on the original image of the CI frame and the reconstructed image corresponding to the CI. Calculate the distortion value loss D corresponding to the CI frame n The calculation method can refer to the existing technology and is not limited in this application.

[0262] Step 6. Get the loss value loss of the compression model.

[0263] Exemplarily, the training device may calculate the loss value corresponding to the compression model in this iteration based on formula (2) or formula (3).

[0264] Step 7. Update the compression model parameters based on the loss value loss.

[0265] Exemplarily, the training device can determine whether the training cutoff condition is met based on the loss value loss of the compression model. In an embodiment of the present application, the training cutoff condition includes but is not limited to: the loss function reaches the loss function threshold, the number of training times reaches the training times threshold, and / or the fluctuation value of the loss function remains within a certain range. The above thresholds can be set according to actual needs and are not limited in this application.

[0266] In one example, if the training deadline is met, the training is stopped and the trained compressed model is saved.

[0267] In another example, Fig.14 As shown, if the training cutoff condition is not met, the weight value of the compression model (ie, the parameters of the compression model) is updated, and the training is iterated again, that is, training steps 1 to 7 are repeated.

[0268] In a possible implementation, in order to make the CI frame have higher quality (ie, the quality of the reconstructed image is higher), when calculating the loss value loss of the CI frame, its corresponding β value can be a multiple of the β value corresponding to other P frames, and the multiple is greater than 1.

[0269] In another possible implementation, the length of the GOP in each iteration may be gradually lengthened, which is not limited in this application.

[0270] In another possible implementation, the training image sequence used in each iteration process may be the same or different, which is not limited in this application.

[0271] In another possible implementation, the trained compression model can be deployed on a target electronic device, which may be a terminal or other device, which is not limited in this application. The compression model can compress (i.e., encode or decode) images captured by the electronic device. In actual use, the first codec model in the compression model can be used to compress the first I frame of each sequence. The second codec model can be used to compress the P frames in the sequence. The third codec model can be used to compress at least one intermediate CI frame in each sequence. The specific usage can be set according to actual needs, which is not limited in this application.

[0272] Fig.15 For example, based on Fig.14 The RD performance test results of the trained compression model are shown in . Fig.15 As shown, in the test scenario, the compression model tests the video HEVC test sequence. The video HEVC test sequence may include multiple test sequences, such as test sequence 1, test sequence 2, or test sequence 3. Optionally, the test sequence may also be class B-D, which is not limited in this application.

[0273] like Fig.15 As shown, in the result diagram corresponding to each test sequence, the solid line represents Fig.14 The dotted line shows the test results of the compression model in the embodiment of the present application, and the dotted line shows the test results of the traditional codec model. As can be seen from the figure, the RD performance of the compression model in the embodiment of the present application during the test is significantly better than the RD performance of the traditional codec model. In other words, by adding the training method of CI frames, the RD performance of the trained compression model can be significantly improved.

[0274] Fig.16 For example, based on Fig.14 The test results of the trained compression model are shown in . Fig.16 As shown, the solid line represents Fig.14 The dotted line shows the test results of the compression model in the embodiment of the present application, and the dotted line shows the test results of the traditional codec model. As can be seen from the figure, the compression model in the embodiment of the present application is better than the traditional compression model in terms of loss, PSNR and bpp during the test.

[0275] Cause analysis: This application replaces the I frame with the CI frame. Since the CI frame predicts the current frame based on the reference frame during the compression process, the CI frame can reduce inter-frame redundancy, so that the code rate of the CI frame is lower than that of the I frame. In addition, during the training process, the reconstructed image of the reference frame of the CI frame (for example, the previous P frame) (wherein the reconstructed image of the previous P frame can also be understood as a reference frame in the compression process of the CI frame) is used for the compression process of the CI frame. Therefore, in order to make the loss of the CI frame smaller and to make the overall (i.e., compression model) loss smaller, the quality of the reference frame of the CI frame (i.e., the reconstructed image of the previous P frame) needs to be improved. Therefore, in the process of repeated iterations, in order to make the loss of the compression model smaller, the smaller the loss of the CI frame, the lower its corresponding code rate, and accordingly, the higher the quality of the reconstructed image of each P frame, thereby achieving a stable constraint on the quality of the reconstructed image of the P frame. In this way, when the trained compression model is put into the test scene, the quality of the reconstructed image corresponding to each compressed image frame is higher than the quality of the reconstructed image of the traditional compression model, and the loss value is smaller.

[0276] The following is a detailed description of the training method for adding virtual P frames described above. Fig.17 The following is a schematic diagram of the training image sequence. Fig.17 In this example, the GOP of the training image sequence is set to n. For example, n can be 6 or 7, that is, the GOP includes 6 image frames. Among them, the first frame x of the GOP 0 For I frame, other frames are x 1 ,x 2 …,x n-1 The structure of the GOP is P frame, that is, the GOP is an IPPPP structure. In the training process, the GOP is defined by an I frame, that is, the first frame of each GOP is an I frame, and each GOP includes only one I frame. The GOP structure in the embodiment of the present application is only an illustrative example to better illustrate the difference between an I frame and a CI frame. In practical applications, a GOP may also include but is not limited to multiple I frames (for example, Figure 8 The GOP structure shown in FIG. 1 is not limited in this application.

[0277] Still refer to Fig.17, in the training image sequence, k (k>1) virtual P frames are added to the GOP. In one example, the training device may copy the I frame to obtain k copied virtual P frames, and insert them into any position in the GOP. Optionally, the k virtual P frames are inserted before the n P frames in the original GOP, so that the P frame can be closer to the position in the GOP in the actual test environment. In another example, the training device may copy the P frame to obtain k copied virtual P frames, and insert them into the GOP. In another example, the training device may also copy the I frame and the P frame to obtain k virtual P frames, wherein some of the k virtual P frames are obtained by copying the I frame, and the other part is obtained by copying the P frame, which is not limited in this application.

[0278] like Fig.17 As shown, in an embodiment of the present application, the first coding model in the compression model compresses the I frame and outputs the compression information of the I frame (including but not limited to the bit rate and reconstructed image, etc.). The second coding model compresses the P frames (including n-1 P frames and k virtual P frames) one by one, wherein the reconstructed image of the reference frame of each P frame (for example, the I frame or a previous P frame) is used for the compression process of the P frame. The second coding model outputs the compression information corresponding to a single P frame (including but not limited to the bit rate and reconstructed image, etc.).

[0279] Fig.18 For an exemplary training flow diagram, please refer to Fig.18 , the specific training process is as follows:

[0280] Step 1. Compress the I frame through the first encoding and decoding model, and output the bit rate and reconstructed image of the I frame.

[0281] For example, the I frame in the training image sequence (denoted as x 0 ) as the input of the first codec model, the first codec model compresses the I frame and outputs the bit rate R of the I frame 0 and reconstruct the image

[0282] Step 2: Compress the k virtual P frames one by one through the second encoding and decoding model, and output the bit rate R of a single virtual P frame i ′ and reconstructed image

[0283] For example, Fig.18 As shown, with virtual P frame x 1 ′ (also called x 1 'frame, hereinafter referred to as x 1 ′, other frames are similar and will not be described below) as an example. 1 ' is used as the input of the second codec model, and the second codec model is used to1 'Compress and output x 1 ' (i.e. virtual P frame) code rate R 1 ′ and reconstructed image Among them, the reconstructed image of I frame For the second codec model pair x 1 The specific compression process can be found in Figure 7 The compression process of the residual encoding and decoding model in will not be repeated here.

[0284] Still refer to Fig.18 , x 2 ' is used as the input of the second codec model, and the second codec model is used to 2 'Compress and output x 2 ' (i.e. virtual P frame) code rate R 2 ′ and reconstructed image Among them, x 1 'Reconstructed image of the frame For the second codec model pair x 2 The specific compression process can be found in Figure 7 The compression process of the residual encoding and decoding model in will not be repeated here.

[0285] Exemplarily, as described above, the bit rate R of each virtual P frame is i ′Including optical flow rate and residual bit rate For example, virtual P frame x 1 The code rate R 1 ′Including optical flow rate and residual bit rate Other virtual P frames are similar and will not be described one by one here.

[0286] Exemplarily, the second encoding and decoding model compresses each virtual P frame according to the above process, and the bit rate R corresponding to each virtual P frame can be obtained. i ′ and reconstructed image It should be noted that in the subsequent loss calculation process, the bit rate of the virtual P frame does not participate in the loss calculation, but in the compression process, the compression model still outputs the corresponding bit rate.

[0287] Step 3: Compress n-1 P frames one by one through the second encoding and decoding model, and output the bit rate R of a single P frame. i and reconstruct the image

[0288] For example, Fig.18 As shown, P frame x 1 (Also called x 1 Frame, hereinafter referred to as x1 , other frames are similar and will not be described below) as an example. 1 As the input of the second codec model, the second codec model processes x 1 Compress and output x 1 (i.e. P frame) bit rate R 1 and reconstruct the image Among them, the reconstructed image of I frame For the second codec model pair x 1 The specific compression process can be found in Figure 7 The compression process of the residual encoding and decoding model in will not be repeated here.

[0289] Still refer to Fig.18 , x 2 As the input of the second codec model, the second codec model processes x 2 Compress and output x 2 (i.e. P frame) bit rate R 2 and reconstruct the image Among them, x 1 Reconstructed image of frame For the second codec model pair x 2 The specific compression process can be found in Figure 7 The compression process of the residual encoding and decoding model in will not be repeated here.

[0290] Exemplarily, as described above, the bit rate R of each P frame i Including optical flow bit rate and residual bit rate For example, P frame x 1 The code rate R 1 Including optical flow bit rate and residual bit rate Other P frames are similar and will not be described one by one here.

[0291] Exemplarily, the second encoding and decoding model compresses each P frame according to the above process, and the bit rate R corresponding to each P frame can be obtained. i and reconstruct the image

[0292] Step 4. Reconstructed image based on n-1 P frames and P frames Calculate the distortion value loss D corresponding to each P frame i ,i=1,2,…,n-1.

[0293] For example, the training device may be based on the original image of each P frame and the reconstructed image corresponding to the P frame. Calculate the distortion value loss D corresponding to each P frame iThe calculation method can refer to the existing technology and is not limited in this application.

[0294] Step 5. Get the loss value loss of the compression model.

[0295] Exemplarily, the training device may calculate the loss value corresponding to the compression model in this iteration based on formula (2). When calculating the loss, the bit rate of the k virtual P frames does not participate in the loss calculation. In other words, the training device calculates the loss of the compression model based on the bit rate and distortion value corresponding to the I frame and n-1 P frames.

[0296] Step 6. Update the compression model parameters based on the loss value loss.

[0297] The detailed description can be found above and will not be repeated here.

[0298] In a possible implementation, k can be set based on the length of the image sequence in the test scene. For example, usually, the length of the image sequence in the test scene may reach 100, while the length of the original GOP in the training scene (i.e., n) is usually 6 (only for illustrative example). Accordingly, the value of k can be selected from 1 to 94 (i.e., 100 minus 6).

[0299] In the test scenario, based on Fig.18 The trained compression model shown (i.e., including the first encoding model and the second encoding model) tests the video HEVC test sequence, wherein the test sequence includes test sequences corresponding to different resolutions. After the compression model in the embodiment of the present application tests the image sequences corresponding to different resolutions, the corresponding test results show that the compression model in the embodiment of the present application has a smaller BD-rate than the traditional compression model, which can be improved by an average of 2.63%. It can be seen that in the embodiment of the present application, by adding a virtual training sequence, the P frame in the original image can simulate the position in the GOP in the actual test scene, thereby enhancing the model's ability to reconstruct the entire GOP and reducing the cumulative error during the test.

[0300] Fig.19 For an exemplary training flow diagram, please refer to Fig.19 In the embodiment of the present application, the training of the compression model can be combined with Fig.14 and Fig.18The training method is to add CI frames to the GOP and add virtual P frames (also called inserted frames, which are not limited in this application). Specifically, during the training process, the first codec model compresses the I frame, the second codec model compresses k virtual P frames and n-1 P frames, and the third codec model compresses the CI frame, and obtains the loss value of the compression model, so as to further update the parameters of the compression model based on the loss value of the compression model, and iterate the training until the training cutoff condition is met. The specific training process can be referred to Fig.14 and Fig.18 The relevant description of is not repeated here. Fig.19 The compression model (including the first codec model, the second codec model and the third codec model) trained by the training method shown can effectively reduce the cumulative error and improve the RD performance in actual use. In addition, in the embodiment of the present application, the various indicators of the compression model can be effectively improved through the training process without affecting the test complexity in the test scenario.

[0301] Fig. 20 FIG. 2 is a schematic diagram of the structure of the training device 2000. Fig. 20 , specifically comprising: a first acquisition module 2001, a second acquisition module 2002 and an update module 2003. The first acquisition module 2001 is used to input the training image sequence into the compression model to obtain the compression information of the training image sequence output by the compression model. The second acquisition module 2002 is used to obtain the loss value of the compression model based on the compression information corresponding to each image in the training image sequence. The update module 2003 is used to update the weight value of the compression model based on the loss value of the compression model. The training image sequence includes a first image frame, a plurality of intermediate frames and a second image frame. The compression information of the intermediate frame includes first compression information, which is obtained by the compression model based on the residual feature parameters of the intermediate frame, and the residual feature parameters are obtained based on the reference frame of the intermediate frame; the compression information of the second image frame includes second compression information, which is obtained by the compression model based on the coding probability of the image feature of the second image frame, and the coding probability of the image feature of the second image frame is obtained based on the reference frame of the second image frame.

[0302] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0303] In a possible implementation, the first image frame is an I frame, the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

[0304] In a possible implementation manner, the first image frame is a first frame, and at least one inserted frame is after the first image frame.

[0305] In one possible implementation, the compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes compression information of the first image frame, compression information of at least one P frame, and compression information of the second image frame; wherein, at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

[0306] In one possible implementation, the compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress at least one intermediate frame, and the third neural network model is used to compress the second image frame.

[0307] In a possible implementation, the compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes a bit rate of the first image frame.

[0308] In a possible implementation, the compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame, and the reconstructed image of the intermediate frame.

[0309] In a possible implementation, the compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes second compression information, the optical flow code rate of the second image frame, and the reconstructed image of the second image frame.

[0310] In a possible implementation, the second acquisition module is specifically used to: acquire the distortion rate of the intermediate frame based on the reconstructed image of the intermediate frame and the intermediate frame; acquire the loss value of the intermediate frame based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame and the first compression information of the intermediate frame.

[0311] In a possible implementation, the second acquisition module is specifically used to: acquire the distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; acquire the loss value of the second image frame based on the distortion rate of the second image frame, the optical flow code rate of the second image frame and the second compression information of the second image frame.

[0312] In a possible implementation, the second acquisition module is specifically used to acquire the loss value of the compression model based on the bit rate of the first image frame, the loss value of each intermediate frame in at least one intermediate frame, and the loss value of the second image frame.

[0313] In one possible implementation, the update module is specifically used to: when the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training times threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

[0314] In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application-specific integrated circuit (application-specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware coding processor to be executed, or the hardware and software modules in the coding processor are combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0315] The memory mentioned in the above embodiments may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0316] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0317] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0318] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0319] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0320] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0321] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to make a computer device (personal computer, server, or network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. Or the computer software product can be transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD), a read-only memory (ROM), a random access memory (RAM)), etc.

[0322] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiment of the present application, which should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A training method, characterized in that: include: Inputting a training image sequence into a compression model, and obtaining compression information of the training image sequence output by the compression model; Obtaining a loss value of the compression model based on compression information corresponding to each image in the training image sequence; Based on the loss value of the compression model, updating the weight value of the compression model; Wherein, the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; The compression information of the intermediate frame includes first compression information, where the first compression information is obtained by the compression model based on residual feature parameters of the intermediate frame, where the residual feature parameters are obtained based on a reference frame of the intermediate frame; The compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on a reference frame of the second image frame.

2. The method according to claim 1, characterized in that The first image frame is an I frame, and the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

3. The method according to claim 2, characterized in that The first image frame is a first frame, and the at least one inserted frame is after the first image frame.

4. The method according to claim 3, characterized in that The compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of the at least one P frame and the compression information of the second image frame; wherein the at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

5. The method according to any one of claims 1 to 4, characterized in that: The compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress the at least one intermediate frame, and the third neural network model is used to compress the second image frame.

6. The method according to claim 5, characterized in that The compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes the bit rate of the first image frame.

7. The method according to claim 6, characterized in that The compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame and the reconstructed image of the intermediate frame.

8. The method according to claim 7, characterized in that The compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame and the reconstructed image of the second image frame.

9. The method according to claim 8, characterized in that The obtaining, based on compression information corresponding to each image in the training image sequence, a loss value corresponding to the training image sequence, comprises: Based on the reconstructed image of the intermediate frame and the intermediate frame, obtaining a distortion rate of the intermediate frame; A loss value of the intermediate frame is acquired based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame, and the first compression information of the intermediate frame.

10. The method according to claim 8 or 9, characterized in that: The obtaining the loss value of the compression model based on the compression information corresponding to each image in the training image sequence includes: acquiring a distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; A loss value of the second image frame is acquired based on the distortion rate of the second image frame, the optical flow code rate of the second image frame, and the second compression information of the second image frame.

11. The method according to any one of claims 8 to 10, characterized in that The obtaining the loss value of the compression model based on the compression information corresponding to each image in the training image sequence includes: A loss value of the compression model is obtained based on the bit rate of the first image frame, the loss value of each intermediate frame in the at least one intermediate frame, and the loss value of the second image frame.

12. The method according to any one of claims 1 to 11, characterized in that The updating of the weight value of the compression model based on the loss value of the compression model includes: When the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training times threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

13. A training device, characterized in that: include: A first acquisition module, used for inputting a training image sequence into the compression model, and acquiring compression information of the training image sequence output by the compression model; A second acquisition module, used to acquire a loss value of the compression model based on compression information corresponding to each image in the training image sequence; An updating module, used for updating a weight value of the compression model based on a loss value of the compression model; Wherein, the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; The compression information of the intermediate frame includes first compression information, where the first compression information is obtained by the compression model based on residual feature parameters of the intermediate frame, where the residual feature parameters are obtained based on a reference frame of the intermediate frame; The compression information of the second image frame includes second compression information, which is obtained by the compression model based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on a reference frame of the second image frame.

14. The device according to claim 13, characterized in that The first image frame is an I frame, and the multiple intermediate frames include at least one P frame and at least one inserted frame, and the at least one inserted frame is obtained by copying the I frame or the P frame.

15. The device according to claim 14, characterized in that The first image frame is a first frame, and the at least one inserted frame is after the first image frame.

16. The device according to claim 15, characterized in that The compression information of the training image sequence is obtained after the compression model compresses the training image sequence, and the compression information of the training image sequence includes the compression information of the first image frame, the compression information of the at least one P frame and the compression information of the second image frame; wherein the at least one inserted frame is used for the compression process of the compression model on the training image sequence, and does not participate in the process of obtaining the loss value of the compression model.

17. The device according to any one of claims 13 to 16, characterized in that The compression model includes a first neural network model, a second neural network model and a third neural network model, the first neural network model is used to compress the first image frame, the second neural network model is used to compress the at least one intermediate frame, and the third neural network model is used to compress the second image frame.

18. The device according to claim 17, characterized in that The compression information of the first image frame is output by the first neural network model, and the compression information of the first image frame includes the bit rate of the first image frame.

19. The device according to claim 18, characterized in that The compression information of the intermediate frame is output by the second neural network model, and the compression information of the intermediate frame includes the first compression information, the optical flow code rate of the intermediate frame and the reconstructed image of the intermediate frame.

20. The device according to claim 19, characterized in that The compression information of the second image frame is output by the third neural network model, and the compression information of the second image frame includes the second compression information, the optical flow code rate of the second image frame and the reconstructed image of the second image frame.

21. The device according to claim 20, characterized in that The second acquisition module is specifically used for: Based on the reconstructed image of the intermediate frame and the intermediate frame, obtaining a distortion rate of the intermediate frame; A loss value of the intermediate frame is acquired based on the distortion rate of the intermediate frame, the optical flow code rate of the intermediate frame, and the first compression information of the intermediate frame.

22. The device according to claim 20 or 21, characterized in that The second acquisition module is specifically used for: acquiring a distortion rate of the second image frame based on the reconstructed image of the second image frame and the second image frame; A loss value of the second image frame is acquired based on the distortion rate of the second image frame, the optical flow code rate of the second image frame, and the second compression information of the second image frame.

23. The device according to any one of claims 20 to 22, characterized in that The second acquisition module is specifically used for: A loss value of the compression model is obtained based on the bit rate of the first image frame, the loss value of each intermediate frame in the at least one intermediate frame, and the loss value of the second image frame.

24. The device according to any one of claims 13 to 23, characterized in that The update module is specifically used for: When the loss value of the compression model reaches a first threshold, or the fluctuation of the loss value of the compression model during multiple training processes reaches a second threshold, or the number of training times reaches a training times threshold, stop updating the weight value of the compression model based on the loss value of the compression model.

25. A coding method, characterized in that: The method comprises: Acquiring video data or image data; Encoding the video data or the image data based on a coding network to obtain a code stream; wherein the coding network is trained based on a training image sequence, and the training image sequence includes a first image frame, a plurality of intermediate frames, and a second image frame; compression information corresponding to each image in the training image sequence is used to obtain a loss value of the coding network, and the loss value of the coding network is used to update a weight value of the coding network; The compression information of the intermediate frame includes first compression information, where the first compression information is obtained by the encoding network based on residual feature parameters of the intermediate frame, where the residual feature parameters are obtained based on a reference frame of the intermediate frame; The compression information of the second image frame includes second compression information, which is obtained by the encoding network based on the encoding probability of the image features of the second image frame, and the encoding probability of the image features of the second image frame is obtained based on the reference frame of the second image frame.

26. A decoding method, characterized in that: The method comprises: Receive code stream; The bitstream is decoded based on a decoding network to obtain reconstructed video data; wherein the decoding network is obtained based on the training method described in any one of claims 1 to 12.

27. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 12.

28. A coding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the encoding device executes the method according to claim 25.

29. A decoding device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the decoding device performs the method according to claim 26.

30. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed on a computer or a processor, the computer or the processor executes the method according to any one of claims 1 to 12.

31. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 12.

Images