Image coding method and apparatus

Abstract

The application provides an image coding method and device. The image coding method comprises the following steps: obtaining a first image feature to be processed; performing nonlinear transformation processing on the first image feature to obtain a processed image feature, wherein the nonlinear transformation processing comprises a first nonlinear operation, convolution processing and point-by-point multiplication operation in sequence; and performing coding according to the processed image feature to obtain a code stream. The application can avoid the limitation of convolution parameters, so as to realize efficient nonlinear transformation processing in a coding / decoding network, and further improve the rate-distortion performance in an image / video compression algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image coding method and device. BACKGROUND

[0002] With the performance of convolution neural network (CNN) far exceeding traditional algorithms in computer vision tasks such as image recognition and target detection, more and more researchers have begun to explore image / video compression methods based on deep learning. Some researchers design an end-to-end deep learning image / video compression algorithm, for example, the coding network, the entropy estimation network, the entropy coding network, the entropy decoding network, the decoding network and other modules are optimized as a whole, wherein the coding network and the decoding network can also be called transformation module and inverse transformation module, which are generally composed of convolution layers and nonlinear transformation units.

[0003] The nonlinear transformation unit is one of the basic components of the image / video compression network, and the pros and cons of its nonlinear characteristics directly affect the rate-distortion performance of the compression algorithm. Therefore, designing a more efficient nonlinear transformation unit is the key to further improving the rate-distortion performance of the image / video compression algorithm. SUMMARY

[0004] The embodiments of the present application provide an image coding method and device, which can realize efficient nonlinear transformation processing in the coding / decoding network and further improve the rate-distortion performance of the image / video compression algorithm.

[0005] In a first aspect, the embodiments of the present application provide an image coding method, comprising: obtaining a first image feature to be processed; performing nonlinear transformation processing on the first image feature to obtain a processed image feature, wherein the nonlinear transformation processing comprises, in sequence, first nonlinear operation, convolution processing and point-by-point multiplication operation; and encoding the processed image feature to obtain a bitstream.

[0006] The first image feature is obtained by converting the image to be processed from the image domain to the feature domain at the encoding end, and the conversion can include but is not limited to: 1. Convolution processing, using convolution layers to extract features, with local receptive field and weight sharing mechanism (i.e. each filter slides the input feature). 2. Using MLP or fully connected layer to extract features, with global receptive field characteristics, and weight sharing. 3. Transformer processing, which includes matrix multiplication, MLP and normalization processing, with global receptive field characteristics and strong long-distance dependence capturing ability.

[0007] In a possible implementation, a non-linear operation is performed on each feature value in the first image feature to obtain a second image feature; a convolution operation is performed on the second image feature to obtain a third image feature, and the feature values in the third image feature correspond to the feature values in the first image feature; and a point-by-point multiplication operation is performed on the corresponding feature values in the first image feature and the third image feature to obtain a processed image feature.

[0008] The first non-linear operation is an operation performed on each feature value in the first image feature, and can include an absolute value operation, a ReLU series, a Sigmoid, a Tanh, or the like.

[0009] After the first non-linear operation, the first image feature is converted into the second image feature, and the first image feature and the second image feature can be represented in the form of a matrix. Since the first non-linear operation is performed on each feature value in the first image feature, each feature value in the first image feature has a corresponding feature value in the second image feature. Therefore, the matrix corresponding to the second image feature has the same size as the matrix corresponding to the first image feature, and the feature values (matrix elements) at the same positions correspond to each other. For example, if the first image feature is represented in the form of a 3x3 matrix, the second image feature can also be represented in the form of a 3x3 matrix. However, since the first image feature and the second image feature have undergone the first non-linear operation, the feature values in the first image feature and the second image feature are not completely the same, and correspondingly, the element values in the matrices corresponding to the first image feature and the second image feature are also not completely the same.

[0010] The convolution operation is performed on the second image feature to output the third image feature, which can be regarded as a local response (i.e., a corrected value) of the second image feature. That is, the third image feature is a response signal obtained by performing the convolution operation on the second image feature. Since the receptive field of the convolution operation is limited, the response value at each position in the image feature output by the convolution operation is only related to the input feature values at the positions adjacent to the position, and therefore the response is called a local response.

[0011] Therefore, the local attention mechanism is realized through the above nonlinear operation, convolution processing and point-by-point multiplication operation. The local refers to that the nonlinear operation is performed point by point, and each feature value of the input only needs its own characteristics to obtain the output value, without considering the influence of the surrounding feature values. The attention mechanism refers to that all feature values in the first image feature are important or redundant. The output of the convolution processing can be the weight of each feature value in the image feature, which can correct the important feature value and suppress the redundant feature value. The point-by-point multiplication operation corrects the value of each feature value on the first image feature using the foregoing local information, without requiring the convolution parameter to be a positive number, avoiding the limitation of the value space of the convolution parameter, and obtaining more optimal convolution parameters in a wider value space, thereby achieving better compression performance on the image.

[0012] In a possible implementation, the nonlinear transformation processing further includes a point-by-point addition operation after the point-by-point multiplication operation.

[0013] In a possible implementation, the nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; the convolution processing is performed on the second image feature to obtain a third image feature, and the plurality of feature values in the third image feature correspond to the plurality of feature values in the first image feature; the point-by-point multiplication operation is performed on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, and the plurality of feature values in the fourth image feature correspond to the plurality of feature values in the first image feature; and the point-by-point addition operation is performed on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain the processed image feature.

[0014] The point-by-point addition operation is a residual structure with a first addition, which can make the coding and decoding network using the above processing process more easily converge during training. The convolution processing conv1(x) and the convolution processing conv2(x) are similar, and the difference lies in that the bias parameter β of the convolution in the convolution processing conv2(x) is additionally added by 1. In this way, the conversion between the two implementation manners can be realized by fine-tuning the bias parameter β of the convolution in the convolution, that is, if the nonlinear transformation unit does not include the point-by-point addition operation, the convolution processing conv1(x) can be used; if the nonlinear transformation unit includes the point-by-point addition operation, the convolution processing conv2(x) can be used. After obtaining the processed image feature, the encoding side can continue to perform convolution processing thereon, or perform nonlinear transformation processing on the feature output by the convolution processing again after the convolution processing, and then encode the result feature of the foregoing processing to obtain the code stream.

[0015] The embodiments of the present application can change the nonlinear transformation processing in the encoding network, so that each input feature value can obtain an output value only by its own characteristics, without considering the influence of the surrounding feature values, and the original feature values are modified to highlight important feature values and suppress redundant feature values. In addition, the value of each feature value in the first image feature can be corrected, avoiding the limitation of convolution parameters, thereby realizing efficient nonlinear transformation processing in the encoding network and further improving the rate-distortion performance of the image / video compression algorithm.

[0016] The nonlinear operation is performed on each feature value in the first image feature, and can include a piecewise linear mapping method, which can be absolute value (taking the absolute value of the input feature value), or a rectified linear unit (ReLU) or a leaky rectified linear unit (LeakyReLU). ReLU is a piecewise linear mapping method, which outputs 0 for feature values less than 0 and outputs the same for feature values greater than or equal to 0. LeakyReLU is a piecewise linear mapping method that scales the input feature value less than 0 by a pre-set weight, which is usually 0.01. In addition, the nonlinear operation can also include other methods, such as sigmoid, which is not limited in the embodiments of the present application.

[0017] In a possible implementation, the nonlinear transformation processing further includes a second nonlinear operation between the convolution processing and the point-by-point multiplication operation, and the second nonlinear operation is the same as or different from the first nonlinear operation. For example, the first nonlinear operation can be an absolute value operation, and the second nonlinear operation can still be an absolute value operation. The second nonlinear operation can also be a piecewise linear mapping or other nonlinear operation.

[0018] In a possible implementation, a nonlinear transformation unit in a training phase can also be constructed, which includes a nonlinear operation layer, a convolution processing layer, a point-by-point multiplication operation layer, and a point-by-point addition operation layer.

[0019] The trained nonlinear transformation unit is obtained by training the training data in advance, and is used to implement the nonlinear transformation processing.

[0020] In a second aspect, the embodiments of the present application provide an image decoding method, including: obtaining a first image feature to be processed; performing nonlinear transformation processing on the first image feature to obtain a processed image feature, the nonlinear transformation processing including a first nonlinear operation, a convolution processing, and a point-by-point multiplication operation in sequence; and obtaining a reconstructed image according to the processed image feature.

[0021] In a possible implementation, the non-linear transformation processing of the first image feature to obtain a processed image feature comprises: performing the non-linear operation on each feature value in the first image feature to obtain a second image feature; performing convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature corresponding to a plurality of feature values in the first image feature; and performing the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

[0022] In a possible implementation, the non-linear transformation processing further comprises a point-by-point addition operation.

[0023] In a possible implementation, the non-linear transformation processing of the first image feature to obtain a processed image feature comprises: performing the non-linear operation on each feature value in the first image feature to obtain a second image feature; performing convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature corresponding to a plurality of feature values in the first image feature; performing the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, a plurality of feature values in the fourth image feature corresponding to a plurality of feature values in the first image feature; and performing the point-by-point addition operation on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain the processed image feature.

[0024] In a possible implementation, the non-linear operation comprises an absolute value operation or a piecewise linear mapping.

[0025] The technical effects of the image decoding method provided in the second aspect and possible implementation manners thereof can refer to the technical effects of the image encoding method provided in the first aspect and corresponding possible implementation manners thereof, which will not be described herein again.

[0026] In a third aspect, an encoding apparatus is provided, comprising: an acquisition module configured to acquire a first image feature to be processed; a transformation module configured to perform non-linear transformation processing on the first image feature to obtain a processed image feature, the non-linear transformation processing comprising, in sequence, a first non-linear operation, convolution processing, and a point-by-point multiplication operation; and an encoding module configured to perform encoding according to the processed image feature to obtain a bitstream.

[0027] In a possible implementation, the transformation module is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature corresponding to a plurality of feature values in the first image feature; and perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

[0028] In a possible implementation, the nonlinear transformation processing further includes a point-by-point addition operation after the point-by-point multiplication operation.

[0029] In a possible implementation, the transformation module is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature corresponding to a plurality of feature values in the first image feature; perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, a plurality of feature values in the fourth image feature corresponding to the plurality of feature values in the first image feature; and perform the point-by-point addition operation on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain the processed image feature.

[0030] In a possible implementation, the nonlinear transformation processing further includes a second nonlinear operation between the convolution processing and the point-by-point multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation. For example, the first nonlinear operation can be an absolute value operation, and the second nonlinear operation can still be an absolute value operation, or the second nonlinear operation can be a piecewise linear mapping or other nonlinear operation.

[0031] In a possible implementation, the first nonlinear operation includes an absolute value operation or a piecewise linear mapping.

[0032] In a possible implementation, the method further includes: training a nonlinear transformation unit in a training stage, the nonlinear transformation unit in the training stage including a first nonlinear operation layer, a convolution processing layer, a point-by-point multiplication operation layer, and a point-by-point addition operation layer; and training a trained nonlinear transformation unit according to pre-acquired training data, the trained nonlinear transformation unit being used to implement the nonlinear transformation processing.

[0033] In a fourth aspect, an embodiment of the present application provides a decoding apparatus, comprising: an acquisition module configured to acquire a first image feature to be processed; a transformation module configured to perform a non-linear transformation on the first image feature to obtain a processed image feature, wherein the non-linear transformation comprises a first non-linear operation, a convolution processing, and a point-by-point multiplication operation; and a reconstruction module configured to acquire a reconstructed image according to the processed image feature.

[0034] In a possible implementation, the transformation module is specifically configured to perform the first non-linear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, wherein a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; and perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

[0035] In a possible implementation, the non-linear transformation further comprises a point-by-point addition operation after the point-by-point multiplication operation.

[0036] In a possible implementation, the transformation module is specifically configured to perform the first non-linear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, wherein a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, wherein a plurality of feature values in the fourth image feature correspond to the plurality of feature values in the first image feature; and perform the point-by-point addition operation on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain the processed image feature.

[0037] In a possible implementation, the non-linear transformation further comprises a second non-linear operation between the convolution processing and the point-by-point multiplication operation, wherein the second non-linear operation is the same as or different from the first non-linear operation.

[0038] In a possible implementation, the first non-linear operation comprises an absolute value operation or a piecewise linear mapping, and the second non-linear operation comprises an absolute value operation or a piecewise linear mapping.

[0039] In a possible implementation, the method further includes: training a training stage nonlinear transformation unit, the training stage nonlinear transformation unit including a first nonlinear operation layer, a convolution processing layer, a point-by-point multiplication operation layer, and a point-by-point addition operation layer; and training a trained nonlinear transformation unit according to pre-acquired training data, the trained nonlinear transformation unit being used to implement the nonlinear transformation processing.

[0040] In a fifth aspect, an embodiment of the present application provides an encoder, including: one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, wherein the programming, when executed by the processors, causes the encoder to carry out the method according to any one of the first aspect.

[0041] In a sixth aspect, an embodiment of the present application provides a decoder, including: one or more processors; and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, wherein the programming, when executed by the processors, causes the decoder to carry out the method according to any one of the second aspect.

[0042] In a seventh aspect, an embodiment of the present application provides a computer program product including program code for carrying out the method according to any one of the first to second aspects when the program code is executed on a computer or processor.

[0043] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to carry out the method according to any one of the first to second aspects. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1A is a schematic block diagram of an exemplary coding system 10;

[0045] Figure 1A is an explanatory diagram of an example of a video coding system 40;

[0046] Figure 2 is a schematic diagram of a video coding device 400 provided by an embodiment of the present application;

[0047] Figure 3 is a simplified block diagram of an apparatus 500 provided by an exemplary embodiment;

[0048] Figure 4 is an example diagram of an end-to-end deep learning image coding and decoding framework;

[0049] Figure 5 is an example diagram of an end-to-end deep learning video coding and decoding framework;

[0050] Figure 6 An example diagram of an application scenario of embodiments of the present application;

[0051] Figure 7 An example diagram of an application scenario of embodiments of the present application;

[0052] Figure 8 An example diagram of an application scenario of embodiments of the present application;

[0053] Figure 9 A flowchart of process 900 of an image encoding method according to embodiments of the present application;

[0054] Figure 10a A structural diagram of a nonlinear transformation unit with local attention mechanism;

[0055] Figure 10b A structural diagram of a residual nonlinear transformation unit with local attention mechanism;

[0056] Figure 10c A structural diagram of a residual nonlinear transformation unit with attention mechanism;

[0057] Figure 10d A structural diagram of a nonlinear transformation unit with attention mechanism

[0058] Figure 11 An example diagram of a convolution process;

[0059] Figure 12 A structural diagram of an encoding network;

[0060] Figure 13 A flowchart of process 1300 of an image decoding method according to embodiments of the present application;

[0061] Figure 14 A structural diagram of a decoding network;

[0062] Figure 15a An example structural diagram of ResAU;

[0063] Figure 15b An example structural diagram of ResAU;

[0064] Figure 16a Overall performance of ResAU on 24 images of Kodak test set;

[0065] Figure 16b Overall performance of ResAU on 24 images of Kodak test set;

[0066] Figure 17 An example structural diagram of an encoding device 1700 according to embodiments of the present application;

[0067] Figure 18 FIG. 18 shows an exemplary structure of a decoding device 1800 according to an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of them. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0069] The terms "first", "second", etc. in the description of the embodiments of the present application and the claims and drawings are only used for distinguishing description purposes, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying sequence. In addition, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, comprising a series of steps or units. The method, system, product or device does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0070] It should be understood that in the present application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association between the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0071] The following is a description of the terms related to the embodiments of the present application:

[0072] 1. Code rate: In image compression, the average encoding length required by the encoding unit pixel.

[0073] 2. Rate-distortion performance: An index used to measure the performance of a compression algorithm, which takes into account both the code rate and the distortion of the decoded image.

[0074] 3、Attention mechanism: a means of filtering high-value information from a large amount of information using limited attention resources. It can make the neural network pay more attention to relevant parts of the input and less attention to irrelevant parts.

[0075] 4、Residual structure: a commonly used connection structure in neural networks, the calculation method can be represented as H(x) = x + f(x). This structure can prevent the problems of gradient disappearance and gradient explosion that may occur when the network depth increases.

[0076] 5、Nonlinear transformation unit: a network unit containing nonlinear operations (such as ReLU, Sigmoid, Tanh, etc.), the overall calculation method of the unit does not conform to the linear characteristic.

[0077] Since the embodiments of the present application involve the application of neural networks, in order to facilitate understanding, the related terms or terms used in the embodiments of the present application will be explained first:

[0078] 1、Neural network

[0079] The neural network (neural network, NN) is a machine learning model, and the neural network can be composed of neural units, which can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0080]

[0081] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce non-linear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a nonlinear function such as ReLU. The neural network is a network formed by connecting many single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0082] 2、Multi-layer perception (MLP)

[0083] MLP is a simple deep neural network (DNN) (fully connected between different layers), also called multi-layer neural network, which can be understood as a neural network with many hidden layers, where "many" is not a special measure. From the DNN divided by the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, and the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex, simply speaking, it is as follows: where, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is only to obtain the output vector from the input vector by such a simple operation. Since the number of DNN layers is large, the number of coefficients W and offset vectors is also large. These parameters are defined in the DNN as follows: taking the coefficient W as an example: assuming in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as It should be noted that the input layer has no W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is to learn the weight matrix, and the ultimate goal is to obtain all the weight matrices of the trained deep neural network (weight matrix formed by many layers of vectors W).

[0084] 3. Convolutional neural network

[0085] A convolutional neural network (CNN) is a deep neural network with convolutional structures. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron responds to an input image. A CNN contains a feature extractor consisting of convolutional layers and pooling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as performing convolution with a trainable filter and an input image or a convolutional feature map.

[0086] The convolution layer refers to a neuron layer in a convolutional neural network that performs convolution processing on an input signal. The convolution layer can include a plurality of convolution operators, also referred to as kernels, which function as filters that extract specific information from an input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. During the convolution operation on the image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix produces a single-depth-dimension convolution output, but most cases do not use a single weight matrix, but apply a plurality of weight matrices of the same size (rows x columns), i.e., a plurality of homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as being determined by the "plurality" described above. Different weight matrices can be used to extract different features in the image, such as a weight matrix for extracting image edge information, another weight matrix for extracting specific colors of the image, and another weight matrix for blurring unwanted noise in the image, etc. The plurality of weight matrices are of the same size (rows x columns), and the feature maps extracted by the plurality of weight matrices of the same size are also of the same size. The plurality of feature maps of the same size are combined to form the output of the convolution operation. The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. The weight matrices formed by the trained weight values can be used to extract information from the input image, thereby enabling the convolutional neural network to make correct predictions. When the convolutional neural network has multiple convolution layers, the initial convolution layer often extracts more general features, which can also be referred to as low-level features. As the depth of the convolutional neural network increases, the features extracted by the later convolution layers become more and more complex, such as high-level semantic features, etc. The higher the semantic level, the more suitable the features are for the problem to be solved.

[0087] As it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after a convolution layer, which can be a convolution layer followed by a pooling layer, or a plurality of convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer can include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain a smaller size image. The average pooling operator can calculate the average value of the pixel values in the image within a certain range as the result of the average pooling. The maximum pooling operator can take the maximum value of the pixels in the range as the result of the maximum pooling. In addition, just like the size of the weight matrix in the convolution layer should be related to the size of the image, the operator in the pooling layer should also be related to the size of the image. The size of the image output by the pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0088] After the convolution layer / pooling layer processing, the convolutional neural network is not enough to output the required output information. Because, as mentioned earlier, the convolution layer / pooling layer only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network needs to use the neural network layer to generate an output of one or a group of the number of required classes. Therefore, the neural network layer can include a plurality of hidden layers, and the parameters contained in the plurality of hidden layers can be pre-trained according to the related training data of a specific task type, for example, the task type can include image recognition, image classification, image super-resolution reconstruction, etc.

[0089] Optionally, after the plurality of hidden layers in the neural network layer, an output layer of the entire convolutional neural network is further included, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network is completed, the back propagation starts to update the weight values and biases of the layers mentioned above to reduce the loss of the convolutional neural network and the error between the result output by the convolutional neural network through the output layer and the ideal result.

[0090] 4. Recurrent neural network

[0091] Recurrent neural networks (RNN) are used to process sequential data. In traditional neural network models, it is from the input layer to the hidden layer and then to the output layer, and the layers are fully connected, while the nodes between each layer are not connected. Although this ordinary neural network has solved many difficult problems, it is still powerless for many problems. For example, you want to predict the next word of a sentence, you usually need to use the previous words, because the previous and next words in a sentence are not independent. RNN is called recurrent neural network because the current output of a sequence is also related to the previous output. The specific form of expression is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer unconnected but connected, and the input of the hidden layer not only includes the output of the input layer but also the output of the previous hidden layer. In theory, RNN can process sequential data of any length. The training of RNN is the same as the training of traditional CNN or DNN. Similarly, the error back propagation algorithm is used, but there is a difference: that is, if the RNN is expanded into a network, the parameters such as W are shared; but the traditional neural network as described above is not. And in the use of gradient descent algorithm, the output of each step depends not only on the current network but also on the state of the previous several steps of network. This learning algorithm is called back propagation through time (BPTT).

[0092] Since there is a convolutional neural network, why do we still need a recurrent neural network? The reason is simple: in a convolutional neural network, there is a premise assumption that elements are independent of each other, and input and output are also independent, such as cats and dogs. But in the real world, many elements are connected to each other, such as the change of stock over time, and again, a person said: I like traveling, and the most favorite place is Yunnan, and I will definitely go there if I have the opportunity. Fill in the blank, humans should all know to fill in "Yunnan". Because humans will infer according to the content of the context, but how can machines do this? RNN was born at the right moment. RNN aims to give machines the ability to remember like humans. Therefore, the output of RNN needs to depend on the current input information and historical memory information.

[0093] 5、Loss function

[0094] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the truly intended target value by comparing the two, (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer of the deep neural network are pre-configured), for example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize this loss.

[0095] 6. Back propagation algorithm

[0096] The convolutional neural network can use the back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model in the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward propagation of the input signal until the output will produce an error loss, and the initial super-resolution model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0097] 7. Generative adversarial network

[0098] A generative adversarial network (GAN) is a kind of deep learning model. The model includes at least two modules: one module is a generative model (Generative Model), and the other module is a discriminative model (Discriminative Model), which learn from each other through the two modules to generate better output. The generative model and the discriminative model can both be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: taking a GAN for generating pictures as an example, suppose there are two networks, G (Generator) and D (Discriminator), where G is a network for generating pictures, which receives a random noise z and generates a picture through the noise, denoted as G(z); D is a discriminative network for discriminating whether a picture is real or not. Its input parameter is x, which represents a picture, and the output D(x) represents the probability that x is a real picture. If it is 1, it means that 130% is a real picture, and if it is 0, it means that it is not a real picture. During the training of the generative adversarial network, the goal of the generative network G is to generate as real a picture as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the picture generated by G from the real picture as much as possible. In this way, G and D constitute a dynamic "game" process, that is, the "adversarial" in the "generative adversarial network". Finally, the result of the game is that, in an ideal state, G can generate a picture G(z) that is sufficient to "deceive the real", and D cannot determine whether the picture generated by G is real or not, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate pictures.

[0099] With the performance of convolution neural network (CNN) far exceeding traditional algorithms in image recognition, target detection and other computer vision tasks, more and more researchers have begun to explore image / video compression methods based on deep learning. Some researchers have designed end-to-end deep learning image / video compression algorithms, for example, the coding network, entropy estimation network, entropy encoding network, entropy decoding network, decoding network and other modules are optimized as a whole, wherein the coding network and the decoding network can also be called transformation module and inverse transformation module, which are generally composed of convolution layers and nonlinear transformation units.

[0100] Figure 1AThis is a schematic block diagram of an exemplary decoding system 10, such as a video decoding system 10 (or simply decoding system 10) that can utilize the technology of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 represent devices, etc., that can be used to perform the various technologies according to the various examples described in this application.

[0101] like Figure 1A As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, such as encoded images, to a destination device 14 for decoding the encoded image data 21.

[0102] The source device 12 includes an encoder 20, and optionally may include an image source 16, a preprocessor (or preprocessing unit) 18 such as an image preprocessor, and a communication interface (or communication unit) 22.

[0103] Image source 16 may include or may be any type of image capture device for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer animation images or any type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images, and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device storing any of the images described above.

[0104] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as the raw image (or raw image data) 17.

[0105] The preprocessor 18 receives (raw) image data 17 and preprocesses the image data 17 to obtain a preprocessed image (or preprocessed image data) 19. For example, the preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component.

[0106] Video encoder (or encoder) 20 is used to receive preprocessed image data 19 and provide encoded image data 21 (hereinafter referred to as...) Figure 2 (and so on, for further description).

[0107] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send encoded image data 21 (or other arbitrarily processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0108] The target device 14 includes a decoder 30, and optionally may include a communication interface (or communication unit) 28, a post-processor (or post-processing unit) 32 and a display device 34.

[0109] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or other processed versions) directly from the source device 12 or from any other source device such as a storage device, for example, the storage device is an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0110] Communication interfaces 22 and 28 can be used to send or receive encoded image data (or encoded data 21) through a direct communication link between source device 12 and destination device 14, such as a direct wired or wireless connection, or through any type of network, such as a wired network, a wireless network or any combination thereof, any type of private network and public network or any combination thereof.

[0111] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it on a communication link or communication network.

[0112] Communication interface 28 corresponds to communication interface 22. For example, it can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0113] Both communication interface 22 and communication interface 28 can be configured as follows: Figure 1A The arrow pointing from the source device 12 to the corresponding communication channel 13 of the destination device 14 indicates a one-way or two-way communication interface, which can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission, such as encoded image data transmission, etc.

[0114] Video decoder (or decoder) 30 is used to receive encoded image data 21 and provide decoded image data (or decoded image data) 31 (hereinafter referred to as...). Figure 3 (and so on, for further description).

[0115] The post-processor 32 is configured to post-process the decoded image data 31, e.g. a decoded image such as a reconstructed image, to obtain post-processed image data 33, e.g. a post-processed image. The post-processing performed by the post-processing unit 32 can include, for example, color format conversion (e.g. from YCbCr to RGB), toning, pruning or resampling, or any other processing for producing decoded image data 31 for display by the display device 34 or the like.

[0116] The display device 34 is configured to receive the post-processed image data 33 for displaying an image to a user or viewer or the like. The display device 34 can be or comprise any type of display for representing a reconstructed image, e.g. an integrated or external display screen or monitor. For example, the display screen can comprise a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP) or any other type of display screen.

[0117] The coding system 10 further comprises a training engine 25 configured to train the encoder 20 or the decoder 30 for implementing the conversion between the image domain and the feature domain.

[0118] The training data in the embodiments of the present application can be stored in a database (not shown), and the training engine 25 trains the encoding / decoding network based on the training data. It should be noted that the source of the training data in the embodiments of the present application is not limited, for example, the training data can be obtained from the cloud or other places for model training.

[0119] Although Figure 1A Although the source device 12 and the destination device 14 are shown as separate devices, the device embodiments can also comprise both the source device 12 and the destination device 14 or the functionalities of the source device 12 and the destination device 14, i.e. both the source device 12 or the corresponding functionalities and the destination device 14 or the corresponding functionalities. In these embodiments, the source device 12 or the corresponding functionalities and the destination device 14 or the corresponding functionalities can be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0120] According to the description, Figure 1A It will be apparent to the skilled person that the presence and (precise) division of the different units or functionalities in the source device 12 and / or the destination device 14 shown can vary depending on the actual device and application.

[0121] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30) or both can be transmitted via, for example, Figure 1B The processing circuitry shown can be implemented as, for example, one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, video encoding dedicated processors, or any combination thereof. Encoder 20 can be implemented via processing circuitry 46 to include reference... Figure 2 Encoder 20 refers to various modules discussed herein and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to include references. Figure 3 Decoder 30 may include various modules discussed herein and / or any other decoder system or subsystem described herein. The processing circuitry 46 may be used to perform various operations discussed below. Figure 8 As shown, if some of the technology is implemented in software, the device can store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors, thereby performing the technology of the present invention. One of the video encoder 20 and video decoder 30 can be integrated into a single device as part of a combined codec (encoder / decoder, CODEC), such as... Figure 1B As shown.

[0122] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or notebook computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (e.g., content service server or content distribution server), broadcast receiving device, broadcast transmitting device, etc., and may or may not use an operating system of any type. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0123] In some cases, Figure 1AThe illustrated video coding system 10 is merely exemplary. The techniques provided herein can be applied to video encoding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local storage, transmitted over a network, etc. A video encoding device can encode data and store the data into a memory, and / or a video decoding device can retrieve data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but rather only encode data to a memory and / or retrieve and decode data from a memory.

[0124] Figure 1B FIG. 1 is a diagram illustrating an example of a video coding system 40. The video coding system 40 can include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memory stores 44, and / or a display device 45.

[0125] As shown, the imaging device 41, the antenna 42, the processing circuitry 46, the video encoder 20, the video decoder 30, the processor 43, the memory store 44, and / or the display device 45 can be in communication with each other. In different examples, the video coding system 40 can include only the video encoder 20 or only the video decoder 30. Figure 1B

[0126] In some examples, the antenna 42 can be used to transmit or receive encoded bitstreams of video data. Additionally, in some examples, the display device 45 can be used to present video data. The processing circuitry 46 can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general purpose processor, etc. The video coding system 40 can also include an optional processor 43, which similarly can include application-specific integrated circuit (ASIC) logic, a graphics processor, a general purpose processor, etc. Additionally, the memory store 44 can be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), etc. In non-limiting examples, the memory store 44 can be implemented by cache memory. In other examples, the processing circuitry 46 can include memory (e.g., a cache, etc.) for implementing an image buffer, etc. ​

[0127] In some examples, video encoder 20 implemented by logic circuitry can include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented by processing circuitry 46). The graphics processing unit can be communicatively coupled to the image buffer. The graphics processing unit can include video encoder 20 implemented by processing circuitry 46 to implement various modules discussed with respect to Figure 2 and / or any other encoder system or subsystem described herein. The logic circuitry can be used to perform various operations discussed herein.

[0128] In some examples, video decoder 30 can be implemented in a similar manner by processing circuitry 46 to implement various modules discussed with respect to Figure 3 and / or any other decoder system or subsystem described herein. In some examples, video decoder 30 implemented by logic circuitry can include an image buffer (implemented by processing circuitry 46 or memory storage 44) and a graphics processing unit (implemented by processing circuitry 46). The graphics processing unit can be communicatively coupled to the image buffer. The graphics processing unit can include video decoder 30 implemented by processing circuitry 46 to implement various modules discussed with respect to Figure 3 and / or any other decoder system or subsystem described herein.

[0129] In some examples, antenna 42 can be used to receive an encoded bitstream of video data. As discussed, the encoded bitstream can include data, indicators, index values, mode selection data, etc. discussed herein related to encoding video frames, such as data related to encoding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining encoding partitions). Video coding system 40 can also include video decoder 30 coupled to antenna 42 and used to decode the encoded bitstream. Display device 45 is used to present video frames.

[0130] It should be understood that video decoder 30 can be used to perform the inverse processes described with respect to reference video encoder 20 in the examples of the present embodiments. With respect to signaling syntax elements, video decoder 30 can be used to receive and parse such syntax elements and decode the related video data accordingly. In some examples, video encoder 20 can entropy encode syntax elements into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode the related video data accordingly.

[0131] For ease of description, embodiments of the present application are described with reference to the Versatile Video Coding (VVC) reference software or High-Efficiency Video Coding (HEVC) developed by the Joint Collaboration Team on Video Coding (JCT-VC) of ITU-T Video Coding Experts Group (VCEG) and ISO / IEC Motion Picture Experts Group (MPEG). It is understood by those of ordinary skill in the art that embodiments of the present application are not limited to HEVC or VVC.

[0132] Figure 2 A schematic diagram of a video coding device 400 provided by embodiments of the present application is shown. The video coding device 400 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 400 can be a decoder, such as the video decoder 30 in Figure 1A , or an encoder, such as the video encoder 20 in Figure 1A .

[0133] The video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data; a processor, logic unit or central processing unit (CPU) 430 for processing data; for example, the processor 430 here can be a neural network processor 430; a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting data; a memory 460 for storing data. The video coding device 400 can further include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440 and the egress port 450 for egress or ingress of optical signals or electrical signals.

[0134] The processor 430 is implemented by hardware and software. The processor 430 can be implemented as one or more processor chips, cores (e.g., a multi-core processor), FPGAs, ASICs, and DSPs. The processor 430 is in communication with the ingress port 410, the receiving unit 420, the sending unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network, NN-based coding module 470). The coding module 470 implements the embodiments disclosed above. For example, the coding module 470 performs, processes, prepares, or provides various coding operations. Thus, the coding module 470 provides substantial improvement to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented in instructions stored in the memory 460 and executed by the processor 430.

[0135] The memory 460 includes one or more disks, tape drives, and solid-state drives, can be used as an overflow data storage device, to store programming when it is selected for execution, and to store instructions and data that are read during program execution. The memory 460 can be volatile and / or non-volatile, can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0136] Figure 3 A simplified block diagram of an apparatus 500 is provided for the exemplary embodiments, which can be used as either or both of the source device 12 and the destination device 14 in FIG. 1. Figure 1A

[0137] The processor 502 in the apparatus 500 can be a central processing unit. Alternatively, the processor 502 can be any other type of device, either existing or developed in the future, capable of manipulating or processing information. While a single processor 502 as shown in the figure can be used to implement the disclosed implementations, it is faster and more efficient to use more than one processor.

[0138] ​In one implementation, the memory 504 in the apparatus 500 can be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be utilized as the memory 504. The memory 504 can include code and data 506 that is accessed by the processor 502 via the bus 512. The memory 504 can also include an operating system 508 and an application program 510 including at least one program for allowing the processor 502 to perform the methods described herein. For example, the application program 510 can include applications 1 through N, including a video coding application that performs the methods described herein.

[0139] The apparatus 500 can also include one or more output devices, such as a display 518. In one example, the display 518 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 518 can be coupled to the processor 502 via the bus 512.

[0140] Although the bus 512 in the apparatus 500 is described herein as a single bus, the bus 512 can include multiple buses. Also, the secondary storage can be directly coupled to the other components of the apparatus 500 or accessed via a network, and can include a single integrated unit, or multiple units of storage such as memory cards. Thus, the apparatus 500 can have a wide variety of configurations.

[0141] Figure 4 An example diagram of an end-to-end deep learning image coding framework is shown in FIG. 1, which includes an encoding end: an encoder, a quantization module, an entropy coding network; a decoding end: an entropy decoding network, a decoder, and an entropy estimation network. Figure 4

[0142] At the encoding end, the original image is transformed from the image domain to the feature domain by the encoder, and the transformed image features are encoded into a bitstream to be transmitted or stored by the quantization module and the entropy coding network. At the decoding end, the bitstream is decoded into image features by the entropy decoding network, and the image features are transformed from the feature domain to the image domain by the decoder, thereby obtaining the reconstructed image. The entropy estimation network estimates the estimated probability value of each feature element according to the image features, which is used for the processing of the entropy coding network and the entropy decoding network.

[0143] In this embodiment, both the encoder and the decoder have a nonlinear transformation unit.

[0144] Figure 5 An example diagram of an end-to-end deep learning video coding framework is shown in FIG. 2, which includes an encoding end: an encoder, a quantization module, an entropy coding network; a decoding end: an entropy decoding network, a decoder, and an entropy estimation network. Figure 5 ​As shown, the video codec framework includes a prediction model and a residual compress model,

[0145] The prediction model predicts a current frame using a reconstructed image of a previous frame to obtain a predicted image, and the residual compress model compresses a residual between an original image of the current frame and the predicted image, and decompresses to obtain a reconstructed residual, and the reconstructed residual and the predicted image are summed to obtain a reconstructed image of the current frame. The coding sub-network and the decoding sub-network in the prediction model and the residual compress model both have a nonlinear transformation unit.

[0146] In this embodiment, the prediction model and the residual compress model both have a nonlinear transformation unit.

[0147] Figure 6 An example diagram of an application scenario of the embodiment is shown in FIG. 1. Figure 6 As shown, the application scenario can be a service involving image / video acquisition, storage or transmission in a terminal, a cloud server or video monitoring, for example, terminal photographing / video recording, a photo album, a cloud photo album, video monitoring, etc.

[0148] At the encoding end, a camera captures images / videos. An artificial intelligence (AI) image / video coding network extracts features from the images / videos to obtain image features with low redundancy, and then compresses the image features to obtain a code stream / image file.

[0149] At the decoding end, when an image / video needs to be output, an AI image / video decoding network decompresses the code stream / image file to obtain image features, and then performs inverse feature extraction on the image features to obtain a reconstructed image / video.

[0150] A storage / transmission module stores or transmits the code stream / image file obtained by compression for different services (for example, terminal photographing, video monitoring, cloud server, etc.).

[0151] Figure 7 An example diagram of an application scenario of the embodiment is shown in FIG. 1. Figure 7 As shown, the application scenario can be a service involving image / video acquisition, storage or transmission in a terminal, a cloud server or video monitoring, for example, terminal photographing / video recording, a photo album, a cloud photo album, video monitoring, etc.

[0152] Encoding side: the encoding network transforms the image / video into image features with lower redundancy, which usually contains nonlinear transformation units with nonlinear characteristics. The entropy estimation network is responsible for calculating the encoding probability of each data in the image features. The entropy encoding network losslessly encodes the image features according to the probability corresponding to each data to obtain the code stream / image file, further reducing the data transmission amount in the image compression process.

[0153] Decoding side: the entropy decoding network losslessly decodes the code stream / image file according to the probability corresponding to each data to obtain the reconstructed image features. The decoding network inversely transforms the image features output by the entropy decoding to analyze them into images / videos. Corresponding to the encoding network, it usually contains nonlinear transformation units with nonlinear characteristics. The saving module saves the code stream / image file to the corresponding storage location of the terminal. The loading module loads the code stream / image file from the corresponding storage location of the terminal and inputs it to the entropy decoding network.

[0154] Figure 8 For the example of the application scenario of the embodiments of the present application, as shown in Figure 8 , the application scenario can be a service involving image / video acquisition, storage or transmission in the cloud, video monitoring, such as cloud album, video monitoring, live broadcast, etc.

[0155] Encoding side: locally acquire images / videos, encode the images / videos to obtain compressed images / videos, and then send the compressed images / videos to the cloud. The cloud decodes the compressed images / videos to obtain images / videos, and then compresses the images / videos to obtain code streams / image files and stores them.

[0156] Decoding side: when the local needs to acquire images / videos from the cloud, the cloud decompresses the code streams / image files to obtain images / videos, then encodes the images / videos to obtain compressed images / videos, and sends the compressed images / videos to the local. The local decodes the compressed images / videos to obtain images / videos. The structure of the cloud and the purposes of the various modules can refer to the structure of Figure 7 and the purposes of the various modules, which will not be described here in detail.

[0157] Based on the above encoding / decoding network and application scenario, the embodiments of the present application provide an image encoding / decoding method to realize efficient nonlinear transformation processing and improve the rate-distortion performance in the image / video compression algorithm.

[0158] Figure 9 The flowchart of the process 900 of the image encoding method of the embodiments of the present application. The process 900 can be performed by the encoding side in the above embodiments. The process 900 is described as a series of steps or operations, and it should be understood that the process 900 can be executed in various orders and / or simultaneously, and is not limited toFigure 9 The execution order shown. Process 900 includes the following steps:

[0159] Step 901, obtaining a first image feature to be processed.

[0160] The first image feature is obtained after the encoding end converts the image to be processed from the image domain to the feature domain. The conversion can include: 1, convolution processing, using convolution layers to extract features, with local receptive field, weight sharing mechanism (i.e. each filter sliding processing input feature). 2, using MLP or fully connected layer to extract features, with global receptive field characteristics, weight sharing. 3, transformer processing, which includes matrix multiplication, MLP and normalization processing, with global receptive field characteristics, strong ability to capture long distance dependence.

[0161] The first image feature can be represented in the form of a two-dimensional matrix (LxC, L represents length, C represents channel) or a three-dimensional matrix (CxHxW, C represents the number of channels, H represents height, W represents width). The specific form is associated with the aforementioned conversion method, for example, the first image feature obtained by convolution processing or MLP extraction generally corresponds to a three-dimensional matrix, and the first image feature obtained by transformer processing generally corresponds to a two-dimensional matrix.

[0162] For example, the first image feature is represented as a two-dimensional matrix:

[0163]

[0164] The two-dimensional matrix A is a 3x3 matrix containing 9 elements, each element a(i,j) corresponds to a feature value of the first image feature, where i represents the length corresponding to the element a(i,j), and j represents the channel where the element a(i,j) is located.

[0165] For example, the first image feature is represented as a three-dimensional matrix:

[0166]

[0167] The three-dimensional matrix B is a 3x3x2 matrix containing 18 elements, each element a(i,j,l) corresponds to a feature value of the first image feature, where i represents the row where the element a(i,j,l) is located, j represents the column where the element a(i,j,l) is located, and l represents the channel where the element a(i,j,l) is located.

[0168] It should be noted that the present application does not limit the manner of obtaining the first image feature.

[0169] The to-be-processed image can be a picture, a frame of image in a video, or an image block segmented from the picture or the frame, and no specific limitation is made to this.

[0170] At step 902, the first image feature is subjected to nonlinear transformation processing to obtain a processed image feature.

[0171] In a possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; convolution processing is performed on the second image feature to obtain a third image feature, and multiple feature values in the third image feature correspond to multiple feature values in the first image feature; and point-by-point multiplication operation is performed on the corresponding multiple feature values in the first image feature and the third image feature to obtain the processed image feature.

[0172] Figure 10a A structure diagram of the nonlinear transformation unit with the attention mechanism is shown in FIG. 1. Figure 10a As shown in FIG. 1, in an embodiment of the present application, the nonlinear transformation unit is used to implement the nonlinear transformation processing, and includes a first nonlinear operation, convolution processing, and point-by-point multiplication operation.

[0173] The first nonlinear operation is an operation performed on each feature value in the first image feature, and can include absolute value operation, ReLU series, Sigmoid, Tanh, etc.

[0174] The absolute value operation refers to taking the absolute value of the input feature value. This can be represented by the following formula:

[0175]

[0176] The piecewise linear mapping includes a rectified linear unit (ReLU) or a leaky rectified linear unit (LeakyReLU), wherein the ReLU is a piecewise linear mapping mode, and for the input feature value, the feature value less than 0 is output as 0, and the feature value greater than or equal to 0 is output as a constant. This can be represented by the following formula:

[0177]

[0178] The LeakyReLU is a piecewise linear mapping mode, which scales the input feature value less than 0 by a pre-set weight on the basis of the ReLU, and the weight is usually 0.01. This can be represented by the following formula:

[0179]

[0180] Wherein, a represents a pre-set value, and is usually set to 0.01.

[0181] The Sigmoid can be represented by the following operation:

[0182]

[0183] The tanh can be expressed as follows:

[0184]

[0185] It should be noted that the piecewise linear mapping can also use other variations of ReLU, or other new first nonlinear operation, and the embodiments of the present application do not make specific limitations.

[0186] After the above first nonlinear operation, the first image feature is converted into the second image feature, which can also be represented as a matrix form as the first image feature. Since the first nonlinear operation is performed on each feature value in the first image feature, each feature value in the first image feature has a corresponding feature value in the second image feature. Therefore, the matrix corresponding to the second image feature and the matrix corresponding to the first image feature have the same size, and the feature values (matrix elements) at the same position correspond to each other. For example, the first image feature is represented as a 3x3 matrix as in the above matrix A, and the second image feature can also be represented as a 3x3 matrix. However, since the first image feature and the second image feature have undergone the first nonlinear operation, the feature values in the first image feature and the second image feature are not exactly the same, and the corresponding elements in the matrices corresponding to the two are also not exactly the same.

[0187] The second image feature is subjected to convolution processing to output a third image feature, which can be considered as a local response (i.e. corrected value) of the second image feature. That is, the third image feature is a response signal obtained by convolution processing of the second image feature. Since the receptive field of the convolution processing is limited, the response value at each position in the image feature output by the convolution processing is only related to the input feature values of the adjacent positions of the position, and therefore it is called a local response. The convolution processing can be expressed as follows:

[0188] conv1(x) = β + ∑γ×x

[0189] Where γ represents the weight of the convolution layer, and β represents the bias parameter of the convolution layer.

[0190] After convolution processing, the matrix corresponding to the third image feature and the matrix corresponding to the first image feature also have the same size. Figure 11 A schematic diagram of the convolution processing is shown in FIG. 2. Figure 11As shown, an input 1*2 matrix is processed by a convolution layer to output a 1*4 matrix, the convolution layer includes two filters, one is a 2*50 matrix W1, and the other is a 50*4 matrix W2. The input matrix is first convolved by the matrix W1 to obtain a 1*50 matrix, and then the matrix is convolved by the matrix W2 to obtain a 1*4 output matrix.

[0191] The corresponding feature values in the first image feature and the third image feature are point-by-point multiplied, that is, the initial input of the nonlinear transformation unit and the output of the convolution processing are point-by-point multiplied, which can be expressed as the following formula:

[0192] c(i,j)=a(i,j)×b(i,j)

[0193] Where (i,j) represents the index of the feature value in the image feature, a(i,j) represents the feature value of the first image feature, b(i,j) represents the feature value of the second image feature, and c(i,j) represents the feature value of the processed image feature.

[0194] As can be seen, the size of the matrix corresponding to the processed image feature and the matrix corresponding to the first image feature is also the same.

[0195] It should be noted that in the embodiments of the present application, the corresponding multiple feature values in the two image features can mean that the values of the elements at the same position in the two matrices have an operation relationship after the two image features are represented as matrices, and the two are corresponding. For example, the first image feature and the third image feature are both represented in the form of the above matrix A, and the elements at the same position of a 0,2 correspond.

[0196] In addition, the convolution processing can also obtain a third image feature with a size different from that of the first image feature, which depends on the internal structure of the convolution processing, especially the length, width and channel number of the filter when the convolution processing is performed. If the size of the third image feature is different from that of the first image feature, the elements in the matrix corresponding to the third image feature and the elements in the matrix corresponding to the first image feature are not one-to-one corresponding, at this time, the multiple elements in the matrix corresponding to the first image feature can be multiplied with the same element in the matrix corresponding to the third image feature. For example, the channel number of the matrix corresponding to the first image feature is 3, and the channel number of the matrix corresponding to the third image feature is 1, the elements in the matrix corresponding to the third image feature can be multiplied with the elements at the same position in each channel of the matrix corresponding to the first image feature. The embodiments of the present application do not make specific limitation on this.

[0197] The above first nonlinear operation, convolution processing and point-by-point multiplication operation can be expressed as the following formula (1):

[0198]

[0199] wherein x represents an input feature value, y represents an output feature value, represents a first nonlinear operation, γ represents a weight of a convolution layer, and β represents a bias parameter of the convolution layer.

[0200] Since the multiplication operation is used in the above formula, compared with the convolution processing of the related art (GDN division normalization), the division operation of the above formula, on the one hand, does not require the convolution parameters γ and β to be positive numbers in order to accommodate the limitation that the square root must be a positive number, and on the other hand, does not require the denominator to be non-zero in order to accommodate the limitation that the division cannot be zero, thereby limiting the values of the convolution parameters γ and β.

[0201] The local attention mechanism is realized through the above first nonlinear operation, convolution processing, and point-by-point multiplication operation. The local refers to the fact that the first nonlinear operation is performed point by point, and each input feature value only needs its own characteristics to obtain the output value, without considering the influence of the surrounding feature values. The attention mechanism refers to the fact that all feature values in the first image feature are important or redundant, and the output of the convolution processing can be the weight of each feature value in the image feature, which can modify the original feature value, highlight the important feature value, and suppress the redundant feature value. The point-by-point multiplication operation corrects the value of each feature value in the first image feature using the aforementioned local information, thereby avoiding the limitation on the convolution parameters.

[0202] In one possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; a convolution processing is performed on the second image feature to obtain a third image feature, and the plurality of feature values in the third image feature correspond to the plurality of feature values in the first image feature; a point-by-point multiplication operation is performed on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, and the plurality of feature values in the fourth image feature correspond to the plurality of feature values in the first image feature; and a point-by-point addition operation is performed on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain a processed image feature.

[0203] Figure 10b The structure diagram of the residual nonlinear transformation unit with the attention mechanism is shown in FIG. 1. Figure 10b As shown in FIG. 1, in the embodiments of the present application, the nonlinear transformation unit is used to implement the nonlinear transformation processing described above, and includes a first nonlinear operation, a convolution processing, a point-by-point multiplication operation, and a point-by-point addition operation.

[0204] In the embodiments of the present application, the first nonlinear operation, the convolution processing, and the point-by-point multiplication operation can all refer to the description in the above implementation, which will not be described here again.

[0205] After obtaining the fourth image feature, the corresponding feature values in the first image feature and the fourth image feature are subjected to point-by-point addition operation, that is, the initial input of the nonlinear transformation unit and the output of the point-by-point multiplication operation are subjected to point-by-point addition, which can be expressed as the following formula:

[0206] sum(i,j) = a(i,j) + c(i,j)

[0207] where (i,j) represents the index of the feature value in the image feature, a(i,j) represents the feature value of the first image feature, c(i,j) represents the feature value of the fourth image feature, and sum(i,j) represents the feature value of the processed image feature.

[0208] The point-by-point addition operation is a first-order addition residual structure, which can make the codec network using the above processing process more easily converge during training.

[0209] The above first nonlinear operation, convolution processing, point-by-point multiplication operation and point-by-point addition operation can be expressed as the following formula (2):

[0210]

[0211] where x represents the input feature value, y represents the output feature value, represents the first nonlinear operation, γ represents the weight of the convolution layer, and β represents the bias parameter of the convolution layer.

[0212] The above formula (2) can be transformed to obtain: In comparison with formula (1), it can be obtained that the convolution processing can be expressed as the following formula:

[0213] conv2(x) = β + 1 + ∑γ×x

[0214] As can be seen, the convolution processing conv1(x) and the convolution processing conv2(x) are similar, and the difference lies in that the convolution parameter in the convolution processing conv2(x) is additionally added by 1. In this way, the conversion between the above two implementation manners can be realized by fine-tuning the convolution parameter β in the convolution processing, that is, if the nonlinear transformation unit does not contain the point-by-point addition operation, the convolution processing conv1(x) can be used; if the nonlinear transformation unit contains the point-by-point addition operation, the convolution processing conv2(x) can be used.

[0215] In a possible implementation, on the basis of the first implementation, the third image feature can be subjected to a second nonlinear operation, and the output processed third image feature and the first image feature can be subjected to a point-by-point multiplication operation, or on the basis of the second implementation, the third image feature can be subjected to a second nonlinear operation, and the output processed third image feature and the first image feature can be subjected to a point-by-point multiplication operation. That is, a second nonlinear operation is added in the nonlinear transformation processing, and the input of the second nonlinear operation is the output of the convolution processing in the nonlinear transformation processing, and the output of the second nonlinear operation is one input of the point-by-point multiplication operation.

[0216] Figure 10c As shown in a structural diagram of a residual nonlinear transformation unit with an attention mechanism, Figure 10c In an embodiment of the present application, the nonlinear transformation unit is used to implement the nonlinear transformation processing, and includes a first nonlinear operation, convolution processing, a second nonlinear operation, and a point-by-point multiplication operation.

[0217] The first nonlinear operation, the convolution processing, and the point-by-point multiplication operation can refer to the description of the embodiment shown in Figure 10a The second nonlinear operation can use the same operation method as the first nonlinear operation, or can use a different operation method. The second nonlinear operation can include taking an absolute value, ReLU, LeadyReLU, etc., and the present application is not limited in this regard.

[0218] The first nonlinear operation, the convolution processing, the second nonlinear operation, and the point-by-point multiplication operation can be represented by the following formula (3):

[0219]

[0220] where x represents an input feature value, y represents an output feature value, represents the first nonlinear operation, represents the second nonlinear operation, γ represents a weight of a convolution layer, and β represents a bias parameter of the convolution layer.

[0221] Figure 10d As shown in a structural diagram of a nonlinear transformation unit with an attention mechanism, Figure 10d In an embodiment of the present application, the nonlinear transformation unit is used to implement the nonlinear transformation processing, and includes a first nonlinear operation, convolution processing, a second nonlinear operation, a point-by-point multiplication operation, and a point-by-point addition operation.

[0222] The first nonlinear operation, the convolution processing, the point-by-point multiplication operation, and the point-by-point addition operation can refer to the description of the embodiment shown in Figure 10bThe second nonlinear operation can use the same operation method as the first nonlinear operation, or can use a different operation method, which can include taking an absolute value, ReLU, LeadyReLU, etc., and embodiments of the application are not limited in this regard.

[0223] The first nonlinear operation, the convolution processing, the second nonlinear operation, the point-by-point multiplication operation, and the point-by-point addition operation described above can be represented by the following formula (4):

[0224]

[0225] where x represents an input feature value, y represents an output feature value, represents the first nonlinear operation, represents the second nonlinear operation, γ represents the weight of the convolution layer, and β represents the bias parameter of the convolution layer.

[0226] Step 903: encoding according to the processed image features to obtain a bitstream.

[0227] After obtaining the processed image features, the encoding side can continue to perform convolution processing thereon, or can perform nonlinear transformation processing on the output of the convolution processing again after the convolution processing, and then perform entropy encoding on the result of the foregoing processing to obtain a bitstream. The entropy encoding can be implemented by using the entropy encoding network as shown in Figure 4-8 Embodiments, which will not be described here. Other encoding methods can also be used to encode the result of the foregoing processing to obtain a bitstream, and the application is not limited in this regard.

[0228] Figure 12 The structure diagram of the encoding network is as shown in Figure 12 The encoding network includes four convolution layers (conv) and three nonlinear transformation units, and the convolution layers and the nonlinear transformation units are arranged alternately, that is, the input image is sequentially subjected to convolution processing, nonlinear transformation processing, convolution processing, nonlinear transformation processing, convolution processing, nonlinear transformation processing, and convolution processing to obtain an output image feature. The nonlinear transformation unit can use the structure of the embodiments as shown in Figure 10a or Figure 10b The output image feature is then subjected to entropy encoding, which will not be described here.

[0229] In this embodiment, by modifying the nonlinear transformation processing in the decoding network, the output value can be obtained by only considering the characteristics of each input feature value, without considering the influence of surrounding feature values. The original feature values ​​are corrected, highlighting important feature values ​​and suppressing redundant feature values. In addition, the value of each feature value on the first image feature can be corrected, avoiding the limitation of convolution parameters. This achieves efficient nonlinear transformation processing in the encoding network and further improves the rate-distortion performance in image / video compression algorithms.

[0230] Figure 13 This is a flowchart of process 1300 of the image decoding method according to an embodiment of this application. Process 1300 can be executed by the decoding end in the above embodiment. Process 1300 is described as a series of steps or operations. It should be understood that process 1300 can be executed in various orders and / or occur simultaneously, and is not limited to... Figure 13 The execution order is shown. Process 1300 includes the following steps:

[0231] Step 1301: Obtain the first image features to be processed.

[0232] The decoding end corresponds to the encoding end. The first image feature can be the result of entropy decoding of the bitstream by the decoding end. Entropy decoding can be achieved by... Figure 4-8 The entropy decoding network implementation in the illustrated embodiment will not be described in detail here. It is then processed through convolution, deconvolution, transposed convolution, interpolation + convolution, and Transformer processing. It should be understood that after the aforementioned processing, the size of the output first image feature will be restored (it is mirror-symmetric to the encoding end), and the size and number of channels of the input image feature may also change, etc., which is not specifically limited. The aforementioned processing and... Figure 9 The conversion method in step 901 of the illustrated embodiment is reversed.

[0233] Similarly, the first image feature can also be represented as a two-dimensional matrix or a three-dimensional matrix. The principle is described in step 901 and will not be repeated here.

[0234] Step 1302: Perform nonlinear transformation on the first image features to obtain the processed image features.

[0235] In one possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; the second image feature is convolved to obtain a third image feature, wherein multiple feature values ​​in the third image feature correspond to multiple feature values ​​in the first image feature; and the corresponding multiple feature values ​​in the first image feature and the third image feature are multiplied point by point to obtain the processed image feature.

[0236] This embodiment can be referred to.Figure 10a The embodiments shown here will not be repeated.

[0237] In a possible implementation, the first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; the second image feature is subjected to convolution processing to obtain a third image feature, and the plurality of feature values in the third image feature correspond to the plurality of feature values in the first image feature; the plurality of corresponding feature values in the first image feature and the third image feature are subjected to point-by-point multiplication operation to obtain a fourth image feature, and the plurality of feature values in the fourth image feature correspond to the plurality of feature values in the first image feature; and the plurality of corresponding feature values in the first image feature and the fourth image feature are subjected to point-by-point addition operation to obtain the processed image feature.

[0238] The embodiments can refer to Figure 10b The embodiments shown here will not be repeated.

[0239] At step 1303, the reconstructed image is obtained according to the processed image feature.

[0240] After the processed image feature is obtained, the decoding side can also continue to perform convolution processing thereon, or can perform nonlinear transformation processing on the output of the convolution processing again, so as to convert it from the feature domain to the image domain and obtain the reconstructed image.

[0241] Figure 14 The structure diagram of the decoding network is shown in Figure 14 The decoding network includes four deconvolution layers (Deconv) and three nonlinear transformation units, and the deconvolution layers and the nonlinear transformation units are arranged alternately, that is, the input image is subjected to deconvolution processing, nonlinear transformation processing, deconvolution processing, nonlinear transformation processing, deconvolution processing, nonlinear transformation processing, and deconvolution processing in sequence to obtain the reconstructed image. The nonlinear transformation unit can adopt the structure of the embodiments shown in Figure 10a or Figure 10b After that, the output image feature is subjected to entropy decoding, which will not be repeated here.

[0242] In the embodiments, the nonlinear transformation processing in the decoding network is changed, so that each input feature value only needs its own characteristics to obtain the output value, without considering the influence of the surrounding feature values, and the original feature value is modified to highlight the important feature value and suppress the redundant feature value. In addition, the value of each feature value in the first image feature is corrected, avoiding the limitation of the convolution parameter, so as to realize the efficient nonlinear transformation processing in the encoding network and further improve the rate-distortion performance in the image / video compression algorithm.

[0243] It should be noted that this application embodiment also provides a training method for an encoding / decoding network, which may include: firstly, constructing an end-to-end encoding / decoding network, including an encoder, a decoder, and an entropy estimation unit. During training, the encoding / decoding network is treated as a whole. Training data (images or videos) is input into the encoder to obtain feature data; the feature data is processed by the entropy estimation unit to calculate the encoding bitrate overhead, resulting in a bitrate loss; the feature data is also processed by the decoder to output reconstructed data. The reconstructed data and the input data are compared to calculate the distortion level, resulting in a distortion loss. The backpropagation algorithm updates the learnable parameters in the model using a weighted loss composed of the bitrate loss and the distortion loss. After training is complete, the parameters of all sub-modules in the model are fixed.

[0244] Subsequently, the encoder and entropy estimation unit are assigned to the encoding end to encode the data to be encoded into a bitstream file; the decoder and entropy estimation unit are assigned to the decoding end to reconstruct the data from the bitstream file.

[0245] The above image encoding method is described below through several specific embodiments. In the following text, the nonlinear transformation unit provided in the embodiments of this application is referred to as ResAU.

[0246] Example 1

[0247] Figure 15a Here is an example structural diagram of ResAU, such as Figure 15a As shown, ResAU uses Figure 10b The structure shown includes a first nonlinear operation, convolution processing, pointwise multiplication, and pointwise addition, where the first nonlinear operation uses absolute value (abs). This ResAU can be applied to... Figure 12 The encoding network shown or Figure 14 The decoding network shown.

[0248] Based on formula (2), the ResAU of this embodiment can be expressed as the following formula:

[0249] y i =x i *(β i +∑ j γ ij |x j |)+x i

[0250] right Figure 15a The ResAU shown is used for compression performance testing.

[0251] Test set: Kodak test set, which contains 24 portable network graphics (PNG) images with a resolution of 768×512 or 512×768.

[0252] Experiment: Using Gaussian mixture hyperprior entropy estimation in encoding / decoding network architectures Figure 15a The ResAU shown.

[0253] The corresponding performance results of the experiment: Figure 16a The overall performance of ResAU on 24 images in the Kodak test set is shown. Compared with the currently mainstream GDN nonlinear unit, the GMM network using ResAU has better RD performance at the same computational complexity. More specifically, to achieve the same decoding and reconstruction quality, using ResAU in the GMM encoder / decoder network can save about 12% of the coding rate overhead compared to ReLU, and about 5% compared to GDN.

[0254] Performance Analysis: In the ResAU provided in this application embodiment, introducing nonlinear characteristics by taking the absolute value is a feasible solution. On the basic GMM structure, using ResAU can achieve better rate-distortion performance than using GDN.

[0255] Example 2

[0256] Figure 15b Here is an example structural diagram of ResAU, such as Figure 15b As shown, ResAU uses Figure 10b The structure shown includes a first nonlinear operation, convolution processing, pointwise multiplication, and pointwise addition, where the first nonlinear operation uses ReLU. This ResAU can be applied to... Figure 12 The encoding network shown or Figure 14 The decoding network shown.

[0257] Based on formula (2), the ResAU of this embodiment can be expressed as the following formula:

[0258]

[0259] right Figure 15b The ResAU shown is used for compression performance testing.

[0260] Test set: 24 Kodak test images.

[0261] Experiment: Using Gaussian mixture hyperprior entropy estimation in encoding / decoding network architectures Figure 15bThe ResAU is shown. In this experiment, the nonlinear operation in ResAU was replaced with ReLU and LeakyReLU, respectively. The comparative experiments included the Identity scheme without nonlinear operation and the ResAU scheme with absolute value as described in Example 1.

[0262] The corresponding performance results of the experiment: Figure 16b This shows the overall performance of ResAU on 24 images in the Kodak test set. The RD curves show that, compared to the Identity scheme without non-linear operations, using pointwise non-linear operations can significantly improve the rate-distortion performance of the image compression network. Furthermore, ReLU-type non-linear operations perform slightly better than absolute value-based non-linear operations.

[0263] Figure 17 This is an exemplary structural diagram of the encoding device 1700 according to an embodiment of this application, as shown below. Figure 17 As shown, the device 1700 of this embodiment can be applied to the encoding end. The device 1700 may include: an acquisition module 1701, a transformation module 1702, an encoding module 1703, and a training module 1704. Wherein,

[0264] The acquisition module 1701 is used to acquire the first image features to be processed; the transformation module 1702 is used to perform nonlinear transformation processing on the first image features to obtain processed image features, wherein the nonlinear transformation processing includes a first nonlinear operation, convolution processing and point-by-point multiplication operation in sequence; and the encoding module 1703 is used to encode the processed image features to obtain a bitstream.

[0265] In one possible implementation, the transformation module 1702 is specifically used to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; to perform the convolution processing on the second image feature to obtain a third image feature, wherein multiple feature values ​​in the third image feature correspond to multiple feature values ​​in the first image feature; and to perform the point-by-point multiplication operation on the corresponding multiple feature values ​​of the first image feature and the third image feature to obtain the processed image feature.

[0266] In one possible implementation, the nonlinear transformation process further includes a point-by-point addition operation following the point-by-point multiplication operation.

[0267] In a possible implementation, the transformation module 1702 is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, a plurality of feature values in the fourth image feature correspond to the plurality of feature values in the first image feature; and perform the point-by-point addition operation on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain the processed image feature.

[0268] In a possible implementation, the nonlinear transformation processing further includes a second nonlinear operation between the convolution processing and the point-by-point multiplication operation, the second nonlinear operation is the same as or different from the first nonlinear operation.

[0269] In a possible implementation, the first nonlinear operation includes an absolute value operation or a piecewise linear mapping.

[0270] In a possible implementation, the apparatus further includes a training module 1704 configured to construct a training stage nonlinear transformation unit, the training stage nonlinear transformation unit including a first nonlinear operation layer, a convolution processing layer, a point-by-point multiplication operation layer, and a point-by-point addition operation layer; and train a trained nonlinear transformation unit according to pre-acquired training data, the trained nonlinear transformation unit being used to implement the nonlinear transformation processing.

[0271] The apparatus of the embodiment can be used to perform the method of the embodiment. Figure 9 The technical solutions of the method embodiment are similar in principle and technical effects to the apparatus embodiment, and are not described here again.

[0272] Figure 18 An example structure of a decoding apparatus 1800 of the embodiment is shown in FIG. 18. As shown in FIG. 18, the apparatus 1800 of the embodiment can be applied to a decoding end. The apparatus 1800 can include an acquisition module 1801, a transformation module 1802, a reconstruction module 1803, and a training module 1804. Specifically, Figure 18

[0273] The acquisition module 1801 is configured to acquire a first image feature to be processed. The transformation module 1802 is configured to perform nonlinear transformation processing on the first image feature to obtain a processed image feature, the nonlinear transformation processing including, in sequence, a first nonlinear operation, convolution processing, and a point-by-point multiplication operation. The reconstruction module 1803 is configured to acquire a reconstructed image according to the processed image feature. ​

[0274] In a possible implementation, the transformation module 1802 is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; and perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

[0275] In a possible implementation, the nonlinear transformation processing further includes a point-by-point addition operation after the point-by-point multiplication operation.

[0276] In a possible implementation, the transformation module 1802 is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain a fourth image feature, a plurality of feature values in the fourth image feature correspond to a plurality of feature values in the first image feature; and perform the point-by-point addition operation on the corresponding plurality of feature values in the first image feature and the fourth image feature to obtain the processed image feature.

[0277] In a possible implementation, the nonlinear transformation processing further includes a second nonlinear operation between the convolution processing and the point-by-point multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation.

[0278] In a possible implementation, the first nonlinear operation includes an absolute value operation or a piecewise linear mapping.

[0279] In a possible implementation, the training module 1804 is configured to construct a nonlinear transformation unit in a training stage, the nonlinear transformation unit in the training stage including a first nonlinear operation layer, a convolution processing layer, a point-by-point multiplication operation layer, and a point-by-point addition operation layer; and train a trained nonlinear transformation unit according to pre-acquired training data, the trained nonlinear transformation unit being used to implement the nonlinear transformation processing.

[0280] The apparatus of the embodiment can be used to perform the method of the embodiment. Figure 13 The technical solutions of the method embodiment are similar in principle and technical effects to the apparatus embodiment, and will not be described here.

[0281] In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits of hardware in the processor or instructions in the form of software. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware coding executed by the processor to complete, or executed by a combination of hardware and software modules in the coding processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, and other mature storage media in the art. The storage medium is located in the storage, and the processor reads the information in the storage, and combines the hardware to complete the steps of the above method.

[0282] The memory mentioned in each of the above embodiments can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct Rambus RAM (DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but not be limited to, these and any other suitable types of memory.

[0283] Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of the present application.

[0284] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0285] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the division of the above-described device embodiments is only a logical function division, and there can be another division manner for actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, or the among different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0286] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0287] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically, or two or more units can be integrated into one unit.

[0288] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (personal computer, server, network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. Alternatively, the computer software product can be transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD), a read-only memory (ROM), a random access memory (RAM), etc.).

[0289] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. An image coding method characterized by, The method comprises: obtaining a first image feature to be processed; performing nonlinear transformation processing on the first image feature to obtain a processed image feature, the nonlinear transformation processing comprising, in sequence, first nonlinear operation, convolution processing and point-by-point multiplication operation; encoding the processed image feature to obtain a code stream; wherein the nonlinear transformation processing on the first image feature to obtain the processed image feature comprises: performing the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing the convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature corresponding to a plurality of feature values in the first image feature; performing the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

2. The method of claim 1, wherein, The nonlinear transformation processing further comprises second nonlinear operation between the convolution processing and the point-by-point multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation.

3. The method according to claim 1 or 2, characterized in that, The first nonlinear operation comprises absolute value operation, ReLU series, Sigmoid or Tanh.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: constructing a nonlinear transformation unit in a training stage, the nonlinear transformation unit in the training stage comprising a first nonlinear operation layer, a convolution processing layer and a point-by-point multiplication operation layer; training a trained nonlinear transformation unit according to pre-acquired training data, the trained nonlinear transformation unit being used to implement the nonlinear transformation processing.

5. An image decoding method characterized by, The method comprises: obtaining a first image feature to be processed; performing nonlinear transformation processing on the first image feature to obtain a processed image feature, the nonlinear transformation processing comprising, in sequence, first nonlinear operation, convolution processing and point-by-point multiplication operation; obtaining a reconstructed image according to the processed image feature. wherein the nonlinear transformation processing on the first image feature to obtain the processed image feature comprises: performing the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing the convolution processing on the second image feature to obtain a third image feature, a plurality of feature values in the third image feature corresponding to a plurality of feature values in the first image feature; performing the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

6. The method of claim 5, wherein, The nonlinear transformation processing further comprises second nonlinear operation between the convolution processing and the point-by-point multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation.

7. The method according to claim 5 or 6, characterized in that, The first nonlinear operation comprises absolute value operation, ReLU series, Sigmoid or Tanh.

8. The method according to any one of claims 5-7, characterized in that, The method further comprises: constructing a nonlinear transformation unit in a training stage, the nonlinear transformation unit in the training stage comprising a first nonlinear operation layer, a convolution processing layer and a point-by-point multiplication operation layer; The trained nonlinear transformation unit is trained according to pre-acquired training data, and is used to implement the nonlinear transformation process.

9. An encoding apparatus, comprising: Comprise: The acquisition module is configured to acquire a first image feature to be processed; The transformation module is configured to perform nonlinear transformation processing on the first image feature to obtain a processed image feature, wherein the nonlinear transformation processing comprises, in sequence, first nonlinear operation, convolution processing, and point-by-point multiplication operation; The encoding module is configured to encode the processed image feature to obtain a bitstream; The transformation module is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, wherein a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; and perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

10. The apparatus of claim 9, wherein, The nonlinear transformation processing further comprises second nonlinear operation between the convolution processing and the point-by-point multiplication operation, and the second nonlinear operation is the same as or different from the first nonlinear operation.

11. The apparatus of claim 9 or 10, wherein, The first nonlinear operation comprises absolute value operation, ReLU series, Sigmoid, or Tanh.

12. The apparatus of any one of claims 9-11, wherein, Further comprise: The training module is configured to construct a nonlinear transformation unit in a training stage, wherein the nonlinear transformation unit in the training stage comprises a first nonlinear operation layer, a convolution processing layer, and a point-by-point multiplication operation layer; and a trained nonlinear transformation unit is trained according to pre-acquired training data, and is used to implement the nonlinear transformation process.

13. A decoding apparatus, characterized by comprising: Comprise: The acquisition module is configured to acquire a first image feature to be processed; The transformation module is configured to perform nonlinear transformation processing on the first image feature to obtain a processed image feature, wherein the nonlinear transformation processing comprises, in sequence, first nonlinear operation, convolution processing, and point-by-point multiplication operation; The reconstruction module is configured to acquire a reconstructed image according to the processed image feature; The transformation module is specifically configured to perform the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; perform the convolution processing on the second image feature to obtain a third image feature, wherein a plurality of feature values in the third image feature correspond to a plurality of feature values in the first image feature; and perform the point-by-point multiplication operation on the corresponding plurality of feature values in the first image feature and the third image feature to obtain the processed image feature.

14. The apparatus of claim 13, wherein, The nonlinear transformation processing further comprises second nonlinear operation between the convolution processing and the point-by-point multiplication operation, and the second nonlinear operation is the same as or different from the first nonlinear operation.

15. The apparatus of claim 13 or 14, wherein, The first nonlinear operation comprises absolute value operation, ReLU series, Sigmoid, or Tanh.

16. The apparatus of any one of claims 13-15, wherein, The device further comprises: The training module is configured to construct a nonlinear transformation unit of a training stage, the nonlinear transformation unit of the training stage comprising a first nonlinear operation layer, a convolution processing layer and a point-by-point multiplication operation layer; and a trained nonlinear transformation unit is obtained by training according to pre-acquired training data, the trained nonlinear transformation unit being configured to implement the nonlinear transformation processing.

17. An encoder comprising: Comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, wherein the programming, when executed by the processors, causes the encoder to carry out the method according to any one of claims 1-4.

18. A decoder, comprising: Comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, wherein the programming, when executed by the processors, causes the encoder to carry out the method according to any one of claims 1-4.

19. A computer program product, characterised in that, Comprising program code for performing the method according to any one of claims 1-8 when the program code is executed on a computer or processor.

20. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to carry out the method according to any one of claims 1-8.

Images