Method and apparatus for image encoding and decoding

Abstract

This application provides an encoding and decoding method and apparatus.SOLUTION: An image encoding method in this application includes obtaining a to-be-processed first image feature, performing nonlinear conversion processing on the first image feature to obtain a processed image feature, where the nonlinear conversion processing sequentially includes a first nonlinear operation, convolution processing, and an elementwise multiplication operation, and performing encoding based on the processed image feature to obtain a bitstream. This application can avoid a limitation on a convolution parameter, to implement efficient non-linear transformation processing in an encoding / decoding network, and further improve rate-distortion performances of an image / video compression algorithm.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] This application claims priority to Chinese Patent Application No. 202111470979.5, entitled "Method and Apparatus for Image Encoding and Decoding," filed with the State Intellectual Property Office of China on December 3, 2021, the entirety of which is incorporated herein by reference.

[0002] Technical Field The present application relates to the field of image processing technology, and in particular to a method and apparatus for image encoding and decoding. [Background technology]

[0003] As convolutional neural networks (CNNs) outperform traditional algorithms in computer vision tasks such as image recognition and target detection, more and more researchers are beginning to explore deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms, for example, in which modules such as an encoding network, an entropy estimation network, an entropy encoding network, an entropy decoding network, and a decoding network are optimized as a whole. The encoding network and the decoding network, sometimes called the transform module and the inverse transform module, generally include a convolutional layer and a nonlinear transform unit.

[0004] Nonlinear transform units are one of the fundamental components of image / video compression networks. The nonlinear characteristics of nonlinear transform units directly affect the rate-distortion performance of compression algorithms. Therefore, designing more efficient nonlinear transform units is the key to further improving the rate-distortion performance of image / video compression algorithms. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide an image encoding and decoding method and apparatus for implementing efficient nonlinear transform processing in the encoding / decoding network to further improve the rate-distortion performance of image / video compression algorithms. [Means for solving the problem]

[0006] According to a first aspect, an embodiment of the present application provides an image encoding method, including: obtaining first image features to be processed; performing a nonlinear transformation process on the first image features to obtain processed image features, the nonlinear transformation process sequentially including a first nonlinear operation, a convolution operation, and an element-wise multiplication operation; and performing encoding based on the processed image features to obtain a bitstream.

[0007] The first image feature is obtained after the encoder side obtains the image to be processed and transforms the image to be processed from the image domain to the feature domain. The transformation in this specification can include, but is not limited to: 1. convolution processing, in which a convolution layer is used to extract features, and the convolution processing has a local receptive field and a weight sharing mechanism (i.e., each filter processes the input features sequentially); 2. extracting features by using a multi-layer perceptron (MLP) or a fully connected layer, where the feature extraction by using an MLP or a fully connected layer has global receptive field features and no weight sharing; 3. transformer processing, including matrix multiplication, MLP, and normalization processing, where the transformer processing has global receptive field features and has strong long-range capture dependency ability.

[0008] In one possible implementation, a nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature, a convolution operation is performed on the second image feature to obtain a third image feature, where multiple feature values ​​in the third image feature correspond to multiple feature values ​​in the first image feature, and an element-wise multiplication operation is performed on multiple corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0009] The first nonlinear operation is an operation performed on each feature value in the first image feature, and may include an absolute value operation, a rectified linear unit (ReLU) system, a sigmoid, a Tanh, a piecewise linear (PWL) operation, etc. The rectified linear unit is also called a rectified linear unit.

[0010] After the first nonlinear operation, the first image feature is transformed into a second image feature. The second image feature and the first image feature may be represented in the form of a matrix. Because the first nonlinear operation is performed on each feature value in the first image feature, each feature value in the first image feature corresponds to one feature value in the second image feature. Therefore, the matrix corresponding to the second image feature and the matrix corresponding to the first image feature have the same size, and feature values ​​(matrix element values) at the same position correspond to each other. For example, if the first image feature is represented as a 3x3 matrix, the second image feature may also be represented as a 3x3 matrix. However, because the first nonlinear operation is performed on the first image feature and the second image feature, the feature values ​​in the first image feature and the second image feature are not completely the same. Correspondingly, the element values ​​in the matrices corresponding to the first image feature and the second image feature, respectively, are not completely the same.

[0011] A convolution process is performed on the second image feature to output a third image feature. The third image feature may be considered as a local response (i.e., refined value) of the second image feature. That is, the third image feature is a response signal obtained by performing a convolution process on the second image feature. Because the receptive field of the convolution process is limited, the response value of each position in the output image feature after the convolution process is related only to the input feature values ​​of positions adjacent to that position. This is called a local response.

[0012] It can be seen that a local attention mechanism is implemented after the aforementioned nonlinear operation, convolution, and element-wise multiplication. Local means that the nonlinear operation is performed element-wise, and for each input feature value, an output value is obtained based solely on the characteristics of the input feature value, without considering the influence of surrounding features. The attention mechanism means that some of all feature values ​​in the first image feature are important and some are redundant. The output of the convolution process may be a weight for each feature value in the image feature. The original feature values ​​may be refined, highlighting important feature values ​​and suppressing redundant feature values. In the element-wise multiplication operation, the value of each feature value in the first image feature is refined by using the aforementioned local information, and the convolution parameters do not need to be positive. This avoids limitations on the value range of the convolution parameters and allows for better convolution parameters in a wider value range. In this way, better image compression performance is achieved.

[0013] In one possible implementation, the nonlinear transformation process further comprises an element-wise multiplication operation followed by an element-wise addition operation.

[0014] In one possible implementation, a nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; a convolution operation is performed on the second image feature to obtain a third image feature, where the feature values ​​in the third image feature correspond to the feature values ​​in the first image feature; an element-wise multiplication operation is performed on corresponding feature values ​​in the first and third image features to obtain a fourth image feature, where the feature values ​​in the fourth image feature correspond to the feature values ​​in the first image feature; and an element-wise addition operation is performed on corresponding feature values ​​in the first and fourth image features to obtain a processed image feature.

[0015] The element-wise addition operation is a residual structure to which the head and tail are added, which allows encoding and decoding networks using the above processing process to easily converge during training. The convolution process conv1(x) is similar to the convolution process conv2(x). The difference is that the convolution process conv2(x) adds an additional 1 to the convolution bias parameter β. In this way, the above two implementations can be modified by fine-tuning the convolution bias parameter β in the convolution process. Specifically, if the nonlinear transformation unit includes the convolution process conv1(x) and the element-wise addition operation, the convolution process conv1(x) and the element-wise addition operation may be merged into the convolution process conv2(x), and the element-wise addition operation may be omitted. After the processed image features are obtained, the encoder side may proceed to perform a convolution process on the processed image features, or after the convolution process, perform a non-linear transformation process on the convolved features, and then encode the features obtained after the above process to obtain a bitstream.

[0016] In the present embodiment, the nonlinear transformation process in the encoding network is modified so that for each input feature value, an output value is obtained based solely on the characteristics of that input feature value, without considering the influence of surrounding feature values. The original feature values ​​are also refined to highlight important features and suppress redundant features. Additionally, the value of each feature value in the first image feature may be further refined, thereby avoiding limitations on convolution parameters and implementing an efficient nonlinear transformation process in the encoding network, further improving the rate-distortion performance of image / video compression algorithms.

[0017] The nonlinear operation is an operation performed on each feature value in the first image feature and may include using a piecewise linear mapping method. This method may be obtaining an absolute value (obtaining the absolute value of the input feature value), or may be a rectified linear unit (ReLU), a leaky rectified linear unit (LeakyReLU), or a piecewise linear (PWL) operation. ReLU is a piecewise linear mapping method. For input feature values, feature values ​​less than 0 are output as 0, and feature values ​​greater than or equal to 0 remain unchanged. LeakyReLU is a piecewise linear mapping method. Based on ReLU, input feature values ​​less than 0 are scaled by using a preset weight, which is usually 0.01. PWL operation is also a piecewise linear mapping method, and the amount of segments in PWL operation may be larger. For a specific definition of PWL operation, please refer to the following embodiments. In addition, the nonlinear operation may further include other methods, such as piecewise nonlinear operation, Tanh, or sigmoid. This is not particularly limited in the embodiments of the present application.

[0018] In one possible implementation, the nonlinear transformation process further includes a second nonlinear operation between the convolution process and the element-wise multiplication operation. The second nonlinear operation may be the same as or different from the first nonlinear operation. For example, the first nonlinear operation may be an absolute value operation, and the second nonlinear operation may be an absolute value operation, another piecewise linear mapping method, or another nonlinear operation. Alternatively, the first nonlinear operation may be ReLU, and the second nonlinear operation may be ReLU, sigmoid, or another nonlinear operation. Alternatively, the first nonlinear operation may be LeakyReLU, and the second nonlinear operation may be LeakyReLU, Tanh, or another nonlinear operation. Alternatively, the first nonlinear operation may be PWL, and the second nonlinear operation may be PWL or another nonlinear operation. When the second nonlinear operation is implemented using PWL, the piecewise linear mapping may use different segment amounts. The mapping gradient on each segment may be determined through training or specified directly. A different piecewise linear function may be used for each channel of the input feature image, or the same piecewise linear function may be used for all channels, or the same piecewise linear function may be used to process several channels. In this implementation, after model training is completed, the residual structure is no longer merged with the convolution but with the piecewise linear function. Specifically, the output of the original piecewise linear function is incremented by 1 to form a new piecewise linear function.

[0019] In one possible implementation, the nonlinear transformation unit in the training phase may further include a first nonlinear operation layer, a convolution processing layer, an element-wise multiplication operation layer, and an element-wise addition operation layer.

[0020] Training is performed based on previously obtained training data to obtain a trained nonlinear transformation unit, which is then used to implement a nonlinear transformation process.

[0021] According to a second aspect, an embodiment of the present application provides an image decoding method, including: obtaining first image features to be processed; performing a nonlinear transformation process on the first image features to obtain processed image features, the nonlinear transformation process sequentially including a first nonlinear operation, a convolution operation, and an element-wise multiplication operation; and obtaining a reconstructed image based on the processed image features.

[0022] In one possible implementation, performing a nonlinear transformation process on a first image feature to obtain a processed image feature includes: performing a first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing a convolution process on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain the processed image feature.

[0023] In one possible implementation, the nonlinear transformation process further comprises an element-wise addition operation.

[0024] In one possible implementation, performing a nonlinear transformation process on a first image feature to obtain a processed image feature includes: performing a first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing a convolution process on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, wherein a plurality of feature values ​​in the fourth image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise addition operation on a plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain the processed image feature.

[0025] In one possible implementation, the nonlinear operation includes a piecewise linear mapping, such as ReLU, LeakyReLU, PWL, and Abs. In another possible implementation, the nonlinear operation includes a continuous function, such as Tanh or sigmoid. In another possible implementation, the nonlinear operation includes a piecewise nonlinear operation. For technical effects of the image decoding method provided in the second aspect and possible implementations of the second aspect, please refer to the technical effects of the image encoding method provided in the first aspect and possible implementations corresponding to the first aspect. Details will not be described again in this specification.

[0026] According to a third aspect, an embodiment of the present application provides an encoding device including: an acquisition module configured to acquire first image features to be processed; a transformation module configured to perform a nonlinear transformation operation on the first image features to obtain processed image features, the nonlinear transformation operation sequentially including a first nonlinear operation, a convolution operation, and an element-wise multiplication operation; and an encoding module configured to perform encoding based on the processed image features to obtain a bitstream.

[0027] In one possible implementation, the transformation module is specifically configured to perform the steps of: performing a first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing a convolution operation on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0028] In one possible implementation, the nonlinear transformation process further comprises an element-wise multiplication operation followed by an element-wise addition operation.

[0029] In one possible implementation, the transformation module is specifically configured to perform the following steps: performing a first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing a convolution operation on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, wherein the plurality of feature values ​​in the fourth image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise addition operation on a plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain a processed image feature.

[0030] In one possible implementation, the nonlinear transformation process further includes a second nonlinear operation between the convolution process and the element-wise multiplication operation. The second nonlinear operation may be the same as or different from the first nonlinear operation. For example, the first nonlinear operation may be an absolute value operation, and the second nonlinear operation may also be an absolute value operation, and the second nonlinear operation may be referred to as a piecewise linear mapping or another nonlinear operation.

[0031] In one possible implementation, the nonlinear operation includes a piecewise linear mapping, such as ReLU, LeakyReLU, PWL, and Abs. In another possible implementation, the nonlinear operation includes a continuous function, such as Tanh or Sigmoid. In another possible implementation, the nonlinear operation includes a piecewise nonlinear operation.

[0032] In one possible implementation, the apparatus further includes a training module configured to perform the steps of: constructing a nonlinear transformation unit in a training phase, wherein the nonlinear transformation unit in the training phase includes a first nonlinear operation layer, a convolution processing layer, an element-wise multiplication operation layer, and an element-wise addition operation layer; and performing training based on pre-acquired training data to obtain a trained nonlinear transformation unit, wherein the trained nonlinear transformation unit is used to perform a nonlinear transformation process.

[0033] According to a fourth aspect, an embodiment of the present application provides a decoding device including: an acquisition module configured to acquire first image features to be processed; a transformation module configured to perform a nonlinear transformation process on the first image features to obtain processed image features, the nonlinear transformation process including a first nonlinear operation, a convolution process, and an element-wise multiplication operation; and a reconstruction module configured to obtain a reconstructed image based on the processed image features.

[0034] In one possible implementation, the transformation module is specifically configured to perform the steps of: performing a first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing a convolution operation on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0035] In one possible implementation, the nonlinear transformation process further comprises an element-wise multiplication operation followed by an element-wise addition operation.

[0036] In one possible implementation, the transformation module is specifically configured to perform the following steps: performing a first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing a convolution operation on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, wherein a plurality of feature values ​​in the fourth image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise addition operation on a plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain a processed image feature.

[0037] In one possible implementation, the nonlinear transformation process further includes a second nonlinear operation between the convolution process and the element-wise multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation.

[0038] In one possible implementation, the nonlinear operation includes a piecewise linear mapping, such as ReLU, LeakyReLU, PWL, and Abs. In another possible implementation, the nonlinear operation includes a continuous function, such as Tanh or a sigmoid. In another possible implementation, the nonlinear operation includes a piecewise nonlinear operation.

[0039] In one possible implementation, the apparatus further includes a training module configured to perform the steps of: constructing a nonlinear transformation unit in a training phase, wherein the nonlinear transformation unit in the training phase includes a first nonlinear operation layer, a convolution processing layer, an element-wise multiplication operation layer, and an element-wise addition operation layer; and performing training based on pre-acquired training data to obtain a trained nonlinear transformation unit, wherein the trained nonlinear transformation unit is used to implement a nonlinear transformation process.

[0040] According to a fifth aspect, an embodiment of the present application provides an encoder including one or more processors; and a non-transitory computer-readable storage medium, coupled to the processors, storing a program for execution by the processors, the program, when executed by the processors, enabling the encoder to perform a method according to any one of the first aspects above.

[0041] According to a sixth aspect, an embodiment of the present application provides a decoder including one or more processors; and a non-transitory computer-readable storage medium, coupled to the processors, storing a program for execution by the processors, the program, when executed by the processors, enabling the decoder to perform a method according to any one of the second aspects above.

[0042] According to a seventh aspect, an embodiment of the present application provides a computer program product including program code that, when run on a computer or processor, is configured to perform a method according to any one of the first and second aspects.

[0043] According to an eighth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform a method according to any one of the first and second aspects.

[0044] According to a ninth aspect, an embodiment of the present application provides a bitstream, the bitstream being generated by a processor by performing a method according to any one of the first aspects.

[0045] According to a tenth aspect, an embodiment of the present application provides a bitstream storage device, the device including a receiver and at least one storage medium, the receiver configured to receive a bitstream, and the at least one storage medium configured to store the bitstream, the bitstream being a bitstream generated according to any one of the methods according to the first aspect.

[0046] According to an eleventh aspect, an embodiment of the present application provides a bitstream transmission apparatus, the apparatus including a transmitter and at least one storage medium, the at least one storage medium configured to store a bitstream, the bitstream including a bitstream generated by a processor according to any one of the methods according to the first aspect, and the transmitter configured to transmit the bitstream to another electronic device.

[0047] According to a twelfth aspect, an embodiment of the present application provides a bitstream distribution system, the system including at least one storage medium and a streaming media device, the at least one storage medium configured to store at least one bitstream, the at least one bitstream including a bitstream generated according to any one of the implementations of the first aspect. The streaming media device is configured to retrieve a target bitstream from the at least one storage medium and transmit the target bitstream to a terminal-side device, the streaming media device including a content server or a content distribution server.

[0048] According to a thirteenth aspect, an embodiment of the present application provides a bitstream distribution system, the system including: a communication interface configured to receive a user request to retrieve a target bitstream; and a processor configured to determine a storage location of the target bitstream in response to the user request, the communication interface further configured to transmit the storage location of the target bitstream to a user so that the user retrieves the target bitstream from the storage location of the target bitstream, the target bitstream being generated by the processor by performing a method according to any one of the first aspect. [Brief explanation of the drawings]

[0049] [Figure 1A] 1 is a schematic block diagram of a coding system 10.

[0050] [Figure 1B] 1 is a schematic diagram of a video coding system 40.

[0051] [Figure 2] 4 is a schematic diagram of a video coding device 400 according to an embodiment of the present invention.

[0052] [Figure 3] 5 is a simplified block diagram of an apparatus 500 according to an embodiment.

[0053] [Figure 4] 1 is a schematic diagram of an end-to-end deep learning image encoding and decoding framework.

[0054] [Figure 5] Schematic diagram of an end-to-end deep learning video encoding and decoding framework.

[0055] [Figure 6]1 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0056] [Figure 7] 1 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0057] [Figure 8] 1 is a schematic diagram of an application scenario according to an embodiment of the present application;

[0058] [Figure 9] 9 is a flowchart of a process 900 of an image encoding method according to an embodiment of the present application.

[0059] [Figure 10a] 1 is a schematic diagram of the structure of a nonlinear transformation unit with a local attention mechanism.

[0060] [Figure 10b] 1 is a schematic diagram of the structure of a residual nonlinear transformation unit with local attention mechanism;

[0061] [Figure 10c] 1 is a schematic diagram of the structure of a residual nonlinear transformation unit with attention mechanism;

[0062] [Figure 10d] 1 is a schematic diagram of the structure of a nonlinear transformation unit with attention mechanism;

[0063] [Figure 11] FIG. 1 is a schematic diagram of a PWL function.

[0064] [Figure 12] FIG. 1 is a schematic diagram of a convolution process.

[0065] [Figure 13] Schematic diagram of the encoding network structure.

[0066] [Figure 14] 13 is a flowchart of a process 1300 of an image decoding method according to an embodiment of the present application.

[0067] [Figure 15] Schematic diagram of the structure of the decoding network.

[0068] [Figure 16a] FIG. 1 is an exemplary schematic diagram of the structure of ResAU.

[0069] [Figure 16b] FIG. 1 is an exemplary schematic diagram of the structure of ResAU.

[0070] [Figure 16c] FIG. 1 is an exemplary schematic diagram of the structure of ResAU.

[0071] [Figure 16d] FIG. 1 is an exemplary schematic diagram of the structure of ResAU.

[0072] [Figure 16e] FIG. 1 is an exemplary schematic diagram of the structure of ResAU.

[0073] [Figure 17a] Overall performance of ResAU on 24 images in the Kodak test set.

[0074] [Figure 17b] Overall performance of ResAU on 24 images in the Kodak test set.

[0075] [Figure 18] 17 is a schematic diagram of the structure of an encoding device 1700 according to an embodiment of the present application.

[0076] [Figure 19] 18 is a schematic diagram of the structure of a decoding device 1800 according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0077] In order to make the objectives, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the accompanying drawings in the present application. It is clear that the described embodiments are only a part of, but not all of, the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0078] In the specification, embodiments, claims, and accompanying drawings of this application, terms such as "first," "second," etc. are intended merely to distinguish and describe, and should not be understood as indicating or implying relative importance or ordering. In addition, the terms "comprise," "have," and any variations thereof are intended to cover non-exclusive inclusions, e.g., including a series of steps or units. For example, a method, system, product, or device is not necessarily limited to explicitly recited steps or units, but may include other steps or units not explicitly recited or inherent in such a process, method, product, or device.

[0079] It should be understood that, in this application, "at least one (item)" refers to one or more, and "multiple" refers to two or more. The term "and / or" is used to describe an association relationship between associated objects and indicates that three relationships may exist. For example, "A and / or B" can represent three cases: only A is present, only B is present, or both A and B are present. Here, A and B may be singular or plural. The symbol " / " generally indicates an "or" relationship between associated objects. The phrase "at least one of the following items" or similar phrases refers to any combination of these items, including any combination of single items or multiple items. For example, "at least one of a, b, or c" may refer to a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c may be singular or plural.

[0080] The terms in the embodiments of the present application are explained below.

[0081] 1. Bit rate: It is the average encoding length required to encode a unit pixel in image compression.

[0082] 2. Rate-Distortion Performance: A metric used to measure the performance of a compression algorithm. It takes into account the bit rate and distortion of the decoded image.

[0083] 3. Attention mechanism: A means of screening high-value information from a large amount of information using limited attention resources. A neural network can pay more attention to relevant parts of the input and less attention to irrelevant parts.

[0084] 4. Residual structure: Residual structure is a common connection structure in neural networks, and the calculation method can be expressed as H(x) = x + f(x). This structure can prevent the gradient vanishing and gradient exploding problems that can occur when the network depth increases.

[0085] 5. Nonlinear Transformation Unit: A nonlinear transformation unit is a network unit that includes nonlinear operations (for example, operations such as ReLU, Sigmoid, Tanh, or PWL), and the overall calculation method of the unit does not fit linear characteristics.

[0086] Since the embodiments of the present application relate to the application of neural networks, for ease of understanding, the following will first explain and describe relevant nouns or terms used in the embodiments of the present application.

[0087] 1. Neural Networks

[0088] A neural network (NN) is a machine learning model. A neural network may contain neurons. A neuron is a set of x s and the intercept of 1 as input, and the output of the calculation unit may be:

number

[0089] s=1, 2, …, or n, where n is a natural number greater than 1, and W s x swhere is the weight of the neuron, and b is the bias of the neuron. f is the activation function of the neuron, which is configured to introduce nonlinear features into the neural network to convert the input signal at the neuron into an output signal. The output signal of the activation function may be used as the input of the next convolutional layer. The activation function may be a nonlinear function such as ReLU. A neural network is a network formed by connecting many single neurons. Specifically, the output of one neuron may be the input of another neuron. The input of each neuron may be connected to the local receptive field of the previous layer to extract features of the local receptive field. The local receptive field may be an area containing several neurons.

[0090] 2. Multi-layer perceptron (MLP)

[0091] MLP is a simple deep neural network (DNN) (different layers are fully connected), also called a multi-layer neural network. MLP can be understood as a neural network with multiple hidden layers. There is no special metric for "multiple" in this specification. DNN is divided based on the position of different layers, and neural networks in DNN can be divided into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layer is a hidden layer. The layers are fully connected. Specifically, any neuron in the i-th layer is always connected to any neuron in the i+1-th layer. DNN looks complicated, but it is not complicated in terms of the work in each layer. Simply put, DNN is a linear relationship as follows:

number

[0092] 3. Convolutional Neural Networks

[0093] A convolutional neural network (CNN) is a deep neural network with a convolutional structure and a deep learning architecture. In a deep learning architecture, multi-layer learning is performed at different levels of abstraction according to a machine learning algorithm. As a deep learning architecture, a CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network may respond to images input to the neural network. A convolutional neural network includes a feature extractor composed of convolutional layers and pooling layers. The feature extractor may be considered as a filter. The convolution process may be considered as using a trainable filter to perform convolution on an input image or a convolutional feature plane (feature map).

[0094] A convolutional layer is a neuron layer in a convolutional neural network where a convolution operation is performed on an input signal. A convolutional layer may contain multiple convolution operators. A convolution operator is also called a kernel. In image processing, a convolution operator functions as a filter that extracts specific information from an input image matrix. A convolution operator may essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, the weight matrix is ​​usually used to process pixels at a granularity of one pixel (or two pixels, depending on the stride value) horizontally in the input image to extract specific features from the image. The size of the weight matrix should be related to the size of the image. Note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. During the convolution operation, the weight matrix spans the entire depth of the input image. Therefore, a convolution output with a single depth dimension is generated through convolution with a single weight matrix. However, in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (rows x columns), i.e., multiple matrices of the same type, are applied. The outputs of the weight matrices are stacked to form the depth dimension of the convolved image. The dimension in this specification may be understood as being determined based on the aforementioned "multiple." Different weight matrices may be used to extract different features from an image. For example, one weight matrix is ​​used to extract edge information of the image, another weight matrix is ​​used to extract specific colors of the image, and yet another weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows x columns). The feature maps extracted from multiple weight matrices with the same size also have the same size, and then the multiple extracted feature maps with the same size are combined to form the output of the convolution operation. In practical applications, the weight values ​​in these weight matrices need to be obtained through extensive training. Each weight matrix formed by using the weight values ​​obtained through training may be used to extract information from the input image, allowing the convolutional neural network to perform correct predictions.When a convolutional neural network has multiple convolutional layers, a relatively large number of general features are usually extracted in the initial convolutional layers. The general features are also called low-level features. As the depth of the convolutional neural network increases, the features extracted in subsequent convolutional layers become more complex, for example, high-level semantic features. Features with higher-level semantic content are more applicable to the problem to be solved.

[0095] The amount of training parameters often needs to be reduced. Therefore, pooling layers often need to be introduced periodically after convolutional layers. A convolutional layer may be followed by a pooling layer, or multiple convolutional layers may be followed by one or more pooling layers. During image processing, pooling layers are used solely to reduce the spatial size of an image. Pooling layers may include average pooling and / or max pooling operators to perform sampling on the input image to obtain a smaller image. An average pooling operator may be used to calculate pixel values ​​within a specific range and generate an average value. The average value is used as the average pooling result. A max pooling operator may be used to select the pixel with the maximum value within a specific range as the max pooling result. Similarly, just as the size of the weight matrix in a convolutional layer needs to be related to the size of the image, the operators in a pooling layer also need to be related to the size of the image. The size of the processed image output from a pooling layer may be smaller than the size of the image input to the pooling layer. Each pixel in the image output from the pooling layer represents the mean or maximum value of the corresponding subregion of the image input to the pooling layer.

[0096] After the processing performed in the convolutional / pooling layers, the convolutional neural network is not yet ready to output the required output information. This is because, as mentioned above, the convolutional / pooling layers only extract features and reduce the parameters resulting from the input image. However, to generate the final output information (required class information or other related information), the convolutional neural network must use neural network layers to generate the output of one required class or a group of required classes. Therefore, the convolutional neural network layers may contain multiple hidden layers. The parameters contained in the multiple hidden layers may be obtained through pre-training based on training data related to a specific task type. For example, task types may include image recognition, image classification, and super-resolution image reconstruction.

[0097] Optionally, in the neural network layer, multiple hidden layers are followed by an output layer of the entire convolutional neural network. The output layer has a loss function similar to categorical cross entropy, and the loss function is specifically used to calculate prediction errors. After forward propagation through the entire convolutional neural network is completed, backward propagation begins to update the weight values ​​and deviations of each of the above layers to reduce the loss of the convolutional neural network and the error between the results output by the convolutional neural network and the ideal results using the output layer.

[0098] 4. Recurrent Neural Networks

[0099] Recurrent neural networks (RNNs) are used to process sequence data. In traditional neural network models, each layer, from the input layer to the hidden layer to the output layer, is fully connected, and the nodes in each layer are not connected to each other. While such general neural networks can solve many problems, they cannot solve many others. For example, predicting the next word in a sentence typically requires using the previous word, because the previous and next words in a sentence are not independent. RNNs are called recurrent neural networks because the current output of a sequence is related to the previous output of the sequence. Specifically, the network memorizes previous information and applies that information to the calculation of the current output. Specifically, the nodes in the hidden layer are connected to each other, and the input of the hidden layer includes not only the output of the input layer but also the output of the hidden layer at the previous point in time. In theory, RNNs can process sequence data of any length. Training for RNNs is similar to training for traditional CNNs and DNNs. The backpropagation algorithm is also used, but with the following differences: When an RNN is expanded, parameters such as W of the RNN are shared. This is different from the traditional neural network described in the example above. Also, when using the gradient descent algorithm, the output of each stage depends not only on the network state in the current stage, but also on the network state of several previous stages. The learning algorithm is called the back propagation through time (BPTT) algorithm.

[0100] Why do we need recurrent neural networks even when convolutional neural networks are available? The reason is simple. Convolutional neural networks assume that elements are independent of each other, and that inputs and outputs are also independent, like cats and dogs. However, in the real world, multiple elements are interconnected. For example, stocks change over time. Another example is when someone says, "I love traveling, and my favorite place is Yunnan. I'll go there if I have the chance." Here, people should know that the person is going to "Yunnan." This is because people perform inference from context. But how does a machine do this? That's where RNNs come in. RNNs are intended to enable machines to remember like humans. Therefore, the output of an RNN must depend on the current input information and stored information from history.

[0101] 5. Loss Function

[0102] In the process of training a deep neural network, since it is expected that the output of the deep neural network will be as close as possible to the actual expected predicted value, the current network predicted value may be compared with the actual expected target value, and then the weight vector of each layer of the neural network is updated based on the difference between the predicted value and the target value (of course, there is usually an initialization process before the first update, specifically, parameters are pre-configured for all layers of the deep neural network). For example, if the network predicted value is large, the weight vector is adjusted to reduce the predicted value, and adjustments are continuously performed until the deep neural network can predict the actual expected target value or a value close to the actual expected target value. Therefore, it is necessary to predefine "how to obtain the difference between the predicted value and the target value through comparison." This is the loss function or objective function. The loss function and objective function are important formulas that measure the difference between the predicted value and the target value. The loss function is used as an example. A higher output value (loss) of the loss function indicates a larger difference. Therefore, training a deep neural network is a process of minimizing the loss as much as possible.

[0103] 6. Backpropagation Algorithm

[0104] A convolutional neural network may use the backpropagation (BP) algorithm to correct the parameter values ​​of the initial super-resolution model during the training process, thereby reducing the error loss in reconstructing the super-resolution model. Specifically, the input signal is forward-transferred until an error loss occurs at the output, and the parameters of the initial super-resolution model are updated based on the backpropagation error loss information, causing the error loss to converge. The backpropagation algorithm is an error-loss-centered backpropagation process intended to obtain parameters such as the weight matrix of the optimal super-resolution model.

[0105] 7. Generative Adversarial Networks

[0106] A generative adversarial network (GAN) is a deep learning model. The model includes at least two modules. One module is a generative model, and the other module is a discriminative model. The two modules are used to learn through competition with each other to generate better outputs. Both the generative model and the discriminative model may be neural networks, specifically, deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Using a GAN for generating pictures as an example, suppose there are two networks, namely, G (generator) and D (discriminator). G is a network that generates pictures. G receives random noise z and generates pictures by using the noise, where the picture is denoted as G(z). D is a discriminator network used to determine whether a picture is "real." The input parameter of D is x, where x represents a picture, and the output D(x) represents the probability that x is a real picture. When D(x) has a value of 1, it indicates that the picture is 100% authentic. When D(x) has a value of 0, it indicates that the picture cannot be authentic. In the process of training a generative adversarial network, the goal of the generative network G is to generate pictures that are as realistic as possible to deceive the discriminator network D, and the goal of the discriminator network D is to distinguish between pictures generated by G and authentic pictures as much as possible. Thus, a dynamic "match" process, specifically the "adversarial" in the "generative adversarial network," exists between G and D. In the final match result, ideally, G may generate an image G(z) that is difficult to distinguish from an authentic image, and D has difficulty determining whether the image generated by G is authentic, specifically, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained and can be used to generate pictures.

[0107] Because convolutional neural networks (CNNs) outperform traditional algorithms in computer vision tasks such as image recognition and target detection, more and more researchers have begun to explore deep learning-based image / video compression methods. Some researchers have designed end-to-end deep learning image / video compression algorithms. For example, modules such as an encoding network, an entropy estimation network, an entropy encoding network, an entropy decoding network, and a decoding network are optimized as a whole. The encoding network and the decoding network, sometimes called the transform module and the inverse transform module, generally include a convolutional layer and a nonlinear transformation unit.

[0108] 1A is a schematic block diagram illustrating an example coding system 10, e.g., video coding system 10 (or coding system 10 for short), that may employ the techniques of the present application. A video encoder 20 (or encoder 20 for short) and a video decoder 30 (or decoder 30 for short) of video coding system 10 represent example devices that may be configured to perform the techniques according to various examples described herein.

[0109] As shown in FIG. 1A, coding system 10 includes a source device 12 configured to, for example, provide encoded image data 21 to a destination device 14 for decoding the encoded image data 21.

[0110] The source device 12 includes an encoder 20 and may further include, i.e., optionally, an image source 16, a preprocessor (or preprocessing unit) 18, e.g., an image preprocessor 18, and a communication interface (or communication unit) 22.

[0111] Image source 16 may include or be any kind of image capture device, e.g., a camera for capturing real-world images, and / or any type of image generation device, e.g., a computer graphics processing unit for generating computer-animated images, or any type of other device for obtaining and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). Image source may also be any type of memory or storage device that stores any of the foregoing images.

[0112] To distinguish the processing performed by the preprocessor (or preprocessing unit) 18, the image (or image data) 17 may also be referred to as raw (or raw image data) 17.

[0113] The pre-processor 18 is configured to receive (raw) image data 17 and perform pre-processing on the image data 17 to obtain a pre-processed image (or pre-processed image data) 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component.

[0114] A video encoder (or encoder) 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 (further details are described below, for example, with reference to FIG. 2).

[0115] The communications interface 22 of the source device 12 may be configured to receive the encoded image data 21 and transmit the encoded image data 21 (or any further processed version thereof) over the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0116] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may further include, optionally, a communications interface (or communications unit) 28, a post-processor 32 (or post-processing unit), and a display device 34.

[0117] The communications interface 28 of the destination device 14 is configured to receive the encoded image data 21 (or any further processed version thereof), for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0118] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded image data (or encoded data) 21 via a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0119] The communications interface 22 may be configured, for example, to package the encoded image data 21 into an appropriate format, e.g., packets, and / or to process the encoded image data using any type of transmission encoding or processing for transmission over a communications link or network.

[0120] Communications interface 28, which forms a counterpart of communications interface 22, may be configured, for example, to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or depackaging to obtain encoded image data 21.

[0121] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in FIG. 1A, or as bidirectional communication interfaces, e.g., configured to send and receive messages, e.g., to set up connections, receive, acknowledge, and exchange communication links and / or any other information related to data transmission, e.g., encoded image data transmission.

[0122] Video decoder (or decoder) 30 is configured to receive encoded image data 21 and provide decoded image data (or decoded image) 31 (further details are described below, e.g., based on FIG. 3).

[0123] The post-processor 32 is configured to post-process the decoded image data 31 (also called reconstructed image data), e.g., a decoded image, to obtain post-processed image data 33, e.g., a post-processed image. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing such as to prepare the decoded image data 31 for display, such as by a display device 34.

[0124] The display device 34 is configured to receive the post-processed image data 33, for example, to display the image to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display may include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0125] The coding system 10 further includes a training engine 25. The training engine 25 is configured to train the encoder 20 or the decoder 30 to implement a transformation between the image domain and the feature domain.

[0126] In the present embodiment, the training data may be stored in a database (not shown), and the training engine 25 performs training based on the training data to obtain an encoding / decoding network. Note that the source of the training data is not limited in the present embodiment. For example, the training data may be obtained from the cloud or other locations to perform model training.

[0127] 1A illustrates source device 12 and destination device 14 as separate devices, a device embodiment may include both source device 12 and destination device 14, or the functionality of both source device 12 and destination device 14, i.e., both source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, by separate hardware and / or software, or by any combination thereof.

[0128] As will be apparent to those skilled in the art based on this description, the presence and (exact) division of functions of different units as shown in FIG. 1A, or within source device 12 and / or destination device 14, may vary depending on the actual device and application.

[0129] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented via processing circuitry such as that shown in FIG. 1B, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video encoding, or any combination thereof. Encoder 20 may be implemented via processing circuitry 46 to implement various modules described with respect to encoder 20 of FIG. 2 and / or any other encoder system or subsystem described herein. Decoder 30 may be implemented via processing circuitry 46 to implement various modules described with respect to decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. Processing circuitry 46 may be configured to perform various operations, as described below. 8, when the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of the present invention. Both video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in FIG. 1B.

[0130] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0131] In some cases, the video coding system 10 shown in FIG. 1A is merely an example, and the techniques herein may be applied to video encoding scenarios (e.g., video encoding or video decoding) that do not necessarily involve data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory, transmitted over a network, etc. A video encoding device may encode data and store the encoded data in memory, and / or a video decoding device may retrieve data from memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve data from memory and decode data.

[0132] 1B is a schematic diagram of a video coding system 40. Video coding system 40 may include an imaging device 41, a video encoder 20, a video decoder 30 (and / or a video encoder / decoder implemented by processing circuitry 46), an antenna 42, one or more processors 43, one or more memories 44, and / or a display device 45.

[0133] 1B, imaging device 41, antenna 42, processing circuitry 46, video encoder 20, video decoder 30, processor 43, memory 44, and / or display device 45 can be in communication with one another. Video coding system 40 may include only video encoder 20 or only video decoder 30, in different examples.

[0134] In some examples, antenna 42 may be configured to transmit or receive an encoded bitstream of video data. Additionally, in some examples, display device 45 may be configured to present the video data. Processing circuitry 46 may include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, etc. Video coding system 40 may also include an optional processor 43. Optional processor 43 may also include application-specific integrated circuit (ASIC) logic, a graphics processing unit, a general-purpose processor, etc. Additionally, memory 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM) or dynamic random access memory (DRAM)) or non-volatile memory (e.g., flash memory). In a non-limiting example, memory 44 may be implemented by a cache memory. In other examples, processing circuitry 46 may include memory (e.g., a cache) for implementing an image buffer.

[0135] In some examples, video encoder 20 implemented using logic circuitry may include an image buffer (e.g., implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 20 implemented using processing circuitry 46 to implement various modules described with reference to FIG. 2 and / or any other encoder system or subsystem described herein. The logic circuitry may be configured to perform various operations described herein.

[0136] In some examples, video decoder 30 may be implemented using processing circuitry 46 in a similar manner to implement various modules described with reference to decoder 30 in FIG. 3 and / or any other decoder system or subsystem described herein. In some examples, video decoder 30 implemented using logic circuitry may include an image buffer (implemented by processing circuitry 46 or memory 44) and a graphics processing unit (e.g., implemented by processing circuitry 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 implemented using processing circuitry 46 to implement various modules described with reference to FIG. 3 and / or any other decoder system or subsystem described herein.

[0137] In some examples, antenna 42 may be configured to receive an encoded bitstream of video data. As described, the encoded bitstream may include data related to the video frame encoding described herein, indicators, index values, mode selection data, etc., such as data related to coding partitions (e.g., transform coefficients or quantized transform coefficients, optional indicators (as described), and / or data defining the coding partitions). Video coding system 40 may further include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0138] In embodiments herein, for examples described with reference to video encoder 20, it should be understood that video decoder 30 may be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 may be configured to receive and parse such syntax elements and correspondingly decode the associated video data. In some examples, video encoder 20 may entropy encode syntax elements into an encoded video bitstream. In such examples, video decoder 30 may parse such syntax elements and decode the associated video data accordingly.

[0139] For ease of explanation, embodiments of the present invention are described herein with reference to reference software for, for example, High-Efficiency Video Coding (HEVC) or Versatile Video Coding (VVC), the next-generation video coding standard developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Joint Collaboration Team on Video Coding (JCT-VC) of the Motion Picture Experts Group (MPEG). Those skilled in the art will understand that embodiments of the present invention are not limited to HEVC or VVC.

[0140] 2 is a schematic diagram of a video coding device 400 according to an embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video coding device 400 may be a decoder, such as the video decoder 30 of FIG. 1A, or an encoder, such as the video encoder 20 of FIG. 1A.

[0141] Video coding device 400 includes an ingress port 410 (or input port 410) and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit (CPU) 430 for processing the data (e.g., processor 430 may be a neural network processing unit 430), a transmitter unit (Tx) 440 and an egress port 450 (or output port 450) for transmitting the data, and a memory 460 for storing the data. Video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to ingress port 410, receiver unit 420, transmitter unit 440, and egress port 450 for the emission or ingress of optical or electrical signals.

[0142] The processor 430 is implemented in hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, or DSPs. The processor 430 communicates with the ingress port 410, the receiving unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470 (e.g., a neural network (NN)-based coding module 470). The coding module 470 implements the disclosed embodiments. For example, the coding module 470 performs, processes, prepares, or provides various encoding operations. Thus, the inclusion of the encoding / decoding module 470 provides substantial improvements to the functionality of the video coding device 400 and affects the switching of the video coding device 400 to different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0143] Memory 460 may include one or more disks, tape drives, and solid-state drives, may be used as an overflow data storage device, may store programs when such programs are selected for execution, and may store instructions and data read during program execution. Memory 460 may be volatile and / or nonvolatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0144] FIG. 3 is a simplified block diagram of an apparatus 500 that may be used as either or both of source device 12 and destination device 14 in FIG. 1A, according to one embodiment.

[0145] Processor 502 in device 500 may be a central processing unit. Alternatively, processor 502 may be any other type of device or devices now existing or later developed that are capable of manipulating or processing information. While the disclosed implementations may be implemented with a single processor as shown, such as processor 502, advantages in speed and efficiency may be achieved using two or more processors.

[0146] The memory 504 in the apparatus 500 may, in some implementations, be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable class of storage device can be used as the memory 504. The memory 504 may include code and data 506 that is accessed by the processor 502 using a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to perform the methods described herein. For example, the application programs 510 may include applications 1-N, which further include a video coding application that performs the methods described herein.

[0147] The apparatus 500 may also include one or more output devices, such as a display 518. The display 518 may, in one example, be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512.

[0148] Although shown here as a single bus, bus 512 of device 500 may be comprised of multiple buses. Additionally, secondary storage may be directly coupled to other components of device 500 or may be accessed over a network, and may include a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, device 500 may be implemented in a wide variety of configurations.

[0149] A schematic diagram of an end-to-end deep learning image encoding and decoding framework is shown in Figure 4. As shown in Figure 4, the image encoding and decoding framework includes an encoder side, which includes an encoding network (encoder), a quantization module, and an entropy encoding network, and a decoder side, which includes an entropy decoding network, a decoding network (decoder), and an entropy estimation network.

[0150] On the encoder side, the original image is processed by the encoding network and then transformed from the image domain to the feature domain. The transformed image features are processed by the quantization module and the entropy encoding network, and then encoded into a bitstream to be transmitted or stored. On the decoder side, the bitstream is processed by the entropy decoding network and then decoded into image features. The image features are processed by the decoding network and then transformed from the feature domain back to the image domain to obtain a reconstructed image. The entropy estimation network estimates and obtains an estimated probability value for each feature element based on the image features, and the probability value is used for the processing of the entropy encoding network and the entropy decoding network.

[0151] In this embodiment, both the encoding network (encoder) and the decoding network (decoder) have nonlinear transformation units.

[0152] Figure 5 is a schematic diagram of an end-to-end deep learning video encoding and decoding framework. As shown in Figure 5, the video encoding and decoding framework includes a prediction module and a residual compress module.

[0153] The prediction module predicts the current frame by using the reconstructed image of the previous frame to obtain a predicted image. On the other hand, the residual compression module compresses the residual between the original image of the current frame and the predicted image. On the other hand, the residual compression module decompresses the residual to obtain a reconstructed residual, and sums the reconstructed residual with the predicted image to obtain a reconstructed image of the current frame. Both the encoding sub-network and the decoding sub-network in the prediction module and the residual compression module have nonlinear transformation units.

[0154] In this embodiment, both the predict model and the residual compress module have nonlinear transform units.

[0155] 6 is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 6, the application scenario may be a service related to image / video capture, storage, or transmission in a terminal, a cloud server, or video surveillance, such as photo taking / video recording by a terminal, an album, a cloud album, or video surveillance.

[0156] Encoder side: A camera captures an image / video. An artificial intelligence (AI) image / video encoding network extracts low-redundancy image features from the image / video, and then performs compression based on the image features to obtain a bitstream / image file.

[0157] Decoder side: When the image / video needs to be output, the AI ​​image / video decoding network decompresses the bitstream / image file to get the image features, and then performs inverse feature extraction on the image features to get the reconstructed image / video.

[0158] The storage / transmission module stores (e.g., terminal capture, video surveillance, or cloud server) or transmits (e.g., cloud service or live broadcast technology) the bitstream / image file obtained by compression for various services.

[0159] 7 is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 7, the application scenario may be a service related to image / video capture, storage or transmission in a terminal or video surveillance, such as a terminal album, video surveillance or live broadcast.

[0160] Encoder side: The encoding network converts the image / video into image features with lower redundancy. The encoding network usually includes a nonlinear transformation unit and has nonlinear features. The entropy estimation network is responsible for calculating the encoding probability of each data in the image features. The entropy encoding network performs lossless encoding on the image features based on the probability corresponding to each data to obtain a bitstream / image file, which further reduces the amount of data transmission in the image compression process.

[0161] Decoder side: The entropy decoding network performs lossless decoding on the bitstream / image file based on the probability corresponding to each data to obtain reconstructed image features. The decoding network performs inverse transformation on the image features output by the entropy decoding network and parses the image features into an image / video. Corresponding to the encoding network, the decoding network usually includes a nonlinear transformation unit and has nonlinear characteristics. The storage module saves the bitstream / image file at the corresponding storage location of the terminal. The load module loads the bitstream / image file from the corresponding storage location of the terminal and inputs the bitstream / image file into the entropy decoding network.

[0162] 8 is a schematic diagram of an application scenario according to an embodiment of the present application. As shown in FIG. 8, the application scenario may be a service related to image / video capture, storage, or transmission in the cloud or video surveillance, such as cloud album, video surveillance, or live broadcast.

[0163] Encoder side: The local end acquires images / videos, encodes the images / videos (JPEG) to obtain compressed images / videos, and then sends the compressed images / videos to the cloud. The cloud performs JPEG decoding on the compressed images / videos to obtain images / videos, then compresses the images / videos to obtain bitstreams / image files, and stores the bitstreams / image files.

[0164] Decoder side: When an image / video needs to be obtained locally from the cloud, the cloud decompresses the bitstream / image file to obtain the image / video, performs JPEG encoding on the image / video to obtain a compressed image / video, and sends the compressed image / video to the local end. The local end performs JPEG decoding on the compressed image / video to obtain the image / video. For the structure of the cloud and the use of each module, please refer to the structure and use of each module in Figure 7. Details will not be described here in the embodiments of this application.

[0165] Based on the aforementioned encoding / decoding network and application scenario, an embodiment of the present application provides an image encoding / decoding method for implementing efficient nonlinear transform processing and improving the rate-distortion performance of image / video compression algorithms.

[0166] 9 is a flowchart of a process 900 of an image encoding method according to an embodiment of the present application. The process 900 may be performed by the encoder side in the aforementioned embodiment. The process 900 is described as a series of steps or operations. It should be understood that the steps or operations of the process 900 may be performed in various orders and / or simultaneously, and are not limited to the order of execution shown in FIG. 9. The process 900 includes the following steps:

[0167] Step 901: Obtain a first image feature to be processed.

[0168] The first image feature is obtained after the encoder side obtains the image to be processed and transforms the image to be processed from the image domain to the feature domain. The transformation here may include: 1. Convolution processing in which a convolution layer is used to extract features, where the convolution processing has a local receptive field and a weight sharing mechanism (i.e., each filter processes the input features sequentially); 2. Extracting features by using an MLP or a fully connected layer, where the feature extraction by using an MLP or a fully connected layer has global receptive field features and no weight sharing; 3. Transformer processing including matrix multiplication, MLP, and normalization processing, where the transformer processing has global receptive field features and has strong long-range capture dependency ability.

[0169] The first image features may be represented in the form of a two-dimensional matrix (L×C, where L represents the length and C represents the channel) or a three-dimensional matrix (C×H×W, where C represents the number of channels, H represents the height, and W represents the width). The specific form is associated with the aforementioned transformation manner. For example, first image features extracted through a convolution process or by using an MLP generally correspond to a three-dimensional matrix, while first image features obtained through processing by using a transformer generally correspond to a two-dimensional matrix.

[0170] For example, the first image feature is represented as a two-dimensional matrix.

number

[0171] The two-dimensional matrix A is a 3x3 matrix and contains nine elements. Each element a(i,j) corresponds to one feature value in the first image feature. i represents the length corresponding to element a(i,j), and j represents the channel in which element a(i,j) is located.

[0172] In another example, the first image feature is represented as a three-dimensional matrix.

number

[0173] The three-dimensional matrix B is a 3x3x2 matrix and contains 18 elements. Each element a(i,j,l) corresponds to a feature value of the first image feature. i represents the row in which the element a(i,j,l) is located, j represents the column in which the element a(i,j,l) is located, and l represents the channel in which the element a(i,j,l) is located.

[0174] It should be noted that the manner of acquiring the first image feature is not particularly limited in the embodiments of the present application.

[0175] The image to be processed may be a picture, or a frame of an image in a video, or an image block obtained by segmenting said picture or image, but is not particularly limited thereto.

[0176] Step 902: Perform a nonlinear transformation process on the first image feature to obtain a processed image feature.

[0177] In one possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature, a convolution operation is performed on the second image feature to obtain a third image feature, where multiple feature values ​​in the third image feature correspond to multiple feature values ​​in the first image feature, and an element-wise multiplication operation is performed on multiple corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0178] Fig. 10a is a schematic diagram of the structure of a nonlinear transformation unit with attention mechanism. As shown in Fig. 10a, in the embodiments of the present application, the nonlinear transformation unit is used to implement the above-mentioned nonlinear transformation process, and includes a first nonlinear operation, a convolution process, and an element-wise multiplication operation.

[0179] The first nonlinear operation is an operation performed on each feature value in the first image feature, and may include an absolute value operation, a ReLU series, a sigmoid, a Tanh, a PWL operation, or the like.

[0180] The absolute value operation is an operation to obtain the absolute value of the input feature value, which can be expressed by using the following formula:

number

[0181] Piecewise linear mappings include rectified linear units (ReLU), leaky rectified linear units (LeakyReLU), or PWL operations. ReLU is a piecewise linear mapping method. For input feature values, feature values ​​less than 0 are output as 0, and feature values ​​greater than or equal to 0 remain unchanged. This can be expressed by using the following formula:

number

[0182] LeakyReLU is a piecewise linear mapping method. Based on ReLU, input feature values ​​less than 0 are scaled by using a preset weight, which is usually 0.01. This can be expressed by using the following formula:

number

[0183] a is a preset value, typically set to 0.01.

[0184] The sigmoid can be expressed as the following operation:

number

[0185] Tanh can be expressed as the following operation:

number

[0186] PWL can be expressed as the following operation:

number

[0187] N represents the number of segments; B L represents the left boundary; B R represents the right boundary; Y P represents the y-axis coordinate values ​​corresponding to the N+1 boundary points; K L represents the slope of the left end; K R represents the slope of the right end; idx represents the index number of the segment to which x belongs; B idx and K. idx are the corresponding left boundary and slope of the segment; d = (B R -B L ) / N represents the segment length. These values ​​are calculated as follows:

number

[0188] Note that equation (1) can be used at different granularities (per layer, or even per channel). When using the equation based on channels, B L , B R , Y P , K. L , K. R is a per-channel parameter. Optionally, the same hyperparameter N may be used in all PWL operations.

[0189] PWL has the following advantages: As a general approximation, PWL can approximate any continuous bounded scalar function. PWL varies continuously with parameters (except for the hyperparameter N), which is very useful for gradient-based optimization. Flexibility is concentrated in the boundary region, resulting in maximum use of learnable parameters. PWL computational efficiency, especially in inference, is high due to the uniform segmentation.

[0190] Figure 11 is a schematic diagram of the PWL function. As shown in Figure 11, the number of segments N is a hyperparameter that affects the fitting ability of the PWL function. A larger number of segments indicates a higher degree of freedom and a larger model capacity. The left and right boundaries B L and B R defines the main effective area where the PWL is focused. [B L ,B R ] is divided equally into N segments to obtain N+1 boundary points. Each boundary point has a corresponding y-coordinate value Y P These coordinates determine the shape of the PWL curve. L ,B R For the area outside of ], the two slopes K L and K. R By using the , the shape of the out-of-boundary area can be controlled.

[0191] It should be noted that the piecewise linear mapping may alternatively use another modified form of ReLU or another new first nonlinear operation form, which is not particularly limited in the embodiments of the present application.

[0192] After the first nonlinear operation, the first image feature is transformed into a second image feature. The second image feature can be represented in the form of a matrix, just like the first image feature. Because the first nonlinear operation is performed on each feature value in the first image feature, each feature value in the first image feature corresponds to one feature value in the second image feature. Therefore, the matrix corresponding to the second image feature and the matrix corresponding to the first image feature have the same size, and feature values ​​(matrix element values) at the same positions correspond to each other. For example, if the first image feature is represented as a 3x3 matrix as matrix A, the second image feature can also be represented as a 3x3 matrix. However, because the first nonlinear operation is performed on the first image feature and the second image feature, the feature values ​​in the first image feature and the second image feature are not completely the same. Correspondingly, the element values ​​in the matrices corresponding to the first image feature and the second image feature, respectively, are not completely the same.

[0193] A convolution process is performed on the second image feature to output a third image feature. The third image feature may be considered as a local response (i.e., refined value) of the second image feature. That is, the third image feature is a response signal obtained by performing a convolution process on the second image feature. Because the receptive field of the convolution process is limited, the response value of each position in the output image feature after the convolution process is related only to the input feature values ​​of positions adjacent to that position. This is called a local response. The convolution process can be expressed as follows: conv1(x)=β+Σγ×x

[0194] γ represents the weights of the convolutional layer, and β represents the offset parameter of the convolutional layer.

[0195] After the convolution process is performed, the size of the matrix corresponding to the third image feature is also the same as the size of the matrix corresponding to the first image feature. Figure 12 is a schematic diagram of the convolution process. As shown in Figure 12, a 1*2 matrix is ​​input, and a 1*4 matrix is ​​output after processing by the convolution layer. The convolution layer includes two filters, one of which is a 2*50 matrix W1 and the other is a 50*4 matrix W2. First, a convolution operation is performed on the input matrix by using matrix W1 to obtain a 1*50 matrix, and then a convolution operation is performed on the matrix by using matrix W2 to obtain a 1*4 output matrix.

[0196] Performing an element-wise multiplication operation on a plurality of corresponding feature values ​​of the first image feature and the third image feature is equivalent to performing an element-wise multiplication on the initial input of the nonlinear transformation unit and the output of the convolution process, which can be expressed by the following equation: c(i,j)=a(i,j)×b(i,j)

[0197] (i,j) represents the index of the feature value in the image feature where the feature value is located, a(i,j) represents the feature value in the first image feature, b(i,j) represents the feature value in the second image feature, and c(i,j) represents the feature value in the processed image feature.

[0198] It can be seen that the matrix corresponding to the processed image feature and the matrix corresponding to the first image feature also have the same size.

[0199] It should be noted that in the embodiments of the present application, the correspondence of multiple feature values ​​in two graphic features to each other may mean that after the two image features are respectively represented as matrices, the values ​​of elements at the same position in the two matrices have an arithmetic relationship, and the two image features correspond to each other. For example, when both the first image feature and the third image feature are represented in the form of the aforementioned matrix A, and the values ​​at the same position a 0,2 The elements in correspond to each other.

[0200] Furthermore, after the convolution process, a third image feature having a specific size different from that of the first image may be obtained. This depends on the internal structure of the convolution process, particularly the length, width, and number of channels of the filter used in the convolution process. If the third image feature and the first image feature have different sizes, the elements in the matrix corresponding to the third image feature do not have a one-to-one correspondence with the elements in the matrix corresponding to the first image feature. In this case, multiple elements in the matrix corresponding to the first image feature may be considered to be multiplied by the same element in the matrix corresponding to the third image feature. For example, if the number of channels in the matrix corresponding to the first image feature is three and the number of channels in the matrix corresponding to the third image feature is one, the elements in the matrix corresponding to the third image feature may be multiplied separately by elements in the same position in each channel of the matrix corresponding to the first image feature. This is not particularly limited in the embodiments of the present application.

[0201] The first nonlinear operation, the convolution process, and the element-wise multiplication operation may be represented by the following equation (1):

number

[0202] x represents the input feature value. y represents the output feature value. φ(x j ) represents the first nonlinear operation. γ represents the weights of the convolutional layer. β represents the bias parameter of the convolutional layer.

[0203] The above formula uses multiplication operations. Related techniques (GDN division normalization)

number

[0204] The local attention mechanism is implemented after the aforementioned first nonlinear operation, convolution process, and element-wise multiplication operation. "Local" means that the first nonlinear operation is performed element-wise, and for each input feature value, an output value is obtained based only on the characteristics of the input feature value, without considering the influence of surrounding feature values. The attention mechanism means that some of all feature values ​​in the first image feature are important and some are redundant. The output of the convolution process may be a weight for each feature value in the image feature. The original feature values ​​may be refined, with important feature values ​​being highlighted and redundant feature values ​​being suppressed. In the element-wise multiplication operation, the value of each feature value in the first image feature is refined by using the aforementioned local information, thereby avoiding restrictions on the convolution parameters.

[0205] In one possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; a convolution operation is performed on the second image feature to obtain a third image feature, where the feature values ​​in the third image feature correspond to the feature values ​​in the first image feature; an element-wise multiplication operation is performed on corresponding feature values ​​in the first and third image features to obtain a fourth image feature; the feature values ​​in the fourth image feature correspond to the feature values ​​in the first image feature; and an element-wise addition operation is performed on corresponding feature values ​​in the first and fourth image features to obtain a processed image feature.

[0206] Fig. 10b is a schematic diagram of the structure of a residual nonlinear transform unit with attention mechanism. As shown in Fig. 10b, in the embodiments of the present application, the nonlinear transform unit is used to perform the above-mentioned nonlinear transform process, and includes a first nonlinear operation, a convolution process, an element-wise multiplication operation, and an element-wise addition operation.

[0207] In the embodiments of the present application, the first nonlinear operation, the convolution process, and the element-wise multiplication operation are described in the above implementation, and the details will not be described again here.

[0208] After the fourth image feature is obtained, an element-wise addition operation is performed on a plurality of corresponding feature values ​​in the first image feature and the fourth image feature. In other words, the element-wise addition is performed on the initial input of the nonlinear transformation unit and the output of the element-wise multiplication operation. This can be expressed by the following equation: sum(i,j)=a(i,j)+c(i,j)

[0209] (i,j) represents the index of the feature value in the image feature where the feature value is located; a(i,j) represents the feature value in the first image feature; c(i,j) represents the feature value in the fourth image feature; sum(i,j) represents the feature value in the processed image feature.

[0210] The element-wise addition operation is the residual structure to which the head and tail are added, allowing encoding and decoding networks using the aforementioned processing process to easily converge during training.

[0211] The first nonlinear operation, the convolution process, the element-wise multiplication operation, and the element-wise addition operation can be expressed by the following equation (2):

number

[0212] x represents the input feature value. y represents the output feature value. φ(x j ) represents the first nonlinear operation. γ represents the weights of the convolutional layer. β represents the bias parameter of the convolutional layer.

[0213] Transforming the above equation (2) gives:

number

[0214] It can be seen that the convolution process conv1(x) is similar to the convolution process conv2(x). The difference is that an additional 1 is added to the convolution parameter in the convolution process conv2(x). In this way, the two implementations described above can be modified by fine-tuning the convolution parameter β in the convolution process. Specifically, if the nonlinear transformation unit does not include an element-wise addition operation, the convolution process conv1(x) can be used. On the other hand, if the nonlinear transformation unit includes an element-wise addition operation, the convolution process conv2(x) can be used.

[0215] In one possible implementation, based on the first implementation described above, a second nonlinear operation may be further performed on the third image feature, and then an element-wise multiplication operation is performed on the output-processed third image feature and the first image feature. Alternatively, based on the second implementation described above, a second nonlinear operation may be performed on the third image feature, and then an element-wise multiplication operation is performed on the output-processed third image feature and the first image feature. That is, a second nonlinear operation is added to the nonlinear transformation process. The input of the second nonlinear operation is the output of the convolution process in the nonlinear transformation process. The output of the second nonlinear operation is used as the input of the element-wise multiplication operation.

[0216] Fig. 10c is a schematic diagram of the structure of a residual nonlinear transform unit with attention mechanism. As shown in Fig. 10c, in the embodiments of the present application, the nonlinear transform unit is used to perform the above-mentioned nonlinear transform process, and includes a first nonlinear operation, a convolution process, a second nonlinear operation, and an element-wise multiplication operation.

[0217] For the first nonlinear operation, the convolution process, and the element-by-element multiplication operation, please refer to the description of the embodiment shown in FIG. 10a. The details will not be described again here. The second nonlinear operation and the first nonlinear operation may use the same operation method or different operation methods. The second nonlinear operation may include absolute value operation, ReLU, LeakyReLU, etc. This is not particularly limited in the embodiment of the present application.

[0218] The first nonlinear operation, the convolution process, the second nonlinear operation, and the element-wise multiplication operation can be expressed by the following equation (3).

number

[0219] x represents the input feature value. y represents the output feature value. φ1(x j ) represents the first nonlinear operation. j ) represents the second nonlinear operation. γ represents the weights of the convolutional layer. β represents the bias parameter of the convolutional layer.

[0220] Fig. 10d is a schematic diagram of the structure of a nonlinear transformation unit with attention mechanism. As shown in Fig. 10d, in the embodiments of the present application, the nonlinear transformation unit is used to perform the aforementioned nonlinear transformation process, and includes a first nonlinear operation, a convolution process, a second nonlinear operation, an element-wise multiplication operation, and an element-wise addition operation.

[0221] For the first nonlinear operation, convolution processing, element-by-element multiplication operation, and element-by-element addition operation, please refer to the description of the embodiment shown in FIG. 10b. Details will not be described again here. The second nonlinear operation and the first nonlinear operation may use the same operation method or different operation methods. The second nonlinear operation may include absolute value operation, ReLU, LeakyReLU, etc. This is not particularly limited in the embodiment of the present application.

[0222] When the second nonlinear operation is implemented using a piecewise linear mapping, the piecewise linear mapping may use a different number of segments. The mapping gradient on each segment may be determined through training or specified directly. A different piecewise linear function may be used for each channel of the input feature image, or the same piecewise linear function may be used for all channels, or the same piecewise linear function may be used to process several channels. In this implementation, after model training is complete, the residual structure is no longer merged with the convolution but may be merged with the piecewise linear function. Specifically, the output of the original piecewise linear function is incremented by 1 to form a new piecewise linear function.

[0223] The first nonlinear operation, the convolution process, the second nonlinear operation, the element-wise multiplication operation, and the element-wise addition operation may be represented by the following equation (4):

number

[0224] x represents the input feature value. y represents the output feature value. φ1(x j ) represents the first nonlinear operation. j ) represents the second nonlinear operation. γ represents the weights of the convolutional layer. β represents the bias parameter of the convolutional layer.

[0225] Step 903: Perform encoding based on the processed image features to obtain a bitstream.

[0226] After the processed image features are obtained, the encoder side may proceed to perform a convolution process on the processed image features, or after the convolution process, may again perform a nonlinear transformation process on the output of the convolution process, and then perform entropy encoding on the result of the above process to obtain a bitstream. The entropy encoding may be implemented by using an entropy encoding network in the embodiments shown in Figures 4 to 8. The details will not be described again here. Alternatively, the result of the above process may be encoded using another encoding method to obtain a bitstream. This is not a limitation in the present application.

[0227] FIG. 13 is a schematic diagram of the structure of an encoding network. As shown in FIG. 13, the encoding network includes four convolution layers (conv) and three nonlinear transformation units. The convolution layers and the nonlinear transformation units are arranged in a cross-sectional manner. Specifically, a convolution process, a nonlinear transformation process, a convolution process, a nonlinear transformation process, a convolution process, a nonlinear transformation process, and a convolution process are sequentially performed on an input image to obtain output image features. The nonlinear transformation units may use the structure of the embodiment shown in FIG. 10a or 10b. Then, entropy encoding is performed on the output image features. Details will not be described again here.

[0228] In the embodiments of the present application, the nonlinear transformation process in the decoding network is modified so that for each input feature value, an output value is obtained based only on the characteristics of the input feature value, without considering the influence of surrounding feature values. The original feature values ​​are also refined, important feature values ​​are highlighted, and redundant feature values ​​are suppressed. In addition, the value of each feature value in the first image feature may be further refined, thereby avoiding the restriction on the convolution parameters, implementing an efficient nonlinear transformation process in the encoding network, and further improving the rate-distortion performance of the image / video compression algorithm.

[0229] 14 is a flowchart of a process 1300 of an image decoding method according to an embodiment of the present application. The process 1300 may be performed by a decoder side in the aforementioned embodiment. The process 1300 is described as a series of steps or operations. It should be understood that the steps or operations of the process 1300 are not limited to the execution sequence shown in FIG. 14 and may be performed in various sequences and / or simultaneously. The process 1300 includes the following steps:

[0230] Step 1301: Obtain a first image feature to be processed.

[0231] The decoder side corresponds to the encoder side. The first image feature may be obtained after the decoder side performs entropy decoding on the bitstream. The entropy decoding may be implemented by using an entropy decoding network in the embodiments shown in FIGS. 4 to 8. Details will not be described again here. Then, convolution processing, deconvolution processing, transposed convolution processing, interpolation + convolution processing, transform processing, etc. are performed. It should be understood that after the above processing, the size of the output first image feature is restored (which is mirror-symmetric with respect to the encoder side), the size of the input image feature may change, the number of channels may change, etc. This is not particularly limited. The above processing is the reverse of the conversion aspect in step 901 in the embodiment shown in FIG. 9.

[0232] Similarly, the first image feature may be expressed in the form of a two-dimensional matrix or a three-dimensional matrix. For the principle, please refer to the description of step 901. The details will not be described again here.

[0233] Step 1302: Perform a nonlinear transformation process on the first image feature to obtain a processed image feature.

[0234] In one possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature, a convolution operation is performed on the second image feature to obtain a third image feature, where multiple feature values ​​in the third image feature correspond to multiple feature values ​​in the first image feature, and an element-wise multiplication operation is performed on multiple corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0235] For this embodiment, please refer to the embodiment shown in Figure 10a, and the details will not be described again here.

[0236] In one possible implementation, a first nonlinear operation is performed on each feature value in the first image feature to obtain a second image feature; a convolution operation is performed on the second image feature to obtain a third image feature, where multiple feature values ​​in the third image feature correspond to multiple feature values ​​in the first image feature; an element-wise multiplication operation is performed on multiple corresponding feature values ​​in the first and third image features to obtain a fourth image feature, where multiple feature values ​​in the fourth image feature correspond to multiple feature values ​​in the first image feature; and an element-wise addition operation is performed on multiple corresponding feature values ​​in the first and fourth image features to obtain a processed image feature.

[0237] For this embodiment, please refer to the embodiment shown in Figure 10b, and the details will not be described again here.

[0238] Step 1303: Obtain a reconstructed image based on the processed image features.

[0239] After the processed image features are obtained, the decoder side may proceed to perform a convolution process on the processed image features, or after the convolution process, may perform the nonlinear transformation process again on the output of the convolution process to transform the processed image features from the feature domain to the image domain and obtain a reconstructed image.

[0240] FIG. 15 is a schematic diagram of the structure of a decoding network. As shown in FIG. 15, the decoding network includes four deconvolution layers (Deconv) and three nonlinear transformation units. The deconvolution layers and the nonlinear transformation units are arranged in a cross-sectional manner. Specifically, the input image is sequentially subjected to a deconvolution process, a nonlinear transformation process, a deconvolution process, a nonlinear transformation process, a deconvolution process, a nonlinear transformation process, and a deconvolution process to obtain a reconstructed image. The nonlinear transformation units may use the structure of the embodiment shown in FIG. 10a or 10b. Then, entropy decoding is performed on the output image features. Details will not be described again here.

[0241] In the embodiments of the present application, the nonlinear transformation process in the decoding network is modified so that for each input feature value, an output value is obtained based only on the characteristics of the input feature value, without considering the influence of surrounding feature values. The original feature values ​​are also refined to highlight important feature values ​​and suppress redundant feature values. In addition, the value of each feature value in the first image feature may be further refined, thereby avoiding the restriction on the convolution parameters, implementing an efficient nonlinear transformation process in the encoding network, and further improving the rate-distortion performance of the image / video compression algorithm.

[0242] Note that embodiments of the present application further provide a training aspect of the encoding / decoding network, which may include first constructing an end-to-end encoding / decoding network including an encoder, a decoder, and an entropy estimation unit. During training, the encoding and decoding network is considered as a whole, and training data (images or videos) is input to the encoder to obtain feature data. On the one hand, the feature data is used to calculate the encoding bitrate overhead by using the entropy estimation unit to obtain the bitrate loss. On the other hand, the feature data is used to output reconstructed data by using the decoder side, and the distortion measure is calculated for the reconstructed data and the input data to obtain the distortion loss. A backpropagation algorithm updates the learnable parameters in the model based on a weighted loss formed by the bitrate loss and the distortion loss. After training is completed, the parameters of all submodules in the model are fixed.

[0243] Then, an encoder and an entropy estimation unit are split on the encoder side and configured to encode the data to be encoded into a bitstream file, and a decoder and an entropy estimation unit are split on the decoder side and configured to reconstruct data from the bitstream file.

[0244] The following describes the aforementioned image encoding method by using some specific embodiments: The nonlinear transformation unit provided in the embodiments of the present application is hereinafter referred to as ResAU. [Example]

[0245] Embodiment 1 FIG. 16A is an exemplary schematic diagram of the structure of the ResAU. As shown in FIG. 16A, the ResAU uses the structure shown in FIG. 10B and includes a first nonlinear operation, a convolution process, an element-wise multiplication operation, and an element-wise addition operation. An absolute value operation (abs) is used for the first nonlinear operation. The ResAU can be used in the encoding network shown in FIG. 13 or the decoding network shown in FIG. 15.

[0246] Based on equation (2), ResAU in this embodiment can be expressed by the following equation: y i =x i *(β i +Σ j gamma ij |x j |)+x i

[0247] Compression performance tests are conducted on the ResAU shown in Fig. 16a.

[0248] Test Set: The Kodak test set contains 24 portable network graphics (PNG) images with a resolution of 768x512 or 512x768.

[0249] Experiments: ResAU, shown in Figure 16a, is used in an encoding / decoding network structure with a hybrid Gaussian super-entropy prior estimation.

[0250] Experimental Performance: Figure 17a shows the overall performance of ResAU for 24 images in the Kodak test set. Compared with the currently mainstream GDN nonlinear unit, the RD performance of the GMM network using ResAU is better at the same computational cost. More specifically, when the same decoding reconstruction quality is achieved, using ResAU in a GMM encoding / decoding network can save approximately 12% of the encoding bitrate overhead compared to using ReLU, and approximately 5% of the encoding bitrate overhead compared to using GDN.

[0251] Performance Analysis: In the ResAU provided in the embodiment of this application, introducing nonlinear features by using absolute value operation is a feasible solution. Based on the basic GMM structure, ResAU can achieve better rate-distortion performance than GDN. [Example]

[0252] Embodiment 2 FIG. 16b is an exemplary schematic diagram of the structure of the ResAU. As shown in FIG. 16b, the ResAU uses the structure shown in FIG. 10b and includes a first nonlinear operation, a convolution process, an element-by-element multiplication operation, and an element-by-element addition operation. The first nonlinear operation uses a ReLU operation. The ResAU can be used in the encoding network shown in FIG. 13 or the decoding network shown in FIG. 15.

[0253] Based on equation (2), ResAU in this embodiment can be expressed by the following equation: y i =x i *(β i +Σ j gamma ij φ(x j ))+x i

[0254] Compression performance tests are performed on the ResAU shown in Fig. 16b.

[0255] Test set: 24 Kodak test images.

[0256] Experiment: ResAU shown in Fig. 16b is used in an encoding / decoding network structure with hybrid Gaussian super-prior entropy estimation. In this experiment, the nonlinear operations in ResAU are replaced with ReLU and LeakyReLU, respectively. The comparison experiment includes the identity solution without nonlinear operations and the ResAU solution using absolute value operations described in the first embodiment.

[0257] Experimental Performance: Figure 17b shows the overall performance of ResAU for 24 images in the Kodak test set. The RD curves show that using element-wise nonlinear operations can significantly improve the rate-distortion performance of image compression networks compared to the identity solution without nonlinear operations. The effect of ReLU-class nonlinear operations is slightly better than that of absolute-value nonlinear operations. [Example]

[0258] Embodiment 3 FIG. 16c is an exemplary schematic diagram of the structure of the ResAU. As shown in FIG. 16c, the ResAU uses the structure shown in FIG. 10d and includes a first nonlinear operation, a convolution process, a second nonlinear operation, an element-wise multiplication operation, and an element-wise addition operation. Both the first nonlinear operation and the second nonlinear operation are PWL operations. The ResAU can be used in the encoding network shown in FIG. 13 or the decoding network shown in FIG. 15.

[0259] ResAU introduces nonlinear features by using a piecewise linear PWL function, and implements a local attention mechanism by using convolution, a piecewise linear PWL function, and element-wise multiplication operations to correct the response on each channel at each location in the feature map using local information. The element-wise multiplication operation can avoid the problem of a limited space of learnable parameter values ​​in GDN. In addition, the residual connection with the head and tail can make the network more likely to converge during training.

[0260] Based on equation (4), ResAU in this embodiment can be expressed by the following equation: y i =x i *φ2(β i +Σ j gamma ij φ(x j ))+x i

[0261] Both φ1(·) and φ2(·) are PWL functions, γ is the weight of the convolutional layer, and β is the offset parameter of the convolutional layer. Both parameters are learnable parameters with unlimited range of values.

[0262] The functionality of the operations included in this structure is as follows:

[0263] First, the PWL operation: This function is a piecewise linear function and can provide nonlinear features for the overall transformation. Under the effect of this operation, each value of the input data is calculated based on the numerical interval in which the value is located and different mapping relationships to obtain an output value. For a certain input feature, feature maps for different channel dimensions can be calculated by using the same piecewise linear mapping function or different piecewise linear mapping functions. The parameters of the PWL function can be preset values ​​or can be obtained through training and learning.

[0264] Convolution: In ResAU, the input of the convolution is the output of a nonlinear operation. By performing a convolution on the input tensor, a fixed-size tensor is obtained, and the output tensor can be considered as a local response to the input tensor.

[0265] Second PWL operation: This function is a piecewise linear function that scales and maps the output of the convolution, providing nonlinear features for the overall transformation. Under the effect of this operation, each value of the input data is calculated based on the numerical interval in which the value falls and different mapping relationships to obtain an output value. For input features, feature maps for different channel dimensions can be calculated by using the same piecewise linear mapping function or different piecewise linear mapping functions. The parameters of the PWL function can be preset values ​​or can be obtained through training and learning.

[0266] Element-wise multiplication: In ResAU, the inputs of the element-wise multiplication are the original input of the unit and the output of the convolution process. The two input tensors have the same size, and the multiplication is performed on the data at corresponding positions. The size of the output tensor is also the same as that of the input tensor. The convolution process and element-wise multiplication are combined to realize a local attention mechanism, in which the local information of the input features is used to correct the response at each position in the feature map. In addition, the element-wise multiplication avoids the problem of the limited space of learnable parameter values ​​of the mainstream nonlinear units in GDNs in current image and video compression networks.

[0267] Element-wise addition operation: In ResAU, the input of the element-wise addition operation is the original input of the unit and the output of the element-wise multiplication operation. This operation is a residual structure with added heads and tails, so encoding and decoding networks using nonlinear units can easily converge during training.

[0268] The residual attention nonlinear unit using the piecewise linear PWL function described in this embodiment can be used in end-to-end image compression networks and video compression networks based on deep neural networks. More specifically, the residual attention nonlinear unit is generally used in the encoding module (encoder) and the decoding module (decoder) in end-to-end image compression networks, and in the encoding module and the decoding module of the prediction sub-network and the residual compression sub-network in end-to-end video compression networks.

[0269] Experiments are performed on a common end-to-end encoding and decoding network based on a super-priori structure. In this embodiment, when the preset value of the first PWL operation is a Leaky ReLU function, the number of segments of the second PWL operation is 6, and the channel dimension grouping granularity is 1 (i.e., features of all channels are grouped by using different piecewise linear functions to calculate output values), the compression ratio-distortion performance can be improved by 0.506% compared to a solution using Tanh to perform nonlinear operations after the convolution operation. In this embodiment, when the preset value of the first PWL operation is a Leaky ReLU function, the number of segments of the second PWL operation is 6, and the channel dimension grouping granularity is 8 (i.e., features are grouped by using every eight channels as a unit, and all channels in the same group use the same piecewise linear function to calculate output values), the compression ratio-distortion performance decreases by 0.488% compared to a solution using Tanh to perform nonlinear operations after the convolution operation. In this embodiment, when the preset value of the first PWL operation is a Leaky ReLU function, the number of segments of the second PWL operation is 6, and the channel dimensions are not grouped (i.e., the same piecewise linear function is used to calculate the output values ​​for the features of all channels), the compression ratio / distortion performance decreases by 0.659% compared to that in the solution where Tanh is used to perform the nonlinear operation after the convolution operation.

[0270] Optionally, FIG. 16d is an exemplary schematic diagram of the structure of the ResAU. As shown in FIG. 16d, the ResAU uses the structure shown in FIG. 10d and includes a first nonlinear operation, a convolution process, a second nonlinear operation, an element-wise multiplication operation, and an element-wise addition operation. The first nonlinear operation is Leaky ReLU, and the second nonlinear operation is Tanh. The convolution process may be conv1×1, where conv1×1 indicates that the size of the convolution kernel (or convolution operator) is 1×1. The ResAU may be used in the encoding network shown in FIG. 13 or the decoding network shown in FIG. 15. [Example]

[0271] Embodiment 4 FIG. 16e is an exemplary schematic diagram of the structure of the ResAU. As shown in FIG. 16e, the ResAU uses the structure shown in FIG. 10c and includes a first nonlinear operation, a convolution process, a second nonlinear operation, and an element-wise multiplication operation. Both the first nonlinear operation and the second nonlinear operation are PWL operations. The ResAU can be used in the encoding network shown in FIG. 13 or the decoding network shown in FIG. 15.

[0272] Based on equation (3), ResAU in this embodiment can be expressed by the following equation: y i =x i *φ2(β i +Σ j gamma ij φ(x j ))

[0273] Both φ1(·) and φ2(·) are PWL functions, γ is the weight of the convolutional layer, and β is the offset parameter of the convolutional layer. Both parameters are learnable parameters with unlimited range of values.

[0274] The functionality of the operations contained in this structure is as follows:

[0275] First, the PWL operation: This function is a piecewise linear function and can provide nonlinear features for the overall transformation. Under the effect of this operation, each value of the input data is calculated based on the numerical interval in which the value is located and different mapping relationships to obtain an output value. For the input features, feature maps for different channel dimensions can be calculated by using the same piecewise linear mapping function or different piecewise linear mapping functions. The parameters of the PWL function can be preset values ​​or can be obtained through training and learning.

[0276] Convolution: In ResAU, the input of the convolution is the output of a nonlinear operation. By performing a convolution on the input tensor, a fixed-size tensor is obtained, and the output tensor can be considered as a local response to the input tensor.

[0277] Second PWL operation: This function is a piecewise linear function that scales and maps the output of the convolution to provide nonlinear features for the overall transformation. Under the effect of this operation, each value of the input data is calculated based on the numerical interval in which the value falls and different mapping relationships to obtain an output value. For input features, feature maps for different channel dimensions can be calculated by using the same piecewise linear mapping function or different piecewise linear mapping functions. The parameters of the PWL function may be preset values ​​or may be obtained through training and learning.

[0278] Element-wise multiplication: In ResAU, the inputs of the element-wise multiplication are the original input of the unit and the output of the convolution process. The two input tensors have the same size, and the multiplication is performed on the data at corresponding positions. The size of the output tensor is also the same as that of the input tensor. The convolution process and element-wise multiplication are combined to realize a local attention mechanism, which uses local information of the input features to correct the response at each position in the feature map. In addition, the element-wise multiplication avoids the problem of the limited space of learnable parameter values ​​of the nonlinear units in the mainstream GDNs in current image and video compression networks.

[0279] The attention nonlinear unit using a residual-free structure and a piecewise linear function described in this embodiment can be used in end-to-end image compression networks and video compression networks based on deep neural networks. More specifically, the attention nonlinear unit is generally used in the encoding module, encoder and decoding module, and decoder in end-to-end image compression networks, and in the encoding module and decoding module of the prediction sub-network and residual compression sub-network of end-to-end video compression networks. Furthermore, the attention nonlinear unit using a residual-free structure and a piecewise linear function described in this embodiment can be obtained through transformation by using a residual attention nonlinear unit using a piecewise linear function. After training is completed, the second PWL operation and the element-wise addition operation in the residual attention nonlinear unit using a piecewise linear function can be combined (i.e., adding 1 to the entire output of the second PWL operation to form a new PWL function and eliminating the element-wise addition operation) to obtain the corresponding attention nonlinear unit using a residual-free structure and a piecewise linear function.

[0280] Based on the super priori structure, experiments are carried out on a common end-to-end encoding and decoding network. After training is completed, the residual-focused nonlinear unit in the experiment corresponding to embodiment 4 is converted into a nonlinear unit using the residual-free structure described in this embodiment, thereby achieving the same encoding and decoding effect as in the solution corresponding to embodiment 4.

[0281] In this embodiment, when the preset value of the first PWL operation is a Leaky ReLU function, the number of segments of the second PWL operation is 6, and the channel dimension grouping granularity is 1 (i.e., the output values ​​are calculated using different piecewise linear functions for the characteristics of all channels), the compression ratio / distortion performance can be improved by 0.506% compared to the solution in which Tanh is used to perform nonlinear operations after the convolution operation. Furthermore, the solution in this embodiment can reduce the calculation time consumption and power consumption in the element-by-element addition operation.

[0282] In this embodiment, when the preset value of the first PWL operation is a Leaky ReLU function, the number of segments of the second PWL operation is 6, and the channel dimension grouping granularity is 8 (i.e., the features are grouped by using every 8 channels as a unit, and all channels in the same group use the same piecewise linear function to calculate the output value), the compression ratio / distortion performance is reduced by 0.488% compared with that in the solution in which Tanh is used to perform nonlinear operations after the convolution operation. Furthermore, the solution in this embodiment can reduce the calculation time consumption and power consumption in the element-wise addition operation.

[0283] In this embodiment, when the preset value of the first PWL operation is a Leaky ReLU function, the number of segments of the second PWL operation is 6, and the channel dimensions are not grouped (i.e., the same piecewise linear function is used to calculate the output values ​​for the features of all channels), the compression ratio / distortion performance decreases by 0.659% compared with that in the solution in which Tanh is used to perform nonlinear operations after the convolution operation. Furthermore, the solution in this embodiment can reduce the calculation time consumption and power consumption in the element-wise addition operation.

[0284] 18 is a schematic diagram of the structure of an encoding device 1700 according to an embodiment of the present application. As shown in FIG. 18, the device 1700 in this embodiment may be used on the encoder side. The device 1700 may include an acquisition module 1701, a transformation module 1702, an encoding module 1703, and a training module 1704. Specifically:

[0285] The acquisition module 1701 is configured to acquire first image features to be processed. The transformation module 1702 is configured to perform a nonlinear transformation process on the first image features to obtain processed image features, where the nonlinear transformation process sequentially includes a first nonlinear operation, a convolution operation, and an element-wise multiplication operation. The encoding module 1703 is configured to perform encoding based on the processed image features to obtain a bitstream.

[0286] In one possible implementation, the transformation module 1702 is specifically configured to perform the steps of: performing a first nonlinear operation on each feature value in a first image feature to obtain a second image feature; performing a convolution operation on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0287] In one possible implementation, the nonlinear transformation process further comprises an element-wise multiplication operation followed by an element-wise addition operation.

[0288] In one possible implementation, the transformation module 1702 is specifically configured to perform the following steps: perform a first nonlinear operation on each feature value in a first image feature to obtain a second image feature; perform a convolution operation on the second image feature to obtain a third image feature, where the plurality of feature values ​​in the third image feature correspond to the plurality of feature values ​​in the first image feature; perform an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, where the plurality of feature values ​​in the fourth image feature correspond to the plurality of feature values ​​in the first image feature; and perform an element-wise addition operation on a plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain a processed image feature.

[0289] In one possible implementation, the nonlinear transformation process further includes a second nonlinear operation between the convolution process and the element-wise multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation.

[0290] In one possible implementation, the nonlinear operation includes a piecewise linear mapping, such as ReLU, LeakyReLU, PWL, and Abs. In another possible implementation, the nonlinear operation includes a continuous function, such as Tanh or Sigmoid. In another possible implementation, the nonlinear operation includes a piecewise nonlinear operation.

[0291] In one possible implementation, the apparatus further includes a training module 1704 configured to perform the steps of: constructing a nonlinear transformation unit in a training phase, wherein the nonlinear transformation unit in the training phase includes a first nonlinear operation layer, a convolution processing layer, an element-wise multiplication operation layer, and an element-wise addition operation layer; and performing training based on pre-acquired training data to obtain a trained nonlinear transformation unit, wherein the trained nonlinear transformation unit is used to perform a nonlinear transformation process.

[0292] The device in this embodiment may be configured to implement the technical solution in the method embodiment shown in Figure 9, and its implementation principle and technical effect are similar. The details will not be described again here.

[0293] 19 is a schematic diagram of the structure of a decoding device 1800 according to an embodiment of the present application. As shown in FIG. 19, the device 1800 in this embodiment may be used on the decoder side. The device 1800 may include an acquisition module 1801, a transformation module 1802, a reconstruction module 1803, and a training module 1804. Specifically:

[0294] The acquisition module 1801 is configured to acquire first image features to be processed. The transformation module 1802 is configured to perform a nonlinear transformation operation on the first image features to obtain processed image features, where the nonlinear transformation operation includes a first nonlinear operation, a convolution operation, and an element-wise multiplication operation. The reconstruction module 1803 is configured to obtain a reconstructed image based on the processed image features.

[0295] In one possible implementation, the transformation module 1802 is specifically configured to perform the steps of: performing a first nonlinear operation on each feature value in a first image feature to obtain a second image feature; performing a convolution operation on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; and performing an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a processed image feature.

[0296] In one possible implementation, the nonlinear transformation process further comprises an element-wise multiplication operation followed by an element-wise addition operation.

[0297] In one possible implementation, the transformation module 1802 is specifically configured to perform the following steps: perform a first nonlinear operation on each feature value in a first image feature to obtain a second image feature; perform a convolution operation on the second image feature to obtain a third image feature, where the plurality of feature values ​​in the third image feature correspond to the plurality of feature values ​​in the first image feature; perform an element-wise multiplication operation on a plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, where the plurality of feature values ​​in the fourth image feature correspond to the plurality of feature values ​​in the first image feature; and perform an element-wise addition operation on a plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain a processed image feature.

[0298] In one possible implementation, the nonlinear transformation process further includes a second nonlinear operation between the convolution process and the element-wise multiplication operation, the second nonlinear operation being the same as or different from the first nonlinear operation.

[0299] In one possible implementation, the nonlinear operation includes a piecewise linear mapping, such as ReLU, LeakyReLU, PWL, and Abs. In another possible implementation, the nonlinear operation includes a continuous function, such as Tanh or Sigmoid. In another possible implementation, the nonlinear operation includes a piecewise nonlinear operation.

[0300] In one possible implementation, the training module 1804 is configured to perform a step of constructing a nonlinear transformation unit in a training phase, where the nonlinear transformation unit in the training phase includes a first nonlinear operation layer, a convolution processing layer, an element-wise multiplication operation layer, and an element-wise addition operation layer, and a step of performing training based on pre-acquired training data to obtain a trained nonlinear transformation unit, where the trained nonlinear transformation unit is used to perform a nonlinear transformation process.

[0301] The device in this embodiment can be configured to implement the technical solution in the method embodiment shown in Figure 14. The implementation principle and technical effect of the device are similar to those of the method embodiment, and the details will not be described again here.

[0302] An embodiment of the present application provides a bitstream, which is generated by a processor by performing any of the encoding methods in the above embodiments.

[0303] An embodiment of the present application provides a bitstream storage device, the device including: a receiver configured to receive a bitstream; and at least one storage medium configured to store the bitstream, the bitstream being generated by any of the encoding methods in the aforementioned embodiments.

[0304] An embodiment of the present application provides a bitstream transmission device, the device including a transmitter and at least one storage medium, the at least one storage medium configured to store a bitstream, the bitstream being generated by a processor by performing any encoding method in the aforementioned embodiments, and the transmitter configured to transmit the bitstream to another electronic device. Optionally, the bitstream transmission device further includes a receiver and a processor, the receiver configured to receive a user request, and the processor configured to select a target bitstream from the storage medium and instruct the transmitter to send the target bitstream in response to the user request.

[0305] An embodiment of the present application provides a bitstream distribution system, the system including at least one storage medium and a streaming media device, the at least one storage medium configured to store at least one bitstream, the at least one bitstream including a bitstream generated according to any one of the implementations of the first aspect. The streaming media device is configured to obtain a target bitstream from the at least one storage medium and transmit the target bitstream to a terminal-side device, the streaming media device including a content server or a content distribution server.

[0306] An embodiment of the present application provides a bitstream distribution system, the system including: a communication interface configured to receive a user request to acquire a target bitstream; and a processor configured to determine a storage location of the target bitstream in response to the user request. The communication interface is further configured to transmit the storage location of the target bitstream to the user, whereby the user acquires the target bitstream from the storage location of the target bitstream. Here, the target bitstream is generated by the processor by performing any of the encoding methods in the aforementioned embodiments.

[0307] In one implementation process, the steps in the aforementioned method embodiments may be implemented by using hardware integrated logic circuitry in a processor or by using instructions in the form of software. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the method disclosed in the embodiments of this application may be directly presented as being performed and completed by a hardware encoding processor, or may be performed and completed by a combination of hardware and software modules in the encoding processor. The software modules may be located in a storage medium well-established in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory, and the processor reads information in the memory and completes the steps in the aforementioned method in combination with the processor's hardware.

[0308] The memory in the above embodiments may be volatile or nonvolatile memory, or may include both volatile and nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM may be used, such as static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlink DRAM, SLDRAM), and direct rambus dynamic random access memory (direct rambus RAM, DR RAM). It should be noted that memory in the systems and methods described herein includes, but is not limited to, these and any other suitable types of memory.

[0309] Those skilled in the art may recognize that, in combination with the examples described in the embodiments disclosed herein, the units and algorithm steps may be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to go beyond the scope of the embodiments of the present application.

[0310] It will be clearly understood by those skilled in the art that for the sake of convenience, the detailed operation processes of the aforementioned systems, devices and units may be referred to the corresponding procedures in the aforementioned method embodiments, and the details will not be described again in this specification.

[0311] It should be understood that the disclosed systems, devices, and methods in some embodiments provided in the present application may be implemented in other ways. For example, the described device embodiments are merely examples. For example, the division into units is merely a logical division of function, and other divisions may be used in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0312] The units described as separate parts may or may not be physically separate, and the parts shown as units may or may not be physical units, and may be located in one location or distributed over multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions of the embodiments.

[0313] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0314] When a function is implemented in the form of a software functional unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application may essentially be implemented in the form of a software product, or a portion contributing to the prior art may be implemented in the form of a software product. A computer software product is stored in a storage medium and includes instructions for instructing a computer device (a personal computer, a server, or a network device) to perform all or some of the steps of the method described in the embodiments of the present application. Alternatively, a computer software product may be transferred from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, fiber optic, or digital subscriber line (DSL)) or wireless (e.g., infrared, radio, or microwave) method. A computer-readable storage medium may be any available medium accessible by a computer, or a data storage device integrating one or more available media, such as a server or a data center. The available media may be magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., Solid State Disk (SSD), read-only memory (ROM), or random access memory (RAM)).

[0315] The above description is merely a specific implementation of the embodiments of the present application. However, the protection scope of the embodiments of the present application is not limited thereto. Any modifications or replacements that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of the present application shall fall within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. 1. A method of encoding an image, comprising: obtaining a first image feature from the image as the image feature to be processed; performing a nonlinear transformation process to obtain processed image features, wherein the first image features are input to the nonlinear transformation process, the nonlinear transformation process sequentially comprising a first nonlinear operation, a convolution operation, and an element-wise multiplication operation, and the first nonlinear operation, the convolution operation, and the element-wise multiplication operation are sequentially performed as recited; performing encoding based on the processed image features to obtain a bitstream; method.

2. The step of performing a nonlinear transformation process on the first image feature to obtain a processed image feature comprises: performing the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing the convolution process on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing the element-wise multiplication operation on the plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain the processed image feature. The method of claim 1.

3. The method of claim 1 , wherein the nonlinear transformation process further comprises an element-wise addition operation after the element-wise multiplication operation.

4. The step of performing a nonlinear transformation process on the first image feature to obtain a processed image feature comprises: performing the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing the convolution process on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing the element-wise multiplication operation on the corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, wherein the feature values ​​in the fourth image feature correspond to the feature values ​​in the first image feature; performing the element-wise addition operation on the plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain the processed image feature. The method of claim 3.

5. The method of claim 1 , wherein the first nonlinear operation comprises an activation function of a rectified linear unit system, a sigmoid, a Tanh, or a piecewise linear mapping.

6. The method further comprises: constructing a nonlinear transformation unit in a training phase, the nonlinear transformation unit in the training phase including a first nonlinear operation layer, a convolution processing layer, and an element-wise multiplication operation layer; performing training based on previously obtained training data to obtain a trained nonlinear transformation unit, the trained nonlinear transformation unit being used to implement the nonlinear transformation process; The method of claim 1.

7. 1. A method of decoding an image, comprising: obtaining a first image feature from the bitstream as the image feature to be processed; performing a nonlinear transformation process to obtain processed image features, wherein the first image features are input to the nonlinear transformation process, the nonlinear transformation process sequentially comprising a first nonlinear operation, a convolution operation, and an element-wise multiplication operation, and the first nonlinear operation, the convolution operation, and the element-wise multiplication operation are sequentially performed as recited; obtaining a reconstructed image based on the processed image features. method.

8. The step of performing a nonlinear transformation process on the first image feature to obtain a processed image feature comprises: performing the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing the convolution process on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing the element-wise multiplication operation on the plurality of corresponding feature values ​​in the first image feature and the third image feature to obtain the processed image feature. The method of claim 7.

9. The method of claim 7 , wherein the nonlinear transformation process further comprises an element-wise addition operation after the element-wise multiplication operation.

10. The step of performing a nonlinear transformation process on the first image feature to obtain a processed image feature comprises: performing the first nonlinear operation on each feature value in the first image feature to obtain a second image feature; performing the convolution process on the second image feature to obtain a third image feature, wherein a plurality of feature values ​​in the third image feature correspond to a plurality of feature values ​​in the first image feature; performing the element-wise multiplication operation on the corresponding feature values ​​in the first image feature and the third image feature to obtain a fourth image feature, wherein the feature values ​​in the fourth image feature correspond to the feature values ​​in the first image feature; performing the element-wise addition operation on the plurality of corresponding feature values ​​in the first image feature and the fourth image feature to obtain the processed image feature.

10. The method of claim 9.

11. The method of claim 7 , wherein the first nonlinear operation comprises an activation function of a rectified linear unit system, a sigmoid, a Tanh, or a piecewise linear mapping.

12. The method further comprises: constructing a nonlinear transformation unit in a training phase, the nonlinear transformation unit in the training phase including a first nonlinear operation layer, a convolution processing layer, and an element-wise multiplication operation layer; performing training based on previously obtained training data to obtain a trained nonlinear transformation unit, the trained nonlinear transformation unit being used to implement the nonlinear transformation process; The method of claim 7.

13. 1. An encoding device comprising: an acquisition module configured to acquire a first image feature from the image as the image feature to be processed; a transformation module configured to perform a nonlinear transformation process on the first image feature to obtain a processed image feature, the first image feature being input to the nonlinear transformation process, the nonlinear transformation process sequentially including a first nonlinear operation, a convolution operation, and an element-wise multiplication operation, the first nonlinear operation, the convolution operation, and the element-wise multiplication operation being sequentially performed as recited; an encoding module configured to perform encoding based on the processed image features to obtain a bitstream. Device.

14. 1. A decoding device comprising: an acquisition module configured to acquire a first image feature from the bitstream as the image feature to be processed; a transformation module configured to perform a nonlinear transformation process on the first image feature to obtain a processed image feature, the first image feature being input to the nonlinear transformation process, the nonlinear transformation process including a first nonlinear operation, a convolution operation, and an element-wise multiplication operation, the first nonlinear operation, the convolution operation, and the element-wise multiplication operation being performed sequentially as recited; a reconstruction module configured to obtain a reconstructed image based on the processed image features. Device.

15. An encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the processor and storing a program for execution by the processor, the program, when executed by the processor, configuring the encoder to perform the method of any one of claims 1 to 6. Encoder.

16. A decoder comprising: one or more processors; a non-transitory computer readable storage medium coupled to the processor and storing a program for execution by the processor, the program, when executed by the processor, configuring the decoder to perform the method of any one of claims 7 to 12. decoder.

17. A computer program product for causing a computer or processor to carry out a method according to any one of claims 1 to 6 or claims 7 to 12.

18. 13. A computer readable storage medium containing instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 6 or claims 7 to 12.

19. A computer-readable storage medium storing a bitstream obtained according to the method of any one of claims 1 to 6.

Images