Decoding and coding method, device and equipment

Abstract

The invention provides a decoding and coding method, device and equipment thereof. The decoding and coding method comprises the following steps: decoding a first code stream corresponding to a current image block to obtain a coefficient hyper-parameter feature; determining a probability distribution parameter based on the coefficient hyper-parameter feature, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a reconstruction feature; inputting the reconstruction features into a synthesis transformation network to obtain a reconstruction image block; the synthesis transformation network comprises an attention module, and the attention module is a cascade mixed attention module or a parallel mixed attention module; the cascaded mixed attention module comprises a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are serial; the parallel mixed attention module comprises a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are parallel. Through the scheme of the invention, the decoding performance is improved, and the complexity is reduced.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding and coding method, device and equipment thereof. Background Art

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, prediction can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to predict the current pixel using the pixels of the adjacent encoded images to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to predict the current pixel using the pixels of the encoded block of the current frame image to remove the redundancy in the video space domain.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor decoding performance and high complexity. Summary of the invention

[0004] The present application provides a decoding and encoding method, apparatus and device thereof, which can improve decoding performance and reduce complexity.

[0005] The present application provides a decoding method, which is applied to a decoding end, and the method includes:

[0006] Decoding a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block;

[0007] Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode the second bitstream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block;

[0008] Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block;

[0009] Wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules; or, the parallel hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two parallel sub-modules.

[0010] The present application provides an encoding method, which is applied to an encoding end, and the method includes:

[0011] Decoding a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block;

[0012] Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode the second bitstream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block;

[0013] Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block;

[0014] Wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules; or, the parallel hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two parallel sub-modules.

[0015] The present application provides a decoding device, applied to a decoding end, the device comprising:

[0016] A decoding module, configured to decode a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode a second bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block;

[0017] A processing module, used for inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0018] The present application provides an encoding device, applied to an encoding end, the device comprising:

[0019] A decoding module, configured to decode a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode a second bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block;

[0020] A processing module, used for inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0021] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0022] The processor is used to execute machine executable instructions to implement the above decoding method.

[0023] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0024] The processor is used to execute machine executable instructions to implement the above encoding method.

[0025] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0026] The processor is used to execute machine executable instructions to implement the above decoding method or encoding method.

[0027] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the above-mentioned decoding method or encoding method.

[0028] The present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned decoding method or encoding method is implemented.

[0029] It can be seen from the above technical solutions that in the embodiments of the present application, a synthetic transformation network with an attention mechanism is proposed for the neural network-based coding and decoding technology. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module. When the synthetic transformation network is realized by the cascaded hybrid attention module or the parallel hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, and the decoding performance is improved. While reducing the complexity, the quality of the reconstructed image block is effectively guaranteed, and the encoding performance and decoding performance are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;

[0031] Figure 2 is a schematic diagram of pixel reorganization in one embodiment of the present application;

[0032] Figure 3 is a flowchart of a decoding method in one embodiment of the present application;

[0033] Figure 4 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0035] Figure 6A-6D is a schematic diagram of the structure of a synthetic transformation network in one embodiment of the present application;

[0036] Figure 7A-7C is a schematic diagram of the structure of a synthetic transformation network in one embodiment of the present application;

[0037] Figure 8A-8H is a schematic diagram of the structure of a synthetic transformation network in one embodiment of the present application;

[0038] Fig. 9A is a hardware structure diagram of a decoding end device in one embodiment of the present application;

[0039] Fig. 9B It is a hardware structure diagram of an encoding end device in one implementation of the present application. DETAILED DESCRIPTION

[0040] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at ... ", or "when ... ", or "in response to determination".

[0041] In an embodiment of the present application, a decoding method is proposed, which may involve the following concepts:

[0042] Entropy coding: Entropy coding is the coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information of the source (a measure of uncertainty). Entropy coding methods may include but are not limited to: Shannon coding, Huffman coding, and arithmetic coding.

[0043] Neural Network (NN): Neural network refers to artificial neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the outside world; output units realize the output of processing results; hidden units are units between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural network, and imitate the information processing function of human brain nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), fully connected network, etc.

[0044] Convolutional neural network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large-scale image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, the positional relationship between it and other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU (Rectified Linear Unit) function, Leaky-ReLU function, PReLU (Parametric Rectified Linear Unit) function, GDN (Generalized Divisive Normalization) function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping surface share weights, the number of free parameters of the network is reduced.

[0045] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through local connections, weight sharing and other methods.

[0046] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input through padding (of course, it can also remain the same). If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0047] Depthwise Separable Convolution: Depthwise separable convolution requires two convolution operations. In the first convolution operation, deep_wise convolution is performed first (i.e., the features of each layer are collected), kernel_size = K*K*1, and the total number of parameters for the first convolution is K*K*Cin. In the second convolution operation, in order to obtain the output of the Cout dimension, kernel_size = 1*1*Cin, and the total number of parameters for the second convolution is 1*1*Cin*Cout. The output of the second convolution can be the output of the depthwise separable convolution, and the input of the first convolution can be the input of the depthwise separable convolution.

[0048] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to fresh samples. The purpose of learning is to learn the rules hidden behind the data pairs. For data outside the learning set with the same rules, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0049] Feature: A feature is a three-dimensional feature matrix or tensor of C*W*H, see Figure 1 As shown, it is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of a neural network or the output of a neural network.

[0050] Pixel shuffle: Pixel shuffle can also be called feature position shuffle, which is to obtain high-resolution feature maps by reorganizing low-resolution features across multiple channels. Figure 2 As shown, it is a schematic diagram of pixel reorganization. Figure 2 In the figure, a schematic diagram of pixel reorganization is given when the upsampling factor is 2. Obviously, a feature with 4 channels and a resolution of 3*3 becomes a feature with 1 channel and a resolution of 6*6 after passing through the feature position reorganization layer.

[0051] Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J (mode) = D + λ * R, where D represents Distortion, which can usually be measured using the SSE (sum-square error) indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to consider the cost, the SAD (Sum of Absolute Difference) indicator can also be used. SAD refers to the sum of the absolute values ​​of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make comparison decisions on the encoding mode, the best encoding performance can usually be guaranteed.

[0052] A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal mode combination for different content to obtain the best coding performance. The decoding end can obtain relevant mode information by directly parsing the flag information, and the complexity impact is small.

[0053] In the general framework of end-to-end image coding, it mainly includes the feature main information part and the super prior side information part. The feature main information part includes analysis transformation network, quantization, normal entropy coding, normal entropy decoding and synthesis transformation network, and the super prior side information includes super prior analysis network, quantization, factored entropy coding, factored entropy decoding and super prior synthesis network. The image component x is compressed and encoded and reconstructed and restored by the analysis transformation network and synthesis transformation network of the feature main information part respectively; the super prior side information part is mainly used to model the probability of the feature main information and guide the entropy encoding and decoding of the feature main information. In the general framework of end-to-end image coding, there are problems such as high computational complexity and poor decoding performance.

[0054] In view of the above findings, in this embodiment, a synthetic transformation network with an attention mechanism is proposed by utilizing the characteristics of an end-to-end image coding framework. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module. When the synthetic transformation network is implemented by a cascaded hybrid attention module or a parallel hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, thereby improving the decoding performance.

[0055] The decoding method in the embodiments of the present application is described in detail below in conjunction with several specific embodiments.

[0056] Example 1: A decoding method is proposed in the embodiment of the present application, see Figure 3 As shown, it is a flowchart of the decoding method, which can be applied to a decoding end (also called a video decoder), and the method may include:

[0057] Step 301: decode the first code stream corresponding to the current image block to obtain coefficient hyper-parameter features of the current image block.

[0058] Step 302: Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode the second code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block.

[0059] For example, a code stream corresponding to the current image block (such as the first code stream) can be a code stream encoding the coefficient hyper-parameter features corresponding to the current image block; another code stream corresponding to the current image block (such as the second code stream) can be a code stream encoding the residual features corresponding to the current image block.

[0060] Step 303, input the reconstructed features to the synthetic transformation network to obtain a reconstructed image block corresponding to the current image block. Wherein, the synthetic transformation network includes at least an attention module, and the attention module can be a cascaded hybrid attention module or a parallel hybrid attention module. Wherein, the cascaded hybrid attention module can include a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules, for example, the output feature of the first attention submodule is the input feature of the second attention submodule. Alternatively, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules, for example, the input feature of the first attention submodule is parallel to the input feature of the second attention submodule, and the output feature of the first attention submodule is parallel to the output feature of the second attention submodule.

[0061] Exemplarily, if the attention module is a cascaded hybrid attention module, the first attention submodule can be used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, and the output features of the first attention submodule are the input features of the second attention submodule; wherein the first processing can be at least one of layer normalization, convolution operation and dimension transformation operation. The second attention submodule can be used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule; wherein the second processing can be at least one of depth separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing can be at least one of depth separable convolution operation, layer normalization, and multi-layer perception operation. Alternatively, the second processing can be at least one of layer normalization, linear operation, and activation operation. Alternatively, the second processing can be at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.

[0062] Exemplarily, if the attention module is a parallel hybrid attention module, the input features of the parallel hybrid attention module can also be split into two sub-features in the channel dimension, and the two sub-features can be used as the input features of the first attention sub-module and the input features of the second attention sub-module respectively. The input features of the first attention sub-module can be first processed by the first attention sub-module to obtain the output features of the first attention sub-module; wherein the first processing can be at least one of layer normalization, convolution operation and dimension transformation operation. And, the input features of the second attention sub-module can also be second processed by the second attention sub-module to obtain the output features of the second attention sub-module; wherein the second processing can be at least one of depth separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing can be at least one of depth separable convolution operation, layer normalization, and multi-layer perception operation. Or, the second processing can be at least one of layer normalization, linear operation, and activation operation. Or, the second processing can be at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation. Then, the output features of the first attention submodule and the output features of the second attention submodule can be concatenated in the channel dimension to obtain the output features of the parallel hybrid attention module.

[0063] In a possible implementation, a first processing is performed on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule, including but not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; performing a dimension transformation operation on the query vector, the key vector, and the value vector, respectively, to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; determining an attention weight based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determining a modified feature corresponding to the input feature based on the attention weight and the dimensionally transformed value vector; and determining the output features of the first attention submodule based on the input features and the modified features.

[0064] In a possible implementation, a first processing is performed on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule, which may include but is not limited to: layer normalization is performed on the input features of the first attention submodule to obtain layer-normalized features; window division is performed on the layer-normalized features to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to multiple small-size features may be spliced ​​to obtain a query vector; the small-size key features corresponding to multiple small-size features may be spliced ​​to obtain a key vector; the small-size value features corresponding to multiple small-size features may be spliced ​​to obtain a value vector. Then, the query vector, the key vector and the value vector are respectively transformed in dimension to obtain a transformed query vector, a transformed key vector and a transformed value vector; the attention weight is determined based on the transformed query vector and the transformed key vector, and the correction feature corresponding to the input feature is determined based on the attention weight and the transformed value vector; the output feature of the first attention submodule is determined based on the input feature and the correction feature.

[0065] In a possible implementation, a first processing is performed on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determining an attention weight based on the query vector and the key vector, determining an intermediate feature based on the attention weight and the value vector, performing layer normalization on the intermediate feature to obtain a corrected feature corresponding to the input feature; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0066] In a possible implementation, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; performing window division on the layer normalized features to obtain multiple small-size features; for each small-size feature, a three-way convolution operation may be performed on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature. Then, the small-size query features corresponding to the multiple small-size features may be spliced ​​to obtain a query vector; the small-size key features corresponding to the multiple small-size features may be spliced ​​to obtain a key vector; the small-size value features corresponding to the multiple small-size features may be spliced ​​to obtain a value vector. Based on the query vector and the key vector, an attention weight is determined, and based on the attention weight and the value vector, an intermediate feature is determined, and the intermediate feature is layer normalized to obtain a correction feature corresponding to the input feature. Then, the output features of the first attention submodule may be determined based on the input feature and the correction feature.

[0067] In a possible implementation, a second processing is performed on the input features of the second attention submodule by the second attention submodule to obtain output features of the second attention submodule, which may include but is not limited to: performing a depth-separable convolution operation on the input features of the second attention submodule to obtain convolved features; performing layer normalization on the convolved features to obtain layer-normalized features; performing a multi-layer perception operation on the layer-normalized features to obtain multi-layer perceived features; and determining the output features of the second attention submodule based on the input features and the multi-layer perceived features.

[0068] In a possible implementation, a first processing is performed on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determining an attention weight based on the query vector and the key vector, and determining a modified feature corresponding to the input feature based on the attention weight and the value vector; and determining the output features of the first attention submodule based on the input features and the modified features.

[0069] In a possible implementation, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; performing window division on the layer normalized features to obtain multiple small-size features; for each small-size feature, performing a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; splicing the small-size query features corresponding to multiple small-size features to obtain a query vector; splicing the small-size key features corresponding to multiple small-size features to obtain a key vector; splicing the small-size value features corresponding to multiple small-size features to obtain a value vector; determining the attention weight based on the query vector and the key vector, and determining the correction feature corresponding to the input feature based on the attention weight and the value vector. Then, the output features of the first attention submodule can be determined based on the input features and the correction features.

[0070] In a possible implementation, a second processing is performed on the input features of the second attention submodule by the second attention submodule to obtain output features of the second attention submodule, which may include but is not limited to: layer normalization may be performed on the input features of the second attention submodule to obtain layer-normalized features; a first linear operation may be performed on the layer-normalized features to obtain features after the first linear operation; an activation operation may be performed on the features after the first linear operation to obtain activated features; a second linear operation may be performed on the activated features to obtain features after the second linear operation; and the output features of the second attention submodule may be determined based on the input features and the features after the second linear operation.

[0071] In a possible implementation, the second attention submodule performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, which may include but is not limited to: the input features of the second attention submodule may be downsampled to obtain the downsampled features; the downsampled features may be subjected to residual convolution to obtain the residual convolution features; the residual convolution features may be upsampled to obtain the upsampled features; the upsampled features may be activated to obtain the activated features; and the input features may be subjected to residual convolution to obtain the convolution features after the residual convolution. On this basis, the output features of the second attention submodule may be determined based on the input features, the activated features, and the convolution features.

[0072] In a possible implementation, the synthetic transformation network may further include at least one deconvolution layer; wherein: the attention module may be located behind any deconvolution layer. Alternatively, the attention module may be located behind each deconvolution layer; wherein the attention modules located behind different deconvolution layers are exactly the same, or the attention modules located behind different deconvolution layers are not exactly the same. Alternatively, the attention module may be located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers); wherein the attention modules located behind different deconvolution layers are exactly the same, or the attention modules located behind different deconvolution layers are not exactly the same.

[0073] Exemplarily, the synthetic transformation network may further include a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer. Wherein, the synthetic transformation network may include an attention module, the attention module is located behind the first deconvolution layer, or the attention module is located behind the second deconvolution layer, or the attention module is located behind the third deconvolution layer. Alternatively, the synthetic transformation network may include an attention module located behind the first deconvolution layer, an attention module located behind the second deconvolution layer, and an attention module located behind the third deconvolution layer. Alternatively, the synthetic transformation network may include an attention module located behind the first deconvolution layer, and an attention module located behind the second deconvolution layer. Alternatively, the synthetic transformation network may include an attention module located behind the first deconvolution layer, and an attention module located behind the third deconvolution layer. Alternatively, the synthetic transformation network may include an attention module located behind the first deconvolution layer, and an attention module located behind the third deconvolution layer. Alternatively, the synthetic transformation network may include an attention module located behind the second deconvolution layer, and an attention module located behind the third deconvolution layer. Alternatively, the synthetic transformation network may include an attention module located behind the second deconvolution layer, and an attention module located behind the third deconvolution layer.

[0074] Exemplarily, the synthetic transformation network includes at least a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, and a second deconvolution layer. The synthetic transformation network includes at least one attention module, wherein one attention module is located behind the first deconvolution layer.

[0075] Based on the same application concept as the above-mentioned decoding method, an encoding method is also proposed in an embodiment of the present application. The method can be applied to an encoding end (also called a video encoder), and the method may include: decoding a first code stream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, and decoding a second code stream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; inputting the reconstruction features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0076] Exemplarily, the implementation process of the encoding method is similar to that of the decoding method, and the repeated contents are not repeated here.

[0077] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0078] It can be seen from the above technical solutions that in the embodiments of the present application, a synthetic transformation network with an attention mechanism is proposed for the neural network-based coding and decoding technology. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module. When the synthetic transformation network is realized by the cascaded hybrid attention module or the parallel hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, and the decoding performance is improved. While reducing the complexity, the quality of the reconstructed image block is effectively guaranteed, and the encoding performance and decoding performance are improved.

[0079] Example 2: For the processing process of the encoding end, please refer to Figure 4 As shown, of course, Figure 4 This is just an example of the processing process at the encoding end. There is no limitation on the processing process at the encoding end. The encoding process can be implemented by using a neural network.

[0080] After obtaining the current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoder may analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Among them, performing feature transformation on the current image block x through the analysis transformation network means: transforming the current image block x to an image feature y in the latent domain, thereby facilitating all subsequent processes to operate on the latent domain.

[0081] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0082] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of the hyperparameter encoding network, as long as the coefficient hyperparameter feature transformation can be performed on the image feature y. The image feature y in the latent domain obtains the super prior latent information z after passing through the hyperparameter encoding network.

[0083] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 4 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, i.e. Figure 4 The AE operation in represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.

[0084] After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to the subsequent embodiments.

[0085] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 4 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0086] For the encoding process of Bitstream#1, the encoding method of the fixed probability density model can be adopted, and for the decoding process of Bitstream#1, the decoding method of the fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0087] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block (the determination process of the reconstruction feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstruction feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain a residual by subtracting from the original feature and to obtain the reconstruction feature y_hat by adding the decoded residual.

[0088] In another embodiment, the mean prediction network can also obtain the prediction value corresponding to the current image block based only on the coefficient hyperparameter feature z_hat of the current image block, that is, the prediction value corresponding to the current image block is obtained without using the reconstruction feature y_hat of the previous image block, which can greatly reduce the complexity of obtaining the prediction value and speed up the encoding and decoding process. The following prediction value acquisition method on the encoding side can adopt one of the above two solutions, which will not be described one by one.

[0089] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0090] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature y, the encoder can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0091] After obtaining the image feature s, the encoder can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 4 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e. Figure 4 The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.

[0092] After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing of Bitstream#2 corresponding to the current image block by the decoder, refer to the subsequent embodiments.

[0093] After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the figure represents the decoding process. Then, the encoder can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0094] After obtaining the image feature s', the encoder can perform feature recovery (i.e., the inverse process of feature processing) on ​​the image feature s'. There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the reconstructed feature y_hat based on the residual feature r_hat and the predicted value mu. The reconstructed feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the predicted value mu is used as the reconstructed feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the predicted value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery (i.e., the inverse process of feature processing) on ​​the image feature s' to obtain the reconstructed feature y_hat. The reconstructed feature y_hat and the image feature y can be the same or different. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0095] After obtaining the reconstructed feature y_hat, the encoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0096] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0097] To obtain the probability distribution model, continue to refer to Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network, and it is sufficient to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0098] In a possible implementation, the processing process at the encoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without any limitation on the encoding process.

[0099] Example 3: For the processing process of the decoding end, please refer to Figure 5 As shown, of course, Figure 5 This is just an example of the processing process at the decoding end. There is no limitation on the processing process at the decoding end. The decoding process can be implemented by using a neural network.

[0100] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 5 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 5 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoding end can decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat. Among them, for the decoding process of Bitstream#1, a decoding method of a fixed probability density model can be used, and there is no restriction on this.

[0101] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0102] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block (the determination process of the reconstruction feature y_hat is shown in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstruction feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0103] In another embodiment, after obtaining the coefficient hyperparameter feature z_hat of the current image block, the decoding end can directly obtain the prediction value mu corresponding to the current image block based on the coefficient hyperparameter feature z_hat, that is, it does not rely on the reconstruction feature y_hat of the previous image block to obtain the prediction value, which can greatly reduce the complexity of obtaining the prediction value and speed up the encoding and decoding process. The following prediction value acquisition method on the decoding side can adopt one of the above two solutions, which will not be repeated one by one.

[0104] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 5 The dashed box in indicates that the mean prediction network is optional.

[0105] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can decode Bitstream#2 to obtain the image quantization feature, that is, Figure 5 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 5 The IQ operation in may be an inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoder may decode the Bitstream#2 to obtain the image feature s' without involving an inverse quantization process of the image quantization feature.

[0106] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoder determines the reconstructed feature y_hat based on the residual feature r_hat and the prediction value mu, and the reconstructed feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the reconstructed feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the prediction value mu is provided by the mean prediction network. Alternatively, after obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the reconstructed feature y_hat, which is the same as or different from the image feature y. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0107] After obtaining the reconstructed feature y_hat, the decoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0108] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to Figure 5 As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0109] In a possible implementation, the processing process at the above decoding end can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without limiting the decoding process.

[0110] Embodiment 4: For Embodiment 1, Embodiment 2 and Embodiment 3, a mean prediction network may be deployed or not. In order to improve the feature encoding performance, the deployment of the mean prediction network is taken as an example. When the mean prediction network is deployed, in order to obtain an accurate prediction value of the image feature, a context-based prediction may be performed based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block to obtain the prediction value mu corresponding to the current image block; or, the prediction value mu corresponding to the current image block may be obtained based on the coefficient hyperparameter feature z_hat of the current image block.

[0111] The processing process at the encoding end may include the following steps:

[0112] Step S11, after obtaining the original image, determine whether to divide the original image into blocks based on the image resolution. For example, if the image resolution is greater than the threshold, the original image is divided into blocks, otherwise, the original image is not divided into blocks. If yes, the original image is divided into a number of image blocks x, and there are overlapping parts between adjacent image blocks. If not, the image block x is the original image, that is, the original image has only one image block. For the convenience of description, an image block is taken as an example, and this image block can be recorded as the current image block x, and the processing process of the current image block x is taken as an example to explain.

[0113] Step S12: After obtaining the current image block x, the current image block x is analyzed and transformed through the analysis transformation network to obtain the image feature y (i.e., feature block) corresponding to the current image block x. For example, the current image block x is transformed into the image feature y in the latent domain through the analysis transformation network, so that all subsequent processes are convenient to operate on the latent domain.

[0114] Step S13, perform coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x. For example, the image feature y can be input into a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network performs coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x.

[0115] Step S14, encode the coefficient hyperparameter feature z to the first bitstream (Bitstream#1) corresponding to the current image block. For example, the coefficient hyperparameter feature z is quantized to obtain the hyperparameter quantization feature, and the hyperparameter quantization feature is encoded to obtain the first bitstream corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z is directly encoded to obtain the first bitstream corresponding to the current image block. That is, the first bitstream corresponding to the current image block is a bitstream encoded with the coefficient hyperparameter feature z corresponding to the current image block x. After obtaining the first bitstream corresponding to the current image block, the first bitstream corresponding to the current image block is sent to the decoding end.

[0116] Step S15: After obtaining the first bitstream corresponding to the current image block, the first bitstream is decoded to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process.

[0117] Step S16, determine the probability distribution parameter based on the coefficient hyperparameter feature z_hat. For example, perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, input the coefficient hyperparameter feature z_hat to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on the probability distribution parameter p.

[0118] Step S17: Input the coefficient hyperparameter feature z_hat to the mean prediction network, and the mean prediction network performs context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat of the previous image block to obtain the predicted value mu. Alternatively, input the coefficient hyperparameter feature z_hat to the mean prediction network, and the mean prediction network obtains the predicted value mu based on the coefficient hyperparameter feature z_hat.

[0119] Step S18, based on the image feature y and the mean feature mu (i.e., the predicted value mu), determine the residual feature r, such as taking the difference between the image feature y and the mean feature mu as the residual feature r. Perform feature processing on the residual feature r to obtain the residual feature r after feature processing, and there is no restriction on the feature processing process. Feature processing is an optional step, and feature processing may not be performed on the residual feature r.

[0120] Step S19, quantize the residual feature r (or the residual feature r after feature processing) to obtain the image quantization feature, and encode the image quantization feature to obtain the second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r (or the residual feature r after feature processing) can be directly encoded to obtain the second bitstream corresponding to the current image block. That is, the second bitstream corresponding to the current image block is a bitstream encoded with the residual feature r corresponding to the current image block x. After obtaining the second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is sent to the decoding end.

[0121] Exemplarily, when encoding the image quantization feature or the residual feature r, the probability distribution model corresponding to the probability distribution parameter p may be used to encode the image quantization feature or the residual feature r to obtain a second code stream.

[0122] Step S20: After obtaining the second bitstream corresponding to the current image block, the probability distribution model corresponding to the probability distribution parameter p is used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat, or, after dequantizing the image quantization features, the residual features r_hat are directly obtained without involving a feature recovery process. Alternatively, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain residual features r_hat, or, after decoding the second bitstream, the residual features r_hat are directly obtained.

[0123] Step S21: After obtaining the residual feature r_hat, determine the reconstructed feature y_hat based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu is used as the reconstructed feature y_hat.

[0124] Step S22: input the reconstructed feature y_hat to the synthesis transformation network, and the synthesis transformation network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstruction feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0125] Exemplarily, taking the deployment of a mean prediction network as an example, the processing process at the decoding end may include the following steps:

[0126] Step S31, after obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process.

[0127] Step S32, determine the probability distribution parameter based on the coefficient hyperparameter feature z_hat. For example, perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, input the coefficient hyperparameter feature z_hat to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on the probability distribution parameter p.

[0128] Step S33, after obtaining the second bitstream corresponding to the current image block, decode the second bitstream corresponding to the current image block based on the probability distribution parameter p to obtain the reconstruction feature corresponding to the current image block. For example, the residual feature r_hat corresponding to the current image block can be determined, and the prediction value mu corresponding to the current image block can be determined, and the reconstruction feature y_hat can be determined based on the residual feature r_hat and the mean feature mu, for example, the sum of the residual feature r_hat and the mean feature mu is used as the reconstruction feature y_hat.

[0129] Among them, in order to determine the prediction value mu corresponding to the current image block, after obtaining the coefficient hyperparameter feature z_hat, the coefficient hyperparameter feature z_hat can be input into the mean prediction network, and the mean prediction network performs context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat of the previous image block to obtain the prediction value mu corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z_hat can be input into the mean prediction network, and the mean prediction network obtains the prediction value mu corresponding to the current image block based on the coefficient hyperparameter feature z_hat.

[0130] In order to determine the residual feature r_hat corresponding to the current image block, the probability distribution model corresponding to the probability distribution parameter p can be used to decode the second bitstream to obtain the image quantization feature, dequantize the image quantization feature, and perform feature recovery on the dequantized feature to obtain the residual feature r_hat, or, after dequantizing the image quantization feature, the residual feature r_hat is directly obtained without involving the feature recovery process. Alternatively, after decoding the second bitstream, the decoded feature is feature recovered to obtain the residual feature r_hat, or, after decoding the second bitstream, the residual feature r_hat is directly obtained.

[0131] Step S34: input the reconstruction feature y_hat to the synthesis transformation network, and the synthesis transformation network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstruction feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0132] Embodiment 5: Embodiment 1, Embodiment 2, Embodiment 3 and Embodiment 4 all involve a synthetic transformation network. The implementation of this embodiment is the same as that of Embodiment 1, Embodiment 2, Embodiment 3 or Embodiment 4, and the repeated contents are not repeated. The synthetic transformation network involved therein is described in detail below.

[0133] An example of this synthetic transformer network can be found in Fig. 6AAs shown, the synthetic transformation network may include a residual layer, a deconvolution layer, a cropping layer, a residual activation layer, a deconvolution layer, a cropping layer, a residual activation layer, a deconvolution layer, an attention model, a cropping layer, a residual activation layer, a deconvolution layer, a cropping layer, etc. The input feature of the synthetic transformation network may be a reconstruction feature y_hat, and the output feature of the synthetic transformation network may be a reconstructed image block x_hat. That is, after the reconstruction feature y_hat is input to the synthetic transformation network, it is processed by the above network layers in sequence to obtain a reconstructed image block x_hat.

[0134] For example, an example of a residual layer in a synthetic transformation network can be found in Figure 6B As shown, the residual layer may include a convolution layer, a rectified linear unit (ReLU), a convolution layer, and a stacking layer (used to add two features) in sequence. The first convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer may be selected. The second convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer may be selected.

[0135] For example, an example of a residual activation layer in a synthetic transformation network can be found in Figure 6C As shown, the residual activation layer may include an activation layer (such as a LeakyRelu activation layer), a convolution layer, an activation layer (such as a tanh activation layer), a multiplication layer (for multiplying two features), and an overlay layer (for adding two features). The convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no limitation to this. For example, a 1x1 convolution layer may be selected.

[0136] For example, the attention model in the synthetic transformation network can be a residual non-local attention block (RNAB), that is, RNAB is introduced as an attention model, RNAB is used to obtain non-local information of the image, and uses the non-local information as attention weight to improve decoding performance. An example of RNAB can be found in Fig.6D As shown, RNAB can include residual blocks, residual blocks, downsampling convolution layers (such as 2x downsampling convolution layers, such as 3x3 convolution layers), residual blocks, residual blocks, upsampling convolution layers (such as 2x upsampling convolution layers, such as 3x3 convolution layers), residual blocks, residual blocks, convolution layers (such as 3x3 convolution layers), S-shaped activation functions (Sigmoid), residual blocks, residual blocks, residual blocks, multiplication layers (for realizing feature multiplication) and superposition layers (for realizing feature addition) in sequence.

[0137] For the residual block, the residual block can be found in Figure 6B As shown, the residual block is Figure 6B Similar to the residual layer in , it can also include a convolutional layer, a rectified linear unit (ReLU) (such as a leaky rectified linear unit LeakyReLU), a convolutional layer, and a stacking layer (for adding features). In this way, the input features of the residual block are extracted by the convolutional layer, the leaky rectified linear unit, and the convolutional layer, and then added to the input features of the residual block to obtain the final output features.

[0138] The sub-module consisting of a residual block, a residual block, a downsampling convolution layer, a residual block, a residual block, an upsampling convolution layer, a residual block, a residual block, a convolution layer and an S-shaped activation function can be a non-local attention extraction sub-module, and the sub-module consisting of 3 residual blocks can be a feature extraction sub-module. The input features of RNAB are multiplied by the output of the feature extraction sub-module after the non-local attention extraction sub-module generates attention weights, and are added to the input features of RNAB to obtain the final output features of RNAB.

[0139] Embodiment 6: Embodiment 1, Embodiment 2, Embodiment 3 and Embodiment 4 all involve a synthetic transformation network. The implementation of this embodiment is the same as that of Embodiment 1, Embodiment 2, Embodiment 3 or Embodiment 4, and the repeated contents are not repeated. The synthetic transformation network involved therein is described in detail below.

[0140] The synthetic transformation network may include at least an attention module, and the attention module may be a cascaded hybrid attention module or a parallel hybrid attention module. Of course, in addition to the attention module, the synthetic transformation network may also include other network layers. In this embodiment, the structure of the synthetic transformation network is not limited, as long as the synthetic transformation network includes the attention module.

[0141] In a possible implementation, in addition to the attention module, the synthetic transformation network may also include at least one deconvolution layer. Of course, in addition to the attention module and at least one deconvolution layer, the synthetic transformation network may also include other network layers, which is not limited. When the synthetic transformation network includes at least one deconvolution layer, the attention module may be located behind any deconvolution layer. Alternatively, the attention module may be located behind each deconvolution layer; wherein the attention modules located behind different deconvolution layers may be exactly the same, or the attention modules located behind different deconvolution layers may not be exactly the same. Alternatively, the attention module may be located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers); wherein the attention modules located behind different deconvolution layers may be exactly the same, or the attention modules located behind different deconvolution layers may not be exactly the same.

[0142] In a possible implementation, in addition to the attention module, the synthetic transformation network may include at least one of the following network layers: residual layer, first deconvolution layer, first cropping layer, first residual activation layer, second deconvolution layer, second cropping layer, second residual activation layer, third deconvolution layer, third cropping layer, third residual activation layer, fourth deconvolution layer, fourth cropping layer, that is, the synthetic transformation network may include some or all of the above network layers. Of course, in addition to the above network layers, the synthetic transformation network may also include other network layers, which is not limited, and the above network layers are used as an example for illustration.

[0143] See also Fig. 7A As shown, it is an example of a synthetic transformation network, that is, the synthetic transformation network may sequentially include a residual layer, a deconvolution layer 1 (i.e., the first deconvolution layer), a cropping layer 1 (i.e., the first cropping layer), a residual activation layer 1 (i.e., the first residual activation layer), a deconvolution layer 2 (i.e., the second deconvolution layer), a cropping layer 2 (i.e., the second cropping layer), a residual activation layer 2 (i.e., the second residual activation layer), a deconvolution layer 3 (i.e., the third deconvolution layer), a cropping layer 3 (i.e., the third cropping layer), a residual activation layer 3 (i.e., the third residual activation layer), a deconvolution layer 4 (i.e., the fourth deconvolution layer), and a cropping layer 4 (i.e., the fourth cropping layer). Exemplarily, the input feature of the synthetic transformation network may be a reconstruction feature y_hat, and the output feature of the synthetic transformation network may be a reconstructed image block x_hat, that is, after the reconstruction feature y_hat is input to the synthetic transformation network, the synthetic transformation network may obtain a reconstructed image block x_hat after being processed by the above network layers in sequence, and output the reconstructed image block x_hat.

[0144] exist Fig. 7A, three attention model positions are shown, namely, attention model position 1, attention model position 2, and attention model position 3. Exemplarily, if the synthetic transformation network includes only one attention module (i.e., a cascaded hybrid attention module or a parallel hybrid attention module), the attention module can be located behind the deconvolution layer 1, that is, the attention module is located at the attention model position 1. Alternatively, the attention module can be located behind the deconvolution layer 2, that is, the attention module is located at the attention model position 2. Alternatively, the attention module can be located behind the deconvolution layer 3, that is, the attention module is located at the attention model position 3, for example, the attention module is placed behind the deconvolution layer 3. Of course, the above are just a few examples of the positions of the attention modules, and there is no limitation to this, and they can be located at any position of the synthetic transformation network.

[0145] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 1, an attention module 2 located behind the deconvolution layer 2, and an attention module 3 located behind the deconvolution layer 3, that is, the attention module 1 is located at the attention model position 1, the attention module 2 is located at the attention model position 2, and the attention module 3 is located at the attention model position 3. Of course, the attention module may also be located at other positions of the synthetic transformation network, and there is no limitation on this.

[0146] Among them, attention module 1, attention module 2 and attention module 3 can be completely the same, that is, attention module 1, attention module 2 and attention module 3 can adopt the same network structure. Alternatively, attention module 1, attention module 2 and attention module 3 may not be completely the same, for example, the network structure of attention module 1 may be the same as the network structure of attention module 2, and the network structure of attention module 1 may be different from the network structure of attention module 3; or, the network structure of attention module 1 may be the same as the network structure of attention module 3, and the network structure of attention module 1 may be different from the network structure of attention module 2; or, the network structure of attention module 2 may be the same as the network structure of attention module 3, and the network structure of attention module 2 may be different from the network structure of attention module 1; or, the network structure of attention module 1 may be different from the network structure of attention module 2, and the network structure of attention module 1 may be different from the network structure of attention module 2, and the network structure of attention module 2 may be different from the network structure of attention module 3.

[0147] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 1, and an attention module 2 located behind the deconvolution layer 2, that is, the attention module 1 is located at the attention model position 1, and the attention module 2 is located at the attention model position 2. Of course, the attention module may also be located at other positions of the synthetic transformation network, and there is no limitation on this.

[0148] Among them, attention module 1 and attention module 2 can be completely the same, for example, attention module 1 and attention module 2 can use the same network structure. Alternatively, attention module 1 and attention module 2 can be not completely the same, for example, the network structure of attention module 1 can be different from the network structure of attention module 2.

[0149] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 1, and an attention module 2 located behind the deconvolution layer 3, that is, the attention module 1 is located at the attention model position 1, and the attention module 2 is located at the attention model position 3. Of course, the attention module may also be located at other positions of the synthetic transformation network, and there is no limitation on this.

[0150] Among them, attention module 1 and attention module 2 can be completely the same, for example, attention module 1 and attention module 2 can use the same network structure. Alternatively, attention module 1 and attention module 2 can be not completely the same, for example, the network structure of attention module 1 can be different from the network structure of attention module 2.

[0151] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 2, and an attention module 2 located behind the deconvolution layer 3, that is, the attention module 1 is located at the attention model position 2, and the attention module 2 is located at the attention model position 3. Of course, the attention module may also be located at other positions of the synthetic transformation network, and there is no limitation on this.

[0152] Among them, attention module 1 and attention module 2 can be completely the same, for example, attention module 1 and attention module 2 can use the same network structure. Alternatively, attention module 1 and attention module 2 can be not completely the same, for example, the network structure of attention module 1 can be different from the network structure of attention module 2.

[0153] For the residual layer in the synthetic transformation network, an example of the residual layer can be found in Figure 6B As shown, the residual layer may include a convolution layer, a rectified linear unit (such as a leaky rectified linear unit LeakyReLU), a convolution layer, and a stacking layer in sequence, that is, after the input features are extracted by the convolution layer, the rectified linear unit, and the convolution layer, they are added to the input features to obtain the final output features. Among them, the first convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this, such as a 3x3 convolution layer. In addition, the second convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this, such as a 3x3 convolution layer can be selected.

[0154] For the residual activation layer in the synthetic transformation network, an example of the residual activation layer can be found in Figure 6C As shown, the residual activation layer may include an activation layer (such as a LeakyRelu activation layer), a convolution layer, an activation layer (such as a tanh activation layer), a multiplication layer, and a stacking layer in sequence, and there is no restriction on the structure of the residual activation layer. Among them, the convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this, such as a 1x1 convolution layer.

[0155] Example 7: This example further describes the attention module involved in the above examples. In Example 6, the attention module in the synthetic transformation network is involved. The attention module can be a cascaded hybrid attention module. The RNAB is replaced by the cascaded hybrid attention module to greatly reduce the computational complexity of decoding while ensuring the decoding performance as much as possible. Exemplarily, the cascaded hybrid attention module can extract the attention weight of the input feature, and determine the output feature of the cascaded hybrid attention module based on the attention weight and the input feature.

[0156] Exemplarily, the cascaded hybrid attention module may include a first attention submodule (the first attention submodule may also be referred to as a transformer attention submodule, and the transformer attention submodule is recorded as a Transformer-basedAttention Module) and a second attention submodule (the second attention submodule may also be referred to as an upgraded convolution submodule, and is recorded as a ConvNext Block). The first attention submodule and the second attention submodule may be two serial submodules, that is, the output feature of the first attention submodule is the input feature of the second attention submodule. For example, see Figure 7B As shown in the figure, it is a schematic diagram of a cascaded hybrid attention module, where the input feature (or input image) of the cascaded hybrid attention module is used as the first attention submodule ( Figure 7B The input features of the first attention submodule are processed by the first attention submodule based on the input features to obtain the output features of the first attention submodule. The output features of the first attention submodule are used as the input features of the second attention submodule ( Figure 7B The input features of the second attention submodule are processed by the second attention submodule based on the input features to obtain the output features of the second attention submodule. The output features of the second attention submodule are used as the output features of the cascaded hybrid attention module, that is, the final output features.

[0157] Exemplarily, after the input feature of the first attention submodule is input to the first attention submodule, the first attention submodule performs a first processing on the input feature to obtain the output feature of the first attention submodule. The first processing may include but is not limited to: at least one of layer normalization, convolution operation and dimension transformation operation, and there is no limitation on the first processing, which is related to the network structure of the first attention submodule, and the network structure of the first attention submodule can be arbitrarily configured.

[0158] Exemplarily, after the input features of the second attention submodule are input to the second attention submodule, the second attention submodule performs a second processing on the input features to obtain the output features of the second attention submodule. The second processing may include but is not limited to: at least one of a depthwise separable convolution operation, layer normalization, a multi-layer perception operation, a linear operation, an activation operation, a downsampling operation, a residual convolution operation, and an upsampling operation. There is no limitation on the second processing, which is related to the network structure of the second attention submodule, and the network structure of the second attention submodule can be arbitrarily configured.

[0159] For example, the second processing may include at least one of a depthwise separable convolution operation, a layer normalization, and a multi-layer perception operation. Alternatively, the second processing may include at least one of a layer normalization, a linear operation, and an activation operation. Alternatively, the second processing may include at least one of a downsampling operation, a residual convolution operation, an upsampling operation, and an activation operation.

[0160] Example 8: This example further describes the first attention submodule involved in the cascade hybrid attention module based on the above examples. In Example 6, it involves an attention module in a synthetic transformation network, which can be a parallel hybrid attention module. The RNAB is replaced by the parallel hybrid attention module to greatly reduce the computational complexity of decoding while ensuring the decoding performance as much as possible. Exemplarily, the parallel hybrid attention module can extract the attention weight of the input feature, and determine the output feature of the parallel hybrid attention module based on the attention weight and the input feature.

[0161] Exemplarily, the parallel hybrid attention module may include a first attention submodule (the first attention submodule may also be referred to as a transformer attention submodule) and a second attention submodule (the second attention submodule may also be referred to as an upgraded convolution submodule). The first attention submodule and the second attention submodule may be two parallel submodules, for example, the input features of the first attention submodule are parallel to the input features of the second attention submodule, and the output features of the first attention submodule are parallel to the output features of the second attention submodule. For example, see Figure 7CAs shown in FIG. 1 , it is a schematic diagram of a parallel hybrid attention module. The input features (or input images) of the parallel hybrid attention module need to be split into channels, that is, the input features of the parallel hybrid attention module are split into two sub-features (or two sub-images) in the channel dimension. These two sub-features are respectively used as the first attention submodule ( Figure 7B The input features of the transformer attention submodule (taking the transformer attention submodule as an example) and the second attention submodule ( Figure 7B The input features of the upscaled convolution submodule are taken as an example. The two subfeatures can be of equal size or of unequal size. Obviously, since the two subfeatures are the two subfeatures after the channel splitting, the two subfeatures can be parallel, that is, the two subfeatures are output in parallel after the channel splitting, that is, the input features of the first attention submodule and the input features of the second attention submodule are parallel.

[0162] After the channel splitting process, a sub-feature after the channel splitting is used as the input feature of the first attention sub-module, and the input feature of the first attention sub-module passes through the first attention sub-module, and the first attention sub-module processes the input feature to obtain the output feature of the first attention sub-module. In addition, another sub-feature after the channel splitting is used as the input feature of the second attention sub-module, and the input feature of the second attention sub-module passes through the second attention sub-module, and the second attention sub-module processes the input feature to obtain the output feature of the second attention sub-module.

[0163] Then, the output features of the first attention submodule and the output features of the second attention submodule can be spliced ​​in the channel dimension to obtain the output features of the parallel mixed attention module (that is, the spliced ​​features are used as the output features of the parallel mixed attention module), that is, the output features of the two submodules are spliced ​​in the channel dimension to obtain the output features of the parallel mixed attention module. Obviously, since the output features of the two submodules need to be spliced ​​in the channel dimension, the output features of the two submodules are parallel, that is, the output features of the first attention submodule and the output features of the second attention submodule are parallel.

[0164] Exemplarily, after the input feature of the first attention submodule is input to the first attention submodule, the first attention submodule performs a first processing on the input feature to obtain the output feature of the first attention submodule. The first processing may include but is not limited to: at least one of layer normalization, convolution operation and dimension transformation operation, and there is no limitation on the first processing, which is related to the network structure of the first attention submodule, and the network structure of the first attention submodule can be arbitrarily configured.

[0165] Exemplarily, after the input features of the second attention submodule are input to the second attention submodule, the second attention submodule performs a second processing on the input features to obtain the output features of the second attention submodule. The second processing may include but is not limited to: at least one of a depthwise separable convolution operation, layer normalization, a multi-layer perception operation, a linear operation, an activation operation, a downsampling operation, a residual convolution operation, and an upsampling operation. There is no limitation on the second processing, which is related to the network structure of the second attention submodule, and the network structure of the second attention submodule can be arbitrarily configured.

[0166] For example, the second processing may include at least one of a depthwise separable convolution operation, a layer normalization, and a multi-layer perception operation. Alternatively, the second processing may include at least one of a layer normalization, a linear operation, and an activation operation. Alternatively, the second processing may include at least one of a downsampling operation, a residual convolution operation, an upsampling operation, and an activation operation.

[0167] Embodiment 9: In Embodiment 7 and Embodiment 8, the cascade hybrid attention module and the parallel hybrid attention module both include a first attention submodule, and the first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; the query vector, the key vector, and the value vector are respectively dimensionally transformed to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; the attention weight is determined based on the dimensionally transformed query vector and the dimensionally transformed key vector, and the correction feature corresponding to the input feature is determined based on the attention weight and the dimensionally transformed value vector; the output feature of the first attention submodule is determined based on the input feature and the correction feature.

[0168] See also Fig. 8A As shown in FIG. 1 , a schematic diagram of the structure of the first attention submodule is shown. This is only an example of the first attention submodule, and the structure of the first attention submodule is not limited. Fig. 8A The first attention submodule of is taken as an example to illustrate.

[0169] The first attention submodule may include a layer normalization layer, which may perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer normalized features. Layer normalization is a neural network regularization technique used to normalize the input features of each layer of the neural network to improve training and generalization performance. The layer normalization operation may include calculating the mean and variance, standardization, scaling, and translation, so that the output of each hidden unit is within a relatively small range and each hidden unit has a similar distribution in the entire data set.

[0170] After obtaining the layer-normalized features, a three-way convolution operation can be performed on the layer-normalized features to obtain a query vector Q, a key vector K, and a value vector V. For example, the layer-normalized features can be convolved through two convolutional layers to obtain a query vector Q. The first convolutional layer can be a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, and there is no restriction on this. For example, a 1x1 convolutional layer can be selected. The second convolutional layer can be a 3x3 convolutional layer, or a 5x5 convolutional layer, or a 7x7 convolutional layer, and there is no restriction on this. For example, a 3x3 convolutional layer can be selected. The layer-normalized features can be convolved through two convolutional layers to obtain a key vector K. The first convolutional layer can be a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, and there is no restriction on this. For example, a 1x1 convolutional layer can be selected. The second convolution layer can be a 3x3 convolution layer, or a 5x5 convolution layer, or a 7x7 convolution layer, such as a 3x3 convolution layer. The layer-normalized features can be convolved through two convolution layers to obtain a value vector V. The first convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 1x1 convolution layer. The second convolution layer can be a 3x3 convolution layer, or a 5x5 convolution layer, or a 7x7 convolution layer, such as a 3x3 convolution layer.

[0171] The query vector Q, the key vector K and the value vector V are respectively subjected to dimension transformation operations (i.e., R operations) to obtain the transformed query vector Q, the transformed key vector K and the transformed value vector V. The purpose of the dimension transformation operation is to transform the transformed tensor into the dimensional form required by the multi-head self-attention mechanism. The multi-head self-attention mechanism is a technology widely used in natural language processing (NLP) tasks. This technology uses the self-attention mechanism to build connections between context words, thereby reconstructing the representation of the target word based on the context. The multi-head attention mechanism is a combination of multiple groups of self-attention components, which can learn many different types of context influences. By passing the outputs of multiple self-attentions through the parameter matrix to obtain a new output, the multi-head attention mechanism can capture a larger and wider range of correlation features and increase the expressive power of the model.

[0172] The attention weight is determined based on the transformed query vector Q and the transformed key vector K. For example, the transformed query vector Q is matrix multiplied with the transformed key vector K, and the result of the matrix multiplication is passed through a soft maximum function (Softmax) to obtain the attention weight, and there is no restriction on the Softmax processing process.

[0173] The modified features corresponding to the input features are determined based on the attention weights and the value vector V after dimension transformation. For example, the value vector V after dimension transformation is matrix multiplied with the attention weight, the result of the matrix multiplication is dimensionally transformed (i.e., R operation), and the features after the dimension transformation operation are convolved to obtain the modified features corresponding to the input features. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer. For example, a 1x1 convolution layer can be selected.

[0174] The output feature of the first attention submodule is determined based on the input feature and the correction feature of the first attention submodule. For example, the output feature of the first attention submodule is obtained by performing matrix addition on the input feature and the correction feature.

[0175] At this point, the processing process of the first attention submodule is completed, and the output features of the first attention submodule are obtained.

[0176] Embodiment 10: This embodiment further describes the first attention submodule involved in the cascade hybrid attention module on the basis of the above embodiments. In Embodiment 7 and Embodiment 8, the cascade hybrid attention module and the parallel hybrid attention module both include a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are window-divided to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to multiple small-size features are concatenated to obtain a value vector. Perform dimension transformation operations on the query vector, key vector and value vector respectively to obtain a dimensionally transformed query vector, a dimensionally transformed key vector and a dimensionally transformed value vector; determine an attention weight based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determine a correction feature based on the attention weight and the dimensionally transformed value vector; determine an output feature of the first attention submodule based on the input feature and the correction feature.

[0177] For example, in Fig. 8A On the basis of the first attention submodule, a window partitioning layer can be added after the layer normalization layer. There is no restriction on the structure of the first attention submodule. Fig. 8A shown.

[0178] Exemplarily, the first attention submodule may include a layer normalization layer, and the layer normalization layer may be used to perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer normalized features.

[0179] The layer-normalized features can be window-partitioned through the window partitioning layer to obtain multiple small-size features. For example, the layer-normalized features can be divided into multiple small-size features, the width of each small-size feature is wi, and the height of each small-size feature is hi. The widths of different small-size features can be the same or different, and the heights of different small-size features can be the same or different. By dividing the features into multiple small-size features for subsequent operations, the complexity can be reduced.

[0180] For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature and small-size value feature corresponding to the small-size feature. For example, the small-size feature is convolved through two convolutional layers to obtain the small-size query feature, and the first convolutional layer is a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, such as a 1x1 convolutional layer. The second convolutional layer is a 3x3 convolutional layer, or a 5x5 convolutional layer, or a 7x7 convolutional layer, such as a 3x3 convolutional layer. The small-size feature is convolved through two convolutional layers to obtain the small-size key feature, and the first convolutional layer is a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, such as a 1x1 convolutional layer. The second convolution layer is a 3x3 convolution layer, or a 5x5 convolution layer, or a 7x7 convolution layer, such as a 3x3 convolution layer. The small-size feature is convolved through two convolution layers to obtain a small-size value feature. The first convolution layer is a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 1x1 convolution layer. The second convolution layer can be a 3x3 convolution layer, or a 5x5 convolution layer, or a 7x7 convolution layer, such as a 3x3 convolution layer.

[0181] The small-size query features corresponding to all small-size features can be spliced ​​to obtain a query vector Q, and there is no restriction on this feature splicing process. In addition, the small-size key features corresponding to all small-size features can be spliced ​​to obtain a key vector K. In addition, the small-size value features corresponding to all small-size features can be spliced ​​to obtain a value vector V.

[0182] The query vector Q, the key vector K and the value vector V may be subjected to a dimension transformation operation (ie, an R operation) respectively to obtain the query vector Q after dimension transformation, the key vector K after dimension transformation and the value vector V after dimension transformation.

[0183] The attention weight can be determined based on the transformed query vector Q and the transformed key vector K. For example, the transformed query vector Q and the transformed key vector K are matrix multiplied, and the result of the matrix multiplication is passed through a soft maximum function (Softmax) to obtain the attention weight, and there is no restriction on the Softmax processing process.

[0184] The modified features corresponding to the input features can be determined based on the attention weights and the value vector V after dimension transformation. For example, the value vector V after dimension transformation is matrix multiplied with the attention weight, the result of the matrix multiplication is dimensionally transformed (i.e., R operation), and the features after the dimension transformation operation are convolved to obtain the modified features corresponding to the input features. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer. For example, a 1x1 convolution layer can be selected.

[0185] The output feature of the first attention submodule may be determined based on the input feature and the correction feature of the first attention submodule. For example, matrix addition may be performed on the input feature and the correction feature to obtain the output feature of the first attention submodule.

[0186] At this point, the processing process of the first attention submodule is completed, and the output features of the first attention submodule are obtained.

[0187] Embodiment 11: This embodiment further describes the first attention submodule involved in the cascaded hybrid attention module on the basis of the above embodiments. In Embodiment 7 and Embodiment 8, the cascaded hybrid attention module and the parallel hybrid attention module both include a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are window-divided to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature. The small-size query features corresponding to multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to multiple small-size features are concatenated to obtain a value vector. An attention weight is determined based on the query vector and the key vector, an intermediate feature is determined based on the attention weight and the value vector, and a layer normalization is performed on the intermediate feature to obtain a correction feature corresponding to the input feature. An output feature of the first attention submodule is determined based on the input feature and the correction feature.

[0188] See also Figure 8B As shown in FIG. 1 , a schematic diagram of the structure of the first attention submodule is shown. This is only an example of the first attention submodule, and the structure of the first attention submodule is not limited. Figure 8B The first attention submodule of is taken as an example to illustrate.

[0189] Exemplarily, the first attention submodule may include a layer normalization layer, and the layer normalization layer may be used to perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer normalized features.

[0190] The first attention sub-module may include a window division layer, and the window division layer may be used to perform window division on the layer-normalized features to obtain multiple small-size features. For example, the layer-normalized features are divided into multiple small-size features through the window division layer, and the width of each small-size feature is wi, and the height of each small-size feature is hi. The widths of different small-size features may be the same or different, and the heights of different small-size features may be the same or different.

[0191] For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature and small-size value feature corresponding to the small-size feature. For example, the small-size query feature is obtained by performing a convolution operation on the small-size feature through a convolution layer (such as a 1x1 convolution layer and a 3x3 convolution layer). The small-size key feature is obtained by performing a convolution operation on the small-size feature through a convolution layer (such as a 1x1 convolution layer and a 3x3 convolution layer). The small-size value feature is obtained by performing a convolution operation on the small-size feature through a convolution layer (such as a 1x1 convolution layer and a 3x3 convolution layer). The small-size query features corresponding to all small-size features are concatenated to obtain a query vector Q, the small-size key features corresponding to all small-size features are concatenated to obtain a key vector K, and the small-size value features corresponding to all small-size features are concatenated to obtain a value vector V.

[0192] The attention weight is determined based on the query vector Q and the key vector K. For example, the query vector Q is matrix multiplied with the key vector K, and the result of the matrix multiplication is passed through the soft maximum function (Softmax) to obtain the attention weight.

[0193] The intermediate feature is determined based on the attention weight and the value vector V. For example, matrix multiplication is performed on the attention weight and the value vector V, and the result of the matrix multiplication is used as the intermediate feature, that is, the intermediate feature corresponding to the input feature.

[0194] The intermediate feature is layer-normalized to obtain a correction feature corresponding to the input feature. That is, after the intermediate feature is layer-normalized, the feature after layer-normalization is used as the correction feature corresponding to the input feature.

[0195] For the obtained attention weight and value vector V, in one example, the correction feature corresponding to the input feature can be directly obtained based on the attention weight and the value vector V. For example, matrix multiplication is performed on the attention weight and the value vector V, and the result of the matrix multiplication is used as the correction feature corresponding to the input feature. That is, Figure 8B The second layer in is the normalization layer.

[0196] The output feature of the first attention submodule may be determined based on the input feature of the first attention submodule and the correction feature. For example, matrix addition may be performed on the input feature and the correction feature to obtain the output feature of the first attention submodule.

[0197] At this point, the processing process of the first attention submodule is completed, and the output features of the first attention submodule are obtained.

[0198] Example 12: This example further describes the first attention submodule involved in the cascaded hybrid attention module on the basis of the above examples. In Examples 7 and 8, the cascaded hybrid attention module and the parallel hybrid attention module both include a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; the attention weight is determined based on the query vector and the key vector, and the intermediate features are determined based on the attention weight and the value vector, and the intermediate features are layer-normalized to obtain the corrected features corresponding to the input features; the output features of the first attention submodule are determined based on the input features and the corrected features of the first attention submodule.

[0199] For example, in Figure 8B Based on the first attention submodule, the window partitioning layer after the layer normalization layer can be removed. There is no restriction on the structure of the first attention submodule. Figure 8B shown.

[0200] Exemplarily, the first attention submodule may include a layer normalization layer, and the layer normalization layer may be used to perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer normalized features.

[0201] The first attention submodule may include a convolutional layer, and a three-way convolution operation may be performed on the layer-normalized features to obtain a query vector Q, a key vector K, and a value vector V. For example, the query vector Q may be obtained by convolving the layer-normalized features through a convolutional layer (such as a 1x1 convolutional layer and a 3x3 convolutional layer). The key vector K may be obtained by convolving the layer-normalized features through a convolutional layer (such as a 1x1 convolutional layer and a 3x3 convolutional layer). The value vector V may be obtained by convolving the layer-normalized features through a convolutional layer (such as a 1x1 convolutional layer and a 3x3 convolutional layer).

[0202] The attention weight is determined based on the query vector Q and the key vector K. For example, the query vector Q is matrix multiplied with the key vector K, and the result of the matrix multiplication is passed through the soft maximum function (Softmax) to obtain the attention weight.

[0203] The intermediate feature is determined based on the attention weight and the value vector V. For example, matrix multiplication is performed on the attention weight and the value vector V, and the result of the matrix multiplication is used as the intermediate feature, that is, the intermediate feature corresponding to the input feature.

[0204] The intermediate feature is layer-normalized to obtain a correction feature corresponding to the input feature. That is, after the intermediate feature is layer-normalized, the feature after layer-normalization is used as the correction feature corresponding to the input feature.

[0205] The output feature of the first attention submodule may be determined based on the input feature of the first attention submodule and the correction feature. For example, matrix addition may be performed on the input feature and the correction feature to obtain the output feature of the first attention submodule.

[0206] For the obtained attention weight and value vector V, in one example, the correction feature corresponding to the input feature can be directly obtained based on the attention weight and the value vector V. For example, matrix multiplication is performed on the attention weight and the value vector V, and the result of the matrix multiplication is used as the correction feature corresponding to the input feature. That is, Figure 8B The second layer in is the normalization layer.

[0207] At this point, the processing process of the first attention submodule is completed, and the output features of the first attention submodule are obtained.

[0208] In Example 9 to Example 12, several examples of the first attention sub-module are given. In this embodiment, there is no limitation on the structure of the first attention sub-module, and the input features can be first processed to obtain the output features.

[0209] Example 13: This example further describes the second attention submodule involved in the cascaded hybrid attention module on the basis of the above examples. In Examples 7 and 8, the cascaded hybrid attention module and the parallel hybrid attention module both include a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a depth-separable convolution operation is performed on the input features of the second attention submodule to obtain the convolved features (i.e., the features after the depth-separable convolution operation); the convolved features are layer-normalized to obtain the layer-normalized features; the layer-normalized features are subjected to a multi-layer perception operation to obtain the multi-layer perceived features; and the output features of the second attention submodule are determined based on the input features and the multi-layer perceived features.

[0210] See also Figure 8C As shown in FIG. 1 , a schematic diagram of the structure of the second attention submodule is shown. This is only an example of the second attention submodule, and the structure of the second attention submodule is not limited. Figure 8C The second attention submodule of is taken as an example to illustrate.

[0211] Exemplarily, the second attention sub-module may include a depthwise separable convolutional layer, which may be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without limitation. For example, a 7x7 convolutional layer may be selected, and a depthwise separable convolution operation may be performed on the input features of the second attention sub-module through the depthwise separable convolutional layer, without limitation on the convolution operation, to obtain the features after convolution (i.e., the features after the depthwise separable convolution operation).

[0212] The second attention submodule may include a layer normalization layer, which can perform layer normalization on the convolutional features to obtain layer normalized features. The layer normalization operation includes calculating the mean and variance, standardization, scaling, and translation, so that the output of each hidden unit is within a relatively small range and each hidden unit has a similar distribution in the data set.

[0213] The second attention submodule may include a multi-layer perceptron, which may perform multi-layer perceptron operations on the layer-normalized features to obtain multi-layer perceptual features. Fig.8DAs shown, it is a schematic diagram of the structure of a multilayer perceptron. The multilayer perceptron may include a linear layer, an activation layer, a random dropout layer, a linear layer, and a random dropout layer. Of course, this is just an example of a multilayer perceptron, and there is no restriction on the structure of the multilayer perceptron. Based on this, the features after layer normalization are sequentially passed through a linear layer, an activation layer, a random dropout layer, a linear layer, and a random dropout layer to obtain the features after multilayer perception. Among them, the activation layer may be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, and there is no restriction on the type of the activation layer. The random dropout layer may be a DropOut layer, and there is no restriction on the type of the random dropout layer.

[0214] The output features of the second attention submodule are determined based on the input features of the second attention submodule and the features after multi-layer perception. For example, matrix addition is performed on the input features and the features after multi-layer perception to obtain the output features.

[0215] At this point, the processing process of the second attention submodule is completed, and the output features of the second attention submodule are obtained.

[0216] Example 14: This example further describes the second attention submodule involved in the cascaded hybrid attention module on the basis of the above examples. In Examples 7 and 8, the cascaded hybrid attention module and the parallel hybrid attention module both include a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule may be layer-normalized to obtain layer-normalized features; the layer-normalized features may be subjected to a first linear operation to obtain the features after the first linear operation; the features after the first linear operation may be activated to obtain the activated features; the activated features may be subjected to a second linear operation to obtain the features after the second linear operation; the output features of the second attention submodule may be determined based on the input features of the second attention submodule and the features after the second linear operation.

[0217] See also Fig. 8E As shown in FIG. 1 , a schematic diagram of the structure of the second attention submodule is shown. This is only an example of the second attention submodule, and the structure of the second attention submodule is not limited. Fig. 8E The second attention submodule of is taken as an example to illustrate.

[0218] Exemplarily, the second attention submodule may include a layer normalization layer, and the layer normalization layer may be used to perform layer normalization on the input features of the second attention submodule to obtain layer normalized features. The second attention submodule may include a linear layer 1, and the linear layer 1 may be used to perform a linear operation on the layer normalized features to obtain linearly operated features.

[0219] The second attention submodule may include an activation layer, and the activation layer may be used to perform an activation operation on the features after the linear operation to obtain the activated features. The activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and there is no restriction on the type of the activation layer, such as selecting a relu layer as the activation layer. The second attention submodule may include a linear layer 2, and the linear layer 2 may be used to perform a linear operation on the features after the activation to obtain the features after the linear operation.

[0220] The output features of the second attention submodule are determined based on the input features of the second attention submodule and the features after the linear operation. For example, matrix addition is performed on the input features and the features after the linear operation to obtain the output features.

[0221] At this point, the processing process of the second attention submodule is completed, and the output features of the second attention submodule are obtained.

[0222] Example 15: This example further describes the second attention submodule involved in the cascaded hybrid attention module on the basis of the above examples. In Examples 7 and 8, the cascaded hybrid attention module and the parallel hybrid attention module both include a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule are downsampled to obtain the downsampled features; the downsampled features are subjected to residual convolution to obtain the residual convolution features; the residual convolution features are upsampled to obtain the upsampled features; the upsampled features are activated to obtain the activated features; the input features are subjected to residual convolution to obtain the convolution features after residual convolution. The output features of the second attention submodule are determined based on the input features, the activated features, and the convolution features.

[0223] See also Figure 8F As shown in FIG. 1 , a schematic diagram of the structure of the second attention submodule is shown. This is only an example of the second attention submodule, and the structure of the second attention submodule is not limited. Figure 8F The second attention submodule of is taken as an example to illustrate.

[0224] Exemplarily, the second attention submodule may include a downsampling layer, which may be a convolutional layer with a kernel size of 3x3 and a stride of 2, or a convolutional layer with a kernel size of 3x3 and a stride of 4, or a convolutional layer with a kernel size of 4x4 and a stride of 2, or a convolutional layer with a kernel size of 4x4 and a stride of 4, or a convolutional layer with a kernel size of 5x5 and a stride of 2, or a convolutional layer with a kernel size of 5x5 and a stride of 4, or the downsampling layer may also be an inverse pixel reorganization layer (inverse PixelShuffle), and there is no restriction on the structure of the downsampling layer, such as a convolutional layer with a kernel size of 3x3 and a stride of 2 may be selected as the downsampling layer. On this basis, the input features of the second attention submodule may be downsampled by the downsampling layer, and there is no restriction on the implementation of the downsampling operation, to obtain the downsampled features.

[0225] The second attention submodule may include M residual convolution layers, where M may be a positive integer, such as 1, 2, 3, 4, etc., and the residual convolution operation may be performed on the downsampled features through the M residual convolution layers to obtain the residual convolution features. For example, the residual convolution operation is performed on the downsampled features through the first residual convolution layer, and the result is subjected to the residual convolution operation through the second residual convolution layer, and so on, until the last residual convolution layer outputs the residual convolution features.

[0226] See also Figure 8G As shown, it is a structural schematic diagram of a residual convolution layer, which may include a convolution layer 1, an activation layer and a convolution layer 2. The input features of the residual convolution layer may be convolved by the convolution layer 1 to obtain the features after the convolution processing, the features after the convolution processing may be activated by the activation layer to obtain the features after the activation processing, and the features after the activation processing may be processed by the convolution layer 2 to obtain the features after the convolution processing. Then, the input features of the residual convolution layer and the features after the convolution processing are matrix-added to obtain the output features of the residual convolution layer.

[0227] Convolution layer 1 is a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, or a 5x5 convolution layer, and there is no restriction on this, such as selecting a 3x3 convolution layer. Convolution layer 2 is a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, or a 5x5 convolution layer, and there is no restriction on this, such as selecting a 3x3 convolution layer. The activation layer is a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, and there is no restriction on this, such as selecting a relu layer or a sigmoid layer.

[0228] See also Figure 8HAs shown, it is another structural schematic diagram of the residual convolution layer, which can include a convolution layer and an activation layer. The input features of the residual convolution layer can be convolved by the convolution layer to obtain the features after the convolution process, and the features after the convolution process can be activated by the activation layer to obtain the features after the activation process. Then, the input features of the residual convolution layer and the features after the activation process are matrix-added to obtain the output features of the residual convolution layer.

[0229] The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, a 5x5 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer can be selected. The activation layer can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and there is no restriction on this. For example, a relu layer or a sigmoid layer can be selected.

[0230] The second attention submodule may include an upsampling layer, which may be a deconvolution layer with a kernel size of 3x3 and a stride of 2, or a deconvolution layer with a kernel size of 3x3 and a stride of 4, or a deconvolution layer with a kernel size of 4x4 and a stride of 2, or a deconvolution layer with a kernel size of 4x4 and a stride of 4, or a deconvolution layer with a kernel size of 5x5 and a stride of 2, or a deconvolution layer with a kernel size of 5x5 and a stride of 4, or the upsampling layer may also be a pixel reorganization layer, and there is no restriction on the structure of the upsampling layer, such as a deconvolution layer with a kernel size of 4x4 and a stride of 2 may be selected as the upsampling layer. On this basis, the features after residual convolution (i.e., the features obtained after M residual convolution layers) may be upsampled by the upsampling layer, and there is no restriction on the implementation of the upsampling operation, to obtain the upsampled features.

[0231] The second attention submodule may include an activation layer, which may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. There is no restriction on the structure of the activation layer, such as a relu layer. On this basis, the upsampled features may be activated by the activation layer to obtain activated features.

[0232] The second attention submodule may include N residual convolution layers, where N may be a positive integer, such as 1, 2, 3, 4, etc. N and M may be the same or different. The input features of the second attention submodule may be subjected to residual convolution operations through N residual convolution layers to obtain convolution features after residual convolution. For example, the input features of the second attention submodule are subjected to residual convolution operations through the first residual convolution layer, and the obtained results are subjected to residual convolution operations through the second residual convolution layer, and so on, until the last residual convolution layer outputs the convolution features after residual convolution.

[0233] See also Figure 8G As shown in FIG. 1 , a schematic diagram of the structure of a residual convolution layer is shown. The residual convolution layer may include a convolution layer 1, an activation layer, and a convolution layer 2. Figure 8H As shown, it is another structural schematic diagram of the residual convolution layer. The residual convolution layer may include a convolution layer and an activation layer. The residual convolution layer may be used to perform the residual convolution operation, which will not be described in detail here.

[0234] See also Figure 8F As shown, after obtaining the convolution features after residual convolution (i.e., the features output after N residual convolution layers) and the features after activation (i.e., the features output after the activation layer), the output features of the second attention submodule can be determined based on the input features of the second attention submodule, the features after activation, and the convolution features after residual convolution. For example, the activated features and the convolution features after residual convolution can be matrix multiplied, and the result of the multiplication can be matrix added to the input features of the second attention submodule to obtain the output features of the second attention submodule.

[0235] At this point, the processing process of the second attention submodule is completed, and the output features of the second attention submodule are obtained.

[0236] In Example 13 to Example 15, several examples of the second attention sub-module are given. In this embodiment, there is no limitation on the structure of the second attention sub-module, and the input features can be subjected to a second processing to obtain the output features.

[0237] Embodiment 16: The implementation of this embodiment is the same as that of Embodiment 1, Embodiment 2, Embodiment 3 or Embodiment 4, and the repeated contents are not repeated here. The synthesis transformation network involved therein is described in detail below.

[0238] In Example 1, Example 2, Example 3 and Example 4, a synthetic transformation network is involved, and the synthetic transformation network may include at least an attention module, and the attention module may be a first attention submodule. Of course, in addition to the first attention submodule, the synthetic transformation network may also include other network layers. In this embodiment, there is no restriction on the structure of the synthetic transformation network, as long as the synthetic transformation network includes the first attention submodule. For example, the synthetic transformation network may also include at least one deconvolution layer, and the first attention submodule may be located behind any deconvolution layer. Or, the first attention submodule may be located behind each deconvolution layer. Or, the first attention submodule may be located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers).

[0239] Exemplarily, the first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. In one possible implementation, the structure of the first attention submodule can be shown in Example 9, in another possible implementation, the structure of the first attention submodule can be shown in Example 10, in another possible implementation, the structure of the first attention submodule can be shown in Example 11, and in another possible implementation, the structure of the first attention submodule can be shown in Example 12. Of course, the above are just a few examples of the first attention submodule, and there is no limitation on the structure of the first attention submodule.

[0240] Embodiment 17: The implementation of this embodiment is the same as that of Embodiment 1, Embodiment 2, Embodiment 3 or Embodiment 4, and the repeated contents are not repeated here. The synthesis transformation network involved therein is described in detail below.

[0241] In Example 1, Example 2, Example 3 and Example 4, a synthetic transformation network is involved, and the synthetic transformation network may include at least an attention module, and the attention module may be a second attention submodule. Of course, in addition to the second attention submodule, the synthetic transformation network may also include other network layers. In this embodiment, there is no restriction on the structure of the synthetic transformation network, as long as the synthetic transformation network includes the second attention submodule. For example, the synthetic transformation network may also include at least one deconvolution layer, and the second attention submodule may be located behind any deconvolution layer. Or, the second attention submodule may be located behind each deconvolution layer. Or, the second attention submodule may be located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers).

[0242] Exemplarily, the second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. In one possible implementation, the structure of the second attention submodule can be shown in Example 13, in another possible implementation, the structure of the second attention submodule can be shown in Example 14, and in another possible implementation, the structure of the second attention submodule can be shown in Example 15. Of course, the above are just a few examples of the second attention submodule, and the structure of the second attention submodule is not limited.

[0243] It can be seen from the above technical solutions that in the embodiment of the present application, for the neural network-based coding and decoding technology, a synthetic transformation network with an attention mechanism is proposed, the synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module. When the synthetic transformation network is realized by the cascaded hybrid attention module or the parallel hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, and the decoding performance is improved. While reducing the complexity, the quality of the reconstructed image block is effectively guaranteed, and the encoding performance and decoding performance are improved. By replacing the residual non-local attention module with a hybrid attention module, the computational complexity of the decoder is greatly reduced. For example, the second attention submodule in the hybrid attention module (cascaded hybrid attention module or parallel hybrid attention module) only occupies the computational complexity of 21K floating-point multiplication and addition operations / pixel, and the complexity of the first attention submodule is also much lower than that of the residual non-local attention module. In addition, the second attention submodule plays the role of the attention mechanism in the residual non-local attention module, and due to the introduction of the transformer structure with better performance, it brings excellent performance. The 7*7 convolution operation in the second attention submodule plays the role of introducing non-local information in the residual non-local attention module, and the random inactivation layer in the second attention submodule plays the role of increasing the robustness of the network. Based on the above functions, the cascaded hybrid attention module or the parallel hybrid attention module will not have too much performance loss while reducing the computational complexity.

[0244] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 17 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 17 can be implemented in combination.

[0245] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0246] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The decoding device is applied to the decoding end, and the decoding device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 17, that is, the processing flow of the decoding end.

[0247] For example, in one possible implementation, the decoder is configured to implement:

[0248] Decoding a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block;

[0249] Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode the second bitstream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block;

[0250] Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block;

[0251] Wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules; or, the parallel hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two parallel sub-modules.

[0252] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application. The encoding device is applied to an encoding end, and the encoding device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 17, that is, the processing flow of the encoding end.

[0253] For example, in one possible implementation, the encoder is configured to implement:

[0254] Decoding a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block;

[0255] Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode the second bitstream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block;

[0256] Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block;

[0257] Wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules; or, the parallel hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two parallel sub-modules.

[0258] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig. 9A As shown. It includes: a processor 901 and a machine-readable storage medium 902, the machine-readable storage medium 902 stores machine-executable instructions that can be executed by the processor 901; the processor 901 is used to execute the machine-executable instructions to implement the decoding method of the above embodiments 1-17 of the present application.

[0259] Based on the same application concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig. 9B As shown. It includes: a processor 911 and a machine-readable storage medium 912, the machine-readable storage medium 912 stores machine-executable instructions that can be executed by the processor 911; the processor 911 is used to execute the machine-executable instructions to implement the encoding method of the above-mentioned embodiments 1-17 of the present application.

[0260] Based on the same application concept as the above method, an embodiment of the present application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method or encoding method of the above embodiments 1-17 of the present application.

[0261] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by the processor, the decoding method or encoding method of the above embodiments 1-17 of the present application is implemented, such as the decoding method or encoding method in the above embodiments.

[0262] Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.

[0263] Based on the same application concept as the above method, a decoding device is also proposed in an embodiment of the present application, which can be applied to a decoding end, and the decoding device includes: a decoding module, used to decode a first code stream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode a second code stream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; a processing module, used to input the reconstruction features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0264] Exemplarily, if the attention module is a cascaded hybrid attention module, the processing module is also used to perform a first processing on the input features of the first attention sub-module through the first attention sub-module to obtain the output features of the first attention sub-module, and the output features of the first attention sub-module are the input features of the second attention sub-module; wherein the first processing is at least one of layer normalization, convolution operation and dimensionality transformation operation; and perform a second processing on the input features of the second attention sub-module through the second attention sub-module to obtain the output features of the second attention sub-module; wherein the second processing is at least one of depthwise separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, downsampling operation, residual convolution operation and upsampling operation.

[0265] Exemplarily, if the attention module is a parallel hybrid attention module, the processing module is also used to split the input features of the parallel hybrid attention module into two sub-features in the channel dimension, and the two sub-features are respectively used as the input features of the first attention sub-module and the input features of the second attention sub-module; the input features of the first attention sub-module are first processed by the first attention sub-module to obtain the output features of the first attention sub-module; the first processing is at least one of layer normalization, convolution operation and dimensionality transformation operation, and the input features of the second attention sub-module are second processed by the second attention sub-module to obtain the output features of the second attention sub-module; the second processing is at least one of depthwise separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation; the output features of the first attention sub-module and the output features of the second attention sub-module are feature spliced ​​in the channel dimension to obtain the output features of the parallel hybrid attention module.

[0266] Exemplarily, the processing module performs a first processing on the input features of the first attention submodule through the first attention submodule, and when obtaining the output features of the first attention submodule, it is specifically used to: perform layer normalization on the input features of the first attention submodule to obtain layer-normalized features; perform a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector; perform a dimension transformation operation on the query vector, the key vector and the value vector, respectively, to obtain a dimensionally transformed query vector, a dimensionally transformed key vector and a dimensionally transformed value vector; determine an attention weight based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determine a modified feature corresponding to the input feature based on the attention weight and the dimensionally transformed value vector; determine the output feature of the first attention submodule based on the input feature and the modified feature.

[0267] Exemplarily, the processing module performs a first processing on the input features of the first attention submodule through the first attention submodule to obtain the output features of the first attention submodule, and is specifically used to: perform layer normalization on the input features of the first attention submodule to obtain layer-normalized features; perform window division on the layer-normalized features to obtain multiple small-size features; for each small-size feature, perform a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; and concatenate the small-size query features corresponding to the multiple small-size features to obtain a query vector; The small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector; the query vector, the key vector and the value vector are respectively dimensionally transformed to obtain a dimensionally transformed query vector, a dimensionally transformed key vector and a dimensionally transformed value vector; an attention weight is determined based on the dimensionally transformed query vector and the dimensionally transformed key vector, and a correction feature corresponding to the input feature is determined based on the attention weight and the dimensionally transformed value vector; an output feature of the first attention submodule is determined based on the input feature and the correction feature.

[0268] Exemplarily, the processing module performs a first processing on the input features of the first attention submodule through the first attention submodule, and when obtaining the output features of the first attention submodule, it is specifically used to: perform layer normalization on the input features of the first attention submodule to obtain layer-normalized features; perform a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector; determine an attention weight based on the query vector and the key vector, determine an intermediate feature based on the attention weight and the value vector, perform layer normalization on the intermediate feature to obtain a corrected feature corresponding to the input feature; and determine the output feature of the first attention submodule based on the input feature and the corrected feature.

[0269] Exemplarily, the processing module performs a first processing on the input features of the first attention submodule through the first attention submodule, and when obtaining the output features of the first attention submodule, it is specifically used for: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing window division on the layer-normalized features to obtain multiple small-size features; for each small-size feature, performing a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; splicing the small-size query features corresponding to the multiple small-size features to obtain a query vector; splicing the small-size key features corresponding to the multiple small-size features to obtain a key vector; splicing the small-size value features corresponding to the multiple small-size features to obtain a value vector; determining the attention weight based on the query vector and the key vector, and determining the intermediate features based on the attention weight and the value vector, performing layer normalization on the intermediate features to obtain the corrected features corresponding to the input features; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0270] Exemplarily, the processing module performs a second processing on the input features of the second attention submodule through the second attention submodule to obtain the output features of the second attention submodule, which is specifically used for: performing a depth-separable convolution operation on the input features of the second attention submodule to obtain convolved features; performing layer normalization on the convolved features to obtain layer-normalized features; performing a multi-layer perception operation on the layer-normalized features to obtain multi-layer perceived features; and determining the output features of the second attention submodule based on the input features and the multi-layer perceived features.

[0271] Exemplarily, the processing module performs a second processing on the input features of the second attention submodule through the second attention submodule to obtain the output features of the second attention submodule, and is specifically used for: performing layer normalization on the input features of the second attention submodule to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0272] Exemplarily, the processing module performs a second processing on the input features of the second attention sub-module through the second attention sub-module to obtain the output features of the second attention sub-module, and is specifically used for: downsampling the input features of the second attention sub-module to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolution features; upsampling the residual convolution features to obtain upsampled features; activating the upsampled features to obtain activated features; performing a residual convolution operation on the input features to obtain convolution features after residual convolution, and determining the output features of the second attention sub-module based on the input features of the second attention sub-module, the activated features and the convolution features.

[0273] Exemplarily, the synthetic transformation network also includes at least one deconvolution layer; wherein: the attention module is located behind any deconvolution layer; or, the attention module is located behind each deconvolution layer; the attention modules located behind different deconvolution layers are exactly the same, or, the attention modules located behind different deconvolution layers are not exactly the same.

[0274] Exemplarily, the synthetic transformation network also includes a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer; the synthetic transformation network includes an attention module, which is located behind the first deconvolution layer, or behind the second deconvolution layer, or behind the third deconvolution layer; or, the synthetic transformation network includes an attention module located behind the first deconvolution layer, an attention module located behind the second deconvolution layer, and an attention module located behind the third deconvolution layer.

[0275] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application, which is applied to the encoding end, and the device includes: a decoding module, which is used to decode the first code stream corresponding to the current image block to obtain the coefficient hyperparameter feature corresponding to the current image block; determine the probability distribution parameter based on the coefficient hyperparameter feature, and decode the second code stream corresponding to the current image block based on the probability distribution parameter to obtain the reconstruction feature corresponding to the current image block; a processing module, which is used to input the reconstruction feature to the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0276] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

[0277] For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image decoding method, characterized in that: The method comprises: Decoding a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, and decoding the bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; Wherein, the synthetic transformation network comprises at least an attention module and at least one deconvolution layer; The attention module is located behind one of the deconvolution layers.

2. The method according to claim 1, characterized in that The synthetic transformation network includes at least one attention module, at least one residual layer, at least one deconvolution layer, at least one residual activation layer and at least one cropping layer.

3. The method according to claim 1 or 2, characterized in that: The synthetic transformation network includes a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, an attention module, a third cropping layer, a third residual activation layer, a third deconvolution layer, and a fourth cropping layer.

4. The method according to any one of claims 1 to 3, characterized in that: The attention module is a cascade hybrid attention module, the cascade hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; Among them, the output features of the first attention sub-module are the input features of the second attention sub-module, and the output features of the second attention sub-module are used to determine the reconstructed image block.

5. The method according to claim 4, characterized in that Performing a first processing on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule; The first processing includes at least one of layer normalization, convolution operation and dimensionality transformation operation.

6. The method according to claim 4, characterized in that performing a second processing on the input features of the second attention submodule by the second attention submodule to obtain output features of the second attention submodule; The second processing includes at least one of layer normalization, activation operation and convolution operation.

7. An image encoding method, characterized in that: The method comprises: Decoding a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, and decoding the bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; Wherein, the synthetic transformation network comprises at least an attention module and at least one deconvolution layer; The attention module is located behind one of the deconvolution layers.

8. An image decoding device, characterized in that: The device comprises: A decoding module, configured to decode a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode the bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; A processing module, used for inputting the reconstructed features into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthesis transformation network comprises at least an attention module and at least one deconvolution layer; The attention module is located behind one of the deconvolution layers.

9. An image encoding device, characterized in that: The device comprises: A decoding module, configured to decode a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode the bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; A processing module, used for inputting the reconstructed features into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthesis transformation network comprises at least an attention module and at least one deconvolution layer; The attention module is located behind one of the deconvolution layers.

10. An image decoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 6.

11. An image encoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 7.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images