A decoding, encoding method, apparatus and device thereof

Abstract

This application provides a decoding and encoding method, apparatus, and device, comprising: decoding a first bitstream corresponding to a current image block to obtain coefficient hyperparameter features; determining probability distribution parameters based on the coefficient hyperparameter features; decoding a second bitstream of the current image block based on the probability distribution parameters to obtain reconstructed features; and inputting the reconstructed features into a synthetic transform network to obtain a reconstructed image block. The synthetic transform network includes an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, which are sequential; the parallel hybrid attention module includes a first attention submodule and a second attention submodule, which are parallel. This application's solution improves decoding performance and reduces complexity.

Description

Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a decoding and encoding method, apparatus and device thereof. Background Technology

[0002] To save space, video images are encoded before transmission. Complete video encoding includes processes such as prediction, transform, quantization, entropy coding, and filtering. The prediction process can be divided into intra-frame prediction and inter-frame prediction. Inter-frame prediction utilizes temporal correlation to predict the current pixel using pixels from neighboring encoded images, effectively removing temporal redundancy. Intra-frame prediction utilizes spatial correlation to predict the current pixel using pixels from the encoded blocks of the current frame, removing spatial redundancy.

[0003] With the rapid development of deep learning, it has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning is also gradually being applied in the field of encoding and decoding, where neural networks can be used to encode and decode images. Although neural network-based encoding and decoding methods have shown great performance potential, they still suffer from problems such as poor decoding performance and high complexity. Summary of the Invention

[0004] This application provides a decoding and encoding method, apparatus, and device that can improve decoding performance and reduce complexity.

[0005] This application provides a decoding method applied at a decoding end, the method comprising:

[0006] Decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block;

[0007] Based on the coefficient hyperparameter features, the probability distribution parameters are determined, and based on the probability distribution parameters, the second bitstream corresponding to the current image block is decoded to obtain the reconstructed features corresponding to the current image block.

[0008] The reconstructed features are input into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block;

[0009] The synthetic transformation network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules. Alternatively, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0010] This application provides an encoding method applied at an encoding end, the method comprising:

[0011] Decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block;

[0012] Based on the coefficient hyperparameter features, the probability distribution parameters are determined, and based on the probability distribution parameters, the second bitstream corresponding to the current image block is decoded to obtain the reconstructed features corresponding to the current image block.

[0013] The reconstructed features are input into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block;

[0014] The synthetic transformation network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules. Alternatively, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0015] This application provides a decoding device applied at a decoding end, the device comprising:

[0016] The decoding module is used to decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine the probability distribution parameters based on the coefficient hyperparameter features; and decode the second bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block.

[0017] A processing module is used to input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0018] This application provides an encoding device applied at an encoding end, the device comprising:

[0019] The decoding module is used to decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine the probability distribution parameters based on the coefficient hyperparameter features; and decode the second bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block.

[0020] A processing module is used to input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0021] This application provides a decoding end device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0022] The processor is used to execute machine-executable instructions to implement the above-described decoding method.

[0023] This application provides an encoding terminal device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0024] The processor is used to execute machine-executable instructions to implement the above-described encoding method.

[0025] This application provides an electronic device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0026] The processor is used to execute machine-executable instructions to implement the above-described decoding or encoding methods.

[0027] This application provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, are used to implement the above-described decoding or encoding method.

[0028] This application provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described decoding method or encoding method.

[0029] As can be seen from the above technical solutions, in this application embodiment, for neural network-based encoding and decoding technology, an attention mechanism-based synthetic transformation network is proposed. The synthetic transformation network includes an attention module, which can be a cascaded hybrid attention module or a parallel hybrid attention module. When the synthetic transformation network is implemented through the cascaded hybrid attention module or the parallel hybrid attention module, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, and the decoding performance is improved. While reducing complexity, the quality of the reconstructed image blocks is effectively guaranteed, and the encoding and decoding performance is improved. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of a three-dimensional feature matrix in one embodiment of this application;

[0031] Figure 2 This is a schematic diagram of pixel recombination in one embodiment of this application;

[0032] Figure 3 This is a flowchart of a decoding method in one embodiment of this application;

[0033] Figure 4 This is a schematic diagram of the processing procedure of the encoding end in one embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the processing procedure at the decoding end in one embodiment of this application;

[0035] Figures 6A-6D This is a schematic diagram of the structure of the synthetic transformation network in one embodiment of this application;

[0036] Figures 7A-7C This is a schematic diagram of the structure of the synthetic transformation network in one embodiment of this application;

[0037] Figures 8A-8H This is a schematic diagram of the structure of the synthetic transformation network in one embodiment of this application;

[0038] Figure 9A This is a hardware structure diagram of the decoding end device in one embodiment of this application;

[0039] Figure 9B This is a hardware structure diagram of the encoding end device in one embodiment of this application. Detailed Implementation

[0040] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments and claims of this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any or all possible combinations including one or more of the associated listed items. It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this application, and similarly, second information may also be referred to as first information, depending on the context. Furthermore, the word “if” as used can be interpreted as “when,” “in response to a determination,” or “when…”.

[0041] This application proposes a decoding method, which may involve the following concepts:

[0042] Entropy coding: Entropy coding is a coding process that follows the principle of entropy without losing any information. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods can include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0043] Neural Networks (NNs): Neural networks are artificial neural networks, a computational model composed of numerous interconnected nodes (or neurons). In a neural network, neurons can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units; the representation and processing of information are reflected in the connections between processing units. Neural networks are a non-programmed, brain-like information processing method. The essence of a neural network is to obtain a parallel and distributed information processing function through the transformations and dynamic behavior of the network, mimicking the information processing function of the human brain's nervous system to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and fully connected networks.

[0044] Convolutional Neural Networks (CNNs): A CNN is a type of feedforward neural network and one of the most representative network structures in deep learning. The artificial neurons in a CNN can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. The basic structure of a CNN consists of two layers: the first is the feature extraction layer (also called the convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, extracting the features of that local area. Once the local feature is extracted, its positional relationship with other features is determined. The second is the feature mapping layer (also called the activation layer). Each computational layer of the neural network consists of multiple feature maps, each of which is a plane where all neurons have equal weights. Feature mapping structures can use functions such as the Sigmoid function, ReLU (Rectified Linear Unit), Leaky-ReLU, PReLU (Parametric Rectified Linear Unit), and GDN (Generalized Divisive Normalization) as activation functions for the convolutional network. Furthermore, because neurons on a single feature map share weights, the number of free parameters in the network is reduced.

[0045] For example, one advantage of convolutional neural networks (CNNs) over image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. Another advantage of CNNs over ordinary neural networks is that ordinary neural networks use fully connected layers, meaning all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, however, avoid this difficulty through local connectivity and weight sharing.

[0046] Deconvolution: Also known as transposed convolution, deconvolution layers work similarly to convolutional layers. The main difference is that deconvolution layers use padding to make the output larger than the input (though they can also remain the same). If the stride is 1, the output size equals the input size; if the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0047] Depthwise Separable Convolution: Depthwise separable convolution requires two convolution operations. In the first operation, a deepwise convolution is performed (collecting features from each layer), with a kernel size of K*K*1 and a total of K*K*Cin parameters. In the second operation, to obtain the Cout dimension output, the kernel size is 1*1*Cin, and the total number of parameters is 1*1*Cin*Cout. The output of the second convolution can be the output of the depthwise separable convolution, and the input of the first convolution can be the input of the depthwise separable convolution.

[0048] Generalization ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns hidden behind data pairs. The trained network can also give appropriate outputs for data outside the learning set that have the same pattern. This ability can be called generalization ability.

[0049] Feature: A feature is a three-dimensional feature matrix or tensor of size C*W*H. See also Figure 1 The diagram shows a schematic of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be either the input or the output of a neural network.

[0050] Pixel shuffle: Also known as feature location reorganization, pixel shuffle involves reorganizing low-resolution features across multiple channels to obtain a high-resolution feature map. See also Figure 2 The image shown is a schematic diagram of pixel recombination. Figure 2 The diagram illustrates pixel recombination with an upsampling factor of 2. Clearly, a feature with 4 channels and a resolution of 3x3 becomes a feature with 1 channel and a resolution of 6x6 after passing through a feature location recombination layer.

[0051] Rate-Distortion Optimized (RDBEM): Two main metrics for evaluating coding efficiency are bitrate and PSNR (Peak Signal-to-Noise Ratio). A smaller bitrate results in a higher compression ratio, and a higher PSNR results in better reconstructed image quality. In mode selection, the discrimination formula is essentially a comprehensive evaluation of these two metrics. For example, the cost corresponding to a mode is: J(mode) = D + λ*R, where D represents Distortion, which is typically measured using the SSE (sum-square error) metric. SSE refers to the sum of the mean squares of the differences between the reconstructed image patch and the source image. To consider the cost, the SAD (Sum of Absolute Difference) metric can also be used, which is the sum of the absolute values ​​of the differences between the reconstructed image patch and the source image. λ is a Lagrange multiplier, and R is the actual number of bits required to encode the image patch in that mode, including the total number of bits required for encoding mode information, motion information, residuals, etc. When selecting a coding mode, using the rate-distortion principle to compare and decide on coding modes can usually guarantee the best coding performance.

[0052] Numerous encoding tools have been proposed for various modules at the encoding end, and each tool often has multiple modes. The optimal encoding tool for different video sequences often differs. Therefore, during encoding, Rate-Distortion Optimization (RDO) is typically used to compare the encoding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information is transmitted by encoding marker information in the bitstream. Although this method introduces higher encoding complexity, it can adaptively select the optimal mode combination for different content to achieve the best encoding performance. The decoding end can obtain the relevant mode information by directly parsing the marker information, with minimal impact from complexity.

[0053] The end-to-end general framework for image coding mainly consists of a main feature information part and a priori edge information part. The main feature information part includes an analysis transform network, quantization, normal entropy coding, normal entropy decoding, and a synthesis transform network. The priori edge information part includes a priori analysis network, quantization, factorized entropy coding, factorized entropy decoding, and a priori synthesis network. The image component x is compressed, encoded, and reconstructed by the analysis transform network and the synthesis transform network of the main feature information part, respectively. The priori edge information part is mainly used to model the probability of the main feature information, guiding the entropy encoding and decoding of the main feature information. The end-to-end general framework for image coding suffers from high computational complexity and poor decoding performance.

[0054] In response to the above findings, this embodiment proposes a synthetic transformation network with an attention mechanism, leveraging the characteristics of an end-to-end image coding framework. This synthetic transformation network includes attention modules, which can be cascaded hybrid attention modules or parallel hybrid attention modules. By implementing the synthetic transformation network through cascaded or parallel hybrid attention modules, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, thereby improving decoding performance.

[0055] The decoding method in the embodiments of this application will be described in detail below with reference to several specific embodiments.

[0056] Example 1: This application proposes a decoding method, see [link to example]. Figure 3 The diagram shown illustrates the flowchart of this decoding method, which can be applied to the decoding end (also known as a video decoder). This method may include:

[0057] Step 301: Decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features of the current image block.

[0058] Step 302: Determine the probability distribution parameters based on the hyperparameter features of the coefficients, and decode the second bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block.

[0059] For example, a bitstream corresponding to the current image patch (such as the first bitstream) can be a bitstream that encodes the coefficient hyperparameter features corresponding to the current image patch; another bitstream corresponding to the current image patch (such as the second bitstream) can be a bitstream that encodes the residual features corresponding to the current image patch.

[0060] Step 303: Input the reconstructed features into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block. The synthetic transformation network includes at least an attention module, which can be a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module can include a first attention submodule and a second attention submodule, which are serially connected; for example, the output features of the first attention submodule are the input features of the second attention submodule. Alternatively, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, which are parallelly connected; for example, the input features of the first attention submodule and the second attention submodule are parallel, and their output features are also parallel.

[0061] For example, if the attention module is a cascaded hybrid attention module, the input features of the first attention submodule can be processed by the first attention submodule to obtain the output features of the first attention submodule, and the output features of the first attention submodule become the input features of the second attention submodule; wherein, the first processing can be at least one of layer normalization, convolution operation, and dimension transformation operation. The input features of the second attention submodule can be processed by the second attention submodule to obtain the output features of the second attention submodule; wherein, the second processing can be at least one of depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing can be at least one of depthwise separable convolution operation, layer normalization, and multilayer perceptron operation. Or, the second processing can be at least one of layer normalization, linear operation, and activation operation. Or, the second processing can be at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.

[0062] For example, if the attention module is a parallel hybrid attention module, the input features of the parallel hybrid attention module can be further split into two sub-features along the channel dimension. These two sub-features can be used as the input features of the first attention sub-module and the second attention sub-module, respectively. The first attention sub-module can perform a first processing on its input features to obtain the output features of the first attention sub-module; wherein the first processing can be at least one of layer normalization, convolution operation, and dimensionality transformation operation. Furthermore, the second attention sub-module can perform a second processing on its input features to obtain the output features of the second attention sub-module; wherein the second processing can be at least one of depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing can be at least one of depthwise separable convolution operation, layer normalization, and multilayer perceptron operation. Or, the second processing can be at least one of layer normalization, linear operation, and activation operation. Or, the second processing can be at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation. Then, the output features of the first attention submodule and the second attention submodule can be concatenated along the channel dimension to obtain the output features of the parallel hybrid attention module.

[0063] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule, including but not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; performing dimensionality transformation operations on the query vector, key vector, and value vector respectively to obtain dimension-transformed query vector, dimension-transformed key vector, and dimension-transformed value vector; determining attention weights based on the dimension-transformed query vector and dimension-transformed key vector; determining the corrected features corresponding to the input features based on the attention weights and dimension-transformed value vector; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0064] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; dividing the layer-normalized features into windows to obtain multiple small-sized features; performing a three-way convolution operation on each small-sized feature to obtain a small-sized query feature, a small-sized key feature, and a small-sized value feature corresponding to that small-sized feature; concatenating the small-sized query features corresponding to multiple small-sized features to obtain a query vector; concatenating the small-sized key features corresponding to multiple small-sized features to obtain a key vector; and concatenating the small-sized value features corresponding to multiple small-sized features to obtain a value vector. Then, dimensional transformation operations are performed on the query vector, the key vector, and the value vector respectively to obtain the dimension-transformed query vector, the dimension-transformed key vector, and the dimension-transformed value vector; attention weights are determined based on the dimension-transformed query vector and the dimension-transformed key vector, and the corrected features corresponding to the input features are determined based on the attention weights and the dimension-transformed value vector; the output features of the first attention submodule are determined based on the input features and the corrected features.

[0065] In one possible implementation, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; performing a three-way convolution operation on the layer normalized features to obtain a query vector, a key vector, and a value vector; determining attention weights based on the query vector and the key vector, and determining intermediate features based on the attention weights and the value vector; performing layer normalization on the intermediate features to obtain the corrected features corresponding to the input features; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0066] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; dividing the layer-normalized features into windows to obtain multiple small-sized features; and performing a three-way convolution operation on each small-sized feature to obtain a small-sized query feature, a small-sized key feature, and a small-sized value feature corresponding to that small-sized feature. Then, the small-sized query features corresponding to multiple small-sized features are concatenated to obtain a query vector; the small-sized key features corresponding to multiple small-sized features are concatenated to obtain a key vector; and the small-sized value features corresponding to multiple small-sized features are concatenated to obtain a value vector. Attention weights are determined based on the query vector and the key vector, and intermediate features are determined based on the attention weights and the value vector. These intermediate features are then layer-normalized to obtain the corrected features corresponding to the input features. Finally, the output features of the first attention submodule are determined based on the input features and the corrected features.

[0067] In one possible implementation, the input features of the second attention submodule are subjected to a second processing to obtain the output features of the second attention submodule. This processing may include, but is not limited to: performing a depthwise separable convolution operation on the input features of the second attention submodule to obtain convolutional features; performing layer normalization on the convolutional features to obtain layer-normalized features; performing a multilayer perception operation on the layer-normalized features to obtain multilayer perception features; and determining the output features of the second attention submodule based on the input features and the multilayer perception features.

[0068] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determining attention weights based on the query vector and the key vector, and determining the corrected features corresponding to the input features based on the attention weights and the value vector; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0069] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; dividing the layer-normalized features into windows to obtain multiple small-sized features; performing a three-way convolution operation on each small-sized feature to obtain a small-sized query feature, a small-sized key feature, and a small-sized value feature corresponding to that small-sized feature; concatenating the small-sized query features corresponding to multiple small-sized features to obtain a query vector; concatenating the small-sized key features corresponding to multiple small-sized features to obtain a key vector; concatenating the small-sized value features corresponding to multiple small-sized features to obtain a value vector; determining attention weights based on the query vector and the key vector, and determining the correction features corresponding to the input features based on the attention weights and the value vector. Then, the output features of the first attention submodule can be determined based on the input features and the correction features.

[0070] In one possible implementation, the input features of the second attention submodule are subjected to a second processing to obtain the output features of the second attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the second attention submodule to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0071] In one possible implementation, the input features of the second attention submodule are subjected to a second processing to obtain the output features of the second attention submodule. This processing may include, but is not limited to: downsampling the input features of the second attention submodule to obtain downsampled features; performing residual convolution on the downsampled features to obtain residual convolutional features; upsampling the residual convolutional features to obtain upsampled features; activating the upsampled features to obtain activated features; and performing residual convolution on the input features to obtain convolutional features. Based on this, the output features of the second attention submodule can be determined based on the input features, the activated features, and the convolutional features.

[0072] In one possible implementation, the synthetic transform network may further include at least one deconvolutional layer; wherein: the attention module may be located after any deconvolutional layer. Alternatively, the attention module may be located after each deconvolutional layer; wherein the attention modules located after different deconvolutional layers are identical, or the attention modules located after different deconvolutional layers are not identical. Alternatively, the attention module may be located after a subset of deconvolutional layers (e.g., K deconvolutional layers out of all deconvolutional layers, where K is greater than 1 and K is less than the total number of deconvolutional layers); wherein the attention modules located after different deconvolutional layers are identical, or the attention modules located after different deconvolutional layers are not identical.

[0073] For example, the synthetic transform network may further include a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, a second deconvolution layer, a second pruning layer, a second residual activation layer, a third deconvolution layer, a third pruning layer, a third residual activation layer, a fourth deconvolution layer, and a fourth pruning layer. The synthetic transform network may include an attention module located after the first deconvolution layer, or after the second deconvolution layer, or after the third deconvolution layer. Alternatively, the synthetic transform network may include attention modules located after the first, second, and third deconvolution layers. Alternatively, the synthetic transform network may include attention modules located after the first and second deconvolution layers. Alternatively, the synthetic transform network may include attention modules located after the first and third deconvolution layers. Alternatively, the synthetic transform network may include attention modules located after the second and third deconvolution layers.

[0074] For example, the synthetic transform network includes at least a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, and a second deconvolution layer. The synthetic transform network includes at least one attention module, wherein one attention module is located after the first deconvolution layer.

[0075] Based on the same concept as the decoding method described above, this application also proposes an encoding method that can be applied to the encoding end (also known as a video encoder). This method may include: decoding a first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding a second bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block; and inputting the reconstructed features into a synthesis transform network to obtain the reconstructed image block corresponding to the current image block. The synthesis transform network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, which are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, which are two parallel submodules.

[0076] For example, the implementation process of the encoding method is similar to that of the decoding method, and the repeated content will not be described again.

[0077] For example, the above execution order is merely an example for ease of description. In practical applications, the execution order between steps can be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the method may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments.

[0078] As can be seen from the above technical solutions, in this application embodiment, for neural network-based encoding and decoding technology, an attention mechanism-based synthetic transformation network is proposed. The synthetic transformation network includes an attention module, which can be a cascaded hybrid attention module or a parallel hybrid attention module. When the synthetic transformation network is implemented through the cascaded hybrid attention module or the parallel hybrid attention module, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, and the decoding performance is improved. While reducing complexity, the quality of the reconstructed image blocks is effectively guaranteed, and the encoding and decoding performance is improved.

[0079] Example 2: For the processing procedure at the encoding end, please refer to... Figure 4 As shown, of course, Figure 4 This is just one example of the processing at the encoding end. There are no restrictions on the processing at the encoding end; a neural network can be used to implement the encoding process.

[0080] After obtaining the current image block x (which can be the original image block x, i.e., the input image block), the encoding end can perform analysis and transformation on the current image block x through an analysis and transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Specifically, performing feature transformation on the current image block x through the analysis and transformation network means transforming the current image block x to image features y in the latent domain, thereby facilitating all subsequent processes to be performed in the latent domain.

[0081] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the encoding and decoding process for image blocks can also be directly applied to the image.

[0082] After obtaining image features y, the encoder performs a coefficient hyperparameter feature transformation on y to obtain coefficient hyperparameter features z. For example, image features y can be input into a hyperparameter coding network (i.e., a neural network), which then performs the coefficient hyperparameter feature transformation on y to obtain coefficient hyperparameter features z. The hyperparameter coding network can be a trained neural network, and its training process is not restricted, as long as it can perform the coefficient hyperparameter feature transformation on image features y. The latent domain image features y, after passing through the hyperparameter coding network, yield the hyper-prior latent information z.

[0083] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the quantized hyperparameter feature corresponding to the coefficient hyperparameter feature z, i.e. Figure 4 The Q-operation in the code represents the quantization process. After obtaining the hyperparameter quantization features corresponding to the coefficient hyperparameter features z, these features are encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image patch. Figure 4 The AE operation in the code represents the encoding process, such as entropy encoding. Alternatively, the encoder can directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image patch. The hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.

[0084] After obtaining the Bitstream#1 corresponding to the current image block, the encoding end can send the Bitstream#1 corresponding to the current image block to the decoding end. For the processing of the Bitstream#1 corresponding to the current image block by the decoding end, please refer to the following embodiments.

[0085] After obtaining Bitstream#1 corresponding to the current image block, the encoding end can also decode Bitstream#1 to obtain the hyperparameter quantization features, i.e. Figure 4 In the diagram, AD represents the decoding process. Then, the hyperparameter quantization features are dequantized to obtain the coefficient hyperparameter features z_hat. The coefficient hyperparameter features z_hat and the coefficient hyperparameter features z can be the same or different. Figure 4 The IQ operation in the code is an inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder can decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0086] For the encoding process of Bitstream#1, a fixed probability density model encoding method can be used, and for the decoding process of Bitstream#1, a fixed probability density model decoding method can be used. There are no restrictions on the encoding and decoding processes.

[0087] After obtaining the hyperparameter feature z_hat, the encoder can perform context-based prediction based on the hyperparameter feature z_hat of the current image patch and the reconstructed feature y_hat of the previous image patch (the determination process of the reconstructed feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image patch. For example, the hyperparameter feature z_hat and the reconstructed feature y_hat can be input into the mean prediction network, which determines the predicted value mu based on the hyperparameter feature z_hat and the reconstructed feature y_hat. This prediction process is not restricted. Specifically, for the context-based prediction process, the input includes the hyperparameter feature z_hat and the decoded reconstructed feature y_hat. The two are input together to obtain a more accurate predicted value mu. The predicted value mu is used to subtract the original feature to obtain the residual and add it to the decoded residual to obtain the reconstructed feature y_hat.

[0088] In another implementation, the mean prediction network can also obtain the predicted value of the current image patch based solely on the coefficient hyperparameter feature z_hat of the current image patch, without utilizing the reconstructed features y_hat of previous image patches. This significantly reduces the complexity of obtaining the predicted value and accelerates the encoding and decoding process. The methods for obtaining the predicted value on the encoding side can all adopt one of the two schemes described above, and will not be elaborated further.

[0089] It is important to note that the mean prediction network is an optional neural network; that is, it is not necessary to use a mean prediction network to determine the predicted value mu. Figure 4 The dashed box in the image indicates that the mean prediction network is optional.

[0090] After obtaining image features y, the encoder can determine residual features r based on image features y and predicted values ​​mu, such as using the difference between image features y and predicted values ​​mu as residual features r. Then, feature processing is performed on the residual features r to obtain image features s. This feature processing method is not restricted and can be any method. In this case, a mean prediction network needs to be deployed to provide the predicted values ​​mu. Alternatively, after obtaining image features y, the encoder can perform feature processing on image features y to obtain image features s. This feature processing method is not restricted and can be any method. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0091] After obtaining image features s, the encoding end can quantize image features s to obtain the quantized image features corresponding to image features s, i.e. Figure 4 The Q operation in the code represents the quantization process. After obtaining the quantized image features corresponding to image features s, the encoder can encode these quantized features to obtain Bitstream #2 (i.e., the second bitstream) corresponding to the current image block. Figure 4 The AE operation in the code represents the encoding process, such as entropy encoding. Alternatively, the encoding end can directly encode the image features s to obtain the Bitstream#2 corresponding to the current image block, without involving the quantization process of image features s.

[0092] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can send the Bitstream#2 corresponding to the current image block to the decoding end. For the processing of the Bitstream#2 corresponding to the current image block by the decoding end, please refer to the following embodiments.

[0093] After obtaining Bitstream#2 corresponding to the current image block, the encoding end can also decode Bitstream#2 to obtain the image quantization features, i.e. Figure 4 In this context, AD represents the decoding process. Then, the encoding end can dequantize the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. Figure 4 The IQ operation in the code is the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder can decode Bitstream#2 to obtain image features s' without involving the dequantization process of image quantization features.

[0094] After obtaining image features s', the encoder can perform feature recovery (the inverse of feature processing) on ​​s'. This feature recovery process is not restricted and can be any method, resulting in residual features r_hat. Residual features r_hat and r can be the same or different. After obtaining residual features r_hat, the encoder determines reconstructed features y_hat based on r_hat and the predicted value mu. Reconstructed features y_hat and y can be the same or different from image features y. For example, the sum of residual features r_hat and the predicted value mu can be used as reconstructed features y_hat. In this case, a mean prediction network needs to be deployed to provide the predicted value mu. Alternatively, after obtaining image features s', the encoder can perform feature recovery (the inverse of feature processing) on ​​s' to obtain reconstructed features y_hat. Reconstructed features y_hat and y can be the same or different from image features y. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0095] After obtaining the reconstructed feature y_hat, the encoder can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat can be input into the synthetic transformation network, which will perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0096] In one possible implementation, when the encoding end encodes the image quantization features or image features s to obtain Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization features or image features s based on the probability distribution model. Furthermore, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0097] To obtain the probability distribution model, please refer to [link / reference]. Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat can be input into a probabilistic hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probabilistic hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform the inverse hyperparameter feature transformation on z_hat.

[0098] In one possible implementation, the above-mentioned encoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without any restrictions on the encoding process.

[0099] Example 3: For the processing procedure at the decoding end, please refer to... Figure 5 As shown, of course, Figure 5 This is just an example of the decoding process. There are no restrictions on the decoding process; a neural network can be used to implement the decoding.

[0100] After obtaining Bitstream#1 corresponding to the current image block, the decoding end can decode Bitstream#1 to obtain the hyperparameter quantization features, i.e. Figure 5 In the diagram, AD represents the decoding process. Then, the hyperparameter quantization features are dequantized to obtain the coefficient hyperparameter features z_hat. The coefficient hyperparameter features z_hat and the coefficient hyperparameter features z can be the same or different. Figure 5 The IQ operation in the code is an inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoding end can decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat, without involving the inverse quantization process of the coefficient hyperparameter feature z_hat. The decoding process for Bitstream#1 can employ a decoding method with a fixed probability density model, and there are no restrictions on this.

[0101] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the decoding process for the image block can also be directly applied to the image.

[0102] After obtaining the coefficient hyperparameter feature z_hat, the decoder can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image patch and the reconstructed feature y_hat of the previous image patch (the determination process of the reconstructed feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image patch. For example, the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat can be input into the mean prediction network, which determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat. This prediction process is not restricted. Specifically, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstructed feature y_hat; the combined input of these two features yields a more accurate predicted value mu.

[0103] In another implementation, after obtaining the coefficient hyperparameter feature z_hat of the current image patch, the decoder can directly obtain the predicted value mu corresponding to the current image patch based on the coefficient hyperparameter feature z_hat, that is, it does not rely on the reconstruction feature y_hat of the previous image patch to obtain the predicted value. This can greatly reduce the complexity of obtaining the predicted value and accelerate the encoding and decoding process. The methods for obtaining the predicted value on the decoding side can all adopt one of the above two schemes, and will not be described in detail.

[0104] It is important to note that the mean prediction network is an optional neural network; that is, it is not necessary to use a mean prediction network to determine the predicted value mu. Figure 5 The dashed box in the image indicates that the mean prediction network is optional.

[0105] After obtaining Bitstream #2 corresponding to the current image block, the decoding end can decode Bitstream #2 to obtain the image quantization features, i.e. Figure 5 In this context, AD represents the decoding process. Then, the decoding end can perform inverse quantization on the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. Figure 5 The IQ operation in the code can be an inverse quantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the decoding end can decode Bitstream#2 to obtain image features s' without involving the inverse quantization process of image quantization features.

[0106] After obtaining image features s', the decoder can perform feature recovery (the inverse process of feature processing) on ​​image features s' to obtain residual features r_hat. Residual features r_hat may be the same as or different from residual features r. After obtaining residual features r_hat, the decoder determines reconstructed features y_hat based on residual features r_hat and predicted values ​​mu. Reconstructed features y_hat may be the same as or different from image features y. For example, the sum of residual features r_hat and predicted values ​​mu can be used as reconstructed features y_hat. In this case, a mean prediction network needs to be deployed to provide predicted values ​​mu. Alternatively, after obtaining image features s', the decoder can perform feature recovery on image features s' to obtain reconstructed features y_hat. Reconstructed features y_hat may be the same as or different from image features y. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0107] After obtaining the reconstructed feature y_hat, the decoder can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat can be input into the synthetic transformation network, which will perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0108] In one possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on that probability distribution model. To obtain the probability distribution model, see [link to documentation]. Figure 5 As shown, after obtaining the hyperparameter feature z_hat, the decoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the hyperparameter feature z_hat can be input into a probabilistic hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probabilistic hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p.

[0109] In one possible implementation, the above-mentioned decoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without limiting the decoding process.

[0110] Example 4: For Examples 1, 2, and 3, a mean prediction network can be deployed or not. To improve feature encoding performance, deploying a mean prediction network is taken as an example. When deploying the mean prediction network, to obtain accurate predicted values ​​for image features, context-based prediction can be performed based on the coefficient hyperparameter feature z_hat of the current image patch and the reconstructed feature y_hat of the previous image patch to obtain the predicted value mu corresponding to the current image patch; alternatively, the predicted value mu corresponding to the current image patch can be obtained based on the coefficient hyperparameter feature z_hat of the current image patch.

[0111] The processing at the encoding end may include the following steps:

[0112] Step S11: After obtaining the original image, determine whether to divide the original image into blocks based on the image resolution. For example, if the image resolution is greater than a threshold, the original image is divided into blocks; otherwise, it is not divided. If so, the original image is divided into several image blocks x, with overlapping portions between adjacent image blocks. If not, image block x is the original image, meaning the original image has only one image block. For ease of description, we will use one image block as an example, which can be denoted as the current image block x. The processing of the current image block x will be used as an example in the following explanation.

[0113] Step S12: After obtaining the current image block x, analyze and transform the current image block x through an analysis and transformation network to obtain the image features y (i.e., feature blocks) corresponding to the current image block x. For example, the analysis and transformation network transforms the current image block x into the image features y in the latent domain, thereby facilitating all subsequent processes to be performed in the latent domain.

[0114] Step S13: Perform a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x. For example, the image feature y can be input into a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network can perform a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x.

[0115] Step S14: Encode the coefficient hyperparameter feature z into the first bitstream (Bitstream#1) corresponding to the current image patch. For example, quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantized feature, and then encode this hyperparameter quantized feature to obtain the first bitstream corresponding to the current image patch. Alternatively, directly encode the coefficient hyperparameter feature z to obtain the first bitstream corresponding to the current image patch. That is, the first bitstream corresponding to the current image patch is a bitstream encoded with the coefficient hyperparameter feature z corresponding to the current image patch x. After obtaining the first bitstream corresponding to the current image patch, send the first bitstream corresponding to the current image patch to the decoding end.

[0116] Step S15: After obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process.

[0117] Step S16: Determine the probability distribution parameters based on the coefficient hyperparameter feature z_hat. For example, perform an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Alternatively, input the coefficient hyperparameter feature z_hat into the probability hyperparameter decoding network, which then performs an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on the probability distribution parameter p.

[0118] Step S17: Input the coefficient hyperparameter feature z_hat into the mean prediction network. The mean prediction network performs context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat of the previous image patch to obtain the predicted value mu. Alternatively, input the coefficient hyperparameter feature z_hat into the mean prediction network, and the mean prediction network obtains the predicted value mu based on the coefficient hyperparameter feature z_hat.

[0119] Step S18: Determine the residual feature r based on the image feature y and the mean feature mu (i.e., the predicted value mu). For example, the difference between the image feature y and the mean feature mu can be used as the residual feature r. Perform feature processing on the residual feature r to obtain the feature-processed residual feature r. There are no restrictions on this feature processing procedure. Feature processing is an optional step, and it is also possible not to perform feature processing on the residual feature r.

[0120] Step S19: Quantize the residual feature r (or the residual feature r after feature processing) to obtain the image quantization feature, and encode the image quantization feature to obtain the second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r (or the residual feature r after feature processing) can be directly encoded to obtain the second bitstream corresponding to the current image block. That is, the second bitstream corresponding to the current image block is a bitstream encoded with the residual feature r corresponding to the current image block x. After obtaining the second bitstream corresponding to the current image block, send the second bitstream corresponding to the current image block to the decoding end.

[0121] For example, when encoding image quantization features or residual features r, the probability distribution model corresponding to the probability distribution parameter p can be used to encode the image quantization features or residual features r to obtain the second bitstream.

[0122] Step S20: After obtaining the second bitstream corresponding to the current image block, the second bitstream is decoded using the probability distribution model corresponding to the probability distribution parameter p to obtain image quantization features. The image quantization features are then dequantized, and feature recovery is performed on the dequantized features to obtain the residual feature r_hat. Alternatively, the residual feature r_hat can be directly obtained after dequantizing the image quantization features without involving the feature recovery process. Or, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain the residual feature r_hat, or the residual feature r_hat can be directly obtained after decoding the second bitstream.

[0123] Step S21: After obtaining the residual feature r_hat, determine the reconstruction feature y_hat based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu can be used as the reconstruction feature y_hat.

[0124] Step S22: Input the reconstructed feature y_hat into the synthesis transform network. The synthesis transform network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstructed feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0125] For example, taking the deployment of a mean prediction network as an example, the processing at the decoding end may include the following steps:

[0126] Step S31: After obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process.

[0127] Step S32: Determine the probability distribution parameters based on the coefficient hyperparameter feature z_hat. For example, perform an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Alternatively, input the coefficient hyperparameter feature z_hat into the probability hyperparameter decoding network, which then performs an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on the probability distribution parameter p.

[0128] Step S33: After obtaining the second bitstream corresponding to the current image patch, decode the second bitstream corresponding to the current image patch based on the probability distribution parameter p to obtain the reconstruction features corresponding to the current image patch. For example, the residual feature r_hat corresponding to the current image patch can be determined, and the predicted value mu corresponding to the current image patch can be determined. The reconstruction feature y_hat can be determined based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu can be used as the reconstruction feature y_hat.

[0129] To determine the predicted value *mu* for the current image patch, after obtaining the hyperparameter feature *z_hat*, it can be input into the mean prediction network. The mean prediction network then performs context-based prediction based on the hyperparameter feature *z_hat* and the reconstructed features *y_hat* of the previous image patch to obtain the predicted value *mu* for the current image patch. Alternatively, the hyperparameter feature *z_hat* can be input into the mean prediction network, which then obtains the predicted value *mu* for the current image patch based on the hyperparameter feature *z_hat*.

[0130] To determine the residual feature r_hat corresponding to the current image block, the second bitstream can be decoded using a probability distribution model corresponding to the probability distribution parameter p to obtain the image quantization features. These quantization features can then be dequantized, and feature recovery can be performed on the dequantized features to obtain the residual feature r_hat. Alternatively, the residual feature r_hat can be directly obtained by dequantizing the image quantization features without involving the feature recovery process. Alternatively, the second bitstream can be decoded, and feature recovery can be performed on the decoded features to obtain the residual feature r_hat. Or, the residual feature r_hat can be directly obtained by decoding the second bitstream.

[0131] Step S34: Input the reconstructed feature y_hat into the synthesis transform network. The synthesis transform network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstructed feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0132] Example 5: Examples 1, 2, 3, and 4 all involve a synthetic transformation network. The implementation method of this example is the same as that of Examples 1, 2, 3, or 4, and repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0133] An example of this synthetic transformation network can be found in [link to example]. Figure 6AAs shown, the synthetic transform network can sequentially include residual layers, deconvolution layers, cropping layers, residual activation layers, deconvolution layers, cropping layers, residual activation layers, deconvolution layers, attention models, cropping layers, residual activation layers, deconvolution layers, and cropping layers. The input feature of this synthetic transform network can be the reconstructed feature y_hat, and the output feature can be the reconstructed image patch x_hat. In other words, after the reconstructed feature y_hat is input into the synthetic transform network, it passes through the aforementioned network layers sequentially to obtain the reconstructed image patch x_hat.

[0134] An example of a residual layer in a synthetic transform network can be found here. Figure 6B As shown, the residual layer can sequentially include a convolutional layer, a rectified linear unit (ReLU), a convolutional layer, and a stacking layer (used to add two features). The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer. The second convolutional layer can also be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer.

[0135] An example of a residual activation layer in a synthetic transform network can be found in [reference needed]. Figure 6C As shown, the residual activation layer can sequentially include an activation layer (such as a LeakyReLU activation layer), a convolutional layer, another activation layer (such as a tanh activation layer), a multiplication layer (used to multiply two features), and a stacking layer (used to add two features). The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer; there are no restrictions, such as choosing a 1x1 convolutional layer.

[0136] For example, the attention model in a synthetic transform network can be a Residual Non-local Attention Block (RNAB). That is, the RNAB is introduced as an attention model to acquire non-local information of the image and use this non-local information as attention weights to improve decoding performance. An example of an RNAB can be found in [link to example]. Figure 6D As shown, RNAB can sequentially include residual blocks, residual blocks, downsampling convolutional layers (such as 2x downsampling convolutional layers, such as 3x3 convolutional layers), residual blocks, residual blocks, upsampling convolutional layers (such as 2x upsampling convolutional layers, such as 3x3 convolutional layers), residual blocks, residual blocks, convolutional layers (such as 3x3 convolutional layers), sigmoid activation function, residual blocks, residual blocks, residual blocks, multiplication layers (used to multiply features) and stacking layers (used to add features).

[0137] For the residual block, see [link to relevant documentation]. Figure 6B As shown, that is, the residual block and Figure 6B Similar to residual layers, they can also sequentially include convolutional layers, rectified linear units (ReLUs) (such as LeakyReLU), convolutional layers, and stacking layers (used to add features). In this way, the input features of the residual block are processed through convolutional layers, LeakyReLUs, and convolutional layers to extract features, and then added to the input features of the residual block to obtain the final output features.

[0138] A submodule consisting of residual blocks, residual blocks, downsampling convolutional layers, residual blocks, residual blocks, upsampling convolutional layers, residual blocks, residual blocks, convolutional layers, and sigmoid activation functions can be a nonlocal attention extraction submodule. A submodule consisting of three residual blocks can be a feature extraction submodule. The input features of RNAB are multiplied by the attention weights generated by the nonlocal attention extraction submodule and then added to the input features of RNAB to obtain the final output features of RNAB.

[0139] Example 6: Examples 1, 2, 3, and 4 all involve a synthetic transformation network. The implementation method of this example is the same as that of Examples 1, 2, 3, or 4, and repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0140] The synthetic transformation network may include at least an attention module, which may be a cascaded hybrid attention module or a parallel hybrid attention module. Of course, in addition to attention modules, the synthetic transformation network may also include other network layers. This embodiment does not limit the structure of the synthetic transformation network, as long as it includes attention modules.

[0141] In one possible implementation, in addition to the attention module, the synthetic transform network may further include at least one deconvolutional layer. Of course, besides including the attention module and at least one deconvolutional layer, the synthetic transform network may also include other network layers, without limitation. When the synthetic transform network includes at least one deconvolutional layer, the attention module can be located after any deconvolutional layer. Alternatively, the attention module can be located after each deconvolutional layer; wherein the attention modules located after different deconvolutional layers can be identical, or the attention modules located after different deconvolutional layers can be non-identical. Alternatively, the attention module can be located after a subset of deconvolutional layers (e.g., K deconvolutional layers out of all deconvolutional layers, where K is greater than 1 and K is less than the total number of deconvolutional layers); wherein the attention modules located after different deconvolutional layers can be identical, or the attention modules located after different deconvolutional layers can be non-identical.

[0142] In one possible implementation, besides the attention module, the synthetic transform network may include at least one of the following network layers: a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, a second deconvolution layer, a second pruning layer, a second residual activation layer, a third deconvolution layer, a third pruning layer, a third residual activation layer, a fourth deconvolution layer, and a fourth pruning layer. That is, the synthetic transform network may include some or all of the above-mentioned network layers. Of course, in addition to the above-mentioned network layers, the synthetic transform network may also include other network layers; this is not limited. The above-mentioned network layers are used as an example for illustration.

[0143] See Figure 7A The diagram illustrates an example of a synthetic transform network, which can sequentially include a residual layer, a deconvolution layer 1 (i.e., the first deconvolution layer), a cropping layer 1 (i.e., the first cropping layer), a residual activation layer 1 (i.e., the first residual activation layer), a deconvolution layer 2 (i.e., the second deconvolution layer), a cropping layer 2 (i.e., the second cropping layer), a residual activation layer 2 (i.e., the second residual activation layer), a deconvolution layer 3 (i.e., the third deconvolution layer), a cropping layer 3 (i.e., the third cropping layer), a residual activation layer 3 (i.e., the third residual activation layer), a deconvolution layer 4 (i.e., the fourth deconvolution layer), and a cropping layer 4 (i.e., the fourth cropping layer). For example, the input feature of this synthetic transform network can be a reconstructed feature y_hat, and the output feature can be a reconstructed image patch x_hat. That is, after the reconstructed feature y_hat is input into the synthetic transform network, it undergoes processing through the aforementioned network layers to obtain and output the reconstructed image patch x_hat.

[0144] exist Figure 7AThe diagram illustrates three attention model positions: attention model position 1, attention model position 2, and attention model position 3. For example, if the synthetic transform network includes only one attention module (i.e., a cascaded hybrid attention module or a parallel hybrid attention module), this attention module can be located after deconvolution layer 1, i.e., at attention model position 1. Alternatively, the attention module can be located after deconvolution layer 2, i.e., at attention model position 2. Or, the attention module can be located after deconvolution layer 3, i.e., at attention model position 3, for example, by placing the attention module after deconvolution layer 3. Of course, these are just a few examples of attention module positions and are not restrictive; they can be located at any position within the synthetic transform network.

[0145] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 1, attention module 2 located after deconvolution layer 2, and attention module 3 located after deconvolution layer 3. That is, attention module 1 is located at attention model position 1, attention module 2 is located at attention model position 2, and attention module 3 is located at attention model position 3. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.

[0146] In this context, attention modules 1, 2, and 3 can be completely identical, meaning they can use the same network structure. Alternatively, attention modules 1, 2, and 3 can be different. For example, the network structures of attention modules 1 and 2 can be the same, but different; or, the network structures of attention modules 1 and 3 can be the same, but different; or, the network structures of attention modules 2 and 3 can be the same, but different; or, the network structures of attention modules 1 and 2 can be different, and the network structures of attention modules 2 and 3 can also be different.

[0147] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 1 and attention module 2 located after deconvolution layer 2, that is, attention module 1 is located at attention model position 1 and attention module 2 is located at attention model position 2. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.

[0148] Attention module 1 and attention module 2 can be completely identical; for example, they can use the same network structure. Alternatively, attention module 1 and attention module 2 can be different; for example, the network structures of attention module 1 and attention module 2 can be different.

[0149] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 1 and attention module 2 located after deconvolution layer 3, that is, attention module 1 is located at attention model position 1 and attention module 2 is located at attention model position 3. Of course, the attention modules can also be located at other positions in the synthetic transformation network, without limitation.

[0150] Attention module 1 and attention module 2 can be completely identical; for example, they can use the same network structure. Alternatively, attention module 1 and attention module 2 can be different; for example, the network structures of attention module 1 and attention module 2 can be different.

[0151] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 2 and attention module 2 located after deconvolution layer 3, that is, attention module 1 is located at attention model position 2 and attention module 2 is located at attention model position 3. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.

[0152] Attention module 1 and attention module 2 can be completely identical; for example, they can use the same network structure. Alternatively, attention module 1 and attention module 2 can be different; for example, the network structures of attention module 1 and attention module 2 can be different.

[0153] For an example of a residual layer in a synthetic transform network, see [link to relevant documentation]. Figure 6B As shown, the residual layer can sequentially include convolutional layers, rectified linear units (such as Leaky ReLU), convolutional layers, and stacking layers. That is, after the input features are processed through convolutional layers, rectified linear units, and convolutional layers to extract features, these features are added to the input features to obtain the final output features. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer. Similarly, the second convolutional layer can also be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer.

[0154] For an example of a residual activation layer in a synthetic transform network, see [link to relevant documentation]. Figure 6C As shown, the residual activation layer can sequentially include an activation layer (such as a LeakyReLU activation layer), a convolutional layer, another activation layer (such as a tanh activation layer), a multiplication layer, and a stacking layer. There are no restrictions on the structure of this residual activation layer. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer; there are no restrictions, such as choosing a 1x1 convolutional layer.

[0155] Example 7: This example further describes the attention module involved in the above examples. In Example 6, an attention module in the synthetic transform network is involved. This attention module can be a cascaded hybrid attention module. By replacing RNAB with a cascaded hybrid attention module, the computational complexity of decoding can be significantly reduced while ensuring decoding performance as much as possible. For example, the cascaded hybrid attention module can extract attention weights from the input features and determine the output features of the cascaded hybrid attention module based on the attention weights and the input features.

[0156] For example, a cascaded hybrid attention module may include a first attention submodule (also called a transformer-based attention submodule, denoted as Transformer-basedAttention Module) and a second attention submodule (also called an upgraded convolutional submodule, denoted as ConvNext Block). The first and second attention submodules can be two sequential submodules, meaning the output features of the first attention submodule are the input features of the second attention submodule. For example, see... Figure 7B The diagram shows a cascaded hybrid attention module, where the input features (or input image) serve as the first attention sub-module. Figure 7B Taking the converter attention submodule as an example, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. The output features of the first attention submodule are used as the input features of the second attention submodule. Figure 7B Taking the upgraded convolutional submodule as an example, the input features of the second attention submodule are processed by the second attention submodule to obtain the output features of the second attention submodule. The output features of the second attention submodule are used as the output features of the cascaded hybrid attention module, which is the final output feature.

[0157] For example, after the input features of the first attention submodule are input to the first attention submodule, the first attention submodule performs a first processing on the input features to obtain the output features of the first attention submodule. The first processing may include, but is not limited to, at least one of layer normalization, convolution operation, and dimension transformation operation. This first processing is not limited and is related to the network structure of the first attention submodule; the network structure of the first attention submodule can be arbitrarily configured.

[0158] For example, after the input features of the second attention submodule are input to the second attention submodule, the second attention submodule performs a second processing on the input features to obtain the output features of the second attention submodule. The second processing may include, but is not limited to, at least one of the following: depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. There are no restrictions on this second processing; it is related to the network structure of the second attention submodule, and the network structure of the second attention submodule can be arbitrarily configured.

[0159] For example, the second processing may include at least one of depthwise separable convolution operation, layer normalization, and multilayer perceptron operation. Alternatively, the second processing may include at least one of layer normalization, linear operation, and activation operation. Or, the second processing may include at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.

[0160] Example 8: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on the above examples. In Example 6, an attention module in the synthetic transform network is involved. This attention module can be a parallel hybrid attention module. By replacing RNAB with a parallel hybrid attention module, the computational complexity of decoding can be significantly reduced while ensuring decoding performance as much as possible. For example, the parallel hybrid attention module can extract attention weights from the input features and determine the output features of the parallel hybrid attention module based on the attention weights and the input features.

[0161] For example, a parallel hybrid attention module may include a first attention submodule (which may also be called a transformer attention submodule) and a second attention submodule (which may also be called an upgraded convolution submodule). The first and second attention submodules can be two parallel submodules; for example, the input features of the first and second attention submodules can be parallel, and the output features of the first and second attention submodules can be parallel. See, for example... Figure 7CThe diagram illustrates a parallel hybrid attention module. The input features (or input image) of the parallel hybrid attention module need to undergo channel splitting, that is, the input features of the parallel hybrid attention module are split into two sub-features (or two sub-images) along the channel dimension. These two sub-features serve as the first attention sub-module. Figure 7B The input features of the transformer attention submodule and the second attention submodule (taking the transformer attention submodule as an example) Figure 7B Taking the upgraded convolutional submodule as an example, the input features of the two sub-features can be two sub-features of equal or unequal size. Obviously, since these two sub-features are two sub-features after channel splitting, they can be parallel, that is, the two sub-features are output in parallel after channel splitting. In other words, the input features of the first attention submodule and the input features of the second attention submodule are parallel.

[0162] After the channel splitting process, one sub-feature resulting from the channel splitting serves as the input feature of the first attention submodule. This input feature is processed by the first attention submodule to obtain its output feature. Similarly, the other sub-feature resulting from the channel splitting serves as the input feature of the second attention submodule. This input feature is processed by the second attention submodule to obtain its output feature.

[0163] Then, the output features of the first attention submodule and the second attention submodule can be concatenated along the channel dimension to obtain the output features of the parallel hybrid attention module (i.e., the concatenated features serve as the output features of the parallel hybrid attention module). In other words, the output features of the two submodules are concatenated along the channel dimension to obtain the output features of the parallel hybrid attention module. Clearly, since the output features of these two submodules need to be concatenated along the channel dimension, the output features of these two submodules are parallel; that is, the output features of the first attention submodule and the second attention submodule are parallel.

[0164] For example, after the input features of the first attention submodule are input to the first attention submodule, the first attention submodule performs a first processing on the input features to obtain the output features of the first attention submodule. The first processing may include, but is not limited to, at least one of layer normalization, convolution operation, and dimension transformation operation. This first processing is not limited and is related to the network structure of the first attention submodule; the network structure of the first attention submodule can be arbitrarily configured.

[0165] For example, after the input features of the second attention submodule are input to the second attention submodule, the second attention submodule performs a second processing on the input features to obtain the output features of the second attention submodule. The second processing may include, but is not limited to, at least one of the following: depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. There are no restrictions on this second processing; it is related to the network structure of the second attention submodule, and the network structure of the second attention submodule can be arbitrarily configured.

[0166] For example, the second processing may include at least one of depthwise separable convolution operation, layer normalization, and multilayer perceptron operation. Alternatively, the second processing may include at least one of layer normalization, linear operation, and activation operation. Or, the second processing may include at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.

[0167] Example 9: In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a first attention submodule. The first attention submodule performs a first processing on the input features to obtain the output features. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; the query vector, the key vector, and the value vector are respectively subjected to dimensionality transformation operations to obtain dimension-transformed query vector, dimension-transformed key vector, and dimension-transformed value vector; attention weights are determined based on the dimension-transformed query vector and the dimension-transformed key vector, and the corrected features corresponding to the input features are determined based on the attention weights and the dimension-transformed value vector; the output features of the first attention submodule are determined based on the input features and the corrected features.

[0168] See Figure 8A The diagram shown is a structural illustration of the first attention submodule. This is merely an example of the first attention submodule, and its structure is not limited. Figure 8A Let's take the first attention submodule as an example for explanation.

[0169] The first attention submodule may include a layer normalization layer. This layer normalizes the input features (or input image) of the first attention submodule, resulting in layer-normalized features. Layer normalization is a neural network regularization technique used to standardize the input features of each layer of the neural network, improving training and generalization performance. Layer normalization operations may include calculating the mean and variance, standardization, scaling, and translation, ensuring that the output of each hidden unit is within a relatively small range and that each hidden unit has a similar distribution across the entire dataset.

[0170] After obtaining the layer-normalized features, a three-way convolution operation can be performed on these features to obtain the query vector Q, key vector K, and value vector V. For example, the query vector Q can be obtained by performing a convolution operation on the layer-normalized features using two convolutional layers. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 1x1 convolutional layer. The second convolutional layer can also be a 3x3, 5x5, or 7x7 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer. (The last sentence is a repetition of the first and can be omitted.) The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; a 3x3 convolutional layer is a good choice. Two convolutional layers can be used to convolve the normalized features to obtain a value vector V. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; a 1x1 convolutional layer is a good choice. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; a 3x3 convolutional layer is a good choice.

[0171] The query vector Q, key vector K, and value vector V are each subjected to a dimensionality transformation operation (i.e., the R operation), resulting in dimensionally transformed query vector Q, dimensionally transformed key vector K, and dimensionally transformed value vector V. The purpose of the dimensionality transformation operation is to transform the transformed tensors into the dimensional form required by the multi-head self-attention mechanism. Multi-head self-attention is a widely used technique in Natural Language Processing (NLP) tasks. This technique constructs connections between context words through self-attention mechanisms, thereby reconstructing the representation of the target word based on the context. Multi-head attention is a combination of multiple self-attention components, capable of learning various types of contextual influences. By passing the outputs of multiple self-attention mechanisms through a parameter matrix to obtain a new output, multi-head attention can capture a wider range of relevant features, increasing the model's expressive power.

[0172] Attention weights are determined based on the dimension-transformed query vector Q and the dimension-transformed key vector K. For example, the dimension-transformed query vector Q and the dimension-transformed key vector K are multiplied by a matrix, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights. There are no restrictions on the processing of this softmax function.

[0173] The corrected features corresponding to the input features are determined based on the attention weights and the value vector V after dimensional transformation. For example, the value vector V after dimensional transformation is multiplied by the attention weights, the result of the matrix multiplication is transformed by dimensional transformation (i.e., the R operation), and the features after dimensional transformation are convolved to obtain the corrected features corresponding to the input features. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer; a 1x1 convolutional layer can be selected.

[0174] The output features of the first attention submodule are determined based on the input features and the modified features. For example, the input features and the modified features are matrix-added to obtain the output features of the first attention submodule.

[0175] This completes the processing of the first attention submodule and yields its output features.

[0176] Example 10: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on the above examples. In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a first attention submodule. The first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are subjected to layer normalization to obtain layer-normalized features; the layer-normalized features are windowed to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature; the small-size query features corresponding to multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to multiple small-size features are concatenated to obtain a key vector; and the small-size value features corresponding to multiple small-size features are concatenated to obtain a value vector. The query vector, key vector, and value vector are transformed into dimension vectors, key vectors, and value vectors respectively. Attention weights are determined based on the transformed query vector and key vector, and correction features are determined based on the attention weights and the transformed value vector. The output features of the first attention submodule are determined based on the input features and correction features.

[0177] For example, in Figure 8A Based on the first attention submodule, a window partitioning layer can be added after the layer normalization layer. The structure of the first attention submodule is not restricted in this regard; please refer to [link to relevant documentation]. Figure 8A As shown.

[0178] For example, the first attention submodule may include a layer normalization layer, which can be used to normalize the input features (or input image) of the first attention submodule to obtain the normalized features.

[0179] The normalized features of a layer can be divided into multiple smaller features by windowing. For example, the normalized features can be divided into multiple smaller features, each with a width of wi and a height of hi. The widths and heights of different smaller features can be the same or different. Dividing the features into multiple smaller features for subsequent calculations can reduce complexity.

[0180] For each small-sized feature, a three-way convolution operation is performed on it to obtain the corresponding small-sized query feature, small-sized key feature, and small-sized value feature. For example, the small-sized query feature is obtained by performing a convolution operation on it using two convolutional layers. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; a 1x1 convolutional layer is chosen. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; a 3x3 convolutional layer is chosen. Similarly, the small-sized key feature is obtained by performing a convolution operation on it using two convolutional layers. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; a 1x1 convolutional layer is chosen. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; for example, a 3x3 convolutional layer can be selected. This small-sized feature is convolved using two convolutional layers to obtain a small-value feature. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; for example, a 1x1 convolutional layer can be selected. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; for example, a 3x3 convolutional layer can be selected.

[0181] The query vector Q can be obtained by concatenating the small-size query features corresponding to all small-size features, and there are no restrictions on the concatenation process. The key vector K can be obtained by concatenating the small-size key features corresponding to all small-size features, and the value vector V can be obtained by concatenating the small-size value features corresponding to all small-size features.

[0182] The query vector Q, key vector K, and value vector V can be transformed by dimensionality transformation (i.e., R operation) to obtain the transformed query vector Q, transformed key vector K, and transformed value vector V.

[0183] Attention weights can be determined based on the dimension-transformed query vector Q and the dimension-transformed key vector K. For example, the dimension-transformed query vector Q and the dimension-transformed key vector K are multiplied by a matrix, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights. There are no restrictions on the processing of this softmax function.

[0184] The corrected features corresponding to the input features can be determined based on the attention weights and the value vector V after dimensional transformation. For example, the value vector V after dimensional transformation is multiplied by the attention weights, the result of the matrix multiplication is transformed by dimensional transformation (i.e., the R operation), and the features after dimensional transformation are convolved to obtain the corrected features corresponding to the input features. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer; a 1x1 convolutional layer can be selected.

[0185] The output features of the first attention submodule can be determined based on the input features and the modified features. For example, the output features of the first attention submodule can be obtained by performing matrix addition on the input features and the modified features.

[0186] This completes the processing of the first attention submodule and yields its output features.

[0187] Example 11: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on the above examples. In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a first attention submodule. The first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are subjected to layer normalization to obtain layer-normalized features; the layer-normalized features are windowed to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature. The small-size query features corresponding to multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to multiple small-size features are concatenated to obtain a value vector. Attention weights are determined based on the query vector and the key vector. Intermediate features are then determined based on the attention weights and the value vector. Layer normalization is performed on these intermediate features to obtain the corrected features corresponding to the input features. The output features of the first attention submodule are then determined based on these input features and the corrected features.

[0188] See Figure 8B The diagram shown is a structural illustration of the first attention submodule. This is merely an example of the first attention submodule, and its structure is not limited. Figure 8B Let's take the first attention submodule as an example for explanation.

[0189] For example, the first attention submodule may include a layer normalization layer, which can be used to normalize the input features (or input image) of the first attention submodule to obtain the normalized features.

[0190] The first attention submodule may include a window partitioning layer, which can partition the normalized features of the layer into multiple small-sized features. For example, the normalized features of the layer can be divided into multiple small-sized features by the window partitioning layer. The width of each small-sized feature is wi, and the height of each small-sized feature is hi. The widths of different small-sized features can be the same or different, and the heights of different small-sized features can be the same or different.

[0191] For each small-sized feature, a three-way convolution operation is performed to obtain the corresponding small-sized query feature, small-sized key feature, and small-sized value feature. For example, convolutional layers (such as 1x1 and 3x3 layers) are used to obtain the small-sized query feature. Convolutional layers (such as 1x1 and 3x3 layers) are used to obtain the small-sized key feature. Convolutional layers (such as 1x1 and 3x3 layers) are used to obtain the small-sized value feature. The small-sized query features corresponding to all small-sized features are concatenated to obtain the query vector Q, the small-sized key features corresponding to all small-sized features are concatenated to obtain the key vector K, and the small-sized value features corresponding to all small-sized features are concatenated to obtain the value vector V.

[0192] Attention weights are determined based on the query vector Q and the key vector K. For example, the query vector Q and the key vector K are multiplied by a matrix, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights.

[0193] Based on the attention weights and value vector V, intermediate features are determined. For example, matrix multiplication is performed on the attention weights and value vector V, and the result of the matrix multiplication is used as the intermediate feature, that is, the intermediate feature corresponding to the input feature.

[0194] The intermediate feature is layer-normalized to obtain the corrected feature corresponding to the input feature. In other words, after the intermediate feature is layer-normalized, the layer-normalized feature is used as the corrected feature corresponding to the input feature.

[0195] Given the obtained attention weights and value vector V, in one example, the corrected features corresponding to the input features can be directly obtained based on these attention weights and value vector V. For instance, performing matrix multiplication on the attention weights and value vector V and using the result as the corrected features corresponding to the input features means that the corrected features can be removed. Figure 8B The second normalization layer in the process.

[0196] The output features of the first attention submodule can be determined based on the input features and the modified features. For example, the output features of the first attention submodule can be obtained by performing matrix addition on the input features and the modified features.

[0197] This completes the processing of the first attention submodule and yields its output features.

[0198] Example 12: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on the above examples. In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a first attention submodule. The first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are subjected to layer normalization to obtain layer normalized features; the layer normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; attention weights are determined based on the query vector and the key vector, and intermediate features are determined based on the attention weights and the value vector; the intermediate features are subjected to layer normalization to obtain the corrected features corresponding to the input features; the output features of the first attention submodule are determined based on the input features and the corrected features of the first attention submodule.

[0199] For example, in Figure 8B Based on the first attention submodule, the window partitioning layer after the layer normalization layer can be removed. The structure of the first attention submodule is not restricted in this regard; see [link to relevant documentation]. Figure 8B As shown.

[0200] For example, the first attention submodule may include a layer normalization layer, which can be used to normalize the input features (or input image) of the first attention submodule to obtain the normalized features.

[0201] The first attention submodule can include convolutional layers, which can perform three-way convolution operations on the normalized features of the layer to obtain the query vector Q, the key vector K, and the value vector V. For example, the query vector Q can be obtained by performing convolution operations on the normalized features of the layer using convolutional layers (such as 1x1 and 3x3 convolutional layers). The key vector K can be obtained by performing convolution operations on the normalized features of the layer using convolutional layers (such as 1x1 and 3x3 convolutional layers). The value vector V can be obtained by performing convolution operations on the normalized features of the layer using convolutional layers (such as 1x1 and 3x3 convolutional layers).

[0202] Attention weights are determined based on the query vector Q and the key vector K. For example, the query vector Q and the key vector K are multiplied by a matrix, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights.

[0203] Based on the attention weights and value vector V, intermediate features are determined. For example, matrix multiplication is performed on the attention weights and value vector V, and the result of the matrix multiplication is used as the intermediate feature, that is, the intermediate feature corresponding to the input feature.

[0204] The intermediate feature is layer-normalized to obtain the corrected feature corresponding to the input feature. In other words, after the intermediate feature is layer-normalized, the layer-normalized feature is used as the corrected feature corresponding to the input feature.

[0205] The output features of the first attention submodule can be determined based on the input features and the modified features. For example, the output features of the first attention submodule can be obtained by performing matrix addition on the input features and the modified features.

[0206] Given the obtained attention weights and value vector V, in one example, the corrected features corresponding to the input features can be directly obtained based on these attention weights and value vector V. For instance, performing matrix multiplication on the attention weights and value vector V and using the result as the corrected features corresponding to the input features means that the corrected features can be removed. Figure 8B The second normalization layer in the process.

[0207] This completes the processing of the first attention submodule and yields its output features.

[0208] In Examples 9-12, several examples of the first attention submodule are given. In this example, the structure of the first attention submodule is not limited. The input features can be processed to obtain the output features.

[0209] Example 13: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on the above examples. In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a second attention submodule. The second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a depthwise separable convolution operation is performed on the input features of the second attention submodule to obtain the convolutional features (i.e., the features after the depthwise separable convolution operation); layer normalization is performed on the convolutional features to obtain the layer-normalized features; multilayer perceptron operation is performed on the layer-normalized features to obtain the multilayer perceptron features; and the output features of the second attention submodule are determined based on the input features and the multilayer perceptron features.

[0210] See Figure 8C The diagram shown is a structural illustration of the second attention submodule. This is merely an example of the second attention submodule, and its structure is not limited. Figure 8C The second attention submodule will be used as an example for explanation.

[0211] For example, the second attention submodule may include a depthwise separable convolutional layer, which can be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without limitation. For example, a 7x7 convolutional layer can be selected. The input features of the second attention submodule can be subjected to depthwise separable convolution operations through the depthwise separable convolutional layer, without limitation on this convolution operation, to obtain the convolutional features (i.e., the features after the depthwise separable convolution operation).

[0212] The second attention submodule may include a layer normalization layer, which normalizes the convolutional features to obtain normalized features. The layer normalization operations include calculating the mean and variance, standardization, scaling, and translation, ensuring that the output of each hidden unit is within a relatively small range and that each hidden unit has a similar distribution in the dataset.

[0213] The second attention submodule may include a multilayer perceptron, which can perform multilayer perceptron operations on the layer-normalized features to obtain multilayer perceptron-processed features. See also Figure 8DThe diagram shows the structure of a multilayer perceptron. A multilayer perceptron can include linear layers, activation layers, random deactivation layers, and so on. This is just an example of a multilayer perceptron, and its structure is not limited. Based on this, the features after layer normalization are sequentially passed through linear layers, activation layers, random deactivation layers, linear layers, and random deactivation layers to obtain the multilayer perceptron features. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a Gelu layer; the type of activation layer is not limited. The random deactivation layer can be a DropOut layer; the type of random deactivation layer is also not limited.

[0214] The output features of the second attention submodule are determined based on the input features and the features after multi-layer perception. For example, the output features are obtained by performing matrix addition on the input features and the features after multi-layer perception.

[0215] This completes the processing of the second attention submodule and yields its output features.

[0216] Example 14: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on the above examples. In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a second attention submodule. The second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule can be layer normalized to obtain layer normalized features; the layer normalized features can be subjected to a first linear operation to obtain features after the first linear operation; the features after the first linear operation can be activated to obtain activated features; the activated features can be subjected to a second linear operation to obtain features after the second linear operation; the output features of the second attention submodule can be determined based on the input features of the second attention submodule and the features after the second linear operation.

[0217] See Figure 8E The diagram shown is a structural illustration of the second attention submodule. This is merely an example of the second attention submodule, and its structure is not limited. Figure 8E The second attention submodule will be used as an example for explanation.

[0218] For example, the second attention submodule may include a layer normalization layer, which can normalize the input features of the second attention submodule to obtain normalized features. The second attention submodule may include a linear layer 1, which can perform linear operations on the normalized features to obtain linearly operated features.

[0219] The second attention submodule may include an activation layer, which can be used to activate the features after linear operations to obtain activated features. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; the type of activation layer is not limited, but a ReLU layer can be selected. The second attention submodule may also include a linear layer 2, which can be used to perform linear operations on the activated features to obtain features after linear operations.

[0220] The output features of the second attention submodule are determined based on the input features and the features after linear operation. For example, the output features are obtained by performing matrix addition on the input features and the features after linear operation.

[0221] This completes the processing of the second attention submodule and yields its output features.

[0222] Example 15: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on the above examples. In Examples 7 and 8, both the cascaded hybrid attention module and the parallel hybrid attention module include a second attention submodule. The second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule are downsampled to obtain downsampled features; the downsampled features are subjected to residual convolution to obtain residual convolution features; the residual convolution features are upsampled to obtain upsampled features; the upsampled features are activated to obtain activated features; and the input features are subjected to residual convolution to obtain convolutional features. The output features of the second attention submodule are determined based on the input features, the activated features, and the convolutional features.

[0223] See Figure 8F The diagram shown is a structural illustration of the second attention submodule. This is merely an example of the second attention submodule, and its structure is not limited. Figure 8F The second attention submodule will be used as an example for explanation.

[0224] For example, the second attention submodule may include a downsampling layer, which can be a convolutional layer with a kernel size of 3x3 and a stride of 2, or a convolutional layer with a kernel size of 3x3 and a stride of 4, or a convolutional layer with a kernel size of 4x4 and a stride of 2, or a convolutional layer with a kernel size of 4x4 and a stride of 4, or a convolutional layer with a kernel size of 5x5 and a stride of 2, or a convolutional layer with a kernel size of 5x5 and a stride of 4. Alternatively, the downsampling layer may also be an inverse pixel shuffle layer. The structure of this downsampling layer is not limited; for example, a convolutional layer with a kernel size of 3x3 and a stride of 2 can be selected as the downsampling layer. Based on this, the input features of the second attention submodule can be downsampled using the downsampling layer. The implementation of this downsampling operation is not limited, resulting in downsampled features.

[0225] The second attention submodule can include M residual convolutional layers, where M can be a positive integer, such as 1, 2, 3, 4, etc. These M residual convolutional layers perform residual convolution operations on the downsampled features to obtain the residual convolutional features. For example, the first residual convolutional layer performs a residual convolution operation on the downsampled features, the result is then passed to the second residual convolutional layer for further residual convolution, and so on, until the last residual convolutional layer outputs the residual convolutional features.

[0226] See Figure 8G The diagram shows a schematic of a residual convolutional layer. This residual convolutional layer can include convolutional layer 1, an activation layer, and convolutional layer 2. Convolutional layer 1 performs convolution processing on the input features of the residual convolutional layer to obtain convolutionally processed features. The activation layer then activates the convolutionally processed features to obtain activated features. Convolutional layer 2 further processes the activated features to obtain convolutionally processed features. Finally, the input features of the residual convolutional layer and the convolutionally processed features are matrix-sumped to obtain the output features of the residual convolutional layer.

[0227] Convolutional layer 1 can be a 1x1, 3x3, 5x5, or 5x5 convolutional layer; there are no restrictions on which type to choose. For example, a 3x3 convolutional layer can be selected. Convolutional layer 2 can also be a 1x1, 3x3, 5x5, or 5x5 convolutional layer; there are no restrictions on which type to choose. For example, a 3x3 convolutional layer can be selected. Activation layers can be ReLU, Leaky ReLU, Sigmoid, Tanh, or GeLU; there are no restrictions on which type to choose. For example, a ReLU or Sigmoid layer can be selected.

[0228] See Figure 8HThe diagram shows another structural schematic of a residual convolutional layer. This residual convolutional layer can include convolutional layers and activation layers. The input features of the residual convolutional layer are convolved by the convolutional layers to obtain convolutionally processed features. The convolutionally processed features are then activated by the activation layers to obtain activated features. Finally, the input features of the residual convolutional layer and the activated features are matrix-summed to obtain the output features of the residual convolutional layer.

[0229] The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 5x5 convolutional layer; there are no restrictions on this, such as choosing a 3x3 convolutional layer. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; there are no restrictions on this, such as choosing a ReLU layer or a sigmoid layer.

[0230] The second attention submodule may include an upsampling layer, which can be a deconvolutional layer with a kernel size of 3x3 and a stride of 2, or a deconvolutional layer with a kernel size of 3x3 and a stride of 4, or a deconvolutional layer with a kernel size of 4x4 and a stride of 2, or a deconvolutional layer with a kernel size of 4x4 and a stride of 4, or a deconvolutional layer with a kernel size of 5x5 and a stride of 2, or a deconvolutional layer with a kernel size of 5x5 and a stride of 4. Alternatively, the upsampling layer may also be a pixel reconstruction layer. The structure of this upsampling layer is not limited; for example, a deconvolutional layer with a kernel size of 4x4 and a stride of 2 can be selected as the upsampling layer. Based on this, the features obtained after residual convolution (i.e., the features obtained after M residual convolutional layers) can be upsampled using the upsampling layer. The implementation of this upsampling operation is not limited, resulting in upsampled features.

[0231] The second attention submodule may include an activation layer, which can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer. The structure of this activation layer is not limited; for example, a ReLU layer can be selected. Based on this, activation operations can be performed on the upsampled features through the activation layer to obtain activated features.

[0232] The second attention submodule can include N residual convolutional layers, where N can be a positive integer, such as 1, 2, 3, 4, etc. N and M can be the same or different. The input features of the second attention submodule can be subjected to residual convolution operations through these N residual convolutional layers to obtain the convolutional features after residual convolution. For example, the input features of the second attention submodule are subjected to residual convolution operations through the first residual convolutional layer, the result is subjected to residual convolution operations through the second residual convolutional layer, and so on, until the last residual convolutional layer outputs the convolutional features after residual convolution.

[0233] See Figure 8G The diagram shown is a schematic of a residual convolutional layer, which may include convolutional layer 1, an activation layer, and convolutional layer 2. (See also...) Figure 8H The diagram shown is another structural schematic of a residual convolutional layer. This residual convolutional layer may include a convolutional layer and an activation layer. The residual convolution operation can be performed using the above-mentioned residual convolutional layer, which will not be described in detail here.

[0234] See Figure 8F As shown, after obtaining the convolutional features after residual convolution (i.e., the features output after N residual convolutional layers) and the activated features (i.e., the features output after the activation layer), the output features of the second attention submodule can be determined based on the input features, activated features, and convolutional features after residual convolution. For example, the activated features and the convolutional features after residual convolution can be multiplied by a matrix, and the result can be added to the input features of the second attention submodule to obtain the output features of the second attention submodule.

[0235] This completes the processing of the second attention submodule and yields its output features.

[0236] In Examples 13-15, several examples of the second attention submodule are given. In this example, the structure of the second attention submodule is not limited. The input features can be processed to obtain the output features.

[0237] Example 16: The implementation method of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0238] In Embodiments 1, 2, 3, and 4, a synthetic transform network is involved. This synthetic transform network can include at least an attention module, and the attention module can be a first attention submodule. Of course, besides the first attention submodule, the synthetic transform network can also include other network layers. This embodiment does not limit the structure of the synthetic transform network, as long as it includes the first attention submodule. For example, the synthetic transform network can also include at least one deconvolution layer, and the first attention submodule can be located after any deconvolution layer. Alternatively, the first attention submodule can be located after each deconvolution layer. Or, the first attention submodule can be located after a subset of deconvolution layers (such as K deconvolution layers out of all deconvolution layers, where K is greater than 1 and K is less than the total number of deconvolution layers).

[0239] For example, the first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. In one possible implementation, the structure of the first attention submodule can be seen in Embodiment 9; in another possible implementation, the structure of the first attention submodule can be seen in Embodiment 10; in another possible implementation, the structure of the first attention submodule can be seen in Embodiment 11; and in another possible implementation, the structure of the first attention submodule can be seen in Embodiment 12. Of course, the above are just a few examples of the first attention submodule, and the structure of the first attention submodule is not limited.

[0240] Example 17: The implementation method of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0241] In Embodiments 1, 2, 3, and 4, a synthetic transform network is involved. This synthetic transform network can include at least an attention module, and the attention module can be a second attention submodule. Of course, besides the second attention submodule, the synthetic transform network can also include other network layers. This embodiment does not limit the structure of the synthetic transform network, as long as it includes a second attention submodule. For example, the synthetic transform network can also include at least one deconvolution layer, and the second attention submodule can be located after any deconvolution layer. Alternatively, the second attention submodule can be located after each deconvolution layer. Or, the second attention submodule can be located after a subset of deconvolution layers (such as K deconvolution layers out of all deconvolution layers, where K is greater than 1 and K is less than the total number of deconvolution layers).

[0242] For example, the second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. In one possible implementation, the structure of the second attention submodule can be seen in Embodiment 13; in another possible implementation, the structure of the second attention submodule can be seen in Embodiment 14; and in yet another possible implementation, the structure of the second attention submodule can be seen in Embodiment 15. Of course, the above are just a few examples of the second attention submodule, and the structure of the second attention submodule is not limited.

[0243] As can be seen from the above technical solutions, in this application embodiment, for neural network-based encoding and decoding technology, an attention mechanism-based synthetic transformation network is proposed. This synthetic transformation network includes attention modules, which can be cascaded hybrid attention modules or parallel hybrid attention modules. When implementing the synthetic transformation network through cascaded or parallel hybrid attention modules, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, thus improving decoding performance. Furthermore, while reducing complexity, the quality of the reconstructed image blocks is effectively guaranteed, improving both encoding and decoding performance. By replacing the residual nonlocal attention module with a hybrid attention module, the computational complexity of the decoder is significantly reduced. For example, the second attention submodule in the hybrid attention module (cascaded or parallel hybrid attention module) only requires 21K floating-point multiply-accumulate operations per pixel, while the complexity of the first attention submodule is also much lower than that of the residual nonlocal attention module. Furthermore, the second attention submodule serves as the attention mechanism in the residual nonlocal attention module, and its superior performance is achieved by introducing a high-performance transformer structure. The 7x7 convolution operation in the second attention submodule introduces nonlocal information into the residual nonlocal attention module, and the random deactivation layer in the second attention submodule increases the robustness of the network. Based on these functions, the cascaded or parallel hybrid attention modules can reduce computational complexity without incurring excessive performance loss.

[0244] For example, the above embodiments can be implemented individually or in combination. For instance, each of embodiments 1-17 can be implemented individually, and at least two embodiments 1-17 can be implemented in combination.

[0245] For example, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0246] Based on the same application concept as the above method, this application also proposes a decoding device, which is applied to a decoding end. The decoding device includes: a memory configured to store video data; and a decoder configured to implement the decoding methods in embodiments 1-17 above, i.e., the processing flow of the decoding end.

[0247] For example, in one possible implementation, the decoder is configured to:

[0248] Decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block;

[0249] Based on the coefficient hyperparameter features, the probability distribution parameters are determined, and based on the probability distribution parameters, the second bitstream corresponding to the current image block is decoded to obtain the reconstructed features corresponding to the current image block.

[0250] The reconstructed features are input into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block;

[0251] The synthetic transformation network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules. Alternatively, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0252] Based on the same application concept as the above method, this application also proposes an encoding device, which is applied to the encoding end. The encoding device includes: a memory configured to store video data; and an encoder configured to implement the encoding methods in embodiments 1-17 above, i.e., the processing flow of the encoding end.

[0253] For example, in one possible implementation, the encoder is configured to:

[0254] Decode the first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block;

[0255] Based on the coefficient hyperparameter features, the probability distribution parameters are determined, and based on the probability distribution parameters, the second bitstream corresponding to the current image block is decoded to obtain the reconstructed features corresponding to the current image block.

[0256] The reconstructed features are input into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block;

[0257] The synthetic transformation network includes at least an attention module, which is either a cascaded hybrid attention module or a parallel hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules. Alternatively, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0258] Based on the same concept as the above method, the decoding device (also known as a video decoder) provided in this application embodiment, from a hardware perspective, its hardware architecture diagram can be found in [reference needed]. Figure 9A As shown, it includes: a processor 901 and a machine-readable storage medium 902, the machine-readable storage medium 902 storing machine-executable instructions that can be executed by the processor 901; the processor 901 is used to execute the machine-executable instructions to implement the decoding method of embodiments 1-17 of this application.

[0259] Based on the same concept as the above method, the encoding end device (also called a video encoder) provided in this application embodiment, from a hardware perspective, its hardware architecture diagram can be found in [reference needed]. Figure 9B As shown, it includes: a processor 911 and a machine-readable storage medium 912, the machine-readable storage medium 912 storing machine-executable instructions that can be executed by the processor 911; the processor 911 is used to execute the machine-executable instructions to implement the encoding methods of embodiments 1-17 of this application.

[0260] Based on the same application concept as the methods described above, this application provides an electronic device. It includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor executes the machine-executable instructions to implement the decoding or encoding methods of embodiments 1-17 of this application described above.

[0261] Based on the same application concept as the above methods, this application embodiment also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, implement the decoding method or encoding method of embodiments 1-17 of this application, such as the decoding method or encoding method in the above embodiments.

[0262] Based on the same concept as the above methods, this application also provides a computer application that, when executed by a processor, can implement the decoding or encoding methods disclosed in the above examples of this application.

[0263] Based on the same application concept as the above method, this application also proposes a decoding device that can be applied to a decoding end. The decoding device includes: a decoding module, used to decode a first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter features; and decode a second bitstream corresponding to the current image block based on the probability distribution parameter to obtain the reconstructed features corresponding to the current image block; and a processing module, used to input the reconstructed features to a synthetic transform network to obtain the reconstructed image block corresponding to the current image block; wherein the synthetic transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0264] For example, if the attention module is a cascaded hybrid attention module, the processing module is further configured to perform a first processing on the input features of the first attention submodule through the first attention submodule to obtain the output features of the first attention submodule, and the output features of the first attention submodule are the input features of the second attention submodule; wherein, the first processing is at least one of layer normalization, convolution operation, and dimension transformation operation; and to perform a second processing on the input features of the second attention submodule through the second attention submodule to obtain the output features of the second attention submodule; wherein, the second processing is at least one of depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation.

[0265] For example, if the attention module is a parallel hybrid attention module, the processing module is further configured to split the input features of the parallel hybrid attention module into two sub-features in the channel dimension, with the two sub-features serving as the input features of the first attention sub-module and the second attention sub-module, respectively; perform a first processing on the input features of the first attention sub-module through the first attention sub-module to obtain the output features of the first attention sub-module; the first processing is at least one of layer normalization, convolution operation, and dimension transformation operation; and perform a second processing on the input features of the second attention sub-module through the second attention sub-module to obtain the output features of the second attention sub-module; the second processing is at least one of depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation; and concatenate the output features of the first attention sub-module and the second attention sub-module in the channel dimension to obtain the output features of the parallel hybrid attention module.

[0266] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performs a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; performs dimensionality transformation operations on the query vector, the key vector, and the value vector respectively to obtain a dimension-transformed query vector, a dimension-transformed key vector, and a dimension-transformed value vector; determines attention weights based on the dimension-transformed query vector and the dimension-transformed key vector, and determines the corrected features corresponding to the input features based on the attention weights and the dimension-transformed value vector; and determines the output features of the first attention submodule based on the input features and the corrected features.

[0267] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; divides the layer-normalized features into windows to obtain multiple small-size features; performs a three-way convolution operation on each small-size feature to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature; and concatenates the small-size query features corresponding to the multiple small-size features to obtain a query vector. The small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector; the query vector, the key vector, and the value vector are subjected to dimensional transformation operations to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; attention weights are determined based on the dimensionally transformed query vector and the dimensionally transformed key vector, and the correction features corresponding to the input feature are determined based on the attention weights and the dimensionally transformed value vector; the output features of the first attention submodule are determined based on the input feature and the correction features.

[0268] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determining attention weights based on the query vector and the key vector, and determining intermediate features based on the attention weights and the value vector; performing layer normalization on the intermediate features to obtain the corrected features corresponding to the input features; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0269] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; divides the layer-normalized features into windows to obtain multiple small-size features; performs a three-way convolution operation on each small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to that small-size feature; concatenates the small-size query features corresponding to the multiple small-size features to obtain a query vector; concatenates the small-size key features corresponding to the multiple small-size features to obtain a key vector; concatenates the small-size value features corresponding to the multiple small-size features to obtain a value vector; determines attention weights based on the query vector and the key vector, and determines intermediate features based on the attention weights and the value vector; performs layer normalization on the intermediate features to obtain a corrected feature corresponding to the input feature; and determines the output features of the first attention submodule based on the input feature and the corrected feature.

[0270] For example, when the processing module performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, it specifically performs the following steps: performing a depthwise separable convolution operation on the input features of the second attention submodule to obtain convolutional features; performing layer normalization on the convolutional features to obtain layer-normalized features; performing a multilayer perception operation on the layer-normalized features to obtain multilayer perception features; and determining the output features of the second attention submodule based on the input features and the multilayer perception features.

[0271] For example, when the processing module performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, it specifically performs the following steps: performing layer normalization on the input features of the second attention submodule to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0272] For example, when the processing module performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, it specifically performs the following steps: performing a downsampling operation on the input features of the second attention submodule to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolutional features; performing an upsampling operation on the residual convolutional features to obtain upsampled features; performing an activation operation on the upsampled features to obtain activated features; performing a residual convolution operation on the input features to obtain convolutional features after residual convolution; and determining the output features of the second attention submodule based on the input features of the second attention submodule, the activated features, and the convolutional features.

[0273] For example, the synthetic transform network further includes at least one deconvolution layer; wherein: the attention module is located after any one of the deconvolution layers; or, the attention module is located after each deconvolution layer; the attention modules located after different deconvolution layers are completely identical, or the attention modules located after different deconvolution layers are not completely identical.

[0274] For example, the synthetic transformation network further includes a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, a second deconvolution layer, a second pruning layer, a second residual activation layer, a third deconvolution layer, a third pruning layer, a third residual activation layer, a fourth deconvolution layer, and a fourth pruning layer; the synthetic transformation network includes an attention module, which is located after the first deconvolution layer, or after the second deconvolution layer, or after the third deconvolution layer; or, the synthetic transformation network includes an attention module located after the first deconvolution layer, an attention module located after the second deconvolution layer, and an attention module located after the third deconvolution layer.

[0275] Based on the same application concept as the above method, this application also proposes an encoding device, which is applied at the encoding end. The device includes: a decoding module, used to decode a first bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter features; and decode a second bitstream corresponding to the current image block based on the probability distribution parameter to obtain the reconstructed features corresponding to the current image block; and a processing module, used to input the reconstructed features to a synthesis transform network to obtain the reconstructed image block corresponding to the current image block; wherein the synthesis transform network includes at least an attention module, which is a cascaded hybrid attention module or a parallel hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; or, the parallel hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two parallel submodules.

[0276] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above descriptions are merely embodiments of this application and are not intended to limit this application.

[0277] Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image decoding method, characterized in that, The method includes: Decode the bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; Based on the coefficient hyperparameter features, the probability distribution parameters are determined, and based on the probability distribution parameters, the bitstream corresponding to the current image block is decoded to obtain the reconstructed features corresponding to the current image block; The reconstructed features are input into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block; The synthetic transformation network includes at least a cascaded hybrid attention module, which is used to extract attention weights from input features and determine the output features of the cascaded hybrid attention module based on the attention weights and the input features. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, which are two serial submodules. The output features of the first attention submodule are the input features of the second attention submodule.

2. The method according to claim 1, characterized in that, The first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule; The first processing includes at least one of layer normalization, convolution operation, and dimension transformation operation.

3. The method according to claim 2, characterized in that, The first processing of the input features of the first attention submodule to obtain the output features of the first attention submodule includes: The input features of the first attention submodule are subjected to layer normalization to obtain the layer normalized features; The normalized features of the layer are subjected to a three-way convolution operation to obtain the query vector, key vector and value vector; Perform dimensional transformation operations on the query vector, the key vector, and the value vector respectively to obtain the dimension-transformed query vector, the dimension-transformed key vector, and the dimension-transformed value vector. Attention weights are determined based on the query vector and key vector after dimensional transformation, and the corrected features corresponding to the input feature are determined based on the attention weights and the value vector after dimensional transformation. The output features of the first attention submodule are determined based on the input features and the correction features.

4. The method according to claim 2, characterized in that, The first processing of the input features of the first attention submodule to obtain the output features of the first attention submodule includes: The input features of the first attention submodule are subjected to layer normalization to obtain the layer normalized features; The normalized features of the layer are divided into windows to obtain multiple small-sized features; For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector. Perform dimensional transformation operations on the query vector, the key vector, and the value vector respectively to obtain the dimension-transformed query vector, the dimension-transformed key vector, and the dimension-transformed value vector. Attention weights are determined based on the query vector and key vector after dimensional transformation, and the corrected features corresponding to the input feature are determined based on the attention weights and the value vector after dimensional transformation. The output features of the first attention submodule are determined based on the input features and the correction features.

5. The method according to claim 2, characterized in that, The first processing of the input features of the first attention submodule to obtain the output features of the first attention submodule includes: The input features of the first attention submodule are subjected to layer normalization to obtain the layer normalized features; The normalized features of the layer are subjected to a three-way convolution operation to obtain the query vector, key vector and value vector; Attention weights are determined based on the query vector and the key vector, and corrected features corresponding to the input feature are determined based on the attention weights and the value vector. The output features of the first attention submodule are determined based on the input features and the correction features.

6. The method according to claim 2, characterized in that, The first processing of the input features of the first attention submodule to obtain the output features of the first attention submodule includes: The input features of the first attention submodule are subjected to layer normalization to obtain the layer normalized features; The normalized features of the layer are divided into windows to obtain multiple small-sized features; For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector. Attention weights are determined based on the query vector and the key vector, and corrected features corresponding to the input feature are determined based on the attention weights and the value vector. The output features of the first attention submodule are determined based on the input features and the correction features.

7. The method according to claim 3 or 4, characterized in that, Determining the corrected feature corresponding to the input feature based on the attention weights and the value vector after dimensional transformation includes: Perform matrix multiplication on the value vector after dimensional transformation and the attention weights to obtain the result of matrix multiplication; Perform a dimension transformation operation on the result of the matrix multiplication to obtain the features after the dimension transformation operation; Perform a convolution operation on the features after the dimensionality transformation operation to obtain the corrected features corresponding to the input features.

8. An image encoding method, characterized in that, The method includes: Decode the bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; Based on the coefficient hyperparameter features, the probability distribution parameters are determined, and based on the probability distribution parameters, the bitstream corresponding to the current image block is decoded to obtain the reconstructed features corresponding to the current image block; The reconstructed features are input into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block; The synthetic transformation network includes at least a cascaded hybrid attention module, which is used to extract attention weights from input features and determine the output features of the cascaded hybrid attention module based on the attention weights and the input features. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, which are two serial submodules. The output features of the first attention submodule are the input features of the second attention submodule.

9. An image decoding device, characterized in that, The device includes: The decoding module is used to decode the bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine the probability distribution parameters based on the coefficient hyperparameter features; and decode the bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block. A processing module is used to input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least a cascaded hybrid attention module, the cascaded hybrid attention module is used to extract attention weights of the input features, and determine the output features of the cascaded hybrid attention module based on the attention weights and the input features; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; wherein the output features of the first attention submodule are the input features of the second attention submodule.

10. An image encoding device, characterized in that, The device includes: The decoding module is used to decode the bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine the probability distribution parameters based on the coefficient hyperparameter features; and decode the bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block. A processing module is used to input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least a cascaded hybrid attention module, the cascaded hybrid attention module is used to extract attention weights of the input features, and determine the output features of the cascaded hybrid attention module based on the attention weights and the input features; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules; wherein the output features of the first attention submodule are the input features of the second attention submodule.

11. An image decoding device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1-7.

12. An image encoding device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of claim 8.

13. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores a plurality of computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-8.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images