Decoding and encoding method, apparatus, and device thereof

Abstract

The present application provides a decoding and encoding method, an apparatus, and a device thereof. The decoding method comprises: decoding a code stream corresponding to a current image block to obtain coefficient hyperparametric features; on the basis of the coefficient hyperparametric features, determining probability distribution parameters, and on the basis of the probability distribution parameters, decoding another stream of the current image block to obtain reconstructed features; inputting the reconstructed features into a synthetic transformation network to obtain reconstructed image blocks; wherein the synthetic transformation network comprises an attention module, and the attention module is a cascaded mixed attention module; the cascaded mixed attention module comprises a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are connected in series.

Description

A decoding and encoding method, apparatus and equipment Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a decoding and encoding method, apparatus and device thereof. Background Technology

[0002] To save space, video images are encoded before transmission. Complete video encoding includes processes such as prediction, transform, quantization, entropy coding, and filtering. The prediction process can be divided into intra-frame prediction and inter-frame prediction. Inter-frame prediction utilizes temporal correlation to predict the current pixel using pixels from neighboring encoded images, effectively removing temporal redundancy. Intra-frame prediction utilizes spatial correlation to predict the current pixel using pixels from the encoded blocks of the current frame, removing spatial redundancy.

[0003] With the rapid development of deep learning, it has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning is also gradually being applied in the field of encoding and decoding, where neural networks can be used to encode and decode images. Although neural network-based encoding and decoding methods have shown great performance potential, they still suffer from problems such as poor decoding performance and high complexity.

[0004] Summary of the Invention

[0005] This application provides a decoding and encoding method, apparatus, and device that can improve decoding performance and reduce complexity.

[0006] This application provides a decoding method applied at a decoding end. The method includes: decoding a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstructed features corresponding to the current image block; and inputting the reconstructed features into a synthesis transform network to obtain a reconstructed image block corresponding to the current image block. The synthesis transform network includes at least an attention module, which is a cascaded hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0007] This application provides an encoding method applied at an encoding end. The method includes: decoding a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstructed features corresponding to the current image block; and inputting the reconstructed features into a synthesis transform network to obtain a reconstructed image block corresponding to the current image block. The synthesis transform network includes at least an attention module, which is a cascaded hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0008] This application provides a decoding device applied at a decoding end. The device includes: a decoding module, configured to decode a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features; and decode another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstructed features corresponding to the current image block; and a processing module, configured to input the reconstructed features into a synthesis transform network to obtain a reconstructed image block corresponding to the current image block; wherein the synthesis transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0009] This application provides an encoding device applied at an encoding end. The device includes: a decoding module, configured to decode a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features; and decode another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstructed features corresponding to the current image block; and a processing module, configured to input the reconstructed features into a synthesis transform network to obtain a reconstructed image block corresponding to the current image block; wherein the synthesis transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0010] This application provides a decoding device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-described decoding method.

[0011] This application provides an encoding terminal device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-described encoding method.

[0012] This application provides an electronic device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-described decoding method or encoding method.

[0013] This application provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, are used to implement the above-described decoding or encoding method.

[0014] This application provides a computer application that, when executed by a processor, implements the above-described decoding or encoding method.

[0015] As can be seen from the above technical solutions, in this application embodiment, for neural network-based encoding and decoding technology, an attention mechanism-based synthetic transformation network is proposed. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module. When the synthetic transformation network is implemented through the cascaded hybrid attention module, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, and the decoding performance is improved. While reducing complexity, the quality of the reconstructed image blocks is effectively guaranteed, and the encoding and decoding performance is improved. Attached Figure Description

[0016] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of this application;

[0017] Figure 2 is a schematic diagram of pixel recombination in one embodiment of this application;

[0018] Figure 3 is a flowchart of a decoding method according to one embodiment of this application;

[0019] Figure 4 is a schematic diagram of the processing procedure of the encoding end in one embodiment of this application;

[0020] Figure 5 is a schematic diagram of the processing procedure of the decoding end in one embodiment of this application;

[0021] Figures 6A-6D are schematic diagrams of the structure of the synthetic transformation network in one embodiment of this application;

[0022] Figures 7A and 7B are schematic diagrams of the structure of the synthetic transformation network in one embodiment of this application;

[0023] Figures 8A-8H are schematic diagrams of the structure of the synthetic transformation network in one embodiment of this application;

[0024] Figure 9A is a hardware structure diagram of a decoding device in one embodiment of this application;

[0025] Figure 9B is a hardware structure diagram of the encoding end device in one embodiment of this application. Detailed Implementation

[0026] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments and claims of this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any or all possible combinations including one or more of the associated listed items. It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this application, and similarly, second information may also be referred to as first information, depending on the context. Furthermore, the word “if” as used can be interpreted as “when,” “in response to a determination,” or “when…”.

[0027] This application proposes a decoding method, which may involve the following concepts:

[0028] Entropy coding: Entropy coding is a coding process that follows the principle of entropy without losing any information. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods can include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0029] Neural Networks (NNs): Neural networks are artificial neural networks, a computational model composed of numerous interconnected nodes (or neurons). In a neural network, neurons (often called processing units) can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. Processing units in a neural network can be categorized into three types: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; hidden units are located between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units; information representation and processing are reflected in the connections between processing units. Neural networks are a non-programmed, brain-like information processing method. The essence of a neural network is to achieve parallel and distributed information processing capabilities through the transformations and dynamics of the network, mimicking the information processing capabilities of the human brain's nervous system to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and fully connected networks.

[0030] Convolutional Neural Networks (CNNs): A CNN is a type of feedforward neural network and one of the most representative network structures in deep learning. The artificial neurons in a CNN can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. The basic structure of a CNN consists of two layers: the first is the feature extraction layer (also called the convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, extracting the features of that local area. Once the local feature is extracted, its positional relationship with other features is determined. The second is the feature mapping layer (also called the activation layer). Each computational layer of the neural network consists of multiple feature maps, each of which is a plane where all neurons have equal weights. Feature mapping structures can use functions such as the Sigmoid function, ReLU (Rectified Linear Unit), Leaky-ReLU, PReLU (Parametric Rectified Linear Unit), and GDN (Generalized Divisive Normalization) as activation functions for the CNN. Furthermore, because neurons on a single mapping plane share weights, the number of free parameters in the network is reduced.

[0031] For example, one advantage of convolutional neural networks (CNNs) over image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. Another advantage of CNNs over ordinary neural networks is that ordinary neural networks use fully connected layers, meaning all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, however, avoid this difficulty through local connections and weight sharing.

[0032] Deconvolution (also known as transposed convolution): The working process of a deconvolutional layer is very similar to that of a convolutional layer. The main difference is that a deconvolutional layer uses padding to make the output larger than the input (or the input can remain the same). If the stride is 1, it means the output size is equal to the input size; if the stride is N, it means the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0033] Depthwise Separable Convolution: This requires two convolution operations. In the first operation, a deepwise convolution is performed (collecting features from each layer), with a kernel size of K*K*1 and a total of K*K*Cin parameters. In the second operation, to obtain the output in the Cout dimension, the kernel size is 1*1*Cin, and the total number of parameters is 1*1*Cin*Cout. The output of the second convolution can be the output of the depthwise separable convolution, and the input of the first convolution can be the input of the depthwise separable convolution.

[0034] Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The goal of learning is to learn the patterns hidden behind the data. The trained network can also give appropriate outputs for data outside the learning set that have the same pattern. This ability can be called generalization ability.

[0035] A feature is a three-dimensional feature matrix or tensor of size C*W*H. See Figure 1 for a schematic diagram of a three-dimensional feature matrix. In this matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be either the input or output of a neural network.

[0036] Pixel shuffle: Also known as feature location reorganization, this technique reorganizes low-resolution features across multiple channels to obtain a high-resolution feature map. Figure 2 illustrates pixel shuffle, showing the result when the upsampling factor is 2. Clearly, a feature with 4 channels and a resolution of 3x3 becomes a feature with 1 channel and a resolution of 6x6 after passing through the feature location reorganization layer.

[0037] Rate-Distortion Optimized (RDBEM) principle: Two main metrics evaluate coding efficiency: bitrate and PSNR (Peak Signal-to-Noise Ratio). A smaller bitrate results in a higher compression ratio; a higher PSNR results in better reconstructed image quality. In mode selection, the discrimination formula is essentially a comprehensive evaluation of these two metrics. For example, the cost corresponding to a mode is: J(mode) = D + λ*R.

[0038] Here, D represents Distortion, which is typically measured using the SSE (sum-square error) metric. SSE refers to the sum of the mean squares of the differences between the reconstructed image patch and the source image. For cost considerations, the SAD (Sum of Absolute Difference) metric can also be used, which is the sum of the absolute values ​​of the differences between the reconstructed image patch and the source image. λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image patch in this mode, including the total number of bits needed for encoding mode information, motion information, residuals, etc. When selecting a mode, using the rate-distortion principle to compare and decide on encoding modes usually ensures optimal coding performance.

[0039] Numerous encoding tools have been proposed for various modules at the encoding end, and each tool often has multiple modes. The optimal encoding tool for different video sequences often differs. Therefore, during encoding, Rate-Distortion Optimization (RDO) is typically used to compare the encoding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information is transmitted by encoding marker information in the bitstream. Although this method introduces higher encoding complexity, it can adaptively select the optimal mode combination for different content to achieve the best encoding performance. The decoding end can obtain the relevant mode information by directly parsing the marker information, with minimal impact from complexity.

[0040] The end-to-end general framework for image coding mainly consists of a main feature information part and a priori edge information part. The main feature information part includes an analysis transform network, quantization, normal entropy coding, normal entropy decoding, and a synthesis transform network. The priori edge information part includes a priori analysis network, quantization, factorized entropy coding, factorized entropy decoding, and a priori synthesis network. The image component x is compressed, encoded, and reconstructed by the analysis transform network and the synthesis transform network of the main feature information part, respectively. The priori edge information part is mainly used to model the probability of the main feature information, guiding the entropy encoding and decoding of the main feature information. The end-to-end general framework for image coding suffers from high computational complexity and poor decoding performance.

[0041] To address the above findings, this embodiment proposes a synthetic transform network with an attention mechanism, leveraging the characteristics of an end-to-end image coding framework. This synthetic transform network includes attention modules, which are cascaded hybrid attention modules. By implementing the synthetic transform network through cascaded hybrid attention modules, the network complexity and computational complexity are effectively reduced while maintaining the quality of the synthesized image, thus improving decoding performance.

[0042] The decoding method in the embodiments of this application will be described in detail below with reference to several specific embodiments.

[0043] Example 1: This application proposes a decoding method. Referring to Figure 3, which is a flowchart of the decoding method, this method can be applied to a decoding end (also called a video decoder). The method may include:

[0044] Step 301: Decode a bitstream corresponding to the current image block to obtain the coefficient hyperparameter features of the current image block.

[0045] Step 302: Determine the probability distribution parameters based on the hyperparameter features of the coefficients, and decode another bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block.

[0046] Step 303: Input the reconstructed features into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block. The synthetic transformation network includes at least an attention module, and this attention module can be a cascaded hybrid attention module. The cascaded hybrid attention module can include a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two sequential submodules. For example, the output features of the first attention submodule are the input features of the second attention submodule.

[0047] Among them, one bitstream corresponding to the current image block can be a bitstream that encodes the coefficient hyperparameter features corresponding to the current image block; the other bitstream corresponding to the current image block can be a bitstream that encodes the residual features corresponding to the current image block.

[0048] For example, if the attention module is a cascaded hybrid attention module, the input features of the first attention submodule can be processed by the first attention submodule to obtain the output features of the first attention submodule, and the output features of the first attention submodule become the input features of the second attention submodule; wherein, the first processing can be at least one of layer normalization, convolution operation, and dimension transformation operation. The input features of the second attention submodule can be processed by the second attention submodule to obtain the output features of the second attention submodule; wherein, the second processing can be at least one of depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing can be at least one of depthwise separable convolution operation, layer normalization, and multilayer perceptron operation. Or, the second processing can be at least one of layer normalization, linear operation, and activation operation. Or, the second processing can be at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.

[0049] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule, including but not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; performing dimensionality transformation operations on the query vector, key vector, and value vector respectively to obtain dimension-transformed query vector, dimension-transformed key vector, and dimension-transformed value vector; determining attention weights based on the dimension-transformed query vector and dimension-transformed key vector; determining the corrected features corresponding to the input features based on the attention weights and dimension-transformed value vector; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0050] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; dividing the layer-normalized features into windows to obtain multiple small-sized features; performing a three-way convolution operation on each small-sized feature to obtain a small-sized query feature, a small-sized key feature, and a small-sized value feature corresponding to that small-sized feature; concatenating the small-sized query features corresponding to multiple small-sized features to obtain a query vector; concatenating the small-sized key features corresponding to multiple small-sized features to obtain a key vector; and concatenating the small-sized value features corresponding to multiple small-sized features to obtain a value vector. Then, dimensional transformation operations are performed on the query vector, the key vector, and the value vector respectively to obtain the dimension-transformed query vector, the dimension-transformed key vector, and the dimension-transformed value vector; attention weights are determined based on the dimension-transformed query vector and the dimension-transformed key vector, and the corrected features corresponding to the input features are determined based on the attention weights and the dimension-transformed value vector; the output features of the first attention submodule are determined based on the input features and the corrected features.

[0051] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determining attention weights based on the query vector and the key vector, and determining the corrected features corresponding to the input features based on the attention weights and the value vector; and determining the output features of the first attention submodule based on the input features and the corrected features.

[0052] In one possible implementation, the input features of the first attention submodule are processed by the first attention submodule to obtain the output features of the first attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; dividing the layer-normalized features into windows to obtain multiple small-sized features; performing a three-way convolution operation on each small-sized feature to obtain a small-sized query feature, a small-sized key feature, and a small-sized value feature corresponding to that small-sized feature; concatenating the small-sized query features corresponding to multiple small-sized features to obtain a query vector; concatenating the small-sized key features corresponding to multiple small-sized features to obtain a key vector; concatenating the small-sized value features corresponding to multiple small-sized features to obtain a value vector; determining attention weights based on the query vector and the key vector, and determining the correction features corresponding to the input features based on the attention weights and the value vector. Then, the output features of the first attention submodule can be determined based on the input features and the correction features.

[0053] In one possible implementation, the input features of the second attention submodule are subjected to a second processing to obtain the output features of the second attention submodule. This processing may include, but is not limited to: performing layer normalization on the input features of the second attention submodule to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0054] In one possible implementation, the input features of the second attention submodule are subjected to a second processing step to obtain the output features of the second attention submodule. This processing may include, but is not limited to: downsampling the input features of the second attention submodule to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolutional features; upsampling the residual convolutional features to obtain upsampled features; performing an activation operation on the upsampled features to obtain activated features; and performing a residual convolution operation on the input features to obtain convolutional features. Based on this, the output features of the second attention submodule can be determined based on the input features, the activated features, and the convolutional features.

[0055] In one possible implementation, the synthetic transform network may further include at least one deconvolutional layer; wherein: the attention module may be located after one of the deconvolutional layers.

[0056] For example, the synthetic transform network may further include a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, a second deconvolution layer, a second pruning layer, a second residual activation layer, a third deconvolution layer, a third pruning layer, a third residual activation layer, a fourth deconvolution layer, and a fourth pruning layer. The synthetic transform network may include at least one attention module, wherein one attention module is located after the first deconvolution layer.

[0057] For example, the synthetic transform network includes at least a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, and a second deconvolution layer. The synthetic transform network includes at least one attention module, wherein one attention module is located after the first deconvolution layer.

[0058] Based on the same concept as the decoding method described above, this application also proposes an encoding method that can be applied to the encoding end (also known as a video encoder). The method may include: decoding a bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain the reconstructed features corresponding to the current image block; and inputting the reconstructed features into a synthesis transform network to obtain the reconstructed image block corresponding to the current image block. The synthesis transform network includes at least an attention module, which is a cascaded hybrid attention module. The cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0059] Among them, one bitstream corresponding to the current image block can be a bitstream that encodes the coefficient hyperparameter features corresponding to the current image block; the other bitstream corresponding to the current image block can be a bitstream that encodes the residual features corresponding to the current image block.

[0060] For example, the implementation process of the encoding method is similar to that of the decoding method, and the repeated content will not be described again.

[0061] For example, the above execution order is merely an example for ease of description. In practical applications, the execution order between steps can be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the method may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments.

[0062] As can be seen from the above technical solutions, in this application embodiment, for neural network-based encoding and decoding technology, an attention mechanism-based synthetic transformation network is proposed. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module. When the synthetic transformation network is implemented through the cascaded hybrid attention module, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, and the decoding performance is improved. While reducing complexity, the quality of the reconstructed image blocks is effectively guaranteed, and the encoding and decoding performance is improved.

[0063] Example 2: The processing procedure at the encoding end can be seen in Figure 4. Of course, Figure 4 is just an example of the processing procedure at the encoding end. There are no restrictions on the processing procedure at the encoding end. The encoding process can be implemented using a neural network.

[0064] After obtaining the current image block x (which can be the original image block x, i.e., the input image block), the encoding end can perform analysis and transformation on the current image block x through an analysis and transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Specifically, performing feature transformation on the current image block x through the analysis and transformation network means transforming the current image block x to image features y in the latent domain, thereby facilitating all subsequent processes to be performed in the latent domain.

[0065] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the encoding and decoding process for image blocks can also be directly applied to the image.

[0066] After obtaining image features y, the encoder performs a coefficient hyperparameter feature transformation on y to obtain coefficient hyperparameter features z. For example, image features y can be input into a hyperparameter coding network (i.e., a neural network), which then performs the coefficient hyperparameter feature transformation on y to obtain coefficient hyperparameter features z. The hyperparameter coding network can be a trained neural network, and its training process is not restricted, as long as it can perform the coefficient hyperparameter feature transformation on image features y. The latent domain image features y, after passing through the hyperparameter coding network, yield the hyper-prior latent information z.

[0067] After obtaining the hyperparameter feature z, the encoder can quantize the hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the hyperparameter feature z. This is represented by the Q operation in Figure 4, which is the quantization process. After obtaining the hyperparameter quantization feature z, the hyperparameter quantization feature is encoded to obtain the Bitstream #1 (which can be called the first bitstream) corresponding to the current image patch. This is represented by the AE operation in Figure 4, which is the encoding process, such as entropy encoding. Alternatively, the encoder can directly encode the hyperparameter feature z to obtain the Bitstream #1 corresponding to the current image patch. The hyperparameter quantization feature or the hyperparameter feature z carried in Bitstream #1 is mainly used to obtain the parameters of the mean and probability distribution models.

[0068] After obtaining the Bitstream#1 corresponding to the current image block, the encoding end can send the Bitstream#1 corresponding to the current image block to the decoding end. For the processing of the Bitstream#1 corresponding to the current image block by the decoding end, please refer to the following embodiments.

[0069] After obtaining Bitstream #1 corresponding to the current image block, the encoder can decode Bitstream #1 to obtain the hyperparameter quantization feature (AD in Figure 4 represents the decoding process). Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat and the coefficient hyperparameter feature z can be the same or different. The IQ operation in Figure 4 represents the dequantization process. Alternatively, after obtaining Bitstream #1 corresponding to the current image block, the encoder can also decode Bitstream #1 to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the hyperparameter quantization feature.

[0070] For the encoding process of Bitstream#1, a fixed probability density model encoding method can be used; for the decoding process of Bitstream#1, a fixed probability density model decoding method can be used. There are no restrictions on the encoding and decoding processes.

[0071] After obtaining the hyperparameter feature z_hat, the encoder can perform context-based prediction based on the hyperparameter feature z_hat of the current image patch and the reconstructed feature y_hat of the previous image patch (the determination process of the reconstructed feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image patch. For example, the hyperparameter feature z_hat and the reconstructed feature y_hat are input into the mean prediction network, which determines the predicted value mu based on the hyperparameter feature z_hat and the reconstructed feature y_hat. This prediction process is not restricted. Specifically, for the context-based prediction process, the input includes the hyperparameter feature z_hat and the decoded reconstructed feature y_hat. The two are jointly input to obtain a more accurate predicted value mu. The predicted value mu is used to subtract the original feature to obtain the residual and add it to the decoded residual to obtain the reconstructed feature y_hat.

[0072] In another implementation, the mean prediction network can also obtain the predicted value of the current image patch based solely on the coefficient hyperparameter feature z_hat of the current image patch, without utilizing the reconstructed features y_hat of previous image patches. This significantly reduces the complexity of obtaining the predicted value and accelerates the encoding and decoding process. The methods for obtaining the predicted value at the encoding end can all adopt one of the two schemes described above, and will not be elaborated further.

[0073] It should be noted that the mean prediction network is an optional neural network, meaning that it can be omitted, i.e., it is not necessary to determine the predicted value mu through the mean prediction network. The dashed box in Figure 4 indicates that the mean prediction network is optional.

[0074] After obtaining image features y, the encoder can determine residual features r based on image features y and predicted values ​​mu, such as using the difference between image features y and predicted values ​​mu as residual features r. Then, feature processing is performed on the residual features r to obtain image features s. This feature processing method is not restricted and can be any method. In this case, a mean prediction network needs to be deployed to provide the predicted values ​​mu. Alternatively, after obtaining image features y, the encoder can perform feature processing on image features y to obtain image features s. This feature processing method is not restricted and can be any method. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0075] After obtaining image features s, the encoder can quantize them to obtain the quantized image features corresponding to s; that is, the Q operation in Figure 4 represents the quantization process. After obtaining the quantized image features s, the encoder can encode these features to obtain Bitstream #2 (which can be called the second bitstream) corresponding to the current image block; that is, the AE operation in Figure 4 represents the encoding process, such as entropy coding. Alternatively, the encoder can directly encode the image features s to obtain Bitstream #2 corresponding to the current image block without involving the quantization process of image features s.

[0076] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can send the Bitstream#2 corresponding to the current image block to the decoding end. For the processing of the Bitstream#2 corresponding to the current image block by the decoding end, please refer to the following embodiments.

[0077] After obtaining Bitstream #2 corresponding to the current image block, the encoding end can decode Bitstream #2 to obtain the image quantization features. In Figure 4, AD represents the decoding process. Then, the encoding end can dequantize the image quantization features to obtain image features s'. Image features s' may be the same as or different from image features s. The IQ operation in Figure 4 represents the dequantization process. Alternatively, after obtaining Bitstream #2 corresponding to the current image block, the encoding end can also decode Bitstream #2 to obtain image features s' without involving the dequantization process of the image quantization features.

[0078] After obtaining image features s', the encoder can perform feature recovery (the inverse of feature processing) on ​​s'. This feature recovery process is not restricted and can be any method, resulting in residual features r_hat. Residual features r_hat and r can be the same or different. After obtaining residual features r_hat, the encoder determines reconstructed features y_hat based on r_hat and the predicted value mu. Reconstructed features y_hat and y can be the same or different from image features y. For example, the sum of residual features r_hat and the predicted value mu can be used as reconstructed features y_hat. In this case, a mean prediction network needs to be deployed to provide the predicted value mu. Alternatively, after obtaining image features s', the encoder can perform feature recovery (the inverse of feature processing) on ​​s' to obtain reconstructed features y_hat. Reconstructed features y_hat and y can be the same or different from image features y. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0079] After obtaining the reconstructed feature y_hat, the encoder can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat can be input into the synthetic transformation network, and the synthetic transformation network can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0080] In one possible implementation, when the encoding end encodes the image quantization features or image features s to obtain Bitstream#2 corresponding to the current image block, it needs to first determine the probability distribution model, and then encode the image quantization features or image features s based on the probability distribution model. Furthermore, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0081] To obtain the probability distribution model, referring to Figure 4, after obtaining the coefficient hyperparameter feature z_hat, the encoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat can be input into the probability hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probability hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform the inverse hyperparameter feature transformation on z_hat.

[0082] In one possible implementation, the above-mentioned encoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without any restrictions on the encoding process.

[0083] Example 3: The processing procedure at the decoding end can be seen in Figure 5. Of course, Figure 5 is just an example of the processing procedure at the decoding end. There are no restrictions on the processing procedure at the decoding end. The decoding process can be implemented using a neural network.

[0084] After obtaining Bitstream #1 corresponding to the current image block, the decoding end can decode Bitstream #1 to obtain the hyperparameter quantization feature, as shown by AD in Figure 5, representing the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat and the coefficient hyperparameter feature z can be the same or different. The IQ operation in Figure 5 represents the dequantization process. Alternatively, after obtaining Bitstream #1 corresponding to the current image block, the decoding end can decode Bitstream #1 to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the hyperparameter quantization feature. The decoding process for Bitstream #1 can employ a fixed probability density model decoding method, without any restrictions.

[0085] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the decoding process for the image block can also be directly applied to the image.

[0086] After obtaining the coefficient hyperparameter feature z_hat, the decoder can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image patch and the reconstructed feature y_hat of the previous image patch (the determination process of the reconstructed feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image patch. For example, the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat can be input into the mean prediction network, which determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat. This prediction process is not restricted. Specifically, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstructed feature y_hat; the combined input of these two features yields a more accurate predicted value mu.

[0087] In another implementation, after obtaining the coefficient hyperparameter feature z_hat of the current image patch, the decoder can directly obtain the predicted value mu corresponding to the current image patch based on the coefficient hyperparameter feature z_hat, that is, it does not rely on the reconstruction feature y_hat of the previous image patch to obtain the predicted value. This can greatly reduce the complexity of obtaining the predicted value and accelerate the encoding and decoding process. The methods for obtaining the predicted value at the decoder can all adopt one of the above two schemes, and will not be described in detail.

[0088] It should be noted that the mean prediction network is an optional neural network, meaning that it can be omitted, i.e., it is not necessary to determine the predicted value mu through the mean prediction network. The dashed box in Figure 5 indicates that the mean prediction network is optional.

[0089] After obtaining Bitstream #2 corresponding to the current image block, the decoding end can decode Bitstream #2 to obtain the image quantization features, i.e., AD in Figure 5 represents the decoding process. Then, the decoding end can dequantize the image quantization features to obtain image features s'. Image features s' may be the same as or different from image features s. The IQ operation in Figure 5 can be the dequantization process. Alternatively, after obtaining Bitstream #2 corresponding to the current image block, the decoding end can decode Bitstream #2 to obtain image features s' without involving the dequantization process of the image quantization features.

[0090] After obtaining image features s', the decoder can perform feature recovery (the inverse process of feature processing) on ​​image features s' to obtain residual features r_hat. Residual features r_hat may be the same as or different from residual features r. After obtaining residual features r_hat, the decoder determines reconstructed features y_hat based on residual features r_hat and predicted values ​​mu. Reconstructed features y_hat may be the same as or different from image features y. For example, the sum of residual features r_hat and predicted values ​​mu can be used as reconstructed features y_hat. In this case, a mean prediction network needs to be deployed to provide predicted values ​​mu. Alternatively, after obtaining image features s', the decoder can perform feature recovery on image features s' to obtain reconstructed features y_hat. Reconstructed features y_hat may be the same as or different from image features y. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0091] After obtaining the reconstructed feature y_hat, the decoder can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat can be input into the synthetic transformation network, and the synthetic transformation network can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0092] In one possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on this probability distribution model. To obtain the probability distribution model, referring to Figure 5, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat can be input into a probability hyperparameter decoding network, which then performs an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network; the training process of this network is not limited, as long as it can perform an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0093] In one possible implementation, the above-mentioned decoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without limiting the decoding process.

[0094] Example 4: For Examples 1, 2, and 3, a mean prediction network can be deployed or not. To improve feature encoding performance, we will take the deployment of a mean prediction network as an example. When deploying a mean prediction network, in order to obtain accurate predicted values ​​of image features, we can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image patch and the reconstructed feature y_hat of the previous image patch to obtain the predicted value mu corresponding to the current image patch; or, we can obtain the predicted value mu corresponding to the current image patch based on the coefficient hyperparameter feature z_hat of the current image patch.

[0095] The processing at the encoding end may include the following steps:

[0096] Step S11: After obtaining the original image, determine whether to divide the original image into blocks based on the image resolution. For example, if the image resolution is greater than a threshold, the original image is divided into blocks; otherwise, it is not divided. If so, the original image is divided into several image blocks x, with overlapping portions between adjacent image blocks; otherwise, image block x is the original image, meaning the original image has only one image block. For ease of description, we will use one image block as an example, denoted as the current image block x, and the processing of the current image block x will be used as an example in the following explanation.

[0097] Step S12: After obtaining the current image block x, analyze and transform the current image block x through an analysis and transformation network to obtain the image features y (i.e., feature blocks) corresponding to the current image block x. For example, the analysis and transformation network transforms the current image block x into the image features y in the latent domain, so that all subsequent processes can be performed in the latent domain.

[0098] Step S13: Perform a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x. For example, the image feature y can be input into a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network can perform a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x.

[0099] Step S14: Encode the coefficient hyperparameter feature z into the first bitstream (Bitstream#1) corresponding to the current image patch. For example, quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantized feature, and then encode this hyperparameter quantized feature to obtain the first bitstream corresponding to the current image patch. Alternatively, directly encode the coefficient hyperparameter feature z to obtain the first bitstream corresponding to the current image patch. That is, the first bitstream corresponding to the current image patch is a bitstream encoded with the coefficient hyperparameter feature z corresponding to the current image patch x. After obtaining the first bitstream corresponding to the current image patch, send the first bitstream corresponding to the current image patch to the decoding end.

[0100] Step S15: After obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process.

[0101] Step S16: Determine the probability distribution parameters based on the coefficient hyperparameter feature z_hat. For example, perform an inverse transformation of the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Alternatively, input the coefficient hyperparameter feature z_hat into the probability hyperparameter decoding network, and have the network perform an inverse transformation to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on it.

[0102] Step S17: Input the hyperparameter feature z_hat into the mean prediction network. The mean prediction network performs context-based prediction based on the hyperparameter feature z_hat and the reconstructed feature y_hat of the previous image patch to obtain the predicted value mu. Alternatively, input the hyperparameter feature z_hat into the mean prediction network, and the mean prediction network obtains the predicted value mu based on the hyperparameter feature z_hat.

[0103] Step S18: Determine the residual feature r based on the image feature y and the mean feature mu (i.e., the predicted value mu). For example, the difference between the image feature y and the mean feature mu can be used as the residual feature r. Perform feature processing on the residual feature r to obtain the feature-processed residual feature r. There are no restrictions on this feature processing procedure. Feature processing is an optional step, and it is also possible not to perform feature processing on the residual feature r.

[0104] Step S19: Quantize the residual feature r (or the residual feature r after feature processing) to obtain the image quantization feature, and encode the image quantization feature to obtain the second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r (or the residual feature r after feature processing) can be directly encoded to obtain the second bitstream corresponding to the current image block. That is, the second bitstream corresponding to the current image block is a bitstream encoded with the residual feature r corresponding to the current image block x. After obtaining the second bitstream corresponding to the current image block, send the second bitstream corresponding to the current image block to the decoding end.

[0105] For example, when encoding image quantization features or residual features r, the probability distribution model corresponding to the probability distribution parameter p can be used to encode the image quantization features or residual features r to obtain the second bitstream.

[0106] Step S20: After obtaining the second bitstream corresponding to the current image block, the second bitstream is decoded using the probability distribution model corresponding to the probability distribution parameter p to obtain image quantization features. The image quantization features are then dequantized, and feature recovery is performed on the dequantized features to obtain the residual feature r_hat. Alternatively, the residual feature r_hat can be directly obtained after dequantizing the image quantization features without involving the feature recovery process. Or, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain the residual feature r_hat, or the residual feature r_hat can be directly obtained after decoding the second bitstream.

[0107] Step S21: After obtaining the residual feature r_hat, determine the reconstruction feature y_hat based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu can be used as the reconstruction feature y_hat.

[0108] Step S22: Input the reconstructed feature y_hat into the synthesis transform network. The synthesis transform network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstructed feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0109] For example, taking the deployment of a mean prediction network as an example, the processing at the decoding end may include the following steps:

[0110] Step S31: After obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process.

[0111] Step S32: Determine the probability distribution parameters based on the coefficient hyperparameter feature z_hat. For example, perform an inverse transformation of the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Alternatively, input the coefficient hyperparameter feature z_hat into the probability hyperparameter decoding network, and have the network perform an inverse transformation to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on it.

[0112] Step S33: After obtaining the second bitstream corresponding to the current image block, decode the second bitstream corresponding to the current image block based on the probability distribution parameter p to obtain the reconstruction features corresponding to the current image block. For example, the residual feature r_hat corresponding to the current image block and the predicted value mu corresponding to the current image block can be determined. The reconstruction feature y_hat is determined based on the residual feature r_hat and the mean feature mu (i.e., the predicted value mu).

[0113] To determine the predicted value *mu* for the current image patch, after obtaining the hyperparameter feature *z_hat*, the hyperparameter feature *z_hat* can be input into the mean prediction network. The mean prediction network then performs context-based prediction based on the hyperparameter feature *z_hat* and the reconstructed features *y_hat* of the previous image patch to obtain the predicted value *mu* for the current image patch. Alternatively, the hyperparameter feature *z_hat* can be input into the mean prediction network, and the mean prediction network obtains the predicted value *mu* for the current image patch based on the hyperparameter feature *z_hat*.

[0114] To determine the residual feature r_hat corresponding to the current image block, the second bitstream can be decoded using a probability distribution model corresponding to the probability distribution parameter p to obtain the image quantization features. These quantization features can then be dequantized, and feature recovery can be performed on the dequantized features to obtain the residual feature r_hat. Alternatively, the residual feature r_hat can be directly obtained by dequantizing the image quantization features without involving the feature recovery process. Alternatively, the second bitstream can be decoded, and feature recovery can be performed on the decoded features to obtain the residual feature r_hat. Or, the residual feature r_hat can be directly obtained by decoding the second bitstream.

[0115] Step S34: Input the reconstructed feature y_hat into the synthesis transform network. The synthesis transform network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstructed feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0116] Example 5: Examples 1, 2, 3, and 4 all involve a synthetic transformation network. The implementation method of this example is the same as that of Examples 1, 2, 3, or 4, and repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0117] An example of this synthetic transform network can be seen in Figure 6A. The synthetic transform network can sequentially include a residual layer, a deconvolution layer, a cropping layer, a residual activation layer, a deconvolution layer, a cropping layer, a residual activation layer, a deconvolution layer, an attention model, a cropping layer, a residual activation layer, a deconvolution layer, and another cropping layer. The input feature of this synthetic transform network can be the reconstructed feature y_hat, and the output feature can be the reconstructed image patch x_hat. In other words, after the reconstructed feature y_hat is input into the synthetic transform network, it is processed sequentially through the aforementioned network layers to obtain the reconstructed image patch x_hat.

[0118] For example, an example of a residual layer in a synthetic transform network can be seen in Figure 6B. The residual layer can sequentially include a convolutional layer, a rectified linear unit (ReLU) (such as a leaky ReLU), a convolutional layer, and a stacking layer (used to add features). The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there is no restriction, such as choosing a 3x3 convolutional layer. The second convolutional layer can also be a 1x1, 3x3, or 5x5 convolutional layer; there is no restriction, such as choosing a 3x3 convolutional layer.

[0119] For example, the residual activation layer in the synthetic transform network can be seen in Figure 6C. That is, the residual activation layer can sequentially include an activation layer (such as a LeakyReLU activation layer), a convolutional layer, an activation layer (such as a tanh activation layer), a multiplication layer (for multiplying features), and a stacking layer (for adding features). The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, without limitation. For example, a 1x1 convolutional layer can be selected.

[0120] For example, the attention model in the synthetic transform network can be a Residual Non-local Attention Block (RNAB). RNAB is introduced as an attention model to acquire non-local information from the image and uses this non-local information as attention weights to improve decoding performance. An example of RNAB can be seen in Figure 6D, where RNAB can sequentially include residual blocks, residual blocks, downsampling convolutional layers (e.g., 2x downsampling convolutional layers, such as 3x3 convolutional layers), residual blocks, residual blocks, upsampling convolutional layers (e.g., 2x upsampling convolutional layers, such as 3x3 convolutional layers), residual blocks, residual blocks, convolutional layers (e.g., 3x3 convolutional layers), sigmoid activation function, residual blocks, residual blocks, residual blocks, multiplication layers (for multiplying features), and stacking layers (for adding features).

[0121] The residual block, as shown in Figure 6B, is similar to the residual layer in Figure 6B. It can sequentially include a convolutional layer, a rectified linear unit (ReLU) (such as LeakyReLU), a convolutional layer, and a stacking layer (used to add features). Thus, the input features of the residual block are processed through the convolutional layer, rectified linear unit, and convolutional layer to extract features, and then added to the input features of the residual block to obtain its output features.

[0122] A submodule consisting of residual blocks, residual blocks, downsampling convolutional layers, residual blocks, residual blocks, upsampling convolutional layers, residual blocks, residual blocks, convolutional layers, and sigmoid activation functions can be a nonlocal attention extraction submodule. A submodule consisting of three residual blocks can be a feature extraction submodule. The input features of RNAB are multiplied by the attention weights generated by the nonlocal attention extraction submodule and then added to the input features of RNAB to obtain the output features of RNAB.

[0123] Example 6: Examples 1, 2, 3, and 4 all involve a synthetic transformation network. The implementation method of this example is the same as that of Examples 1, 2, 3, or 4, and repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0124] The synthetic transformation network may include at least an attention module, and the attention module may be a cascaded hybrid attention module. In addition to the attention module, the synthetic transformation network may also include other network layers. In this embodiment, the structure of the synthetic transformation network is not limited, as long as the synthetic transformation network includes an attention module.

[0125] In one possible implementation, in addition to the attention module, the synthetic transform network may further include at least one deconvolutional layer. Furthermore, besides including the attention module and at least one deconvolutional layer, the synthetic transform network may also include other network layers, without limitation. When the synthetic transform network includes at least one deconvolutional layer, the attention module may be located after any one of the deconvolutional layers. Alternatively, the attention module may be located after each deconvolutional layer; wherein the attention modules located after different deconvolutional layers may be identical, or the attention modules located after different deconvolutional layers may not be identical. Alternatively, the attention module may be located after a subset of deconvolutional layers (e.g., K deconvolutional layers out of all deconvolutional layers, where K is greater than 1 and K is less than the total number of deconvolutional layers); wherein the attention modules located after different deconvolutional layers may be identical, or the attention modules located after different deconvolutional layers may not be identical.

[0126] In one possible implementation, besides the attention module, the synthetic transform network may include at least one of the following network layers: a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, a second deconvolution layer, a second pruning layer, a second residual activation layer, a third deconvolution layer, a third pruning layer, a third residual activation layer, a fourth deconvolution layer, and a fourth pruning layer. That is, the synthetic transform network may include some or all of the above-mentioned network layers. Of course, in addition to the above-mentioned network layers, the synthetic transform network may also include other network layers. This application does not limit this; it only uses the above-mentioned network layers as an example for illustration.

[0127] Referring to Figure 7A, an example of a synthetic transform network is shown. The synthetic transform network can sequentially include a residual layer, a deconvolution layer 1 (i.e., the first deconvolution layer), a cropping layer 1 (i.e., the first cropping layer), a residual activation layer 1 (i.e., the first residual activation layer), a deconvolution layer 2 (i.e., the second deconvolution layer), a cropping layer 2 (i.e., the second cropping layer), a residual activation layer 2 (i.e., the second residual activation layer), a deconvolution layer 3 (i.e., the third deconvolution layer), a cropping layer 3 (i.e., the third cropping layer), a residual activation layer 3 (i.e., the third residual activation layer), a deconvolution layer 4 (i.e., the fourth deconvolution layer), and a cropping layer 4 (i.e., the fourth cropping layer). For example, the input feature of this synthetic transform network can be a reconstructed feature y_hat, and the output feature can be a reconstructed image patch x_hat. That is, after inputting the reconstructed feature y_hat into the synthetic transform network, it is processed sequentially through the above network layers to obtain and output the reconstructed image patch x_hat.

[0128] Figure 7A shows three attention model positions: attention model position 1, attention model position 2, and attention model position 3. For example, if the synthetic transform network includes only one attention module (i.e., a cascaded hybrid attention module), this attention module can be located after deconvolution layer 1, i.e., at attention model position 1. Alternatively, the attention module can be located after deconvolution layer 2, i.e., at attention model position 2. Or, the attention module can be located after deconvolution layer 3, i.e., at attention model position 3, for example, by placing the attention module after deconvolution layer 3. Of course, these are just a few examples of attention module positions and are not restrictive; they can be located at any position within the synthetic transform network.

[0129] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 1, attention module 2 located after deconvolution layer 2, and attention module 3 located after deconvolution layer 3. That is, attention module 1 is located at attention model position 1, attention module 2 is located at attention model position 2, and attention module 3 is located at attention model position 3. Of course, the attention modules may also be located at other positions in the synthetic transformation network, without limitation.

[0130] In this context, attention modules 1, 2, and 3 can be completely identical, meaning they can use the same network structure. Alternatively, attention modules 1, 2, and 3 can be different. For example, the network structures of attention modules 1 and 2 can be the same, but different; or, the network structures of attention modules 1 and 3 can be the same, but different; or, the network structures of attention modules 2 and 3 can be the same, but different; or, the network structures of attention modules 1 and 3 can be different, and the network structures of attention modules 2 and 3 can also be different.

[0131] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 1 and attention module 2 located after deconvolution layer 2, that is, attention module 1 is located at attention model position 1 and attention module 2 is located at attention model position 2. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.

[0132] Attention module 1 and attention module 2 can be completely identical; for example, they can use the same network structure. Alternatively, attention module 1 and attention module 2 can be different; for example, the network structures of attention module 1 and attention module 2 can be different.

[0133] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 1 and attention module 2 located after deconvolution layer 3, that is, attention module 1 is located at attention model position 1 and attention module 2 is located at attention model position 3. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.

[0134] Attention module 1 and attention module 2 can be completely identical; for example, they can use the same network structure. Alternatively, attention module 1 and attention module 2 can be different; for example, the network structures of attention module 1 and attention module 2 can be different.

[0135] For example, the synthetic transformation network may include attention module 1 located after deconvolution layer 2 and attention module 2 located after deconvolution layer 3, that is, attention module 1 is located at attention model position 2 and attention module 2 is located at attention model position 3. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.

[0136] Attention module 1 and attention module 2 can be completely identical; for example, they can use the same network structure. Alternatively, attention module 1 and attention module 2 can be different; for example, the network structures of attention module 1 and attention module 2 can be different.

[0137] An example of a residual layer in a synthetic transform network can be seen in Figure 6B. The residual layer can sequentially include a convolutional layer, a rectified linear unit (such as LeakyReLU), a convolutional layer, and a stacking layer. That is, after the input features are processed through the convolutional layer, rectified linear unit, and convolutional layer to extract features, the residual layer's output features are added to the input features. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer. Similarly, the second convolutional layer can also be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer.

[0138] An example of a residual activation layer in a synthetic transform network can be seen in Figure 6C. The residual activation layer can sequentially include an activation layer (such as a LeakyReLU activation layer), a convolutional layer, another activation layer (such as a tanh activation layer), a multiplication layer, and a stacking layer. There are no restrictions on the structure of this residual activation layer. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer; there are no restrictions, such as choosing a 1x1 convolutional layer.

[0139] Example 7: This example further describes the attention module involved in Example 6. Example 6 involves an attention module in the synthetic transform network, which can be a cascaded hybrid attention module. By replacing RNAB with a cascaded hybrid attention module, the computational complexity of decoding can be significantly reduced while maintaining decoding performance as much as possible. For example, the cascaded hybrid attention module can extract attention weights from the input features and determine the output features of the cascaded hybrid attention module based on the attention weights and the input features.

[0140] For example, a cascaded hybrid attention module may include a first attention submodule (also called a transformer-based attention submodule, denoted as Transformer-based Attention Module) and a second attention submodule (also called an upgraded convolutional submodule, denoted as ConvNext Block). The first and second attention submodules can be two sequential submodules, meaning the output features of the first attention submodule serve as the input features of the second attention submodule. For instance, referring to Figure 7B, which illustrates a cascaded hybrid attention module, the input features (or input image) of the cascaded hybrid attention module serve as the input features of the first attention submodule (using the transformer-based attention submodule as an example in Figure 7B). The input features of the first attention submodule are processed by the first attention submodule to obtain its output features. The output features of the first attention submodule serve as the input features of the second attention submodule (taking the upgraded convolution submodule as an example in Figure 7B). The input features of the second attention submodule are processed by the second attention submodule to obtain the output features of the second attention submodule. The output features of the second attention submodule serve as the output features of the cascaded hybrid attention module.

[0141] For example, after inputting the input features of the first attention submodule into the first attention submodule, the first attention submodule performs a first processing on the input features to obtain the output features of the first attention submodule. The first processing may include, but is not limited to, at least one of: layer normalization, convolution operation, and dimensionality transformation operation. There are no restrictions on the first processing, and it is related to the network structure of the first attention submodule. The network structure of the first attention submodule can be configured arbitrarily.

[0142] For example, after inputting the input features of the second attention submodule into the second attention submodule, the second attention submodule performs a second processing on the input features to obtain the output features of the second attention submodule. The second processing may include, but is not limited to, at least one of the following: depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. There are no restrictions on this second processing, and it is related to the network structure of the second attention submodule. The network structure of the second attention submodule can be configured arbitrarily.

[0143] For example, the second processing may include at least one of depthwise separable convolution operation, layer normalization, and multilayer perceptron operation. Alternatively, the second processing may include at least one of layer normalization, linear operation, and activation operation. Or, the second processing may include at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.

[0144] Example 8: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a first attention submodule, which performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are subjected to layer normalization to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; the query vector, the key vector, and the value vector are subjected to dimensionality transformation operations to obtain dimension-transformed query vector, dimension-transformed key vector, and dimension-transformed value vector; attention weights are determined based on the dimension-transformed query vector and the dimension-transformed key vector, and the corrected features corresponding to the input features are determined based on the attention weights and the dimension-transformed value vector; the output features of the first attention submodule are determined based on the input features and the corrected features.

[0145] Referring to Figure 8A, which is a schematic diagram of the structure of the first attention submodule, this is just an example of the first attention submodule, and there are no restrictions on the structure of this first attention submodule.

[0146] The first attention submodule in Figure 8A will be used as an example for explanation. The first attention submodule may include a layer normalization layer. This layer normalizes the input features (or input image) of the first attention submodule, resulting in normalized features. Layer normalization is a neural network regularization technique used to standardize the input features of each layer of the neural network to improve training and generalization performance. Layer normalization operations may include calculating the mean and variance, standardization, scaling, and translation, ensuring that the output of each hidden unit is within a relatively small range and that each hidden unit has a similar distribution across the entire dataset.

[0147] After obtaining the layer-normalized features, a three-way convolution operation can be performed on these features to obtain the query vector Q, key vector K, and value vector V. For example, the query vector Q can be obtained by performing a convolution operation on the layer-normalized features using two convolutional layers. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 1x1 convolutional layer. The second convolutional layer can also be a 3x3, 5x5, or 7x7 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer. (The last sentence is a repetition of the first and can be omitted.) The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; a 3x3 convolutional layer is a good choice. Two convolutional layers can be used to convolve the normalized features to obtain a value vector V. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; a 1x1 convolutional layer is a good choice. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; a 3x3 convolutional layer is a good choice.

[0148] The query vector Q, key vector K, and value vector V are transformed using dimensionality transformation operations (i.e., R operations), resulting in the dimensionally transformed query vector Q, key vector K, and value vector V. The purpose of the dimensionality transformation operation is to transform the tensors into the dimensional form required by the multi-head self-attention mechanism. Multi-head self-attention is a widely used technique in Natural Language Processing (NLP) tasks. This technique constructs connections between context words through self-attention, thereby reconstructing the representation of the target word based on the context. Multi-head attention is a combination of multiple self-attention components, enabling it to learn various types of contextual influences. By passing the outputs of multiple self-attention mechanisms through a parameter matrix to obtain a new output, multi-head attention can capture a wider range of relevant features, increasing the model's expressive power.

[0149] Attention weights are determined based on the dimension-transformed query vector Q and the dimension-transformed key vector K. For example, matrix multiplication is performed on the dimension-transformed query vector Q and the dimension-transformed key vector K, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights. There are no restrictions on the processing of this softmax function.

[0150] The corrected features corresponding to the input features are determined based on the attention weights and the value vector V after dimensional transformation. For example, matrix multiplication is performed on the value vector V after dimensional transformation and the attention weights. The result of the matrix multiplication is then subjected to a dimensional transformation operation (i.e., the R operation). A convolution operation is then performed on the feature after dimensional transformation to obtain the corrected features corresponding to the input features. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected.

[0151] The output features of the first attention submodule are determined based on the input features and the modified features. For example, the input features and the modified features are matrix-added to obtain the output features of the first attention submodule.

[0152] This completes the processing of the first attention submodule and yields its output features.

[0153] Example 9: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a first attention submodule, which performs a first processing on the input features to obtain the output features. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are windowed to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed to obtain the corresponding small-size query feature, small-size key feature, and small-size value feature; the small-size query features corresponding to multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to multiple small-size features are concatenated to obtain a key vector; and the small-size value features corresponding to multiple small-size features are concatenated to obtain a value vector. The query vector, key vector, and value vector are transformed into dimension vectors, key vectors, and value vectors respectively. Attention weights are determined based on the transformed query vector and key vector, and correction features are determined based on the attention weights and the transformed value vector. The output features of the first attention submodule are determined based on the input features and correction features.

[0154] For example, based on the first attention submodule in Figure 8A, a window partitioning layer can be added after the layer normalization layer. There are no restrictions on the structure of this first attention submodule; for example, an example of the structure of the first attention submodule can be found in Figure 8A.

[0155] For example, the first attention submodule may include a layer normalization layer, which can be used to normalize the input features (or input image) of the first attention submodule to obtain the normalized features.

[0156] The normalized features of a layer can be divided into multiple smaller features by windowing. For example, the normalized features can be divided into multiple smaller features, each with a width of wi and a height of hi. The widths and heights of different smaller features can be the same or different. Subsequent calculations can be performed by dividing the features into multiple smaller features, which can reduce the complexity.

[0157] For each small-sized feature, a three-way convolution operation is performed on it to obtain the corresponding small-sized query feature, small-sized key feature, and small-sized value feature. For example, by performing a convolution operation on the small-sized feature using two convolutional layers, the small-sized query feature is obtained. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; a 1x1 convolutional layer is chosen. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; a 3x3 convolutional layer is chosen. Similarly, by performing a convolution operation on the small-sized feature using two convolutional layers, the small-sized key feature is obtained. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; a 1x1 convolutional layer is chosen. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; for example, a 3x3 convolutional layer can be selected. By performing convolution operations on this small-sized feature using two convolutional layers, a small-value feature is obtained. The first convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; for example, a 1x1 convolutional layer can be selected. The second convolutional layer can be a 3x3, 5x5, or 7x7 convolutional layer; for example, a 3x3 convolutional layer can be selected.

[0158] The query vector Q can be obtained by concatenating the small-size query features corresponding to all small-size features, and there are no restrictions on the concatenation process. The key vector K can be obtained by concatenating the small-size key features corresponding to all small-size features, and the value vector V can be obtained by concatenating the small-size value features corresponding to all small-size features.

[0159] The query vector Q, key vector K, and value vector V can be transformed by dimensionality transformation (i.e., R operation) to obtain the transformed query vector Q, transformed key vector K, and transformed value vector V.

[0160] Attention weights can be determined based on the dimension-transformed query vector Q and the dimension-transformed key vector K. For example, matrix multiplication can be performed on the dimension-transformed query vector Q and the dimension-transformed key vector K, and the result of the matrix multiplication can be passed through a softmax function to obtain the attention weights. There are no restrictions on the processing procedure of the softmax function.

[0161] The corrected features corresponding to the input features can be determined based on the attention weights and the value vector V after dimensional transformation. For example, matrix multiplication is performed on the value vector V after dimensional transformation and the attention weights. The result of the matrix multiplication is then subjected to a dimensional transformation operation (i.e., the R operation). A convolution operation is then performed on the feature after dimensional transformation to obtain the corrected features corresponding to the input features. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected.

[0162] The output features of the first attention submodule can be determined based on the input features and the modified features. For example, the output features of the first attention submodule can be obtained by performing matrix addition on the input features and the modified features.

[0163] This completes the processing of the first attention submodule and yields its output features.

[0164] Example 10: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a first attention submodule, which performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are windowed to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature. The small-size query features corresponding to multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to multiple small-size features are concatenated to obtain a value vector. Attention weights are determined based on the query vector and the key vector, and the correction features corresponding to the input feature are determined based on the attention weights and the value vector. The output features of the first attention submodule are determined based on the input features and the correction features.

[0165] Referring to Figure 8B, which is a schematic diagram of the structure of the first attention submodule, this is just an example of the first attention submodule. The structure of this first attention submodule is not limited; the following explanation uses the first attention submodule in Figure 8B as an example.

[0166] For example, the first attention submodule may include a layer normalization layer, which can be used to normalize the input features (or input image) of the first attention submodule to obtain the normalized features.

[0167] The first attention submodule may include a window partitioning layer, which divides the normalized features of the layer into multiple small-sized features. For example, the window partitioning layer divides the normalized features of the layer into multiple small-sized features. The width of each small-sized feature is wi, and the height of each small-sized feature is hi. The widths of different small-sized features can be the same or different, and the heights of different small-sized features can be the same or different.

[0168] For each small-sized feature, a three-way convolution operation is performed to obtain the corresponding small-sized query feature, small-sized key feature, and small-sized value feature. For example, convolutional layers (such as 1x1 and 3x3 layers) are used to obtain the small-sized query feature. Convolutional layers (such as 1x1 and 3x3 layers) are used to obtain the small-sized key feature. Convolutional layers (such as 1x1 and 3x3 layers) are used to obtain the small-sized value feature. The small-sized query features corresponding to all small-sized features are concatenated to obtain the query vector Q, the small-sized key features corresponding to all small-sized features are concatenated to obtain the key vector K, and the small-sized value features corresponding to all small-sized features are concatenated to obtain the value vector V.

[0169] Attention weights are determined based on the query vector Q and the key vector K. For example, matrix multiplication is performed on the query vector Q and the key vector K, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights.

[0170] In one example, given the attention weights and value vector V, the corrected features corresponding to the input features can be directly obtained based on these attention weights and value vector V. For instance, matrix multiplication can be performed on the attention weights and value vector V, and the result of the matrix multiplication can be used as the corrected features corresponding to the input features. In other words, the second normalization layer in Figure 8B can be removed.

[0171] Furthermore, the output features of the first attention submodule can be determined based on the input features and the modified features. For example, the output features of the first attention submodule can be obtained by performing matrix addition on the input features and the modified features.

[0172] This completes the processing of the first attention submodule and yields its output features.

[0173] Example 11: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a first attention submodule, which performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are subjected to layer normalization to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; attention weights are determined based on the query vector and the key vector, and the corrected features corresponding to the input features are determined based on the attention weights and the value vector; the output features of the first attention submodule are determined based on the input features and the corrected features of the first attention submodule.

[0174] For example, based on the first attention submodule in Figure 8B, the window partitioning layer following the layer normalization layer can be removed. There are no restrictions on the structure of this first attention submodule; for example, an example of its structure can be found in Figure 8B.

[0175] For example, the first attention submodule may include a layer normalization layer, which can be used to normalize the input features (or input image) of the first attention submodule to obtain the normalized features.

[0176] The first attention submodule may include convolutional layers that perform three-way convolution operations on the normalized features to obtain a query vector Q, a key vector K, and a value vector V. For example, the query vector Q can be obtained by performing convolution operations on the normalized features using convolutional layers (such as 1x1 and 3x3 convolutional layers). The key vector K can be obtained by performing convolution operations on the normalized features using convolutional layers (such as 1x1 and 3x3 convolutional layers). The value vector V can be obtained by performing convolution operations on the normalized features using convolutional layers (such as 1x1 and 3x3 convolutional layers).

[0177] Attention weights are determined based on the query vector Q and the key vector K. For example, matrix multiplication is performed on the query vector Q and the key vector K, and the result of the matrix multiplication is passed through a softmax function to obtain the attention weights.

[0178] In one example, given the obtained attention weights and value vector V, the corrected features corresponding to the input features can be directly obtained based on these attention weights and value vector V. For instance, matrix multiplication can be performed on the attention weights and value vector V, and the result of the matrix multiplication can be used as the corrected features corresponding to the input features. In other words, the second normalization layer in Figure 8B can be removed.

[0179] Furthermore, the output features of the first attention submodule can be determined based on the input features and the modified features. For example, the output features of the first attention submodule can be obtained by performing matrix addition on the input features and the modified features.

[0180] This completes the processing of the first attention submodule and yields its output features.

[0181] In Examples 8-11, several examples of the first attention submodule are given. This application does not limit the structure of the first attention submodule; it can be used to perform a first processing on the input features to obtain the output features.

[0182] Example 12: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a second attention submodule, which performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a depthwise separable convolution operation is performed on the input features of the second attention submodule to obtain convolutional features (i.e., features after the depthwise separable convolution operation); layer normalization is performed on the convolutional features to obtain layer-normalized features; multilayer perceptron operation is performed on the layer-normalized features to obtain multilayer perceptron features; and the output features of the second attention submodule are determined based on the input features and the multilayer perceptron features.

[0183] Referring to Figure 8C, which is a schematic diagram of the second attention submodule, this is just an example of the second attention submodule. The structure of this second attention submodule is not limited; the following explanation uses the second attention submodule in Figure 8C as an example.

[0184] For example, the second attention submodule may include a depthwise separable convolutional layer, which may be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without limitation. For example, a 7x7 convolutional layer may be selected. The input features of the second attention submodule can be subjected to depthwise separable convolution operations through the depthwise separable convolutional layer to obtain the convolutional features (i.e., the features after the depthwise separable convolution operation). There is no limitation on this convolution operation.

[0185] The second attention submodule may include a layer normalization layer, which normalizes the convolutional features to obtain normalized features. Layer normalization operations include calculating the mean and variance, standardization, scaling, and translation, ensuring that the output of each hidden unit is within a relatively small range and that each hidden unit has a similar distribution in the dataset.

[0186] The second attention submodule may include a multilayer perceptron (MLP). The MLP can perform multilayer perceptron operations on the layer-normalized features to obtain the multilayer perceptron features. Referring to Figure 8D, which is a schematic diagram of the MLP structure, the MLP may include a linear layer, an activation layer, a random deactivation layer, a linear layer, and a random deactivation layer. Based on this, the layer-normalized features sequentially pass through a linear layer, an activation layer, a random deactivation layer, a linear layer, and a random deactivation layer to obtain the multilayer perceptron features. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a Gelu layer; the type of activation layer is not limited. The random deactivation layer can be a DropOut layer; the type of random deactivation layer is not limited. It should be noted that Figure 8D is only an example of a MLP, and its structure is not limited.

[0187] The output features of the second attention submodule are determined based on the input features and the features after multi-layer perception. For example, the output features are obtained by performing matrix addition on the input features and the features after multi-layer perception.

[0188] This completes the processing of the second attention submodule and yields its output features.

[0189] Example 13: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a second attention submodule, which performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule can be layer-normalized to obtain layer-normalized features; the layer-normalized features can be subjected to a first linear operation to obtain features after the first linear operation; the features after the first linear operation can be activated to obtain activated features; the activated features can be subjected to a second linear operation to obtain features after the second linear operation; and the output features of the second attention submodule can be determined based on the input features of the second attention submodule and the features after the second linear operation.

[0190] Referring to Figure 8E, which is a schematic diagram of the second attention submodule, this is just an example of the second attention submodule. The structure of this second attention submodule is not limited; the following explanation uses the second attention submodule in Figure 8E as an example.

[0191] For example, the second attention submodule may include a layer normalization layer, which can be used to normalize the input features of the second attention submodule to obtain the normalized features. The second attention submodule may include a linear layer 1, which can be used to perform a first linear operation on the normalized features to obtain the features after the first linear operation.

[0192] The second attention submodule may include an activation layer, which can be used to activate the features after the first linear operation to obtain activated features. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; the type of activation layer is not limited, such as choosing a ReLU layer. The second attention submodule may also include a second linear layer, which can be used to perform a second linear operation on the activated features to obtain features after the second linear operation.

[0193] The output features of the second attention submodule are determined based on the input features of the second attention submodule and the features after the second linear operation. For example, the output features are obtained by performing matrix addition on the input features and the features after the second linear operation.

[0194] This completes the processing of the second attention submodule and yields its output features.

[0195] Example 14: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on Examples 6 and 7. In Examples 6 and 7, the cascaded hybrid attention module includes a second attention submodule, which performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule are downsampled to obtain downsampled features; the downsampled features are subjected to residual convolution to obtain residual convolution features; the residual convolution features are upsampled to obtain upsampled features; the upsampled features are activated to obtain activated features; and the input features are subjected to residual convolution to obtain convolutional features. The output features of the second attention submodule are determined based on the input features, the activated features, and the convolutional features.

[0196] Referring to Figure 8F, which is a schematic diagram of the second attention submodule, this is just an example of the second attention submodule. The structure of this second attention submodule is not limited; the following explanation uses the second attention submodule in Figure 8F as an example.

[0197] For example, the second attention submodule may include a downsampling layer, which can be a convolutional layer with a kernel size of 3x3 and a stride of 2, or a convolutional layer with a kernel size of 3x3 and a stride of 4, or a convolutional layer with a kernel size of 4x4 and a stride of 2, or a convolutional layer with a kernel size of 4x4 and a stride of 4, or a convolutional layer with a kernel size of 5x5 and a stride of 2, or a convolutional layer with a kernel size of 5x5 and a stride of 4, or the downsampling layer may also be an inverse pixel shuffle layer. The structure of this downsampling layer is not limited; for example, a convolutional layer with a kernel size of 3x3 and a stride of 2 can be selected as the downsampling layer. Based on this, the input features of the second attention submodule can be downsampled using the downsampling layer to obtain downsampled features. The implementation of this downsampling operation is not limited.

[0198] The second attention submodule can include M residual convolutional layers, where M can be a positive integer, such as 1, 2, 3, 4, etc. These M residual convolutional layers perform residual convolution operations on the downsampled features to obtain the residual convolutional features. For example, the first residual convolutional layer performs a residual convolution operation on the downsampled features, the result is then passed to the second residual convolutional layer for further residual convolution, and so on, until the last residual convolutional layer outputs the residual convolutional features.

[0199] Referring to Figure 8G, which is a schematic diagram of a residual convolutional layer, this residual convolutional layer may include a convolutional layer 1, an activation layer, and a convolutional layer 2. The input features of the residual convolutional layer are subjected to a first convolutional process by convolutional layer 1 to obtain the features after the first convolutional process. The features after the first convolutional process are then activated by the activation layer to obtain the activated features. The activated features are then subjected to a second convolutional process by convolutional layer 2 to obtain the features after the second convolutional process. Finally, the input features of the residual convolutional layer and the features after the second convolutional process are matrix-sumped to obtain the output features of the residual convolutional layer.

[0200] Convolutional layer 1 can be a 1x1, 3x3, 5x5, or 7x7 convolutional layer; there are no restrictions. For example, a 3x3 convolutional layer can be selected. Convolutional layer 2 can also be a 1x1, 3x3, 5x5, or 7x7 convolutional layer; there are no restrictions. For example, a 3x3 convolutional layer can be selected. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; there are no restrictions. For example, a ReLU layer or a sigmoid layer can be selected.

[0201] Referring to Figure 8H, which illustrates another structural diagram of a residual convolutional layer, this residual convolutional layer can include convolutional layers and activation layers. The input features of the residual convolutional layer are convolved by the convolutional layers to obtain convolutionally processed features. The convolutionally processed features are then activated by the activation layers to obtain activated features. Finally, the input features of the residual convolutional layer and the activated features are matrix-summed to obtain the output features of the residual convolutional layer.

[0202] The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer; there are no restrictions on this, such as choosing a 3x3 convolutional layer. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; there are no restrictions on this, such as choosing a ReLU layer or a sigmoid layer.

[0203] The second attention submodule may include an upsampling layer, which can be a deconvolutional layer with a kernel size of 3x3 and a stride of 2, or a deconvolutional layer with a kernel size of 3x3 and a stride of 4, or a deconvolutional layer with a kernel size of 4x4 and a stride of 2, or a deconvolutional layer with a kernel size of 4x4 and a stride of 4, or a deconvolutional layer with a kernel size of 5x5 and a stride of 2, or a deconvolutional layer with a kernel size of 5x5 and a stride of 4, or it may also be a pixel recombination layer. There are no restrictions on the structure of this upsampling layer; for example, a deconvolutional layer with a kernel size of 4x4 and a stride of 2 can be selected as the upsampling layer. Based on this, the features obtained after residual convolution (i.e., the features obtained after residual convolution operations through M residual convolutional layers) can be upsampled using the upsampling layer to obtain upsampled features. There are no restrictions on the implementation of this upsampling operation.

[0204] The second attention submodule may include an activation layer, which can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer. The structure of this activation layer is not limited; for example, a ReLU layer can be selected. Based on this, activation operations can be performed on the upsampled features through the activation layer to obtain activated features.

[0205] The second attention submodule can include N residual convolutional layers, where N can be a positive integer, such as 1, 2, 3, 4, etc. N and M can be the same or different. The input features of the second attention submodule can be subjected to residual convolution operations through these N residual convolutional layers to obtain the convolutional features after residual convolution. For example, the input features of the second attention submodule are subjected to residual convolution operations through the first residual convolutional layer, the result is subjected to residual convolution operations through the second residual convolutional layer, and so on, until the last residual convolutional layer outputs the convolutional features after residual convolution.

[0206] Referring to Figure 8G, which is a schematic diagram of a residual convolutional layer, the residual convolutional layer may include convolutional layer 1, activation layer, and convolutional layer 2. Referring to Figure 8H, which is a schematic diagram of another residual convolutional layer, the residual convolutional layer may include convolutional layer and activation layer. Residual convolution operations can be performed using any of the above residual convolutional layers, which will not be described in detail here.

[0207] Referring to Figure 8F, after obtaining the convolutional features after residual convolution (i.e., the features output after residual convolution operations through N residual convolutional layers) and the activated features (i.e., the features output after activation operations through activation layers), the output features of the second attention submodule can be determined based on the input features, activated features, and convolutional features after residual convolution. For example, the activated features can be multiplied by the convolutional features after residual convolution, and the result of the multiplication can be added to the input features of the second attention submodule to obtain the output features of the second attention submodule.

[0208] This completes the processing of the second attention submodule and yields its output features.

[0209] In Examples 12-14, several examples of the second attention submodule are given. This application does not limit the structure of the second attention submodule; the input features can be processed to obtain the output features.

[0210] Example 15: The implementation method of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0211] In Embodiments 1, 2, 3, and 4, a synthetic transform network is involved. This synthetic transform network can include at least an attention module, and the attention module can be a first attention submodule. Of course, besides the first attention submodule, the synthetic transform network can also include other network layers. This embodiment does not limit the structure of the synthetic transform network, as long as it includes the first attention submodule. For example, the synthetic transform network can also include at least one deconvolution layer, and the first attention submodule can be located after any deconvolution layer. Alternatively, the first attention submodule can be located after each deconvolution layer. Or, the first attention submodule can be located after a subset of deconvolution layers (such as K deconvolution layers out of all deconvolution layers, where K is greater than 1 and K is less than the total number of deconvolution layers).

[0212] For example, the first attention submodule is used to perform a first process on the input features of the first attention submodule to obtain the output features of the first attention submodule. In one possible implementation, the structure of the first attention submodule can be seen in Embodiment 8; in another possible implementation, the structure of the first attention submodule can be seen in Embodiment 9; in another possible implementation, the structure of the first attention submodule can be seen in Embodiment 10; and in another possible implementation, the structure of the first attention submodule can be seen in Embodiment 11. Of course, the above are just a few examples of the first attention submodule, and the structure of the first attention submodule is not limited.

[0213] Example 16: The implementation method of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the repeated content will not be described again. The synthetic transformation network involved is described in detail below.

[0214] In Embodiments 1, 2, 3, and 4, a synthetic transform network is involved. This synthetic transform network may include at least an attention module, and the attention module may be a second attention submodule. Of course, besides the second attention submodule, the synthetic transform network may also include other network layers. This embodiment does not limit the structure of this synthetic transform network, as long as it includes a second attention submodule. For example, the synthetic transform network may also include at least one deconvolution layer, and the second attention submodule may be located after any deconvolution layer. Alternatively, the second attention submodule may be located after each deconvolution layer. Or, the second attention submodule may be located after a subset of deconvolution layers (such as K deconvolution layers out of all deconvolution layers, where K is greater than 1 and K is less than the total number of deconvolution layers).

[0215] For example, the second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. In one possible implementation, the structure of the second attention submodule can be seen in Embodiment 12; in another possible implementation, the structure of the second attention submodule can be seen in Embodiment 13; and in yet another possible implementation, the structure of the second attention submodule can be seen in Embodiment 14. Of course, the above are just a few examples of the second attention submodule, and the structure of the second attention submodule is not limited.

[0216] As can be seen from the above technical solutions, in this application embodiment, for neural network-based encoding and decoding technology, an attention mechanism-based synthetic transformation network is proposed. This synthetic transformation network includes attention modules, and the attention modules are cascaded hybrid attention modules. When implementing the synthetic transformation network through cascaded hybrid attention modules, the network complexity and computational complexity are effectively reduced while ensuring the quality of the synthesized image, thus improving decoding performance. Furthermore, while reducing complexity, the quality of the reconstructed image blocks is effectively guaranteed, improving both encoding and decoding performance. By replacing the residual nonlocal attention module with a hybrid attention module (cascaded hybrid attention module), the computational complexity of the decoder is significantly reduced. For example, the second attention submodule in the hybrid attention module only requires 21K floating-point multiply-accumulate operations per pixel for computational complexity, while the complexity of the first attention submodule is also much lower than that of the residual nonlocal attention module. Furthermore, the second attention submodule functions as the attention mechanism within the residual nonlocal attention module. By introducing a high-performance transformer structure, it delivers superior performance. The 7x7 convolutional operation within the second attention submodule introduces nonlocal information, and the random deactivation layer enhances network robustness. Based on these functionalities, the cascaded hybrid attention module reduces computational complexity without significant performance loss.

[0217] For example, the above embodiments can be implemented individually or in combination. For instance, each of embodiments 1-17 can be implemented individually, and at least two embodiments 1-16 can be implemented in combination.

[0218] For example, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0219] Based on the same concept as the above method, this application also proposes a decoding device, which is applied to a decoding end. The decoding device includes: a memory configured to store video data; and a decoder configured to implement the decoding methods in embodiments 1-16 above, i.e., the processing flow of the decoding end.

[0220] For example, in one possible implementation, the decoder is configured to: decode a bitstream corresponding to the current image patch to obtain coefficient hyperparameter features corresponding to the current image patch; determine probability distribution parameters based on the coefficient hyperparameter features; decode another bitstream corresponding to the current image patch based on the probability distribution parameters to obtain reconstructed features corresponding to the current image patch; input the reconstructed features into a synthesis transform network to obtain a reconstructed image patch corresponding to the current image patch; wherein the synthesis transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0221] Based on the same concept as the above method, this application also proposes an encoding device, which is applied to the encoding end. The encoding device includes: a memory configured to store video data; and an encoder configured to implement the encoding methods in embodiments 1-16 above, i.e., the processing flow of the encoding end.

[0222] For example, in one possible implementation, the encoder is configured to: decode a bitstream corresponding to a current image patch to obtain coefficient hyperparameter features corresponding to the current image patch; determine probability distribution parameters based on the coefficient hyperparameter features; decode another bitstream corresponding to the current image patch based on the probability distribution parameters to obtain reconstructed features corresponding to the current image patch; input the reconstructed features into a synthesis transform network to obtain a reconstructed image patch corresponding to the current image patch; wherein the synthesis transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0223] Based on the same concept as the methods described above, the decoding terminal device (also referred to as a video decoder) provided in this application embodiment, from a hardware perspective, can be specifically referred to in Figure 9A as a hardware architecture diagram. It includes: a processor 901 and a machine-readable storage medium 902. The machine-readable storage medium 902 stores machine-executable instructions that can be executed by the processor 901; the processor 901 is used to execute the machine-executable instructions to implement the decoding methods of embodiments 1-17 of this application described above.

[0224] Machine-readable storage medium 902 can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be random access memory (RAM), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0225] Based on the same concept as the methods described above, the encoding end device (also referred to as a video encoder) provided in this application embodiment, from a hardware perspective, has a hardware architecture schematic diagram as shown in Figure 9B. It includes: a processor 911 and a machine-readable storage medium 912. The machine-readable storage medium 912 stores machine-executable instructions that can be executed by the processor 911; the processor 911 is used to execute the machine-executable instructions to implement the encoding methods of embodiments 1-17 of this application described above.

[0226] Machine-readable storage medium 912 can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be RAM, volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0227] Based on the same concept as the methods described above, this application provides an electronic device. It includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor executes the machine-executable instructions to implement the decoding or encoding methods of embodiments 1-16 of this application described above.

[0228] Based on the same concept as the methods described above, embodiments of this application also provide a machine-readable storage medium storing a plurality of computer instructions. When executed by a processor, these computer instructions implement the decoding or encoding methods of embodiments 1-16 of this application, such as the decoding or encoding methods in the above embodiments. For example, the machine-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0229] Based on the same concept as the above methods, this application also provides a computer application that, when executed by a processor, can implement the decoding or encoding methods disclosed in the above examples of this application.

[0230] Based on the same concept as the above method, this application also proposes a decoding device that can be applied to a decoding end. The decoding device includes: a decoding module, used to decode a bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter features; and decode another bitstream corresponding to the current image block based on the probability distribution parameter to obtain the reconstructed features corresponding to the current image block; and a processing module, used to input the reconstructed features into a synthesis transform network to obtain the reconstructed image block corresponding to the current image block; wherein the synthesis transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module or; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0231] For example, if the attention module is a cascaded hybrid attention module, the processing module is further configured to perform a first processing on the input features of the first attention submodule through the first attention submodule to obtain the output features of the first attention submodule, and the output features of the first attention submodule are the input features of the second attention submodule; wherein, the first processing is at least one of layer normalization, convolution operation, and dimension transformation operation; and to perform a second processing on the input features of the second attention submodule through the second attention submodule to obtain the output features of the second attention submodule; wherein, the second processing is at least one of depthwise separable convolution operation, layer normalization, multilayer perceptron operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation.

[0232] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performs a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; performs dimensionality transformation operations on the query vector, the key vector, and the value vector respectively to obtain a dimension-transformed query vector, a dimension-transformed key vector, and a dimension-transformed value vector; determines attention weights based on the dimension-transformed query vector and the dimension-transformed key vector, and determines the corrected features corresponding to the input features based on the attention weights and the dimension-transformed value vector; and determines the output features of the first attention submodule based on the input features and the corrected features.

[0233] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; divides the layer-normalized features into windows to obtain multiple small-size features; performs a three-way convolution operation on each small-size feature to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature; and concatenates the small-size query features corresponding to the multiple small-size features to obtain a query vector. The small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector; the query vector, the key vector, and the value vector are subjected to dimensional transformation operations to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; attention weights are determined based on the dimensionally transformed query vector and the dimensionally transformed key vector, and the correction features corresponding to the input feature are determined based on the attention weights and the dimensionally transformed value vector; the output features of the first attention submodule are determined based on the input feature and the correction features.

[0234] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performs a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determines attention weights based on the query vector and the key vector, and determines the corrected features corresponding to the input features based on the attention weights and the value vector; and determines the output features of the first attention submodule based on the input features and the corrected features.

[0235] For example, when the processing module performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, it specifically performs the following steps: performs layer normalization on the input features of the first attention submodule to obtain layer-normalized features; divides the layer-normalized features into windows to obtain multiple small-size features; performs a three-way convolution operation on each small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to that small-size feature; concatenates the small-size query features corresponding to the multiple small-size features to obtain a query vector; concatenates the small-size key features corresponding to the multiple small-size features to obtain a key vector; concatenates the small-size value features corresponding to the multiple small-size features to obtain a value vector; determines attention weights based on the query vector and the key vector, and determines the correction features corresponding to the input feature based on the attention weights and the value vector; and determines the output features of the first attention submodule based on the input feature and the correction features.

[0236] For example, when the processing module performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, it specifically performs the following steps: performing layer normalization on the input features of the second attention submodule to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0237] For example, when the processing module performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, it specifically performs the following steps: performing a downsampling operation on the input features of the second attention submodule to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolutional features; performing an upsampling operation on the residual convolutional features to obtain upsampled features; performing an activation operation on the upsampled features to obtain activated features; performing a residual convolution operation on the input features to obtain convolutional features after residual convolution; and determining the output features of the second attention submodule based on the input features of the second attention submodule, the activated features, and the convolutional features.

[0238] For example, the synthetic transform network further includes at least one deconvolution layer; wherein the attention module is located after one of the deconvolution layers.

[0239] For example, the synthetic transformation network further includes a residual layer, a first deconvolution layer, a first pruning layer, a first residual activation layer, a second deconvolution layer, a second pruning layer, a second residual activation layer, a third deconvolution layer, a third pruning layer, a third residual activation layer, a fourth deconvolution layer, and a fourth pruning layer; the synthetic transformation network includes at least one attention module, wherein one attention module is located after the first deconvolution layer.

[0240] Based on the same concept as the above method, this application also proposes an encoding device, which is applied at the encoding end. The device includes: a decoding module, used to decode a bitstream corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter features; and decode another bitstream corresponding to the current image block based on the probability distribution parameter to obtain the reconstructed features corresponding to the current image block; and a processing module, used to input the reconstructed features into a synthesis transform network to obtain the reconstructed image block corresponding to the current image block; wherein the synthesis transform network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.

[0241] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above descriptions are merely embodiments of this application and are not intended to limit this application.

[0242] Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A decoding method, comprising: Applied to a decoding end, the method comprises: Decoding a code stream corresponding to a current image block to obtain a coefficient hyperparameter feature corresponding to the current image block; determining a probability distribution parameter based on the coefficient hyperparameter feature, and decoding another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstructed feature corresponding to the current image block; Inputting the reconstructed feature into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; The synthesis transformation network at least comprises an attention module, and the attention module is a cascaded hybrid attention module. The cascaded hybrid attention module comprises a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two submodules in series.

2. The method of claim 1, wherein, If the attention module is a cascaded hybrid attention module, the method further comprises: performing first processing on the input feature of the first attention submodule through the first attention submodule to obtain the output feature of the first attention submodule, the output feature of the first attention submodule being the input feature of the second attention submodule; wherein the first processing is at least one of layer normalization, convolution operation and dimension transformation operation; performing second processing on the input feature of the second attention submodule through the second attention submodule to obtain the output feature of the second attention submodule; wherein the second processing is at least one of depth separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, down-sampling operation, residual convolution operation and up-sampling operation.

3. The method of claim 2, wherein, The first processing on the input feature of the first attention submodule through the first attention submodule to obtain the output feature of the first attention submodule comprises: performing layer normalization on the input feature of the first attention submodule to obtain a layer-normalized feature; performing three-way convolution operation on the layer-normalized feature to obtain a query vector, a key vector and a value vector; performing dimension transformation operation on the query vector, the key vector and the value vector respectively to obtain a dimension-transformed query vector, a dimension-transformed key vector and a dimension-transformed value vector; determining an attention weight based on the dimension-transformed query vector and the dimension-transformed key vector, and determining a correction feature corresponding to the input feature based on the attention weight and the dimension-transformed value vector; determining the output feature of the first attention submodule based on the input feature and the correction feature.

4. The method of claim 2, wherein, The first processing on the input feature of the first attention submodule through the first attention submodule to obtain the output feature of the first attention submodule comprises: performing layer normalization on the input feature of the first attention submodule to obtain a layer-normalized feature; performing window division on the layer-normalized feature to obtain a plurality of small-size features; For each small-size feature, a three-path convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the plurality of small-size features are spliced to obtain a query vector; the small-size key features corresponding to the plurality of small-size features are spliced to obtain a key vector; and the small-size value features corresponding to the plurality of small-size features are spliced to obtain a value vector; Dimension transformation operations are respectively performed on the query vector, the key vector and the value vector to obtain a dimension-transformed query vector, a dimension-transformed key vector and a dimension-transformed value vector; Based on the dimension-transformed query vector and the dimension-transformed key vector, an attention weight is determined, and based on the attention weight and the dimension-transformed value vector, a modified feature corresponding to the input feature is determined; Based on the input feature and the modified feature, the output feature of the first attention sub-module is determined.

5. The method of claim 2, wherein, The first processing of the input feature of the first attention sub-module by the first attention sub-module to obtain the output feature of the first attention sub-module includes: Layer normalization is performed on the input feature of the first attention sub-module to obtain a layer-normalized feature; A three-path convolution operation is performed on the layer-normalized feature to obtain a query vector, a key vector and a value vector; Based on the query vector and the key vector, an attention weight is determined, and based on the attention weight and the value vector, a modified feature corresponding to the input feature is determined; Based on the input feature and the modified feature, the output feature of the first attention sub-module is determined.

6. The method of claim 2, wherein, The first processing of the input feature of the first attention sub-module by the first attention sub-module to obtain the output feature of the first attention sub-module includes: Layer normalization is performed on the input feature of the first attention sub-module to obtain a layer-normalized feature; Window division is performed on the layer-normalized feature to obtain a plurality of small-size features; For each small-size feature, a three-path convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the plurality of small-size features are spliced to obtain a query vector; the small-size key features corresponding to the plurality of small-size features are spliced to obtain a key vector; and the small-size value features corresponding to the plurality of small-size features are spliced to obtain a value vector; Based on the query vector and the key vector, an attention weight is determined, and based on the attention weight and the value vector, a modified feature corresponding to the input feature is determined; Based on the input feature and the modified feature, the output feature of the first attention sub-module is determined.

7. The method of claim 2, wherein, The second processing of the input feature of the second attention sub-module by the second attention sub-module to obtain the output feature of the second attention sub-module includes: Layer normalization is performed on the input feature of the second attention sub-module to obtain a layer-normalized feature; A first linear operation is performed on the layer-normalized feature to obtain a first linearly operated feature; performing an activation operation on the first linearly operated feature to obtain an activated feature; performing a second linear operation on the activated feature to obtain a second linearly operated feature; determining the output feature of the second attention sub-module based on the input feature and the second linearly operated feature.

8. The method of claim 2, wherein, performing a second processing on the input feature of the second attention sub-module through the second attention sub-module to obtain the output feature of the second attention sub-module, including: performing a down-sampling operation on the input feature of the second attention sub-module to obtain a down-sampled feature; performing a residual convolution operation on the down-sampled feature to obtain a residual convolution operated feature; performing an up-sampling operation on the residual convolution operated feature to obtain an up-sampled feature; and performing an activation operation on the up-sampled feature to obtain an activated feature; performing a residual convolution operation on the input feature to obtain a residual convolution operated convolution feature; determining the output feature of the second attention sub-module based on the input feature, the activated feature and the convolution feature.

9. The method of any one of claims 1-8, wherein: the composite transformation network further comprises at least one deconvolution layer; and the attention module is located behind one of the deconvolution layers.

10. The method of claim 9, wherein, the composite transformation network further comprises a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer. the composite transformation network comprises at least one attention module, and one of the attention modules is located behind the first deconvolution layer.

11. An encoding method, characterized by, applied to an encoding end, the method comprises: decoding a code stream corresponding to a current image block to obtain a coefficient hyper-parameter feature corresponding to the current image block; determining a probability distribution parameter based on the coefficient hyper-parameter feature, and decoding another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstructed feature corresponding to the current image block; inputting the reconstructed feature into a composite transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the composite transformation network comprises at least an attention module, and the attention module is a cascaded hybrid attention module; and the cascaded hybrid attention module comprises a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two sub-modules in series.

12. A decoding apparatus, characterized by comprising: applied to a decoding end, the apparatus comprises: a decoding module configured to decode a code stream corresponding to a current image block to obtain a coefficient hyper-parameter feature corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyper-parameter feature; and decode another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstructed feature corresponding to the current image block. The processing module is configured to input the reconstructed feature into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthesis transformation network comprises at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module comprises a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two submodules in series.

13. An encoding apparatus, comprising: Applied to an encoding end, the apparatus comprises: A decoding module is configured to decode a code stream corresponding to a current image block to obtain a coefficient hyperparameter feature corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter feature; and decode another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstructed feature corresponding to the current image block; The processing module is configured to input the reconstructed feature into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthesis transformation network comprises at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module comprises a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two submodules in series.

14. A decoding-side device, comprising: Comprise: A processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor; The processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-10.

15. An encoding end device, comprising: Comprise: A processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor; The processor is configured to execute the machine executable instructions to implement the method in claim 11.

16. An electronic device, comprising: Comprise: A processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor; The processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-11.

17. A machine-readable storage medium, characterized in that, The machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-11. 18.A computer application program, which when executed by a processor, implements the method in any one of claims 1-11.

Images