Decoding, encoding method, apparatus, and device

Abstract

The present invention provides a decoding and encoding method, apparatus, and device thereof. The decoding method includes the steps of decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features, determining probability distribution parameters based on the coefficient hyperparameter features, decoding another bitstream of the current image block based on the probability distribution parameters to obtain reconstruction features, and inputting the reconstruction features into a composite transformation network to obtain a reconstructed image block, wherein the composite transformation network includes an attention module, the attention module is a cascaded hybrid attention module, the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being in series.

Description

Technical Field

[0001] The present invention relates to the technical field of encoding and decoding, and particularly to decoding, encoding methods, apparatuses, and their devices.

Background Art

[0002] In order to save space, all video images are encoded before transmission, and complete video encoding may include processes such as prediction, transformation, quantization, entropy encoding, filtering, etc. For the prediction process, the prediction may include intra-frame prediction and inter-frame prediction. The inter-frame prediction utilizes the temporal correlation of the video to predict the current pixel using the pixels of adjacent encoded images, thereby achieving the purpose of effectively removing the temporal redundancy of the video. Intra-frame prediction utilizes the spatial correlation of the video to predict the current pixel using the pixels of the encoded blocks of the image in the current frame, thereby achieving the purpose of removing the spatial redundancy of the video.

[0003] With the rapid development of deep learning, deep learning has achieved good results in many high-level computer vision problems such as image classification and object detection. Deep learning has also gradually begun to be applied in the fields of encoding and decoding, that is, it has become possible to encode and decode images using neural networks. Although the encoding and decoding methods based on neural networks show great performance potential, the encoding and decoding methods based on neural networks still have problems such as low decoding performance and high complexity.

Summary of the Invention

[0004] The present invention provides a decoding, encoding method, apparatus, and its device that can improve decoding performance and reduce complexity.

[0005] The present invention is a decoding method applied to the decoding side, The steps include: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis and conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, and provides a decoding method.

[0006] The present invention relates to an encoding method applied to the encoding side, The steps include: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis and conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, and provides an encoding method.

[0007] The present invention relates to a decoding device applied to the decoding side, A decoding module for decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, determining probability distribution parameters based on the coefficient hyperparameter features, decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block, The processing module includes inputting the aforementioned reconstruction features into a composite transformation network and obtaining a reconstructed image block corresponding to the current image block, The aforementioned synthesis and conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module provides a decoding device that includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series.

[0008] The present invention relates to an encoding device applied to the encoding side, A decoding module for decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, determining probability distribution parameters based on the coefficient hyperparameter features, decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block, A processing module for inputting the reconstruction feature into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block, The synthesis transformation network includes at least an attention module, and the attention module is a cascade hybrid attention module, The cascade hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module provide an encoding device that are two serial sub-modules.

[0009] The present invention The steps include decoding the bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, The steps include determining probability distribution parameters based on the coefficient hyperparameter features, decoding the bitstream corresponding to the current image block based on the probability distribution parameters, and obtaining reconstruction features corresponding to the current image block. The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis and conversion network includes at least a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, and the output features of the first attention submodule are the input features of the second attention submodule, providing an image decoding method.

[0010] In one possible embodiment, a first processing is performed on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule. The first process includes at least one of layer normalization, convolution, and dimensionality transformation operations.

[0011] In one possible embodiment, the step of performing a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include performing a convolution operation with three paths on the layer-normalized features to obtain a query vector, a key vector, and a value vector, The steps include performing a dimensional transformation operation on the query vector, the key vector, and the value vector to obtain the dimensionally transformed query vector, the dimensionally transformed key vector, and the dimensionally transformed value vector, The steps include determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determining a modification feature corresponding to the input feature based on the attention weights and the dimensionally transformed value vector, The process includes the step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics.

[0012] )]] In one possible embodiment, the step of performing a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include: performing window division on the layer-normalized features to obtain multiple small-sized features, For each small-size feature, a convolution operation is performed on the small-size feature using three paths to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are combined to obtain a query vector; the small-size key features corresponding to the multiple small-size features are combined to obtain a key vector; and the small-size value features corresponding to the multiple small-size features are combined to obtain a value vector. The steps include performing a dimensional transformation operation on the query vector, the key vector, and the value vector to obtain the dimensionally transformed query vector, the dimensionally transformed key vector, and the dimensionally transformed value vector, The steps include determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determining a modification feature corresponding to the input feature based on the attention weights and the dimensionally transformed value vector, The process includes the step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics.

[0013] In one possible embodiment, the step of performing a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include performing a convolution operation with three paths on the layer-normalized features to obtain a query vector, a key vector, and a value vector, The steps include determining attention weights based on the query vector and the key vector, and determining modification features corresponding to the input features based on the attention weights and the value vector, The process includes the step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics.

[0014] In one possible embodiment, the step of performing a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include: performing window division on the layer-normalized features to obtain multiple small-sized features, For each small-size feature, a convolution operation is performed on the small-size feature using three paths to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are combined to obtain a query vector; the small-size key features corresponding to the multiple small-size features are combined to obtain a key vector; and the small-size value features corresponding to the multiple small-size features are combined to obtain a value vector. The steps include determining attention weights based on the query vector and the key vector, and determining modification features corresponding to the input features based on the attention weights and the value vector, The process includes the step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics.

[0015] In one possible embodiment, the step of determining a modified feature corresponding to the input feature based on the attention weight and the value vector after dimensional transformation is: The steps include performing matrix multiplication on the value vector after dimensional transformation and the attention weights to obtain the result of matrix multiplication, The steps include performing a dimensionality transformation operation on the result of the matrix multiplication and obtaining the characteristics after the dimensionality transformation operation, The process includes the step of performing a convolution operation on the features after the dimensional transformation operation to obtain modified features corresponding to the input features.

[0016] The present invention The steps include decoding the bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, The steps include determining probability distribution parameters based on the coefficient hyperparameter features, decoding the bitstream corresponding to the current image block based on the probability distribution parameters, and obtaining reconstruction features corresponding to the current image block. The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis and conversion network includes at least a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, and the output features of the first attention submodule are the input features of the second attention submodule, providing an image encoding method.

[0017] The present invention The steps include decoding the bitstream corresponding to the current image block to obtain the reconstruction features corresponding to the current image block, The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, the output features of the first attention submodule being the input features of the second attention submodule, and the output features of the second attention submodule being used to determine the reconstructed image block, thereby providing an image decoding method.

[0018] In one possible embodiment, a second processing is performed on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule. The second process includes at least one of layer normalization, activation, and convolution.

[0019] In one possible embodiment, the step of performing a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule is: The steps include determining the input features of the layered normalized network based on the input features of the second attention submodule, A step of performing layer normalization on the input features of the layer normalized network to obtain the output features of the layer normalized network, wherein the output features of the layer normalized network are used to determine the input features of the first convolutional layer. A step of performing a convolution operation on the input features of the first convolutional layer to obtain the output features of the first convolutional layer, wherein the output features of the first convolutional layer are used to determine the input features of the activation layer. A step of performing an activation operation on the input characteristics of the activation layer to obtain the output characteristics of the activation layer, wherein the output characteristics of the activation layer are used to determine the input characteristics of the second convolutional layer. The method includes the step of performing a convolution operation on the input features of the second convolutional layer to obtain the output features of the second convolutional layer, wherein the output features of the second convolutional layer are used to determine the input features of the second attention submodule.

[0020] In one possible embodiment, a second processing is performed on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule. The second process includes at least one of the following: depth-separable convolution, layer normalization, multilayer sensing, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation.

[0021] In one possible embodiment, the step of performing a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule is: The steps include performing layer normalization on the input features of the second attention submodule and obtaining the features after layer normalization, The steps include performing a first linear operation on the features after layer normalization to obtain the features after the first linear operation, The steps include performing an activation operation on the features after the first linear operation to obtain the features after activation, The steps include performing a second linear operation on the activated features to obtain the features after the second linear operation, The process includes the step of determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0022] In one possible embodiment, the step of performing a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule is: The steps include: performing a downsampling operation on the input features of the second attention submodule to obtain the downsampled features; performing a residual convolution operation on the downsampled features to obtain the residual convolutional features; performing an upsampling operation on the residual convolutional features to obtain the upsampled features; and performing an activation operation on the upsampled features to obtain the activated features. The steps include performing a residual convolution operation on the input feature to obtain the convolutional feature after residual convolution, The process includes the step of determining the output features of the second attention submodule based on the input features, the activated features, and the convolutional features.

[0023] In one possible embodiment, the step of decoding the bitstream corresponding to the current image block to obtain the reconstructed features corresponding to the current image block is: The steps include decoding the bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, The process includes the steps of determining probability distribution parameters based on the coefficient hyperparameter features, decoding the bitstream corresponding to the current image block based on the probability distribution parameters, and obtaining reconstruction features corresponding to the current image block.

[0024] The present invention The steps include decoding the bitstream corresponding to the current image block to obtain the reconstruction features corresponding to the current image block, The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, the output features of the first attention submodule being the input features of the second attention submodule, and the output features of the second attention submodule being used to determine the reconstructed image block, thereby providing an image encoding method.

[0025] The present invention The steps include decoding the bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, The steps include determining probability distribution parameters based on the coefficient hyperparameter features, decoding the bitstream corresponding to the current image block based on the probability distribution parameters, and obtaining reconstruction features corresponding to the current image block. The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis transformation network includes at least an attention module and at least one deconvolutional layer. The attention module provides an image decoding method in which the attention module is located after one of the deconvolutional layers among all the deconvolutional layers.

[0026] In one possible embodiment, the composite transformation network includes at least one attention module, at least one residual layer, at least one deconvolution layer, at least one residual activation layer, and at least one crop layer.

[0027] In one possible embodiment, the composite transformation network includes a residual layer, a first deconvolution layer, a first crop layer, a first residual activation layer, a second deconvolution layer, a second crop layer, a second residual activation layer, an attention module, a third crop layer, a third residual activation layer, a third deconvolution layer, and a fourth crop layer.

[0028] In one possible embodiment, the attention module is a cascaded hybrid attention module, which includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series, the output features of the first attention submodule being the input features of the second attention submodule, and the output features of the second attention submodule being used to determine the reconstructed image block.

[0029] In one possible embodiment, a first processing is performed on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule. The first process includes at least one of layer normalization, convolution, and dimensionality transformation operations.

[0030] In one possible embodiment, a second processing is performed on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule. The second process includes at least one of layer normalization, activation, and convolution.

[0031] The present invention The steps include decoding the bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, The steps include determining probability distribution parameters based on the coefficient hyperparameter features, decoding the bitstream corresponding to the current image block based on the probability distribution parameters, and obtaining reconstruction features corresponding to the current image block. The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis transformation network includes at least an attention module and at least one deconvolutional layer. The attention module provides an image coding method in which the attention module is located after one of the deconvolutional layers among all the deconvolutional layers.

[0032] The present invention relates to a decoding device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor provides a decoding device used to execute the above decoding method by executing machine-executable instructions.

[0033] The present invention relates to an encoding device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor provides an encoding device used to execute machine-executable instructions and implement the above-described encoding method.

[0034] The present invention relates to an electronic device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor provides an electronic device used to execute machine-executable instructions to carry out the above-described decoding or encoding method.

[0035] The present invention provides a machine-readable storage medium in which a plurality of computer instructions are stored, wherein when the computer instructions are executed by a processor, the above-described decoding method or encoding method is performed.

[0036] The present invention provides a computer application that, when executed by a processor, performs the above-described decoding or encoding method.

[0037] As can be seen from the above technical proposals, the embodiments of the present invention propose a synthetic transformation network of an attention mechanism for neural network-based coding and decoding technology, the synthetic transformation network includes an attention module, the attention module is a cascaded hybrid attention module, and by realizing the synthetic transformation network with the cascaded hybrid attention module, the quality of the synthesized image is guaranteed, the complexity of the network and computation is effectively reduced, decoding performance is improved, and the quality of the reconstructed image block is effectively guaranteed, coding performance and decoding performance are improved. [Brief explanation of the drawing]

[0038] [Figure 1] This is a schematic diagram of a 3D feature matrix according to one embodiment of the present invention. [Figure 2] This is a schematic diagram of pixel reconstruction according to one embodiment of the present invention. [Figure 3] This is a flowchart of a decoding method according to one embodiment of the present invention. [Figure 4] This is a schematic diagram of the encoding process according to one embodiment of the present invention. [Figure 5] This is a schematic diagram of the decoding process according to one embodiment of the present invention. [Figure 6A] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 6B] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 6C] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 6D] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 7A] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 7B] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8A] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8B] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8C] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8D] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8E] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8F] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8G] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 8H] This is a schematic diagram of the structure of a synthesis-transformation network according to one embodiment of the present invention. [Figure 9A] This is a hardware structure diagram of a decoding device according to one embodiment of the present invention. [Figure 9B] This is a hardware structure diagram of an encoding device according to one embodiment of the present invention. [Modes for carrying out the invention]

[0039] The terms used in the embodiments of this invention are merely for the purpose of describing specific embodiments and are not intended to limit the invention. The singular forms “one kind,” “the said,” and “the” used in the embodiments and claims of this invention are also intended to include the plural form unless the context clearly indicates otherwise. Furthermore, it should be understood that the term “and / or” used in this invention means including any or all possible combinations of one or more related enumerated items. The embodiments of this invention may use terms such as first, second, third, etc. to describe various types of information, but it should be understood that this information is not limited to these terms. These terms are used only to distinguish the same type of information. For example, as long as it does not deviate from the scope of the embodiments of this invention, depending on the context, first information may be called second information, and similarly, second information may be called first information. Furthermore, the word “…case” used herein may be interpreted as “…and,” “…when,” or “in response to a decision.”

[0040] Embodiments of the present invention provide a decoding method, which may relate to the following concepts.

[0041] Entropy coding: Entropy coding is a method of coding in which no information is lost during the coding process, according to the principle of entropy. Information entropy is the average amount of information (degree of uncertainty) of the information source. Entropy coding schemes may include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0042] Neural Network (NN): A neural network refers to an artificial neural network, which is a computational model composed of a large number of nodes (called neurons) connected to one another. In a neural network, neurons (often called processing units) can represent different objects, such as features, alphabets, concepts, or several meaningful abstract modes. There are three types of processing units in a neural network: input units, output units, and hidden units. Input units receive external signals and data, output units realize the output of the processing results, and hidden units are units that are between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information are reflected in the connection relationships of the processing units. A neural network is an unprogrammed, brain-like information processing method, and the essence of a neural network is to acquire parallel and distributed information processing capabilities through the transformation and dynamic actions of the neural network, mimicking the information processing capabilities of the human brain and nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.

[0043] Convolutional Neural Networks (CNNs): Convolutional neural networks are feedforward neural networks and one of the representative network structures in deep learning technology. The artificial neurons in a convolutional neural network can respond to peripheral units within a certain coverage area, and exhibit excellent performance in large-scale image processing. The basic structure of a convolutional neural network consists of two layers: one is a feature extraction layer (also called a convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, and the local features are extracted. Once the local features are extracted, their positional relationship with other features is also determined. The other is a feature mapping layer (also called an activation layer), where each computational layer of the neural network consists of multiple feature mappings, each feature mapping is a plane, and the weights of all neurons in the plane are equal. The feature mapping structure may use functions such as the Sigmoid function, ReLU (Rectified Linear Unit) function, Leaky-ReLU function, PReLU (Parametric Rectified Linear Unit) function, and GDN (Generalized Divisive Normalization) function as activation functions for the convolutional neural network. Furthermore, because neurons on a single mapping plane share weights, the number of free parameters in the network is reduced.

[0044] For example, one advantage of convolutional neural networks compared to image processing algorithms is that they can avoid complex pre-processing processes for images (such as extracting artificial features), directly inputting original images and performing end-to-end learning. Another advantage of convolutional neural networks compared to general neural networks is that while general neural networks employ a fully connected architecture, meaning all neurons from the input layer to the hidden layer are connected, resulting in a huge number of parameters and making network training time-consuming and difficult, convolutional neural networks avoid this difficulty through methods such as local connectivity and weight sharing.

[0045] Deconvolution: Also known as transposed convolution, the operation process of deconvolution and convolution is similar, the main difference being that deconvolution uses padding to make the output larger than the input (although they may be the same). A stride of 1 indicates that the output size is equal to the input size, and a stride of N indicates that the width of the output features is N times the width of the input features, and the height of the output features is N times the height of the input features.

[0046] Depth-separable convolution: This requires two convolution operations. In the first convolution, a deep-wise convolution is performed (i.e., features of each layer are collected), with kernel_size = K*K*1 and the total number of parameters in the first convolution being K*K*Cin. In the second convolution, to obtain a Cout-dimensional output, kernel_size = 1*1*Cin and the total number of parameters in the second convolution being 1*1*Cin*Cout. The output of the second convolution may be the output of a depth-separable convolution, and the input of the first convolution may be the input of a depth-separable convolution.

[0047] Generalization ability refers to a machine learning algorithm's ability to adapt to fresh samples. The goal of learning is to learn the underlying rules of the data, and the trained network can produce appropriate output even for data outside the training set that has the same rules. This ability may also be called generalization ability.

[0048] A feature is a 3D feature matrix or tensor of type C*W*H. Figure 1 is a schematic diagram of a 3D feature matrix, where C represents the number of channels, H represents the feature height, and W represents the feature width. The 3D feature matrix may be the input or output of a neural network.

[0049] Pixel reconstruction (also known as feature position reconstruction) involves reconstructing low-resolution features across multiple channels to obtain high-resolution feature maps. Figure 2 is a schematic diagram of pixel reconstruction, specifically showing the case where the upsampling factor is 2. Clearly, a feature with 4 channels and a resolution of 3*3 becomes a feature with 1 channel and a resolution of 6*6 through feature position reconstruction.

[0050] The Rate-Distortion Optimization principle: Encoding efficiency is evaluated using two metrics: bitrate and PSNR (Peak Signal to Noise Ratio). A smaller bitstream results in greater compression, and a higher PSNR results in better reconstructed image quality. When selecting a mode, the discriminant is essentially a comprehensive evaluation of both. For example, the cost corresponding to a mode is J(mode) = D + λ*R.

[0051] Here, D represents Distortion, which can usually be evaluated using the SSE (sum-square error) metric, where SSE refers to the mean square sum of the differences between the reconstructed image block and the source image. To account for cost, the SAD (Sum of Absolute Difference) metric may also be used, where SAD refers to the sum of the absolute differences between the reconstructed image block and the source image, λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in that mode, including the total number of bits required to encode mode information, motion information, residuals, etc. When selecting a mode, comparing and deciding on the encoding mode using the rate-distortion principle can usually guarantee optimal encoding performance.

[0052] For each module on the encoding side, a great many encoding tools have been proposed, and each tool usually has many modes. The encoding tool that yields optimal encoding performance often differs for different video sequences. Therefore, in the encoding process, Rate-Distortion Optimize (RDO) is typically used to compare the encoding performance of different tools or modes and select the optimal mode. After determining the optimal tool or mode, the decision information is transmitted by encoding mark information into the bitstream. While this method introduces encoding complexity, it allows for the adaptive selection of the optimal mode combination for different content, thereby achieving optimal encoding performance. On the decoding side, the relevant mode information can be obtained by directly analyzing the mark information, minimizing the impact of complexity.

[0053] The end-to-end general-purpose image coding framework primarily consists of a feature main information portion and a hyper-prior side information portion. The feature main information portion includes an analysis-transformation network, quantization, normal entropy coding, normal entropy decoding, and a synthesis-transformation network. The hyper-prior side information portion includes a hyper-prior analysis network, quantization, factorized entropy coding, factorized entropy decoding, and a hyper-prior synthesis network. Each image component x is compressed, coded, and reconstructed by the analysis-transformation network and synthesis-transformation network of the feature main information portion, while the hyper-prior side information portion primarily models the probabilities of the feature main information and is used to guide the entropy coding and decoding of the feature main information. The end-to-end general-purpose image coding framework has problems such as high computational complexity and poor decoding performance.

[0054] In response to the above findings, this embodiment proposes a synthetic transformation network for attention mechanisms, utilizing the features of an end-to-end image coding framework. This synthetic transformation network includes an attention module, which is a cascaded hybrid attention module. Implementing the synthetic transformation network with a cascaded hybrid attention module ensures the quality of the synthesized image, while effectively reducing the complexity of the network and computation, and improving decoding performance. The attention module is an attention network.

[0055] The decoding method according to an embodiment of the present invention will be described in detail below with reference to several specific examples.

[0056] Example 1: An embodiment of the present invention provides a decoding method, Figure 3 is a schematic flowchart of the decoding method, which may be applied to the decoding side (also called a video decoder), and which may include the following steps.

[0057] Step 301: Decode a bitstream corresponding to the current image block to obtain the coefficient hyperparameter features of the current image block.

[0058] Step 302: Based on the coefficient hyperparameter features, a probability distribution parameter is determined, and based on the probability distribution parameter, another bitstream corresponding to the current image block is decoded to obtain a reconstruction feature corresponding to the current image block.

[0059] Step 303, the reconstructed features are input to a composite transformation network to obtain a reconstructed image block corresponding to the current image block. Here, the composite transformation network includes at least an attention module, which may be a cascaded hybrid attention module. The cascaded hybrid attention module may include a first attention submodule and a second attention submodule, which are two submodules in series, for example, the output features of the first attention submodule being the input features of the second attention submodule.

[0060] Here, one bitstream corresponding to the current image block may be a bitstream in which the coefficient hyperparameter features corresponding to the current image block are encoded, and another bitstream corresponding to the current image block may be a bitstream in which the residual features corresponding to the current image block are encoded.

[0061] For example, if the attention module is a cascaded hybrid attention module, a first processing may be performed on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule, the output features of the first attention submodule being the input features of the second attention submodule, and the first processing may be at least one of layer normalization, convolution, and dimensionality transformation operations. A second processing may be performed on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule, and the second processing may be at least one of depthwise separable convolution operation, layer normalization, multilayer sensing operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing may be at least one of depthwise separable convolution operation, layer normalization, and multilayer sensing operation. Or, the second processing may be at least one of layer normalization, linear operation, and activation operation. Alternatively, the second process may be at least one of the following: downsampling, residual convolution, upsampling, or activation.

[0062] In one possible embodiment, the step of performing a first processing on the input features of a first attention submodule via a first attention submodule to obtain the output features of the first attention submodule may include, but is not limited to, the steps of: performing layer normalization on the input features of the first attention submodule to obtain the layer-normalized features; performing a three-path convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; performing a dimensional transformation operation on each of the query vector, key vector, and value vector to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector; determining a modified feature corresponding to the input features based on the attention weights and the dimensionally transformed value vector; and determining the output features of the first attention submodule based on the input features and the modified features.

[0063] In one possible embodiment, the steps of performing a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule include: performing layer normalization on the input features of the first attention submodule to obtain the layer-normalized features; performing window division on the layer-normalized features to obtain a plurality of small-size features; performing a three-path convolution operation on each small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; combining the small-size query features corresponding to the plurality of small-size features to obtain a query vector; and the steps corresponding to the plurality of small-size features The steps may include, but are not limited to, the steps of: combining small-size key features to obtain a key vector; combining small-size value features corresponding to multiple small-size features to obtain a value vector; performing a dimensional transformation operation on the query vector, the key vector, and the value vector to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector; determining modification features corresponding to the input features based on the attention weights and the dimensionally transformed value vector; and determining output features of a first attention submodule based on the input features and modification features.

[0064] In one possible embodiment, the step of performing a first processing on the input features of a first attention submodule via a first attention submodule to obtain the output features of the first attention submodule may include, but is not limited to, the steps of: performing layer normalization on the input features of the first attention submodule to obtain the layer-normalized features; performing a three-path convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector; determining attention weights based on the query vector and the key vector; determining modified features corresponding to the input features based on the attention weights and the value vector; and determining the output features of the first attention submodule based on the input features and the modified features.

[0065] In one possible embodiment, the step of performing a first processing on the input features of a first attention submodule via a first attention submodule to obtain the output features of the first attention submodule may include, but is not limited to, the steps of: performing layer normalization on the input features of the first attention submodule to obtain the layer-normalized features; performing window division on the layer-normalized features to obtain a plurality of small-size features; performing a three-path convolution operation on each small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; combining the small-size query features corresponding to the plurality of small-size features to obtain a query vector; combining the small-size key features corresponding to the plurality of small-size features to obtain a key vector; combining the small-size value features corresponding to the plurality of small-size features to obtain a value vector; determining attention weights based on the query vector and key vector; determining modified features corresponding to the input features based on the attention weights and value vector; and determining the output features of the first attention submodule based on the input features and modified features.

[0066] In one possible embodiment, the step of performing a second process on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule may include, but is not limited to, the steps of: performing layer normalization on the input features of the second attention submodule to obtain the layer-normalized features; performing a first linear operation on the layer-normalized features to obtain the features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain the activated features; performing a second linear operation on the activated features to obtain the features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0067] In one possible embodiment, the step of performing a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule may include, but is not limited to, the steps of: performing a downsampling operation on the input features of the second attention submodule to obtain the downsampled features; performing a residual convolution operation on the downsampled features to obtain the residual convolutional features; performing an upsampling operation on the residual convolutional features to obtain the upsampled features; performing an activation operation on the upsampled features to obtain the activated features; performing a residual convolution operation on the input features to obtain the convolutional features after residual convolution; and determining the output features of the second attention submodule based on the input features, the activated features, and the convolutional features.

[0068] In one possible embodiment, the synthetic transformation network may further include at least one deconvolutional layer, and the attention module may be located after one of these deconvolutional layers.

[0069] Exemplary, the synthetic transformation network may further include a residual layer, a first deconvolution layer, a first crop layer, a first residual activation layer, a second deconvolution layer, a second crop layer, a second residual activation layer, a third deconvolution layer, a third crop layer, a third residual activation layer, a fourth deconvolution layer, and a fourth crop layer. Here, the synthetic transformation network may include at least one attention module, one of which is located after the first deconvolution layer.

[0070] Exemplary, the synthetic transformation network includes at least a residual layer, a first deconvolution layer, a first crop layer, a first residual activation layer, and a second deconvolution layer. The synthetic transformation network includes at least one attention module, one of which is located after the first deconvolution layer.

[0071] Based on a similar concept to the above decoding method, embodiments of the present invention further provide an encoding method which may be applied to an encoding side (also called a video encoder), the method comprising the steps of: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; and inputting the reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, the composite transformation network comprising at least an attention module, the attention module being a cascaded hybrid attention module, the cascaded hybrid attention module comprising a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series.

[0072] Here, one bitstream corresponding to the current image block may be a bitstream in which the coefficient hyperparameter features corresponding to the current image block are encoded, and another bitstream corresponding to the current image block may be a bitstream in which the residual features corresponding to the current image block are encoded.

[0073] For illustrative purposes, the encoding method is similar to the implementation process of the decoding method, and we will omit explanations of overlapping content.

[0074] For illustrative purposes, the above execution order is merely illustrative for the sake of clarity, and in actual applications, the order of execution between steps may be changed and is not limiting. Furthermore, in other embodiments, the steps of the corresponding method may not necessarily be performed in the order shown and described herein, and the method may include more or fewer steps than those described herein. Also, a single step described herein may be broken down into multiple steps in other embodiments, and multiple steps described herein may be combined into a single step in other embodiments.

[0075] As can be seen from the above technical proposals, the embodiments of the present invention propose a synthetic transformation network of an attention mechanism for neural network-based coding and decoding technology, the synthetic transformation network includes an attention module, the attention module is a cascaded hybrid attention module, and by realizing the synthetic transformation network with the cascaded hybrid attention module, the quality of the synthesized image is guaranteed, the complexity of the network and computation is effectively reduced, decoding performance is improved, and the quality of the reconstructed image block is effectively guaranteed, coding performance and decoding performance are improved.

[0076] Example 2: The encoding process is shown in Figure 4. Of course, Figure 4 is just one example of the encoding process, and the encoding process is not limited to this specific process; it is possible to implement the encoding process using a neural network.

[0077] The encoding side may, after obtaining the current image block x (which may be the original image block x, i.e., the input image block), perform an analytical transformation on the current image block x via an analytical transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Here, performing a feature transformation on the current image block x via the analytical transformation network means transforming the current image block x into image features y in the latent domain, so that all subsequent processes can operate in the latent domain.

[0078] The image may be divided into one image block or into multiple image blocks. If the image is divided into one image block, the current image block x may be an image itself, that is, the encoding and decoding processes for the image block may be applied directly to the image.

[0079] The encoding side obtains image features y, then performs a coefficient hyperparameter feature transformation on image features y to obtain coefficient hyperparameter features z. For example, image features y may be input to a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network may perform a coefficient hyperparameter feature transformation on image features y to obtain coefficient hyperparameter features z. The hyperparameter coding network may be a trained neural network, and the training process of this hyperparameter coding network is not limited; it is sufficient that a coefficient hyperparameter feature transformation can be performed on image features y. Here, after the image features y of the latent domain pass through the hyperparameter coding network, hyperprior latent information z is obtained.

[0080] After obtaining the coefficient hyperparameter feature z, the encoding side may quantize the coefficient hyperparameter feature z to obtain the corresponding hyperparameter quantization feature; that is, the Q operation in Figure 4 is a quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 (which may also be called the first bitstream) corresponding to the current image block; that is, the AE operation in Figure 4 represents an encoding process such as an entropy encoding process. Alternatively, the encoding side may directly encode the coefficient hyperparameter feature z to obtain Bitstream#1 corresponding to the current image block. The hyperparameter quantization feature or coefficient hyperparameter feature z contained in Bitstream#1 is mainly used to obtain the mean value and the parameters of the probability distribution model.

[0081] After obtaining Bitstream#1 corresponding to the current image block, the encoding side may send Bitstream#1 corresponding to the current image block to the decoding side. The decoding side's processing process for Bitstream#1 corresponding to the current image block will be described in subsequent embodiments.

[0082] After obtaining Bitstream#1 corresponding to the current image block, the encoding side may decode Bitstream#1 to obtain the hyperparameter quantization feature, i.e., AD in Figure 4 represents the decoding process, and then the hyperparameter quantization feature is inversely quantized to obtain the coefficient hyperparameter feature z_hat, which may be the same as or different from the coefficient hyperparameter feature z, and the IQ operation in Figure 4 is the inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoding side may decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without performing the inverse quantization process of the hyperparameter quantization feature.

[0083] The encoding process for Bitstream#1 may employ a fixed probability density model encoding method, and the decoding process for Bitstream#1 may employ a fixed probability density model decoding method; however, the encoding and decoding processes are not limited to these methods.

[0084] After obtaining the coefficient hyperparameter feature z_hat, the encoding side may perform a context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block (the process for determining the reconstructed feature y_hat will be described in subsequent examples) to obtain a predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat may be input to a mean prediction network, and the mean prediction network may determine the predicted value mu based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat. This prediction process is not limited. Here, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstructed feature y_hat. These two are combined and input to obtain a more accurate predicted value mu. The predicted value mu is used to obtain the residual by calculating the difference from the original feature, and is added to the decoded residual to obtain the reconstructed feature y_hat.

[0085] In another embodiment, the mean prediction network may obtain the predicted value corresponding to the current image block based solely on the coefficient hyperparameter feature z_hat of the current image block; that is, it may obtain the predicted value corresponding to the current image block without using the reconstruction feature y_hat of the previous image block. This significantly reduces the complexity of obtaining the predicted value and accelerates the encoding and decoding processes. The method for obtaining the predicted value on the encoding side can employ one of the two methods described above and will not be explained in detail below.

[0086] Note that the mean prediction network is a selectable neural network; that is, it is not necessary to have a mean prediction network, meaning that it is not necessary to determine the predicted value mu via the mean prediction network. The dashed box in Figure 4 indicates that the mean prediction network is selectable.

[0087] The encoding side can obtain image features y and then determine residual features r based on image features y and predicted values ​​mu. For example, residual features r are defined as the difference between image features y and predicted values ​​mu. Subsequently, feature processing is performed on residual features r to obtain image features s. This feature processing process is not limited to any specific method. In this case, an average value prediction network is required, and the predicted values ​​mu are provided by this network. Alternatively, the encoding side may obtain image features s after obtaining image features y, and this feature processing process is not limited to any specific method. In this case, an average value prediction network is not required, and the dashed box indicates that the residual process is a selectable process.

[0088] After obtaining image features s, the encoding side may quantize the image features s to obtain image quantization features corresponding to the image features s; that is, the Q operation in Figure 4 is a quantization process. After obtaining image quantization features corresponding to image features s, the encoding side may encode these image quantization features to obtain Bitstream #2 (which may also be called the second bitstream) corresponding to the current image block; that is, the AE operation in Figure 4 represents an encoding process such as an entropy encoding process. Alternatively, the encoding side may directly encode the image features s to obtain Bitstream #2 corresponding to the current image block without performing a quantization process of the image features s.

[0089] After obtaining Bitstream#2 corresponding to the current image block, the encoding side may send Bitstream#2 corresponding to the current image block to the decoding side. The decoding side's processing process for Bitstream#2 corresponding to the current image block will be described in subsequent embodiments.

[0090] After obtaining Bitstream#2 corresponding to the current image block, the encoding side may decode Bitstream#2 to obtain image quantization features, i.e., AD in Figure 4 represents the decoding process, and then the encoding side may dequantize the image quantization features to obtain image features s', image features s' may be the same as or different from image features s, and the IQ operation in Figure 4 is the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoding side may decode Bitstream#2 to obtain image features s' without performing the dequantization process of image quantization features.

[0091] The encoding side may, after obtaining image features s', perform feature reconstruction (i.e., the reverse process of feature processing) on ​​image features s'. This feature reconstruction process is not limited to any specific method, and any feature reconstruction method may be used to obtain residual features r_hat, which may be the same as or different from residual features r. After obtaining residual features r_hat, the encoding side determines reconstructed features y_hat based on residual features r_hat and predicted values ​​mu. Reconstructed features y_hat may be the same as or different from image features y. For example, the sum of residual features r_hat and predicted values ​​mu is taken as reconstructed features y_hat. In this case, it is necessary to implement an mean prediction network, which provides the predicted values ​​mu. Alternatively, the encoding side may, after obtaining image features s', perform feature reconstruction (i.e., the reverse process of feature processing) on ​​image features s' to obtain reconstructed features y_hat. Reconstructed features y_hat may be the same as or different from image features y. In this case, it is not necessary to implement an mean prediction network, and the dashed box indicates that the residual process is a selectable process.

[0092] The encoding side may, after obtaining the reconstructed feature y_hat, perform a composite transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat can be input to a composite transformation network, and the composite transformation network will perform a composite transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is complete.

[0093] In one possible embodiment, when the encoding side encodes image quantization features or image features s to obtain Bitstream#2 corresponding to the current image block, it first needs to determine a probability distribution model, and then encode the image quantization features or image features s based on the probability distribution model. Similarly, when decoding Bitstream#2, the encoding side also first needs to determine a probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0094] To obtain a probability distribution model, as shown in Figure 4, the encoding side may, after obtaining the coefficient hyperparameter feature z_hat, perform an inverse coefficient hyperparameter feature transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Alternatively, the coefficient hyperparameter feature z_hat may be input to a probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network may perform an inverse coefficient hyperparameter feature transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Or, after obtaining the probability distribution parameter p, a probability distribution model may be generated based on the probability distribution parameter p. Here, the probabilistic hyperparameter decoding network may be a trained neural network, and the training process of this probabilistic hyperparameter decoding network is not limited; it is sufficient that an inverse coefficient hyperparameter feature transformation can be performed on the coefficient hyperparameter feature z_hat.

[0095] In one possible embodiment, the encoding-side processing process may be performed by a deep learning model or a neural network model to realize an end-to-end image compression and encoding process, but is not limited to this encoding process.

[0096] Example 3: The decoding process is shown in Figure 5. Of course, Figure 5 is merely one example of a decoding process, and the decoding process is not limited to this specific process; a neural network can be used to implement the decoding process.

[0097] After obtaining Bitstream#1 corresponding to the current image block, the decoding side may decode Bitstream#1 to obtain hyperparameter quantization features. That is, AD in Figure 5 represents the decoding process, and then the hyperparameter quantization features are inversely quantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat may be the same as or different from the coefficient hyperparameter feature z, and the IQ operation in Figure 5 is the inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoding side may decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without performing the inverse quantization process of the hyperparameter quantization features. Here, the decoding process for Bitstream#1 may employ a decoding method of a fixed probability density model, and is not limited to this.

[0098] The image may be divided into one image block or into multiple image blocks. If the image is divided into one image block, the current image block x may be an image itself, that is, the decoding process for the image block may be applied directly to the image.

[0099] After obtaining the coefficient hyperparameter feature z_hat, the decoding side may perform a context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block (the process for determining the reconstructed feature y_hat will be described in subsequent examples) to obtain a predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat may be input to a mean prediction network, and the mean prediction network may determine the predicted value mu based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat. This prediction process is not limited. Here, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstructed feature y_hat, and both are combined and input to obtain a more accurate predicted value mu.

[0100] In another embodiment, the decoding side may, after obtaining the coefficient hyperparameter feature z_hat of the current image block, directly obtain the predicted value mu corresponding to the current image block based on the coefficient hyperparameter feature z_hat, that is, obtain the predicted value without depending on the reconstruction feature y_hat of the previous image block. This significantly reduces the complexity of obtaining the predicted value and accelerates the encoding and decoding process. The method for obtaining the predicted value on the decoding side can always be one of the two methods described above and will not be explained in detail below.

[0101] Note that the mean prediction network is a selectable neural network; that is, it is not necessary to have a mean prediction network, meaning that it is not necessary to determine the predicted value mu via the mean prediction network. The dashed box in Figure 5 indicates that the mean prediction network is selectable.

[0102] After obtaining Bitstream#2 corresponding to the current image block, the decoding side may decode Bitstream#2 to obtain image quantization features, i.e., AD in Figure 5 represents the decoding process, and then the decoding side may dequantize the image quantization features to obtain image features s', image features s' may be the same as or different from image features s, and the IQ operation in Figure 5 may be the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the decoding side may decode Bitstream#2 to obtain image features s' without performing the dequantization process of image quantization features.

[0103] The decoding side may, after obtaining image features s', perform feature reconstruction (i.e., the reverse process of feature processing) on ​​image features s' to obtain residual features r_hat, where residual features r_hat may be the same as or different from residual features r. After obtaining residual features r_hat, the decoding side determines reconstructed features y_hat based on residual features r_hat and predicted values ​​mu, where reconstructed features y_hat may be the same as or different from image features y, for example, the sum of residual features r_hat and predicted values ​​mu is taken as reconstructed features y_hat. In this case, it is necessary to set up an mean prediction network, which provides the predicted values ​​mu. Alternatively, the decoding side may, after obtaining image features s', perform feature reconstruction on image features s' to obtain reconstructed features y_hat, where reconstructed features y_hat may be the same as or different from image features y. In this case, it is not necessary to set up an mean prediction network, and the dashed box indicates that the residual process is a selectable process.

[0104] The decoding side may, after obtaining the reconstructed feature y_hat, perform a composite transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat may be input to a composite transformation network, and the composite transformation network may perform a composite transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is complete.

[0105] In one possible embodiment, the decoding side must first determine a probability distribution model when decoding Bitstream#2, and then decode Bitstream#2 based on this probability distribution model. To obtain the probability distribution model, as shown in Figure 5, the decoding side may first obtain the coefficient hyperparameter feature z_hat, then perform an inverse coefficient hyperparameter feature transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat may be input to a probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network may perform an inverse coefficient hyperparameter feature transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. Alternatively, after obtaining the probability distribution parameter p, a probability distribution model may be generated based on the probability distribution parameter p. Here, the probabilistic hyperparameter decoding network may be a trained neural network, and the training process of this probabilistic hyperparameter decoding network is not limited; it is sufficient that the probability distribution parameter p can be obtained by performing an inverse coefficient hyperparameter feature transformation on the coefficient hyperparameter feature z_hat.

[0106] In one possible embodiment, the decoding process may be performed by a deep learning model or a neural network model to realize an end-to-end image compression and encoding process, and is not limited to this decoding process.

[0107] Example 5: For Examples 1, 2, and 3, an average value prediction network may or may not be deployed. An example is given of deploying an average value prediction network to improve feature coding performance. When an average value prediction network is deployed, in order to obtain accurate predicted values ​​of image features, a context-based prediction may be performed based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block to obtain a predicted value mu corresponding to the current image block, or a predicted value mu corresponding to the current image block may be obtained based on the coefficient hyperparameter feature z_hat of the current image block.

[0108] The encoding process may include the following steps:

[0109] In step S11, after obtaining the original image, it is decided whether or not to divide the original image into blocks based on the image resolution. For example, if the image resolution is greater than a threshold, the original image is divided into blocks; otherwise, the original image is not divided into blocks. If this is the case, the original image is divided into multiple image blocks x, and there is an overlapping portion between adjacent image blocks. If there is no overlap, then image block x is the original image, meaning the original image has only one image block. To facilitate explanation, one image block will be used as an example below, and this image block will be referred to as the current image block x. The processing process of the current image block x will be explained below as an example.

[0110] In step S12, after obtaining the current image block x, the analysis-transformation network performs an analysis-transformation on the current image block x to obtain the image feature y (i.e., feature block) corresponding to the current image block x. For example, the analysis-transformation network transforms the current image block x into an image feature y in the latent domain so that all subsequent processes can operate in the latent domain.

[0111] Step S13: A coefficient hyperparameter feature transformation is performed on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x. For example, the image feature y may be input into a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network may perform a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z corresponding to the current image block x.

[0112] Step S14: The coefficient hyperparameter feature z is encoded into a first bitstream (Bitstream #1) corresponding to the current image block. For example, the coefficient hyperparameter feature z is quantized to obtain a hyperparameter quantization feature, and this hyperparameter quantization feature is encoded to obtain the first bitstream corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z is directly encoded to obtain the first bitstream corresponding to the current image block. That is, the first bitstream corresponding to the current image block is a bitstream in which the coefficient hyperparameter feature z corresponding to the current image block x is encoded. After obtaining the first bitstream corresponding to the current image block, the first bitstream corresponding to the current image block is transmitted to the decoding side.

[0113] In step S15, after obtaining the first bitstream corresponding to the current image block, the first bitstream is decoded to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream may be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature may be inversely quantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream may be decoded to obtain the coefficient hyperparameter feature z_hat without performing the inverse quantization process.

[0114] Step S16: The probability distribution parameters are determined based on the coefficient hyperparameter feature z_hat. For example, the coefficient hyperparameter feature z_hat is subjected to an inverse transformation to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into a probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model is generated based on the probability distribution parameter p.

[0115] In step S17, the coefficient hyperparameter feature z_hat is input to the mean prediction network, and the mean prediction network makes a context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat of the previous image block to obtain the predicted value mu. Alternatively, the coefficient hyperparameter feature z_hat is input to the mean prediction network, and the mean prediction network obtains the predicted value mu based on the coefficient hyperparameter feature z_hat.

[0116] In step S18, the residual feature r is determined based on the image feature y and the mean feature mu (i.e., the predicted value mu). For example, the residual feature r is defined as the difference between the image feature y and the mean feature mu. Feature processing is performed on the residual feature r to obtain the residual feature r after feature processing. This feature processing process is not limited. Feature processing is an optional step, and feature processing is not required for the residual feature r.

[0117] Step S19: The residual feature r (or the residual feature r after feature processing) is quantized to obtain an image quantization feature, and this image quantization feature is encoded to obtain a second bitstream (Bitstream #2) corresponding to the current image block. Alternatively, the residual feature r (or the residual feature r after feature processing) may be directly encoded to obtain a second bitstream corresponding to the current image block. That is, the second bitstream corresponding to the current image block is a bitstream in which the residual feature r corresponding to the current image block x is encoded. After obtaining the second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is transmitted to the decoding side.

[0118] For example, when encoding image quantization features or residual features r, a probability distribution model corresponding to the probability distribution parameter p may be used to encode the image quantization features or residual features r to obtain a second bitstream.

[0119] In step S20, after obtaining the second bitstream corresponding to the current image block, the second bitstream is decoded using a probability distribution model corresponding to the probability distribution parameter p to obtain image quantization features, the image quantization features are dequantized, and feature reconstruction is performed on the dequantized features to obtain residual features r_hat, or the image quantization features are dequantized and residual features r_hat are obtained directly without performing the feature reconstruction process. Alternatively, the second bitstream is decoded and feature reconstruction is performed on the decoded features to obtain residual features r_hat, or the second bitstream is decoded and residual features r_hat are obtained directly.

[0120] In step S21, after obtaining the residual feature r_hat, the reconstructed feature y_hat is determined based on the residual feature r_hat and the predicted value mu. For example, the reconstructed feature y_hat is defined as the sum of the residual feature r_hat and the predicted value mu.

[0121] In step S22, the reconstructed feature y_hat is input to the composite transformation network, which determines the reconstructed image block x_hat corresponding to the current image block based on the reconstructed feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0122] As an example, using the deployment of an average prediction network, the decoding process may include the following steps:

[0123] In step S31, after obtaining the first bitstream corresponding to the current image block, the first bitstream is decoded to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream may be decoded to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the hyperparameter quantization feature may be inversely quantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream may be decoded to obtain the coefficient hyperparameter feature z_hat without performing the inverse quantization process.

[0124] Step S32: The probability distribution parameters are determined based on the coefficient hyperparameter feature z_hat. For example, the coefficient hyperparameter feature z_hat is subjected to an inverse transformation to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into a probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model is generated based on the probability distribution parameter p.

[0125] In step S33, after obtaining a second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is decoded based on the probability distribution parameter p to obtain a reconstructed feature corresponding to the current image block. For example, the residual feature r_hat corresponding to the current image block is determined, the predicted value mu corresponding to the current image block is determined, and the reconstructed feature y_hat is determined based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu (i.e., the predicted value mu) is taken as the reconstructed feature y_hat.

[0126] Here, in order to determine the predicted value mu corresponding to the current image block, the coefficient hyperparameter feature z_hat may be obtained, then the coefficient hyperparameter feature z_hat may be input into the mean prediction network, and the mean prediction network may perform a context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat of the previous image block to obtain the predicted value mu corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z_hat may be input into the mean prediction network, and the mean prediction network may obtain the predicted value mu corresponding to the current image block based on the coefficient hyperparameter feature z_hat.

[0127] To determine the residual feature r_hat corresponding to the current image block, the second bitstream is decoded using a probability distribution model corresponding to the probability distribution parameter p to obtain the image quantization feature, the image quantization feature is dequantized, and the residual feature r_hat is obtained by performing feature reconstruction on the dequantized feature, or the residual feature r_hat is obtained directly without dequantizing the image quantization feature and performing the feature reconstruction process. Alternatively, the second bitstream may be decoded and the residual feature r_hat may be obtained by performing feature reconstruction on the decoded feature, or the residual feature r_hat may be obtained directly by decoding the second bitstream.

[0128] In step S34, the reconstructed feature y_hat is input to the composite transformation network, which determines the reconstructed image block x_hat corresponding to the current image block based on the reconstructed feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.

[0129] Example 5: In Examples 1, 2, 3, and 4, a synthesis-transformation network is involved in all cases. The embodiments of this example are the same as those in Examples 1, 2, 3, or 4, and the explanation of redundant content will be omitted. The synthesis-transformation network involved will be described in detail below.

[0130] An example of the composite transformation network is shown in Figure 6A, which may include, in order, a residual layer, a deconvolution layer, a crop layer, a residual activation layer, a deconvolution layer, a crop layer, a residual activation layer, a deconvolution layer, an attention model, a crop layer, a residual activation layer, a deconvolution layer, a crop layer, and so on. Here, the input feature of the composite transformation network may be a reconstructed feature y_hat, and the output feature of the composite transformation network may be a reconstructed image block x_hat, that is, after inputting the reconstructed feature y_hat into the composite transformation network, the reconstructed image block x_hat is obtained by sequentially processing the network layers described above.

[0131] As an example, an example of a residual layer in a composite transformation network is shown in Figure 6B, which may include, in order, a convolutional layer, a rectified linear unit (ReLU) (e.g., LeakyReLU), a convolutional layer, and a superposition layer (for achieving feature addition). The first convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, and is not limited to these; for example, a 3×3 convolutional layer may be selected. The second convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, and is not limited to these; for example, a 3×3 convolutional layer may be selected.

[0132] Exemplary, a residual activation layer in a composite transformation network is shown in Figure 6C, which may include, in order, an activation layer (e.g., a LeakyRelu activation layer), a convolutional layer, another activation layer (e.g., a tanh activation layer), a multiplication layer (for multiplying features), and a superposition layer (for adding features). The convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, and is not limited to these; for example, a 1×1 convolutional layer may be selected.

[0133] For example, the attention model in a synthetic transformation network may be a Residual Non-local Attention Block (RNAB), that is, the RNAB is introduced as an attention model, and the RNAB acquires non-local information from the image and uses this non-local information as attention weights to improve decoding performance. An example of an RNAB is shown in Figure 6D, which may include, in order, a residual block, a residual block, a downsampling convolutional layer (e.g., a 2x downsampling convolutional layer, e.g., a 3x3 convolutional layer), a residual block, a residual block, an upsampling convolutional layer (e.g., a 2x upsampling convolutional layer, e.g., a 3x3 convolutional layer), a residual block, a residual block, a convolutional layer (e.g., a 3x3 convolutional layer), an S-type activation function (Sigmoid), a residual block, a residual block, a residual block, a multiplication layer (for multiplication of features), and a superposition layer (for addition of features).

[0134] Regarding the residual block, it is shown in Figure 6B, and that is, the residual block is similar to the residual layer in Figure 6B and may include, in order, a convolutional layer, a rectified linear unit (ReLU) (e.g., LeakyReLU), a convolutional layer, and a superposition layer (for achieving feature addition). In this way, the input features of the residual block are extracted by the convolutional layer, the rectified linear unit, and the convolutional layer, and then added to the input features of the residual block to obtain the output features of the residual block.

[0135] A submodule consisting of residual blocks, residual blocks, a downsampling convolutional layer, residual blocks, residual blocks, an upsampling convolutional layer, residual blocks, residual blocks, a convolutional layer, and an S-type activation function may be a nonlocal attention extraction submodule, and a submodule consisting of three residual blocks may be a feature extraction submodule. The input features of the RNAB are processed through the nonlocal attention extraction submodule to generate attention weights, which are then multiplied with the output of the feature extraction submodule and added to the input features of the RNAB to obtain the output features of the RNAB.

[0136] Example 6: In Examples 1, 2, 3, and 4, a synthesis-transformation network is involved in all cases. The embodiments of this example are the same as those in Examples 1, 2, 3, or 4, and the explanation of redundant content will be omitted. The synthesis-transformation network involved will be described in detail below.

[0137] The composite translation network may include at least an attention module, which may be a cascaded hybrid attention module. In addition to the attention module, the composite translation network may further include other network layers, and this embodiment does not limit the structure of the composite translation network as long as it includes an attention module.

[0138] In one possible embodiment, in addition to the attention module, the composite transformation network may further include at least one deconvolutional layer, and in addition to the attention module and at least one deconvolutional layer, the composite transformation network may further include other network layers, but is not limited thereto. If the composite transformation network includes at least one deconvolutional layer, the attention module may be located after any of the deconvolutional layers. Alternatively, the attention module may be located after each deconvolutional layer, where the attention modules located after different deconvolutional layers may be exactly the same, or they may not be exactly the same. Alternatively, the attention module may be located after some deconvolutional layers (for example, K deconvolutional layers out of all deconvolutional layers, where K is greater than 1 and K is less than the total number of deconvolutional layers), where attention modules located after different deconvolutional layers may be exactly the same, or they may not be exactly the same.

[0139] In one possible embodiment, in addition to the attention module, the composite transformation network may include at least one network layer from among the residual layer, first deconvolution layer, first crop layer, first residual activation layer, second deconvolution layer, second crop layer, second residual activation layer, third deconvolution layer, third crop layer, third residual activation layer, fourth deconvolution layer, and fourth crop layer, i.e., the composite transformation network may include some or all of the above network layers. Of course, in addition to the above network layers, the composite transformation network may further include other network layers, which are described as examples, but the present invention is not limited thereto.

[0140] Figure 7A shows an example of a composite transformation network, which may include, in order, a residual layer, deconvolution layer 1 (i.e., the first deconvolution layer), crop layer 1 (i.e., the first crop layer), residual activation layer 1 (i.e., the first residual activation layer), deconvolution layer 2 (i.e., the second deconvolution layer), crop layer 2 (i.e., the second crop layer), residual activation layer 2 (i.e., the second residual activation layer), deconvolution layer 3 (i.e., the third deconvolution layer), crop layer 3 (i.e., the third crop layer), residual activation layer 3 (i.e., the third residual activation layer), deconvolution layer 4 (i.e., the fourth deconvolution layer), and crop layer 4 (i.e., the fourth crop layer). For example, the input feature of the composite transformation network may be a reconstructed feature y_hat, and the output feature of the composite transformation network may be a reconstructed image block x_hat. That is, after inputting the reconstructed feature y_hat to the composite transformation network, the network can obtain a reconstructed image block x_hat and output a reconstructed image block x_hat after sequentially going through the processing of the network layer described above.

[0141] Figure 7A shows three attention model positions, namely attention model position 1, attention model position 2, and attention model position 3. Exemplaryly, if the composite transformation network contains only one attention module (i.e., a cascaded hybrid attention module), the attention module may be located after the deconvolutional layer 1, i.e., the attention module is located at attention model position 1. Alternatively, the attention module may be located after the deconvolutional layer 2, i.e., the attention module is located at attention model position 2. Alternatively, the attention module may be located after the deconvolutional layer 3, i.e., the attention module is located at attention model position 3, for example, by placing the attention module after the deconvolutional layer 3. Of course, the above are merely some examples of attention module positions and are not limiting; they may be located at any position in the composite transformation network.

[0142] Exemplary, the composite transformation network may include attention module 1 located after deconvolution layer 1, attention module 2 located after deconvolution layer 2, and attention module 3 located after deconvolution layer 3, i.e., attention module 1 is located at attention model position 1, attention module 2 is located at attention model position 2, and attention module 3 is located at attention model position 3. Of course, attention modules may be located at other positions in the composite transformation network, and are not limited thereto.

[0143] Here, attention module 1, attention module 2, and attention module 3 may be exactly the same; that is, attention module 1, attention module 2, and attention module 3 may employ the same network structure. Alternatively, Attention Module 1, Attention Module 2, and Attention Module 3 do not have to be exactly the same. For example, the network structure of Attention Module 1 and the network structure of Attention Module 2 may be the same, or the network structure of Attention Module 1 and the network structure of Attention Module 3 may be different. Alternatively, the network structure of Attention Module 1 and the network structure of Attention Module 3 may be the same, or the network structure of Attention Module 1 and the network structure of Attention Module 2 may be different. Alternatively, the network structure of Attention Module 1 and the network structure of Attention Module 2 may be different, or the network structure of Attention Module 1 and the network structure of Attention Module 3 may be different.

[0144] Exemplary, the composite transformation network may include an attention module 1 located after the deconvolutional layer 1 and an attention module 2 located after the deconvolutional layer 2, i.e., attention module 1 is located at attention model position 1 and attention module 2 is located at attention model position 2. Of course, the attention modules may be located at other positions in the composite transformation network, and are not limited thereto.

[0145] Here, attention module 1 and attention module 2 may be exactly the same; for example, attention module 1 and attention module 2 may employ the same network structure. Alternatively, attention module 1 and attention module 2 do not have to be exactly the same; for example, the network structure of attention module 1 and the network structure of attention module 2 may be different.

[0146] Exemplary, the composite transformation network may include attention module 1 located after deconvolution layer 1 and attention module 2 located after deconvolution layer 3, i.e., attention module 1 is located at attention model position 1 and attention module 2 is located at attention model position 3. Of course, attention modules may be located at other positions in the composite transformation network, and are not limited thereto.

[0147] Here, attention module 1 and attention module 2 may be exactly the same; for example, attention module 1 and attention module 2 may employ the same network structure. Alternatively, attention module 1 and attention module 2 do not have to be exactly the same; for example, the network structure of attention module 1 and the network structure of attention module 2 may be different.

[0148] Exemplary, the synthetic transformation network may include an attention module 1 located after the deconvolutional layer 2 and an attention module 2 located after the deconvolutional layer 3, i.e., attention module 1 is located at attention model position 2 and attention module 2 is located at attention model position 3. Of course, the attention modules may be located at other positions in the synthetic transformation network, and are not limited thereto.

[0149] Here, attention module 1 and attention module 2 may be exactly the same; for example, attention module 1 and attention module 2 may employ the same network structure. Alternatively, attention module 1 and attention module 2 do not have to be exactly the same; for example, the network structure of attention module 1 and the network structure of attention module 2 may be different.

[0150] An example of a residual layer in a composite transformation network is shown in Figure 6B, where the residual layer may include, in order, a convolutional layer, a modified linear unit (e.g., LeakyReLU), a convolutional layer, and a superposition layer. That is, the input features are extracted by the convolutional layer, the modified linear unit, and the convolutional layer, and then added together to obtain the output features of the residual layer. Here, the first convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, and is not limited to these; for example, a 3×3 convolutional layer may be selected. Similarly, the second convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, and is not limited to these; for example, a 3×3 convolutional layer may be selected.

[0151] An example of a residual activation layer in a composite transformation network is shown in Figure 6C, where the residual activation layer may include, in order, an activation layer (e.g., a LeakyRelu activation layer), a convolutional layer, another activation layer (e.g., a tanh activation layer), a multiplicative layer, and a superposition layer, and the structure of this residual activation layer is not limited. Here, the convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, and is not limited to these; for example, a 1×1 convolutional layer may be selected.

[0152] Example 7: This example further describes the attention module involved, based on Example 6. Example 6 relates to an attention module in a synthetic transform network, which may be a cascade hybrid attention module. By replacing RNABs with a cascade hybrid attention module, decoding performance is guaranteed as much as possible while significantly reducing the computational complexity of decoding. Exemplarily, a cascade hybrid attention module can extract attention weights from input features and determine the output features of the cascade hybrid attention module based on the attention weights and input features.

[0153] Exemplary, a cascade hybrid attention module may include a first attention submodule (which may also be called a transformer-based attention submodule, and a transformer-based attention submodule will be denoted as a Transformer-based Attention Module) and a second attention submodule (which may also be called an upgrade convolution submodule, and will be denoted as a ConvNext Block). Here, the first and second attention submodules may be two submodules in series, that is, the output features of the first attention submodule are the input features of the second attention submodule. For example, Figure 7B is a schematic diagram of a cascade hybrid attention module, where the input features (or input image) of the cascade hybrid attention module are the input features of the first attention submodule (in Figure 7B, a transformer-based attention submodule is used as an example), the input features of the first attention submodule pass through the first attention submodule and are processed by the first attention submodule based on these input features to obtain the output features of the first attention submodule. The output features of the first attention submodule are used as input features of the second attention submodule (in Figure 7B, the upgrade convolution submodule is used as an example), the input features of the second attention submodule pass through the second attention submodule and are processed by the second attention submodule based on these input features to obtain the output features of the second attention submodule, and the output features of the second attention submodule are used as output features of the cascade hybrid attention module.

[0154] Exemplary, after inputting the input features of the first attention submodule into the first attention submodule, the first attention submodule performs a first process on the input features to obtain the output features of the first attention submodule. Here, the first process may include, but is not limited to, layer normalization, convolution, and dimensional transformation operations, and is not limited to this first process, and the first process is related to the network structure of the first attention submodule. The network structure of the first attention submodule can be set arbitrarily.

[0155] Exemplary, after inputting the input features of the second attention submodule into the second attention submodule, the second attention submodule performs a second process on the input features to obtain the output features of the second attention submodule. Here, the second process may include, but is not limited to, at least one of the following: depth-separable convolution, layer normalization, multilayer sensing, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. The second process is related to the network structure of the second attention submodule. The network structure of the second attention submodule can be set arbitrarily.

[0156] For example, the second process may include at least one of depth-separable convolution, layer normalization, or multilayer sensing. Alternatively, the second process may include at least one of layer normalization, linear operation, or activation operation. Alternatively, the second process may include at least one of downsampling, residual convolution, upsampling, or activation operation.

[0157] Example 8: This example further describes the first attention submodule relating to the cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule and to obtain the output features of the first attention submodule. For example, layer normalization is performed on the input features of the first attention submodule to obtain the layer-normalized features, a three-path convolution operation is performed on the layer-normalized features to obtain a query vector, a key vector, and a value vector, a dimensional transformation operation is performed on each of the query vector, key vector, and value vector to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector, attention weights are determined based on the dimensionally transformed query vector and dimensionally transformed key vector, modified features corresponding to the input features are determined based on the attention weights and the dimensionally transformed value vector, and the output features of the first attention submodule are determined based on the input features and modified features.

[0158] Figure 8A is a schematic diagram of the structure of the first attention submodule. This is merely an example of the first attention submodule and does not limit the structure to this particular submodule.

[0159] The following explanation uses the first attention submodule in Figure 8A as an example. The first attention submodule may include a layer normalization layer, which performs layer normalization on the input features (or input images) of the first attention submodule to obtain the layer-normalized features. Here, layer normalization is a neural network regularization technique used to normalize the input features of the neural network in each layer to improve training and generalization performance. Here, the layer normalization operation may include calculating the mean and variance, standardization, scaling, and translation, so that the output of each hidden unit is within a relatively small range and each hidden unit has a similar distribution across the entire dataset.

[0160] After obtaining the layer-normalized features, a convolution operation can be performed on the layer-normalized features using three paths to obtain a query vector Q, a key vector K, and a value vector V. For example, a convolution operation can be performed on the layer-normalized features using two convolutional layers to obtain the query vector Q. The first convolutional layer may be a 1x1, 3x3, or 5x5 convolutional layer, and is not limited to these; for example, a 1x1 convolutional layer may be selected. The second convolutional layer may be a 3x3, 5x5, or 7x7 convolutional layer, and is not limited to these; for example, a 3x3 convolutional layer may be selected. The key vector K may be obtained by performing a convolution operation on the layer-normalized features using two convolutional layers, and the first convolutional layer may be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, for example, a 1x1 convolutional layer may be selected. The second convolutional layer may be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, for example, a 3x3 convolutional layer may be selected. The value vector V may be obtained by performing a convolution operation on the layer-normalized features using two convolutional layers, and the first convolutional layer may be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, for example, a 1x1 convolutional layer may be selected. The second convolutional layer may be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer; for example, a 3x3 convolutional layer may be selected.

[0161] A dimensional transformation operation (i.e., an R operation) is performed on each of the query vector Q, key vector K, and value vector V to obtain the transformed query vector Q, transformed key vector K, and transformed value vector V. The purpose of the dimensional transformation operation is to transform the transformed tensor into a dimensional form that the multi-head self-attention mechanism must satisfy. The multi-head self-attention mechanism is a technique widely used in natural language processing (NLP) tasks. This technique reconstructs the representation of a target word based on context by building relationships between context words using a self-attention mechanism. The multi-head attention mechanism is a combination of multiple sets of self-attention components and can learn multiple different types of contextual influences. By passing the outputs of multiple self-attentions through a parameter matrix to obtain a single new output, the multi-head attention mechanism can capture a wider range of correlation features and improve the expressive power of the model.

[0162] Attention weights are determined based on the dimensionally transformed query vector Q and the dimensionally transformed key vector K. For example, matrix multiplication is performed on the dimensionally transformed query vector Q and the dimensionally transformed key vector K, and the result of the matrix multiplication is passed through the softmax function to obtain the attention weights. The processing method of this softmax is not limited.

[0163] Based on the attention weights and the value vector V after dimensional transformation, a modified feature corresponding to the input feature is determined. For example, matrix multiplication is performed on the value vector V after dimensional transformation and the attention weights, a dimensional transformation operation (i.e., R operation) is performed on the result of the matrix multiplication, a convolution operation is performed on the feature after the dimensional transformation operation to obtain a modified feature corresponding to the input feature, the convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and for example, a 1x1 convolution layer may be selected.

[0164] Based on the input and modification features of the first attention submodule, the output features of the first attention submodule are determined. For example, matrix addition is performed on the input and modification features to obtain the output features of the first attention submodule.

[0165] At this point, the processing of the first attention submodule is complete, and the output characteristics of the first attention submodule are obtained.

[0166] Example 9: This example further describes the first attention submodule relating to the cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule and to obtain the output features of the first attention submodule. For example, layer normalization is performed on the input features of the first attention submodule to obtain the layer-normalized features, window division is performed on the layer-normalized features to obtain multiple small-size features, a convolution operation with three paths is performed on each small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature, a query vector is obtained by combining the small-size query features corresponding to the multiple small-size features, a key vector is obtained by combining the small-size key features corresponding to the multiple small-size features, and a value vector is obtained by combining the small-size value features corresponding to the multiple small-size features. A dimensional transformation operation is performed on the query vector, key vector, and value vector to obtain the dimensionally transformed query vector, dimensionally transformed key vector, and dimensionally transformed value vector. Attention weights are determined based on the dimensionally transformed query vector and dimensionally transformed key vector. Modification features are determined based on the attention weights and the dimensionally transformed value vector. Output features of the first attention submodule are determined based on these input features and modification features.

[0167] For example, a windowing layer may be added after the layer normalization layer, based on the first attention submodule in Figure 8A. The structure of this first attention submodule is not limited; for example, Figure 8A may be used as an example of the structure of the first attention submodule.

[0168] For example, the first attention submodule may include a layer normalization layer, which may perform layer normalization on the input features (or input image) of the first attention submodule to obtain the layer normalized features.

[0169] A windowing layer can be used to perform windowing on the layer-normalized features, thereby obtaining multiple small-sized features. For example, the layer-normalized features can be divided into multiple small-sized features, where each small-sized feature has a width of wi and a height of hi, and the widths of different small-sized features may be the same or different, and the heights of different small-sized features may be the same or different. By dividing the features into multiple small-sized features and performing subsequent calculations, complexity can be reduced.

[0170] For each small-size feature, a convolution operation is performed on the small-size feature using three paths to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature. For example, the small-size feature may be convolved using two convolutional layers to obtain the small-size query feature. The first convolutional layer may be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, and for example, a 1x1 convolutional layer may be selected. The second convolutional layer may be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, and for example, a 3x3 convolutional layer may be selected. For example, a small-size key feature may be obtained by performing a convolution operation on the small-size feature using two convolutional layers, and the first convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, for example, a 1×1 convolutional layer may be selected. The second convolutional layer may be a 3×3 convolutional layer, a 5×5 convolutional layer, or a 7×7 convolutional layer, for example, a 3×3 convolutional layer may be selected. For example, a small-size value feature may be obtained by performing a convolution operation on the small-size feature using two convolutional layers, and the first convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, for example, a 1×1 convolutional layer may be selected. The second convolutional layer may be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer; for example, a 3x3 convolutional layer may be selected.

[0171] A query vector Q may be obtained by combining small query features corresponding to all small features, or, without limiting the feature combining process, a key vector K may be obtained by combining small key features corresponding to all small features, or a value vector V may be obtained by combining small value features corresponding to all small features.

[0172] A dimensional transformation operation (i.e., an R operation) may be performed on the query vector Q, key vector K, and value vector V to obtain the dimensionally transformed query vector Q, the dimensionally transformed key vector K, and the dimensionally transformed value vector V.

[0173] Attention weights may be determined based on the dimensionally transformed query vector Q and the dimensionally transformed key vector K. For example, matrix multiplication may be performed on the dimensionally transformed query vector Q and the dimensionally transformed key vector K, and the results of the matrix multiplication may be passed through a softmax function to obtain the attention weights. The processing method of this softmax function is not limited.

[0174] Modified features corresponding to input features may be determined based on attention weights and the value vector V after dimensional transformation. For example, matrix multiplication may be performed on the value vector V after dimensional transformation and the attention weights, a dimensional transformation operation (i.e., an R operation) may be performed on the result of the matrix multiplication, a convolution operation may be performed on the features after the dimensional transformation operation to obtain modified features corresponding to the input features, and the convolutional layer may be a 1×1 convolutional layer, a 3×3 convolutional layer, or a 5×5 convolutional layer, for example, a 1×1 convolutional layer may be selected.

[0175] The output features of the first attention submodule may be determined based on the input features and modification features of the first attention submodule. For example, matrix addition may be performed on the input features and modification features to obtain the output features of the first attention submodule.

[0176] At this point, the processing of the first attention submodule is complete, and the output characteristics of the first attention submodule are obtained.

[0177] Example 10: This example further describes the first attention submodule relating to the cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule and to obtain the output features of the first attention submodule. For example, layer normalization is performed on the input features of the first attention submodule to obtain the layer-normalized features, window division is performed on the layer-normalized features to obtain multiple small-size features, and for each small-size feature, a convolution operation with three paths is performed on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature. A query vector is obtained by combining the small-size query features corresponding to the multiple small-size features, a key vector is obtained by combining the small-size key features corresponding to the multiple small-size features, and a value vector is obtained by combining the small-size value features corresponding to the multiple small-size features. The attention weights are determined based on the query vector and the key vector, and the modification features corresponding to the input features are determined based on the attention weights and the value vector. The output features of the first attention submodule are determined based on the input features and the modification features.

[0178] Figure 8B is a schematic diagram of the structure of the first attention submodule, and this is merely an example of the first attention submodule. Without limiting the structure of this first attention submodule, the following explanation will use the first attention submodule in Figure 8B as an example.

[0179] For example, the first attention submodule may include a layer normalization layer, which may perform layer normalization on the input features (or input image) of the first attention submodule to obtain the layer normalized features.

[0180] The first attention submodule may include a windowing layer, which may perform windowing on the layer-normalized feature to obtain multiple small-sized features. For example, the windowing layer may divide the layer-normalized feature into multiple small-sized features, where the width of each small-sized feature is wi and the height of each small-sized feature is hi, and the widths of different small-sized features may be the same or different, and the heights of different small-sized features may be the same or different.

[0181] For each small-size feature, a convolution operation is performed on the small-size feature using three paths to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature. For example, a convolution operation is performed on the small-size feature using a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer) to obtain a small-size query feature. A convolution operation is performed on the small-size feature using a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer) to obtain a small-size key feature. A convolution operation is performed on the small-size feature using a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer) to obtain a small-size value feature. The small-size query features corresponding to all small-size features are joined to obtain a query vector Q, the small-size key features corresponding to all small-size features are joined to obtain a key vector K, and the small-size value features corresponding to all small-size features are joined to obtain a value vector V.

[0182] Attention weights are determined based on the query vector Q and the key vector K. For example, matrix multiplication is performed on the query vector Q and the key vector K, and the result of the matrix multiplication is passed through the softmax function to obtain the attention weights.

[0183] In one example, a modified feature corresponding to the input feature may be directly obtained based on the attention weights and value vector V obtained. For example, matrix multiplication may be performed on the attention weights and value vector V, and the result after matrix multiplication may be used as the modified feature corresponding to the input feature, i.e., the second layer normalization layer in Figure 8B may be removed.

[0184] Furthermore, the output features of the first attention submodule may be determined based on the input features and modification features of the first attention submodule. For example, matrix addition may be performed on the input features and modification features to obtain the output features of the first attention submodule.

[0185] At this point, the processing of the first attention submodule is complete, and the output characteristics of the first attention submodule are obtained.

[0186] Example 11: This example further describes the first attention submodule relating to the cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule and to obtain the output features of the first attention submodule. For example, layer normalization is performed on the input features of the first attention submodule to obtain the layer-normalized features, a three-path convolution operation is performed on the layer-normalized features to obtain a query vector, a key vector, and a value vector, attention weights are determined based on the query vector and the key vector, modified features corresponding to the input features are determined based on the attention weights and the value vector, and the output features of the first attention submodule are determined based on the input features and modified features of the first attention submodule.

[0187] For example, the windowing layer after the layer normalization layer may be removed based on the first attention submodule in Figure 8B. The structure of this first attention submodule is not limited; for example, Figure 8B may be used as an example of the structure of the first attention submodule.

[0188] For example, the first attention submodule may include a layer normalization layer, which may perform layer normalization on the input features (or input image) of the first attention submodule to obtain the layer normalized features.

[0189] The first attention submodule may include a convolutional layer, and a convolution operation may be performed on the layer-normalized features in three paths to obtain a query vector Q, a key vector K, and a value vector V. For example, a convolutional operation may be performed on the layer-normalized features using a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer) to obtain the query vector Q. A convolutional operation may be performed on the layer-normalized features using a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer) to obtain the key vector K. A convolutional operation may be performed on the layer-normalized features using a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer) to obtain the value vector V.

[0190] Attention weights are determined based on the query vector Q and the key vector K. For example, matrix multiplication is performed on the query vector Q and the key vector K, and the result of the matrix multiplication is passed through the softmax function to obtain the attention weights.

[0191] In one example, a modified feature corresponding to the input feature may be directly obtained based on the attention weights and value vector V obtained. For example, matrix multiplication may be performed on the attention weights and value vector V, and the result after matrix multiplication may be used as the modified feature corresponding to the input feature, i.e., the second layer normalization layer in Figure 8B may be removed.

[0192] Furthermore, the output features of the first attention submodule may be determined based on the input features and modification features of the first attention submodule. For example, matrix addition may be performed on the input features and modification features to obtain the output features of the first attention submodule.

[0193] At this point, the processing of the first attention submodule is complete, and the output characteristics of the first attention submodule are obtained.

[0194] Examples 8 to 11 show several examples of the first attention submodule, but the present invention does not limit the structure of the first attention submodule; it is sufficient that the first processing is performed on the input features to obtain the output features.

[0195] Example 12: This example further describes a second attention submodule relating to a cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a depth-separable convolution operation is performed on the input features of the second attention submodule to obtain a convolutional feature (i.e., a feature after a depth-separable convolution operation), layer normalization is performed on the convolutional feature to obtain a layer-normalized feature, a multilayer sensing operation is performed on the layer-normalized feature to obtain a multilayer-sensed feature, and the output features of the second attention submodule are determined based on the input features and the multilayer-sensed feature.

[0196] Figure 8C is a schematic diagram of the second attention submodule, and this is merely an example of a second attention submodule. Without limiting the scope of the second attention submodule's structure, the following explanation will use the second attention submodule in Figure 8C as an example.

[0197] For example, the second attention submodule may include a depth-separable convolutional layer, which may be a 1x1, 3x3, 5x5, or 7x7 convolutional layer, and is not limited thereto; for example, a 7x7 convolutional layer may be selected, and a depth-separable convolutional operation may be performed on the input features of the second attention submodule using the depth-separable convolutional layer to obtain the convolutional features (i.e., features after the depth-separable convolutional operation), and is not limited thereto.

[0198] The second attention submodule may include a layer normalization layer, which may perform layer normalization on the convolutional features to obtain the layer normalized features. The layer normalization operation may include calculating the mean and variance, standardization, scaling, and translation, so that the output of each hidden unit is within a relatively small range and each hidden unit has a similar distribution in the dataset.

[0199] The second attention submodule may include a multilayer perceptron, and a multilayer sensing operation may be performed on the layer-normalized features by the multilayer perceptron to obtain the multilayer-sensed features. Figure 8D is a schematic diagram of the structure of a multilayer perceptron, which may include a linear layer, an activation layer, a random deactivation layer, a linear layer, and a random deactivation layer. Based on this, the layer-normalized features are obtained by sequentially passing through a linear layer, an activation layer, a random deactivation layer, a linear layer, and a random deactivation layer to obtain the multilayer-sensed features. Here, the activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and the type of this activation layer is not limited. The random deactivation layer may be a DropOut layer, and the type of this random deactivation layer is not limited. Note that Figure 8D is merely an example of a multilayer perceptron, and the structure of this multilayer perceptron is not limited.

[0200] Based on the input features and multi-layered sensing features of the second attention submodule, the output features of the second attention submodule are determined. For example, matrix addition is performed on the input features and multi-layered sensing features to obtain the output features.

[0201] At this point, the processing of the second attention submodule is complete, and the output characteristics of the second attention submodule are obtained.

[0202] Example 13: This example further describes the second attention submodule relating to the cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule and obtain the output features of the second attention submodule. For example, layer normalization may be performed on the input features of the second attention submodule to obtain the layer-normalized features, a first linear operation may be performed on the layer-normalized features to obtain the features after the first linear operation, an activation operation may be performed on the features after the first linear operation to obtain the activated features, a second linear operation may be performed on the activated features to obtain the features after the second linear operation, and the output features of the second attention submodule may be determined based on the input features and the features after the second linear operation of the second attention submodule.

[0203] Figure 8E is a schematic diagram of the second attention submodule, and this is merely an example of a second attention submodule. Without limiting ourselves to the structure of this second attention submodule, we will explain it below using the second attention submodule in Figure 8E as an example.

[0204] For example, the second attention submodule may include a layer normalization layer, which may perform layer normalization on the input features of the second attention submodule to obtain the layer-normalized features. The second attention submodule may also include a linear layer 1, which may perform a first linear operation on the layer-normalized features to obtain the features after the first linear operation.

[0205] The second attention submodule may include an activation layer, which may perform an activation operation on the features after the first linear operation to obtain the activated features. Here, the activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and the type of this activation layer is not limited; for example, a relu layer may be selected as the activation layer. The second attention submodule may also include a linear layer 2, which may perform a second linear operation on the features after activation to obtain the features after the second linear operation.

[0206] Based on the input features and the features after the second linear operation of the second attention submodule, the output features of the second attention submodule are determined, and for example, matrix addition is performed on the input features and the features after the second linear operation to obtain the output features.

[0207] At this point, the processing of the second attention submodule is complete, and the output characteristics of the second attention submodule are obtained.

[0208] Example 14: This example further describes a second attention submodule relating to a cascade hybrid attention module, based on Examples 6 and 7. In Examples 6 and 7, the cascade hybrid attention module includes a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a downsampling operation is performed on the input features of the second attention submodule to obtain the downsampled features, a residual convolution operation is performed on the downsampled features to obtain the residual convolution features, an upsampling operation is performed on these residual convolution features to obtain the upsampled features, an activation operation is performed on the upsampled features to obtain the activated features, and a residual convolution operation is performed on the input features to obtain the residual convolution features. The output features of the second attention submodule are determined based on the input features, the activated features, and the convolution features.

[0209] Figure 8F is a schematic diagram of the second attention submodule, and this is merely an example of a second attention submodule. Without limiting ourselves to the structure of this second attention submodule, we will explain it below using the second attention submodule in Figure 8F as an example.

[0210] For example, the second attention submodule may include a downsampling layer, which may be a convolutional layer with a kernel size of 3x3 and a stride of 2, or a convolutional layer with a kernel size of 3x3 and a stride of 4, or a convolutional layer with a kernel size of 4x4 and a stride of 2, or a convolutional layer with a kernel size of 5x5 and a stride of 2, or a convolutional layer with a kernel size of 5x5 and a stride of 4, or an inverse pixel shuffle. The structure of this downsampling layer is not limited; for example, a convolutional layer with a kernel size of 3x3 and a stride of 2 may be selected as the downsampling layer. Based on this, the downsampling layer can perform a downsampling operation on the input features of the second attention submodule to obtain the features after downsampling, and the implementation of this downsampling operation is not limited.

[0211] The second attention submodule may include M residual convolution layers, where M is a positive integer, such as 1, 2, 3, 4, etc., and the M residual convolution layers may perform residual convolution operations on the downsampled features to obtain the residual convolutional features. For example, the first residual convolution layer performs residual convolution on the downsampled features, the second residual convolution layer performs residual convolution on the obtained result, ... and the last residual convolution layer outputs the residual convolutional features.

[0212] Figure 8G is a schematic diagram of the residual convolutional layer structure, which may include convolutional layer 1, an activation layer, and convolutional layer 2. Convolutional layer 1 may perform a first convolution on the input features of the residual convolutional layer to obtain the features after the first convolution. The activation layer may perform an activation on the features after the first convolution to obtain the features after the activation. Convolutional layer 2 may perform a second convolution on the features after the activation to obtain the features after the second convolution. Then, matrix addition is performed on the input features of the residual convolutional layer and the features after the second convolution to obtain the output features of the residual convolutional layer.

[0213] Convolutional layer 1 may be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, and is not limited to these; for example, a 3x3 convolutional layer is selected. Convolutional layer 2 may be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, and is not limited to these; for example, a 3x3 convolutional layer is selected. The activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a geleu layer, and is not limited to these; for example, a relu layer or a sigmoid layer is selected.

[0214] Figure 8H is a schematic diagram of another structure of the residual convolutional layer, which may include a convolutional layer and an activation layer. The convolutional layer may perform a convolution operation on the input features of the residual convolutional layer to obtain the features after the convolution operation, and the activation layer may perform an activation operation on the features after the convolution operation to obtain the features after the activation operation. Then, matrix addition is performed on the input features and the features after the activation operation of the residual convolutional layer to obtain the output features of the residual convolutional layer.

[0215] The convolutional layer may be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, and is not limited to these; for example, a 3x3 convolutional layer may be selected. The activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and is not limited to these; for example, a relu layer or a sigmoid layer may be selected.

[0216] The second attention submodule may include an upsampling layer, which may be a deconvolutional layer with a kernel size of 3×3 and a stride of 2, or a deconvolutional layer with a kernel size of 3×3 and a stride of 4, or a deconvolutional layer with a kernel size of 4×4 and a stride of 2, or a deconvolutional layer with a kernel size of 4×4 and a stride of 4, or a deconvolutional layer with a kernel size of 5×5 and a stride of 2, or a pixel reconstruction layer. The structure of this upsampling layer is not limited; for example, a deconvolutional layer with a kernel size of 3×3 and a stride of 2 may be selected as the upsampling layer. Based on this, the upsampling layer can perform an upsampling operation on the residual convolutional feature (i.e., the feature obtained by performing a residual convolution operation with M residual convolutional layers) to obtain the upsampled feature, and the implementation of this upsampling operation is not limited.

[0217] The second attention submodule may include an activation layer, which may be a relu layer, leaky relu layer, sigmoid layer, tanh layer, or gelu layer. The structure of this activation layer is not limited; for example, a relu layer may be selected. Based on this, an activation operation can be performed on the upsampled features using the activation layer to obtain the activated features.

[0218] The second attention submodule may include N residual convolution layers, where N is a positive integer, such as 1, 2, 3, 4, etc., and N and M may be the same or different. The N residual convolution layers may perform a residual convolution operation on the input features of the second attention submodule to obtain the post-residual convolution features. For example, the first residual convolution layer performs a residual convolution operation on the input features of the second attention submodule, the second residual convolution layer performs a residual convolution operation on the obtained result, ... and the last residual convolution layer outputs the post-residual convolution features.

[0219] Figure 8G is a schematic diagram of the residual convolutional layer structure, which may include convolutional layer 1, an activation layer, and convolutional layer 2. Figure 8H is another schematic diagram of the residual convolutional layer structure, which may include a convolutional layer and an activation layer. A residual convolution operation may be performed using any of the above residual convolutional layers, but this will not be explained here.

[0220] As shown in Figure 8F, after obtaining the convolutional features after residual convolution (i.e., the features output after performing a residual convolution operation by N residual convolution layers) and the features after activation (i.e., the features output after performing an activation operation by an activation layer), the output features of the second attention submodule can be determined based on the input features of the second attention submodule, the features after activation, and the convolutional features after residual convolution. For example, matrix multiplication may be performed on the features after activation and the convolutional features after residual convolution, and matrix addition may be performed on the result of the multiplication and the input features of the second attention submodule to obtain the output features of the second attention submodule.

[0221] At this point, the processing of the second attention submodule is complete, and the output characteristics of the second attention submodule are obtained.

[0222] In Examples 12 to 14, some examples of the second attention sub-module were shown. However, the present invention is not limited to the structure of the second attention sub-module, and it is only necessary that the second process can be performed on the input features to obtain the output features.

[0223] Example 15: The implementation form of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the description of overlapping content will be omitted. Hereinafter, the involved synthesis conversion network will be described in detail.

[0224] In Example 1, Example 2, Example 3 and Example 4, the synthesis conversion network is involved in all of them. The synthesis conversion network may include at least an attention module, and the attention module may be a first attention sub-module. Of course, in addition to the first attention sub-module, the synthesis conversion network may further include other network layers. This example is not limited to the structure of this synthesis conversion network, and it is only necessary that the synthesis conversion network includes the first attention sub-module. For example, the synthesis conversion network may further include at least one deconvolution layer, and the first attention sub-module may be located after any deconvolution layer. Or, the first attention sub-module may be located after each deconvolution layer. Or, the first attention sub-module may be located after some deconvolution layers (for example, K deconvolution layers out of all deconvolution layers, where K is greater than 1 and K is less than the total number of deconvolution layers).

[0225] Exemplary, the first attention submodule is used to perform a first processing on the input features of the first attention submodule and to obtain the output features of the first attention submodule. In one possible embodiment, the structure of the first attention submodule may refer to Example 8; in another possible embodiment, the structure of the first attention submodule may refer to Example 9; in yet another possible embodiment, the structure of the first attention submodule may refer to Example 10; and in yet another possible embodiment, the structure of the first attention submodule may refer to Example 11. Of course, the above are just some examples of the first attention submodule and do not limit the structure of this first attention submodule.

[0226] Example 16: The embodiment of this example is the same as that of Example 1, Example 2, Example 3, or Example 4, and the explanation of redundant content will be omitted. The synthesis and transformation network involved will be described in detail below.

[0227] In Examples 1, 2, 3, and 4, a composite transformation network is involved. The composite transformation network may include at least an attention module, and the attention module may be a second attention submodule. Of course, in addition to the second attention submodule, the composite transformation network may further include other network layers, and this embodiment does not limit the structure of this composite transformation network, as long as the composite transformation network includes a second attention submodule. For example, the composite transformation network may further include at least one deconvolution layer, and the second attention submodule may be located after any of the deconvolution layers. Alternatively, the second attention submodule may be located after each deconvolution layer. Alternatively, the second attention submodule may be located after some of the deconvolution layers (for example, K deconvolution layers out of all deconvolution layers, where K is greater than 1 and K is less than the total number of deconvolution layers).

[0228] Exemplary, a second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. In one possible embodiment, the structure of the second attention submodule may refer to Example 12; in another possible embodiment, the structure of the second attention submodule may refer to Example 13; and in yet another possible embodiment, the structure of the second attention submodule may refer to Example 14. Of course, the above are merely some examples of a second attention submodule and do not limit the structure of this second attention submodule.

[0229] As can be seen from the above technical proposals, the embodiments of the present invention propose a synthetic transformation network of attention mechanisms for neural network-based coding and decoding techniques. This synthetic transformation network includes an attention module, which is a cascaded hybrid attention module. By realizing the synthetic transformation network with the cascaded hybrid attention module, the quality of the synthesized image is guaranteed, the complexity of the network and computation are effectively reduced, and decoding performance is improved. Replacing the residual nonlocal attention module with a hybrid attention module (cascaded hybrid attention module) significantly reduces the computational complexity of the decoder. For example, the second attention submodule in the hybrid attention module occupies only 21K floating-point sum-of-accumulate operations per pixel, and the complexity of the first attention submodule is also far lower than that of the residual nonlocal attention module. Furthermore, the second attention submodule plays the role of the attention mechanism in the residual nonlocal attention module and introduces a high-performance transformer structure, resulting in superior performance. The 7x7 convolutional operation in the second attention submodule introduces nonlocal information in the residual nonlocal attention module, and the random deactivation layer in the second attention submodule improves the robustness of the network. Based on the above functions, the cascaded hybrid attention module reduces computational complexity without causing excessive performance loss.

[0230] Exemplary examples, each of the above embodiments may be implemented individually or in combination. For example, each of the embodiments from Embodiments 1 to 16 may be implemented individually, or at least two of the embodiments from Embodiments 1 to 16 may be implemented in combination.

[0231] For example, in each of the above embodiments, the contents of the encoding side may be applied to the decoding side, that is, processed by the decoding side in the same manner, and the contents of the decoding side may be applied to the encoding side, that is, processed by the encoding side in the same manner.

[0232] Based on the same concept as described above, embodiments of the present invention further provide a decoding device which is applied to the decoding side and includes a memory configured to store video data and a decoder configured to carry out the decoding method in embodiments 1 to 16, i.e., the decoding side processing process.

[0233] For example, in one possible embodiment, the decoder is The steps include: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; The system is configured to perform the steps of inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series.

[0234] Based on the same concept as described above, embodiments of the present invention further provide an encoding device which is applied to the encoding side and includes a memory configured to store video data and an encoder configured to perform the encoding method of Embodiments 1 to 16, i.e., the encoding side processing process.

[0235] For example, in one possible embodiment, the encoder is The steps include: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; The system is configured to perform the steps of inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series.

[0236] Based on the same concept as described above, a schematic diagram of the hardware architecture of a decoding device (also called a video decoder) provided by an embodiment of the present invention may be shown in detail in Figure 9A. The decoding device includes a processor 901 and a machine-readable storage medium 902, the machine-readable storage medium 902 stores machine-executable instructions that can be executed by the processor 901, and the processor 901 is used to execute the machine-executable instructions and carry out the decoding methods of embodiments 1 to 16 of the present invention.

[0237] The machine-readable storage medium 902 may be an electronic, magnetic, optical, or other physical storage device capable of storing or remembering information such as executable instructions and data. For example, the machine-readable storage medium may be random access memory (RAM), volatile memory, non-volatile memory, flash memory, storage drives (e.g., hard disk drives), solid-state drives, any type of storage disk (e.g., optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0238] Based on the same concept as described above, a schematic diagram of the hardware architecture of an encoding device (also called a video encoder) provided by an embodiment of the present invention may be shown in detail in Figure 9B. The encoding device includes a processor 911 and a machine-readable storage medium 912, the machine-readable storage medium 912 storing machine-executable instructions that can be executed by the processor 911, and the processor 911 is used to execute the machine-executable instructions and carry out the encoding methods of embodiments 1 to 16 of the present invention.

[0239] The machine-readable storage medium 912 may be an electronic, magnetic, optical, or other physical storage device capable of storing or remembering information such as executable instructions and data. For example, the machine-readable storage medium may be RAM, volatile memory, non-volatile memory, flash memory, storage drives (e.g., hard disk drives), solid-state drives, any type of storage disk (e.g., optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0240] Based on the same concept as described above, embodiments of the present invention provide an electronic device. The electronic device includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions and carry out the decoding or encoding methods of embodiments 1 to 16 of the present invention.

[0241] Based on the same concept as the above method, embodiments of the present invention further provide a machine-readable storage medium storing several computer instructions, and when the computer instructions are executed by a processor, the decoding method or encoding method of the above Embodiments 1 to 16 of the present invention, for example, the decoding method or encoding method in each of the above embodiments, is implemented. For example, the machine-readable storage medium may be a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0242] Based on the same concept as the above method, embodiments of the present invention further provide a computer application, and when the computer application is executed by a processor, the decoding method or encoding method disclosed in the above examples of the present invention may be implemented.

[0243] Based on the same concept as the above method, embodiments of the present invention further provide a decoding device applicable to the decoding side, and the decoding device decodes one bitstream corresponding to the current image block to obtain a coefficient hyperparameter feature corresponding to the current image block, determines a probability distribution parameter based on the coefficient hyperparameter feature, and decodes another bitstream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block, and a decoding module for inputs the reconstruction feature into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block, and a processing module for the synthesis transformation network includes at least an attention module, and the attention module is a cascade hybrid attention module, the cascade hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.

[0244] Exemplary, if the attention module is a cascade hybrid attention module, the processing module is further used to perform a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule, and to perform a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule, wherein the output features of the first attention submodule are the input features of the second attention submodule, the first processing is at least one of layer normalization, convolution operation and dimensional transformation operation, and the second processing is at least one of depth separable convolution operation, layer normalization, multilayer sensing operation, linear operation, activation operation, downsampling operation, residual convolution operation and upsampling operation.

[0245] Exemplary, the processing module is used to perform a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule, to perform layer normalization on the input features of the first attention submodule to obtain the layer-normalized features, to perform a three-path convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector, to perform a dimensional transformation operation on each of the query vector, the key vector and the value vector to obtain a dimensionally transformed query vector, a dimensionally transformed key vector and a dimensionally transformed value vector, to determine attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector, to determine a modified feature corresponding to the input features based on the attention weights and the dimensionally transformed value vector, and to determine the output features of the first attention submodule based on the input features and the modified features.

[0246] For example, when the processing module performs a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule, specifically, it performs layer normalization on the input features of the first attention submodule to obtain the layer normalized features, performs window partitioning on the layer normalized features to obtain a plurality of small-size features, performs a three-path convolution operation on each small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature, combines the small-size query features corresponding to the plurality of small-size features to obtain a query vector, and This is used to determine the output features of the first attention submodule by combining small key features corresponding to multiple small features to obtain a key vector, combining small value features corresponding to the multiple small features to obtain a value vector, performing a dimensional transformation operation on the query vector, the key vector, and the value vector to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector, determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector, determining modification features corresponding to the input features based on the attention weights and the dimensionally transformed value vector, and determining the output features of the first attention submodule based on the input features and the modification features.

[0247] Exemplary, the processing module is used to perform a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule. Specifically, it performs layer normalization on the input features of the first attention submodule to obtain the layer-normalized features, performs a three-path convolution operation on the layer-normalized features to obtain a query vector, a key vector, and a value vector, determines attention weights based on the query vector and the key vector, determines modified features corresponding to the input features based on the attention weights and the value vector, and determines the output features of the first attention submodule based on the input features and the modified features.

[0248] Exemplary, the processing module performs a first processing on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule. Specifically, it performs layer normalization on the input features of the first attention submodule to obtain the layer-normalized features, performs window partitioning on the layer-normalized features to obtain a plurality of small-size features, performs a three-path convolution operation on each small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature, combines the small-size query features corresponding to the plurality of small-size features to obtain a query vector, combines the small-size key features corresponding to the plurality of small-size features to obtain a key vector, combines the small-size value features corresponding to the plurality of small-size features to obtain a value vector, determines attention weights based on the query vector and the key vector, determines modification features corresponding to the input features based on the attention weights and the value vector, and determines the output features of the first attention submodule based on the input features and the modification features.

[0249] Exemplary, the processing module performs a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule. Specifically, it performs layer normalization on the input features of the second attention submodule to obtain the layer-normalized features, performs a first linear operation on the layer-normalized features to obtain the features after the first linear operation, performs an activation operation on the features after the first linear operation to obtain the activated features, performs a second linear operation on the activated features to obtain the features after the second linear operation, and is used to determine the output features of the second attention submodule based on the input features and the features after the second linear operation.

[0250] Exemplary, the processing module performs a second processing on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule. Specifically, it performs a downsampling operation on the input features of the second attention submodule to obtain the downsampled features, a residual convolution operation on the downsampled features to obtain the residual convolution features, an upsampling operation on the residual convolution features to obtain the upsampled features, an activation operation on the upsampled features to obtain the activated features, a residual convolution operation on the input features to obtain the convolution features after residual convolution, and is used to determine the output features of the second attention submodule based on the input features, the activated features, and the convolution features.

[0251] Exemplary, the composite transformation network further includes at least one deconvolutional layer, the attention module located after one of these deconvolutional layers.

[0252] Exemplary, the composite transformation network further comprises a residual layer, a first deconvolution layer, a first crop layer, a first residual activation layer, a second deconvolution layer, a second crop layer, a second residual activation layer, a third deconvolution layer, a third crop layer, a third residual activation layer, a fourth deconvolution layer, and a fourth crop layer, wherein the composite transformation network includes at least one attention module, one of which is located after the first deconvolution layer.

[0253] Based on the same concept as described above, embodiments of the present invention further provide an encoding device applied to the encoding side, the device being A decoding module for decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, determining probability distribution parameters based on the coefficient hyperparameter features, decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block, The processing module includes inputting the aforementioned reconstruction features into a composite transformation network and obtaining a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first and second attention submodules being two submodules in series.

[0254] Those skilled in the art will understand that embodiments of the present invention may be provided as methods, systems, or computer program products. The present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Embodiments of the present invention may take the form of computer program products implemented on one or more computer-compatible storage media (including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-compatible program code. The above are merely embodiments of the present invention and do not limit the present invention.

[0255] To those skilled in the art, the present invention is subject to various modifications and changes. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A decoding method applied to the decoding side, The steps include: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first attention submodule and the second attention submodule being two submodules in series. A decoding method characterized by the following:

2. If the attention module is a cascade hybrid attention module, further, A step of performing a first process on the input features of the first attention submodule via the first attention submodule to obtain the output features of the first attention submodule, wherein the output features of the first attention submodule are the input features of the second attention submodule, and the first process is at least one of layer normalization, convolution, and dimensional transformation. The process includes the step of performing a second process on the input features of the second attention submodule via the second attention submodule to obtain the output features of the second attention submodule, wherein the second process is at least one of a depth separable convolution operation, layer normalization, multilayer sensing operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. The method according to feature 1.

3. The step of performing a first processing on the input characteristics of the first attention submodule via the first attention submodule to obtain the output characteristics of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include performing a convolution operation through three paths on the layer-normalized features to obtain a query vector, a key vector, and a value vector, The steps include performing a dimensional transformation operation on the query vector, the key vector, and the value vector to obtain the dimensionally transformed query vector, the dimensionally transformed key vector, and the dimensionally transformed value vector, The steps include determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determining a modification feature corresponding to the input feature based on the attention weights and the dimensionally transformed value vector, The step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics, The method according to feature 2.

4. The step of performing a first processing on the input characteristics of the first attention submodule via the first attention submodule to obtain the output characteristics of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include: performing window division on the layer-normalized features to obtain multiple small-sized features, For each small-size feature, a convolution operation is performed on the small-size feature using three paths to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are combined to obtain a query vector; the small-size key features corresponding to the multiple small-size features are combined to obtain a key vector; and the small-size value features corresponding to the multiple small-size features are combined to obtain a value vector. The steps include performing a dimensional transformation operation on the query vector, the key vector, and the value vector to obtain the dimensionally transformed query vector, the dimensionally transformed key vector, and the dimensionally transformed value vector, The steps include determining attention weights based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determining a modification feature corresponding to the input feature based on the attention weights and the dimensionally transformed value vector, The step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics, The method according to feature 2.

5. The step of performing a first processing on the input characteristics of the first attention submodule via the first attention submodule to obtain the output characteristics of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include performing a convolution operation through three paths on the layer-normalized features to obtain a query vector, a key vector, and a value vector, The steps include determining attention weights based on the query vector and the key vector, and determining modification features corresponding to the input features based on the attention weights and the value vector, The step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics, The method according to feature 2.

6. The step of performing a first processing on the input characteristics of the first attention submodule via the first attention submodule to obtain the output characteristics of the first attention submodule is: The steps include performing layer normalization on the input features of the first attention submodule and obtaining the features after layer normalization, The steps include: performing window division on the layer-normalized features to obtain multiple small-sized features, For each small-size feature, a convolution operation is performed on the small-size feature using three paths to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are combined to obtain a query vector; the small-size key features corresponding to the multiple small-size features are combined to obtain a key vector; and the small-size value features corresponding to the multiple small-size features are combined to obtain a value vector. The steps include determining attention weights based on the query vector and the key vector, and determining modification features corresponding to the input features based on the attention weights and the value vector, The step of determining the output characteristics of the first attention submodule based on the input characteristics and the modification characteristics, The method according to feature 2.

7. The step of performing a second process on the input characteristics of the second attention submodule via the second attention submodule to obtain the output characteristics of the second attention submodule is: The steps include performing layer normalization on the input features of the second attention submodule and obtaining the features after layer normalization, The steps include performing a first linear operation on the features after layer normalization to obtain the features after the first linear operation, The steps include performing an activation operation on the features after the first linear operation to obtain the features after activation, The steps include performing a second linear operation on the activated features to obtain the features after the second linear operation, The step of determining the output features of the second attention submodule based on the input features and the features after the second linear operation, The method according to feature 2.

8. The step of performing a second process on the input characteristics of the second attention submodule via the second attention submodule to obtain the output characteristics of the second attention submodule is: The steps include: performing a downsampling operation on the input features of the second attention submodule to obtain the downsampled features; performing a residual convolution operation on the downsampled features to obtain the residual convolutional features; performing an upsampling operation on the residual convolutional features to obtain the upsampled features; and performing an activation operation on the upsampled features to obtain the activated features. The steps include performing a residual convolution operation on the input feature to obtain the convolutional feature after residual convolution, The process includes the step of determining the output features of the second attention submodule based on the input features, the activated features, and the convolutional features, The method according to feature 2.

9. The aforementioned composite transformation network further includes at least one deconvolutional layer, The attention module is located after one of the deconvolutional layers. The method according to any one of claims 1 to 8.

10. The aforementioned composite transformation network further includes a residual layer, a first inverse convolution layer, a first crop layer, a first residual activation layer, a second inverse convolution layer, a second crop layer, a second residual activation layer, a third inverse convolution layer, a third crop layer, a third residual activation layer, a fourth inverse convolution layer, and a fourth crop layer. The composite transformation network includes at least one attention module, one of which is located after the first deconvolutional layer. The method according to feature 9.

11. An encoding method applied to the encoding side, The steps include: decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; The steps include inputting the aforementioned reconstruction features into a composite transformation network to obtain a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first attention submodule and the second attention submodule being two submodules in series. An encoding method characterized by the following.

12. A decoding device applied to the decoding side, A decoding module for decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, determining probability distribution parameters based on the coefficient hyperparameter features, decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block, The processing module includes inputting the aforementioned reconstruction features into a composite transformation network and obtaining a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first attention submodule and the second attention submodule being two submodules in series. A decoding device characterized by the following features.

13. An encoding device applied to the encoding side, A decoding module for decoding one bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block, determining probability distribution parameters based on the coefficient hyperparameter features, decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block, The processing module includes inputting the aforementioned reconstruction features into a composite transformation network and obtaining a reconstructed image block corresponding to the current image block, The aforementioned synthesis conversion network includes at least an attention module, and the attention module is a cascaded hybrid attention module. The cascade hybrid attention module includes a first attention submodule and a second attention submodule, the first attention submodule and the second attention submodule being two submodules in series. An encoding device characterized by the following features.

14. A decoding device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor is used to execute machine-executable instructions and carry out the method according to any one of claims 1 to 10. A decoding device characterized by the following features.

15. An encoding device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor, The processor is used to execute machine-executable instructions and carry out the method according to claim 11. An encoding device characterized by the following features.

16. An electronic device comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor is used to execute machine-executable instructions and carry out the method according to any one of claims 1 to 11. An electronic device characterized by the following features.

17. A machine-readable storage medium storing a plurality of computer instructions, wherein when the computer instructions are executed by a processor, the method according to any one of claims 1 to 11 is performed. A machine-readable storage medium characterized by the following features.

18. When executed by a processor, the method according to any one of claims 1 to 11 is performed. A computer application characterized by the following features.

Images