Decoding and encoding method, apparatus, and device thereof
By introducing a synthetic transform network with attention mechanism into the encoding and decoding technology, using the cascading hybrid attention module, the problems of poor decoding performance and high complexity in the neural network encoding and decoding methods are solved, and more efficient image processing is achieved.
Patent Information
- Application Number
- PCT/CN2024/096404
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-06-01
- Filing Date
- 2024-05-30
- Publication Date
- 2024-12-05
AI Technical Summary
The neural network-based encoding and decoding methods have problems with poor decoding performance and high complexity.
A synthetic transformation network using an attention mechanism, including a cascading hybrid attention module, realizes a synthetic transformation network through a cascading hybrid attention module, reducing network complexity and computing complexity, while improving decoding performance.
While ensuring image quality, it reduces network complexity and computing complexity, and improves encoding and decoding performance.
Smart Images

Figure CN2024096404_05122024_PF_FP_ABST
Abstract
Description
A decoding and encoding method, device and equipment thereof Technical Field
[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding and encoding method, device and equipment thereof. Background Art
[0002] To save space, video images are encoded before transmission. Complete video encoding involves prediction, transformation, quantization, entropy coding, and filtering. Prediction can include intra-frame prediction and inter-frame prediction. Inter-frame prediction leverages temporal correlations in the video to predict the current pixel using pixels from adjacent coded images, effectively removing temporal redundancy. Intra-frame prediction leverages spatial correlations in the video to predict the current pixel using pixels from coded blocks in the current frame, effectively removing spatial redundancy.
[0003] With the rapid development of deep learning, it has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning is also gradually being applied to codecs, enabling the use of neural networks to encode and decode images. Although neural network-based codecs demonstrate significant performance potential, they still suffer from issues such as poor decoding performance and high complexity.
[0004] Summary of the Invention
[0005] The present application provides a decoding and encoding method, apparatus and device thereof, which can improve decoding performance and reduce complexity.
[0006] The present application provides a decoding method, which is applied to a decoding end. The method includes: decoding a code stream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, and decoding another code stream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; inputting the reconstruction features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
[0007] The present application provides an encoding method, which is applied to an encoding end. The method includes: decoding a code stream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, and decoding another code stream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; inputting the reconstruction features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
[0008] The present application provides a decoding device, which is applied to a decoding end. The device includes: a decoding module, which is used to decode a code stream corresponding to a current image block to obtain a coefficient hyperparameter feature corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block; a processing module, which is used to input the reconstruction feature into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
[0009] The present application provides an encoding device, which is applied to an encoding end, and the device includes: a decoding module, which is used to decode a code stream corresponding to a current image block to obtain a coefficient hyperparameter feature corresponding to the current image block; determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block; a processing module, which is used to input the reconstruction feature into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
[0010] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method.
[0011] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned encoding method.
[0012] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method or encoding method.
[0013] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the above-mentioned decoding method or encoding method.
[0014] The present application provides a computer application, which, when executed by a processor, implements the above-mentioned decoding method or encoding method.
[0015] It can be seen from the above technical solutions that in the embodiments of the present application, a synthetic transformation network with an attention mechanism is proposed for the neural network-based encoding and decoding technology. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module. When the synthetic transformation network is realized by the cascaded hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, and the decoding performance is improved. While reducing the complexity, the quality of the reconstructed image block is effectively guaranteed, and the encoding performance and decoding performance are improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] FIG1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;
[0017] FIG2 is a schematic diagram of pixel reorganization in one embodiment of the present application;
[0018] FIG3 is a flowchart of a decoding method in one embodiment of the present application;
[0019] FIG4 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;
[0020] FIG5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;
[0021] 6A-6D are schematic diagrams of the structure of a synthesis transformation network in one embodiment of the present application;
[0022] 7A-7B are schematic diagrams of the structure of a synthesis transformation network in one embodiment of the present application;
[0023] 8A-8H are schematic diagrams of the structure of a synthesis transformation network in one embodiment of the present application;
[0024] FIG9A is a hardware structure diagram of a decoding end device in one embodiment of the present application;
[0025] FIG9B is a hardware structure diagram of an encoding terminal device in one embodiment of the present application. DETAILED DESCRIPTION
[0026] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at the time of...", or "when...", or "in response to determination".
[0027] In an embodiment of the present application, a decoding method is proposed, which may involve the following concepts:
[0028] Entropy encoding: Entropy encoding is a method of encoding that does not lose any information during the encoding process according to the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.
[0029] Neural Network (NN): A neural network refers to an artificial neural network. A NN is a computational model composed of a large number of interconnected nodes (or neurons). In a NN, neurons (often called processing units) can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. Processing units in a NN can be divided into three types: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; and hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information is reflected in the connections between processing units. A NN is a non-programmed, brain-like information processing method. Its essence is to achieve parallel and distributed information processing capabilities through the transformations and dynamics of the neural network, mimicking the information processing capabilities of the human brain to varying degrees and levels. In the field of video processing, commonly used NNs include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.
[0030] Convolutional Neural Network (CNN): A convolutional neural network (CNN) is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to surrounding cells within a certain coverage area, making it an excellent choice for processing large images. The basic structure of a CNN consists of two layers. The first is the feature extraction layer (also known as the convolution layer). Each neuron's input is connected to the local receptive field of the previous layer, extracting local features. Once a local feature is extracted, its positional relationship with other features is determined. The second is the feature mapping layer (also known as the activation layer). Each computational layer of the neural network consists of multiple feature maps. Each feature map is a plane where all neurons have equal weights. Feature map structures can use sigmoid functions, ReLU (Rectified Linear Unit) functions, Leaky-ReLU functions, PReLU (Parametric Rectified Linear Unit) functions, and GDN (Generalized Divisive Normalization) functions as activation functions. Furthermore, because neurons within a mapping surface share weights, the number of free parameters in the network is reduced.
[0031] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid complex pre-processing of images (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected. This will result in a huge number of parameters, making network training time-consuming or even difficult to train. Convolutional neural networks avoid this difficulty through methods such as local connections and weight sharing.
[0032] Deconvolution: Also known as a transposed convolution layer, the deconvolution layer works very similarly to the convolution layer. The main difference is that the deconvolution layer uses padding to make the output larger than the input (or the same). If the stride is 1, the output size is equal to the input size; if the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.
[0033] Depthwise Separable Convolution: Two convolution operations are required. In the first convolution operation, deep_wise convolution is performed first (i.e., collecting features of each layer), kernel_size = K*K*1, and the total number of parameters for the first convolution is K*K*Cin. In the second convolution operation, in order to obtain the output of the Cout dimension, kernel_size = 1*1*Cin, and the total number of parameters for the second convolution is 1*1*Cin*Cout. The output of the second convolution can be the output of the depthwise separable convolution, and the input of the first convolution can be the input of the depthwise separable convolution.
[0034] Generalization Ability: refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns hidden behind the data. The trained network can also give appropriate output for data outside the learning set with the same patterns. This ability can be called generalization ability.
[0035] A feature is a three-dimensional feature matrix or tensor of the form C*W*H. Figure 1 shows a schematic diagram of a three-dimensional feature matrix. In this matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. A three-dimensional feature matrix can be either the input or the output of a neural network.
[0036] Pixel shuffle: Also known as feature position shuffling, this technique reassembles low-resolution features across multiple channels to generate high-resolution feature maps. Figure 2 shows a schematic diagram of pixel shuffling with an upsampling factor of 2. Clearly, a feature with 4 channels and a resolution of 3x3 becomes a feature with 1 channel and a resolution of 6x6 after passing through the feature position shuffling layer.
[0037] Rate-Distortion Optimized (RDO): Coding efficiency is evaluated using two main metrics: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit rate, the greater the compression ratio; the greater the PSNR, the better the reconstructed image quality. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode is: J(mode) = D + λ*R,
[0038] Here, D represents distortion, which is typically measured using the sum-square error (SSE), which is the mean squared sum of the differences between the reconstructed image block and the source image. For cost considerations, the sum of absolute differences (SAD) can also be used. λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in that mode, including the bits required for coding mode information, motion information, and residuals. When selecting a mode, using the rate-distortion principle to compare and decide between coding modes generally ensures optimal coding performance.
[0039] Numerous coding tools have been proposed for various modules on the encoding side, and each tool often has multiple modes. Different coding tools often produce the best coding performance for different video sequences. Therefore, during the encoding process, Rate-Distortion Optimization (RDO) is often used to compare the coding performance of different tools or modes to select the optimal mode. After determining the optimal tool or mode, the decision-making information for the tool or mode is conveyed by encoding marker information in the bitstream. While this approach introduces higher coding complexity, it can adaptively select the optimal mode combination for different content, achieving optimal coding performance. The decoding side can obtain relevant mode information by directly parsing the marker information, with minimal complexity impact.
[0040] The general end-to-end image coding framework mainly consists of a feature main information component and a super-prior side information component. The feature main information component includes an analysis transform network, quantization, normal entropy coding, normal entropy decoding, and a synthesis transform network, while the super-prior side information component includes a super-prior analysis network, quantization, factored entropy coding, factored entropy decoding, and a super-prior synthesis network. The image component x is compressed and encoded by the analysis transform network and the synthesis transform network of the feature main information component, respectively, and reconstructed and restored. The super-prior side information component is mainly used to model the probability of the feature main information and guide the entropy encoding and decoding of the feature main information. However, this general end-to-end image coding framework suffers from high computational complexity and poor decoding performance.
[0041] In response to these findings, this embodiment leverages the characteristics of an end-to-end image coding framework to propose a synthetic transform network with an attention mechanism. This synthetic transform network includes an attention module, which is a cascaded hybrid attention module. Implementing this synthetic transform network with cascaded hybrid attention modules effectively reduces network and computational complexity while maintaining synthesized image quality, thereby improving decoding performance.
[0042] The decoding method in the embodiments of the present application is described in detail below with reference to several specific embodiments.
[0043] Embodiment 1: In the embodiment of the present application, a decoding method is proposed. FIG3 is a flowchart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method may include:
[0044] Step 301: Decode a code stream corresponding to the current image block to obtain coefficient hyperparameter features of the current image block.
[0045] Step 302: Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode another code stream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block.
[0046] Step 303: Input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block. The synthetic transformation network includes at least an attention module, which may be a cascaded hybrid attention module. The cascaded hybrid attention module may include a first attention submodule and a second attention submodule, wherein the first attention submodule and the second attention submodule are two serial submodules. For example, the output features of the first attention submodule serve as the input features of the second attention submodule.
[0047] Among them, one code stream corresponding to the current image block can be a code stream encoding the coefficient hyperparameter features corresponding to the current image block; the other code stream corresponding to the current image block can be a code stream encoding the residual features corresponding to the current image block.
[0048] Exemplarily, if the attention module is a cascaded hybrid attention module, the first attention submodule can perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, and the output features of the first attention submodule serve as the input features of the second attention submodule; wherein the first processing can be at least one of layer normalization, convolution operation, and dimensionality transformation operation. The second attention submodule can perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule; wherein the second processing can be at least one of depthwise separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, downsampling operation, residual convolution operation, and upsampling operation. For example, the second processing can be at least one of depthwise separable convolution operation, layer normalization, and multi-layer perception operation. Alternatively, the second processing can be at least one of layer normalization, linear operation, and activation operation. Alternatively, the second processing can be at least one of downsampling operation, residual convolution operation, upsampling operation, and activation operation.
[0049] In a possible embodiment, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, including but not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector; performing a dimension transformation operation on the query vector, the key vector and the value vector respectively to obtain a dimension-transformed query vector, a dimension-transformed key vector and a dimension-transformed value vector; determining an attention weight based on the dimension-transformed query vector and the dimension-transformed key vector, and determining a correction feature corresponding to the input feature based on the attention weight and the dimension-transformed value vector; and determining the output features of the first attention submodule based on the input features and the correction features.
[0050] In a possible implementation, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing window division on the layer-normalized features to obtain multiple small-size features; for each small-size feature, performing a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to multiple small-size features can be spliced to obtain a query vector; the small-size key features corresponding to multiple small-size features can be spliced to obtain a key vector; the small-size value features corresponding to multiple small-size features can be spliced to obtain a value vector. Then, the query vector, the key vector and the value vector are respectively subjected to dimension transformation operations to obtain the dimension-transformed query vector, the dimension-transformed key vector and the dimension-transformed value vector; the attention weight is determined based on the dimension-transformed query vector and the dimension-transformed key vector, and the correction feature corresponding to the input feature is determined based on the attention weight and the dimension-transformed value vector; the output feature of the first attention sub-module is determined based on the input feature and the correction feature.
[0051] In a possible embodiment, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector; determining an attention weight based on the query vector and the key vector, and determining a correction feature corresponding to the input feature based on the attention weight and the value vector; and determining the output feature of the first attention submodule based on the input feature and the correction feature.
[0052] In one possible embodiment, the first attention submodule performs a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule, which may include but is not limited to: performing layer normalization on the input features of the first attention submodule to obtain layer-normalized features; performing window division on the layer-normalized features to obtain multiple small-size features; for each small-size feature, performing a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; splicing the small-size query features corresponding to multiple small-size features to obtain a query vector; splicing the small-size key features corresponding to multiple small-size features to obtain a key vector; splicing the small-size value features corresponding to multiple small-size features to obtain a value vector; determining the attention weight based on the query vector and the key vector, and determining the correction feature corresponding to the input feature based on the attention weight and the value vector. Then, the output features of the first attention submodule can be determined based on the input features and the correction features.
[0053] In a possible embodiment, the second attention submodule performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, which may include but is not limited to: performing layer normalization on the input features of the second attention submodule to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention submodule based on the input features and the features after the second linear operation.
[0054] In one possible implementation, the second attention submodule performs a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule, which may include but is not limited to: downsampling the input features of the second attention submodule to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolution features; upsampling the residual convolution features to obtain upsampled features; activating the upsampled features to obtain activated features; and performing a residual convolution operation on the input features to obtain residual convolution features. On this basis, the output features of the second attention submodule can be determined based on the input features, the activated features, and the convolution features.
[0055] In one possible implementation, the synthetic transformation network may further include at least one deconvolution layer; wherein: the attention module may be located behind one of the deconvolution layers.
[0056] Exemplarily, the synthetic transformation network may further include a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer. The synthetic transformation network may include at least one attention module, wherein one attention module is located after the first deconvolution layer.
[0057] Exemplarily, the synthetic transformation network includes at least a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, and a second deconvolution layer. The synthetic transformation network includes at least one attention module, wherein one attention module is located after the first deconvolution layer.
[0058] Based on the same concept as the above-mentioned decoding method, an encoding method is also proposed in an embodiment of the present application, which can be applied to the encoding end (also called a video encoder). The method may include: decoding a code stream corresponding to the current image block to obtain the coefficient hyperparameter feature corresponding to the current image block; determining the probability distribution parameter based on the coefficient hyperparameter feature, and decoding another code stream corresponding to the current image block based on the probability distribution parameter to obtain the reconstruction feature corresponding to the current image block; inputting the reconstruction feature into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.
[0059] Among them, one code stream corresponding to the current image block can be a code stream encoding the coefficient hyperparameter features corresponding to the current image block; the other code stream corresponding to the current image block can be a code stream encoding the residual features corresponding to the current image block.
[0060] Exemplarily, the implementation process of the encoding method is similar to that of the decoding method, and repeated contents are not repeated here.
[0061] For example, the above execution order is only for the convenience of describing the examples given. In actual applications, the execution order between the steps can also be changed, and this execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.
[0062] It can be seen from the above technical solutions that in the embodiments of the present application, a synthetic transformation network with an attention mechanism is proposed for the neural network-based encoding and decoding technology. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module. When the synthetic transformation network is realized by the cascaded hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, and the decoding performance is improved. While reducing the complexity, the quality of the reconstructed image block is effectively guaranteed, and the encoding performance and decoding performance are improved.
[0063] Example 2: The processing process of the encoding end can be shown in Figure 4. Of course, Figure 4 is only an example of the processing process of the encoding end, and there is no limitation on the processing process of the encoding end. The encoding process can be implemented using a neural network.
[0064] After obtaining the current image block x (which can be the original image block x, i.e., the input image block), the encoder can perform an analysis transformation on the current image block x using an analysis transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Performing feature transformation on the current image block x using the analysis transformation network means transforming the current image block x into image features y in the latent domain, thereby facilitating all subsequent processes to operate in the latent domain.
[0065] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.
[0066] After obtaining the image feature y, the encoding end performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of this hyperparameter encoding network. It only needs to be able to perform a coefficient hyperparameter feature transformation on the image feature y. The image feature y in the latent domain obtains the super-prior latent information z after passing through the hyperparameter encoding network.
[0067] After obtaining the coefficient hyperparameter feature z, the encoding end can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, the Q operation in Figure 4 is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain the Bitstream#1 (which can be called the first code stream) corresponding to the current image block, that is, the AE operation in Figure 4 represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or the coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.
[0068] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the Bitstream#1 corresponding to the current image block by the decoder, please refer to the subsequent embodiments.
[0069] After obtaining Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain super-parameter quantization features. Specifically, AD in Figure 4 represents the decoding process. The super-parameter quantization features are then dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. The IQ operation in Figure 4 represents the dequantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the coefficient super-parameter feature z_hat without dequantizing the super-parameter quantization features.
[0070] For the encoding process of Bitstream#1, an encoding method of a fixed probability density model can be used. For the decoding process of Bitstream#1, a decoding method of a fixed probability density model can be used. There is no restriction on this encoding and decoding process.
[0071] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block (see the subsequent embodiments for the determination process of the reconstruction feature y_hat) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat are input into the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstruction feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain a residual by subtracting the original feature and adding it to the decoded residual to obtain the reconstructed feature y_hat.
[0072] In another embodiment, the mean prediction network can also obtain the prediction value corresponding to the current image block based solely on the coefficient hyperparameter feature z_hat of the current image block, that is, without using the reconstructed feature y_hat of the previous image block to obtain the prediction value corresponding to the current image block. This can greatly reduce the complexity of obtaining the prediction value and accelerate the encoding and decoding process. The following methods for obtaining the prediction value on the encoding end can adopt one of the two solutions described above and will not be described in detail.
[0073] It should be noted that the mean prediction network is an optional neural network, that is, the mean prediction network may not be present, that is, the predicted value mu does not need to be determined through the mean prediction network. The dotted box in Figure 4 indicates that the mean prediction network is optional.
[0074] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the predicted value mu, such as taking the difference between the image feature y and the predicted value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the predicted value mu. Alternatively, after obtaining the image feature y, the encoding end can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.
[0075] After obtaining image feature s, the encoder can quantize image feature s to obtain the image quantization feature corresponding to image feature s. That is, the Q operation in Figure 4 represents the quantization process. After obtaining the image quantization feature corresponding to image feature s, the encoder can encode the image quantization feature to obtain Bitstream #2 (which can be called the second code stream) corresponding to the current image block. That is, the AE operation in Figure 4 represents the encoding process, such as the entropy encoding process. Alternatively, the encoder can directly encode image feature s to obtain Bitstream #2 corresponding to the current image block without involving the quantization process of image feature s.
[0076] After obtaining Bitstream#2 corresponding to the current image block, the encoder can send Bitstream#2 corresponding to the current image block to the decoder. For details about the processing of Bitstream#2 corresponding to the current image block by the decoder, see the subsequent embodiments.
[0077] After obtaining Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain image quantization features. In other words, AD in Figure 4 represents the decoding process. The encoder can then dequantize the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. The IQ operation in Figure 4 represents the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain image features s' without involving the dequantization process of the image quantization features.
[0078] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the reconstructed feature y_hat based on the residual feature r_hat and the predicted value mu. The reconstructed feature y_hat can be the same or different from the image feature y. For example, the sum of the residual feature r_hat and the predicted value mu is used as the reconstructed feature y_hat. In this case, it is necessary to deploy a mean prediction network, which provides the predicted value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the reconstructed feature y_hat. The reconstructed feature y_hat and the image feature y can be the same or different. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.
[0079] After obtaining the reconstructed feature y_hat, the encoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.
[0080] In one possible implementation, when encoding the image quantization feature or image feature s to obtain Bitstream #2 corresponding to the current image block, the encoder needs to first determine a probability distribution model and then encode the image quantization feature or image feature s based on the probability distribution model. Furthermore, when decoding Bitstream #2, the encoder also needs to first determine a probability distribution model and then decode Bitstream #2 based on the probability distribution model.
[0081] In order to obtain the probability distribution model, as shown in Figure 4, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network. There is no restriction on the training process of this probability hyperparameter decoding network. It only needs to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.
[0082] In one possible implementation, the processing at the encoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without any restriction on the encoding process.
[0083] Example 3: The processing process of the decoding end can be shown in Figure 5. Of course, Figure 5 is only an example of the processing process of the decoding end, and there is no limitation on the processing process of the decoding end. The decoding process can be implemented by using a neural network.
[0084] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can decode Bitstream#1 to obtain the super-parameter quantization feature, that is, AD in Figure 5 represents the decoding process, and then dequantize the super-parameter quantization feature to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat and the coefficient super-parameter feature z can be the same or different. The IQ operation in Figure 5 is the dequantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoding end can decode Bitstream#1 to obtain the coefficient super-parameter feature z_hat without involving the dequantization process of the super-parameter quantization feature. Among them, for the decoding process of Bitstream#1, a decoding method of a fixed probability density model can be used, and there is no restriction on this.
[0085] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.
[0086] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block (the determination process of the reconstruction feature y_hat is described in subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat are input into the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstruction feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.
[0087] In another embodiment, after obtaining the coefficient hyperparameter feature z_hat of the current image block, the decoder can directly obtain the prediction value mu corresponding to the current image block based on the coefficient hyperparameter feature z_hat. That is, it does not rely on the reconstruction feature y_hat of the previous image block to obtain the prediction value. This can greatly reduce the complexity of obtaining the prediction value and accelerate the encoding and decoding process. The following methods for obtaining the prediction value on the decoder can adopt one of the two solutions described above and will not be described in detail.
[0088] It should be noted that the mean prediction network is an optional neural network, that is, the mean prediction network may not be present, that is, the predicted value mu does not need to be determined through the mean prediction network. The dotted box in Figure 5 indicates that the mean prediction network is optional.
[0089] After obtaining Bitstream#2 corresponding to the current image block, the decoder can decode Bitstream#2 to obtain the image quantization features. That is, AD in Figure 5 represents the decoding process. The decoder can then dequantize the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. The IQ operation in Figure 5 can be the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the decoder can decode Bitstream#2 to obtain image features s' without involving the dequantization process of the image quantization features.
[0090] After obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoding end determines the reconstructed feature y_hat based on the residual feature r_hat and the predicted value mu. The reconstructed feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the predicted value mu as the reconstructed feature y_hat. In this case, it is necessary to deploy a mean prediction network, which provides the predicted value mu. Alternatively, after obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' to obtain the reconstructed feature y_hat. The reconstructed feature y_hat can be the same as or different from the image feature y. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.
[0091] After obtaining the reconstructed feature y_hat, the decoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.
[0092] In one possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. In order to obtain the probability distribution model, referring to FIG5 , after obtaining the coefficient superparameter feature z_hat, the decoding end can perform a coefficient superparameter feature inverse transformation on the coefficient superparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient superparameter feature z_hat is input into the probability superparameter decoding network, and the probability superparameter decoding network performs a coefficient superparameter feature inverse transformation on the coefficient superparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. The probability superparameter decoding network can be a trained neural network, and there is no restriction on the training process of the probability superparameter decoding network. It is sufficient to perform a coefficient superparameter feature inverse transformation on the coefficient superparameter feature z_hat to obtain the probability distribution parameter p.
[0093] In one possible implementation, the processing at the decoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without limiting the decoding process.
[0094] Example 4: For Example 1, Example 2, and Example 3, a mean prediction network may be deployed or not. To improve feature encoding performance, the deployment of a mean prediction network is taken as an example. When deploying the mean prediction network, in order to obtain an accurate prediction value of the image feature, a context-based prediction may be performed based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block to obtain the prediction value mu corresponding to the current image block; alternatively, the prediction value mu corresponding to the current image block may be obtained based on the coefficient hyperparameter feature z_hat of the current image block.
[0095] The encoding process may include the following steps:
[0096] Step S11: After obtaining the original image, determine whether to segment the original image based on the image resolution. For example, if the image resolution is greater than a threshold, the original image is segmented; otherwise, the original image is not segmented. If the resolution is greater than a threshold, the original image is segmented into a number of image blocks x, with overlapping portions between adjacent image blocks. Otherwise, image block x is the original image, i.e., the original image has only one image block. For ease of description, the following uses one image block as an example, denoted as the current image block x, and the subsequent processing of the current image block x is used as an example.
[0097] Step S12: After obtaining the current image block x, the current image block x is analyzed and transformed by the analysis transformation network to obtain the image features y (i.e., feature blocks) corresponding to the current image block x. For example, the analysis transformation network transforms the current image block x into the image features y in the latent domain, so that all subsequent processes can be performed in the latent domain.
[0098] Step S13: Perform a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z corresponding to the current image block x. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), which performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z corresponding to the current image block x.
[0099] Step S14: Encode the coefficient hyperparameter feature z into the first bitstream (Bitstream#1) corresponding to the current image block. For example, the coefficient hyperparameter feature z is quantized to obtain a hyperparameter quantization feature, and the hyperparameter quantization feature is encoded to obtain the first bitstream corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z is directly encoded to obtain the first bitstream corresponding to the current image block. That is, the first bitstream corresponding to the current image block is a bitstream encoded with the coefficient hyperparameter feature z corresponding to the current image block x. After obtaining the first bitstream corresponding to the current image block, the first bitstream corresponding to the current image block is sent to the decoding end.
[0100] Step S15: After obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the superparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the superparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving a dequantization process.
[0101] Step S16: Determine the probability distribution parameter based on the coefficient hyperparameter feature z_hat. For example, perform an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, input the coefficient hyperparameter feature z_hat into a probability hyperparameter decoding network, which performs an inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on the probability distribution parameter p.
[0102] Step S17: Input the coefficient hyperparameter feature z_hat into the mean prediction network, which performs context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat of the previous image block to obtain the predicted value mu. Alternatively, input the coefficient hyperparameter feature z_hat into the mean prediction network, which obtains the predicted value mu based on the coefficient hyperparameter feature z_hat.
[0103] Step S18: Determine a residual feature r based on the image feature y and the mean feature mu (i.e., the predicted value mu). For example, the difference between the image feature y and the mean feature mu is used as the residual feature r. Feature processing is performed on the residual feature r to obtain a processed residual feature r. There are no restrictions on the feature processing process. Feature processing is an optional step, and feature processing may not be performed on the residual feature r.
[0104] Step S19: quantize the residual feature r (or the residual feature r after feature processing) to obtain an image quantization feature, and encode the image quantization feature to obtain a second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r (or the residual feature r after feature processing) can be directly encoded to obtain a second bitstream corresponding to the current image block. That is, the second bitstream corresponding to the current image block is a bitstream encoded with the residual feature r corresponding to the current image block x. After obtaining the second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is sent to the decoding end.
[0105] Exemplarily, when encoding the image quantization feature or the residual feature r, the probability distribution model corresponding to the probability distribution parameter p may be used to encode the image quantization feature or the residual feature r to obtain a second code stream.
[0106] Step S20: After obtaining the second bitstream corresponding to the current image block, the second bitstream is decoded using the probability distribution model corresponding to the probability distribution parameter p to obtain image quantization features, the image quantization features are dequantized, and feature recovery is performed on the dequantized features to obtain residual features r_hat. Alternatively, the image quantization features are dequantized to directly obtain residual features r_hat without involving a feature recovery process. Alternatively, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain residual features r_hat. Alternatively, the residual features r_hat are directly obtained after decoding the second bitstream.
[0107] Step S21: After obtaining the residual feature r_hat, determine the reconstructed feature y_hat based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu is used as the reconstructed feature y_hat.
[0108] Step S22: input the reconstructed feature y_hat into the synthesis transformation network, and the synthesis transformation network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstruction feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.
[0109] For example, taking the deployment of a mean prediction network as an example, the processing process on the decoding end may include the following steps:
[0110] Step S31: After obtaining the first bitstream corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter feature z_hat corresponding to the current image block x. For example, the first bitstream can be decoded to obtain the superparameter quantization feature corresponding to the coefficient hyperparameter feature z_hat, and the superparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat without involving a dequantization process.
[0111] Step S32: Determine the probability distribution parameter based on the coefficient hyperparameter feature z_hat. For example, perform an inverse coefficient hyperparameter feature transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, input the coefficient hyperparameter feature z_hat into a probability hyperparameter decoding network, which performs an inverse coefficient hyperparameter feature transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, generate a probability distribution model based on the probability distribution parameter p.
[0112] Step S33: After obtaining the second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is decoded based on the probability distribution parameter p to obtain a reconstructed feature corresponding to the current image block. For example, a residual feature r_hat corresponding to the current image block can be determined, and a predicted value mu corresponding to the current image block can be determined. A reconstructed feature y_hat is determined based on the residual feature r_hat and the mean feature mu. For example, the sum of the residual feature r_hat and the mean feature mu (i.e., the predicted value mu) is used as the reconstructed feature y_hat.
[0113] Among them, in order to determine the predicted value mu corresponding to the current image block, after obtaining the coefficient hyperparameter feature z_hat, the coefficient hyperparameter feature z_hat can be input into the mean prediction network. The mean prediction network performs a context-based prediction based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat of the previous image block to obtain the predicted value mu corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z_hat can be input into the mean prediction network, and the mean prediction network obtains the predicted value mu corresponding to the current image block based on the coefficient hyperparameter feature z_hat.
[0114] To determine the residual feature r_hat corresponding to the current image block, the second bitstream can be decoded using a probability distribution model corresponding to the probability distribution parameter p to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain the residual feature r_hat. Alternatively, the image quantization features can be dequantized to directly obtain the residual feature r_hat without involving a feature recovery process. Alternatively, after decoding the second bitstream, feature recovery can be performed on the decoded features to obtain the residual feature r_hat. Alternatively, the residual feature r_hat can be directly obtained after decoding the second bitstream.
[0115] Step S34: input the reconstructed feature y_hat into the synthesis transformation network, and the synthesis transformation network determines the reconstructed image block x_hat corresponding to the current image block based on the reconstruction feature y_hat, and outputs the reconstructed image block x_hat corresponding to the current image block.
[0116] Example 5: In Examples 1, 2, 3, and 4, a composite transformation network is involved. The implementation of this example is the same as that of Examples 1, 2, 3, or 4, and the repeated details are not repeated here. The composite transformation network involved is described in detail below.
[0117] An example of the synthetic transformation network can be shown in FIG6A , that is, the synthetic transformation network can include, in sequence, a residual layer, a deconvolution layer, a cropping layer, a residual activation layer, a deconvolution layer, a cropping layer, a residual activation layer, a deconvolution layer, an attention model, a cropping layer, a residual activation layer, a deconvolution layer, a cropping layer, etc. The input feature of the synthetic transformation network can be a reconstructed feature y_hat, and the output feature of the synthetic transformation network can be a reconstructed image block x_hat. That is, after the reconstructed feature y_hat is input into the synthetic transformation network, it is processed in sequence by the above-mentioned network layers to obtain the reconstructed image block x_hat.
[0118] Exemplarily, an example of a residual layer in a synthetic transformation network can be shown in FIG6B , that is, the residual layer can include a convolution layer, a rectified linear unit (Rectified Linear Unit, ReLU) (such as a leaky rectified linear unit LeakyReLU), a convolution layer, and an overlay layer (for implementing feature addition) in sequence. The first convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer can be selected. The second convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer can be selected.
[0119] For example, the residual activation layer in the synthetic transformation network can be shown in Figure 6C, that is, the residual activation layer can include an activation layer (such as a LeakyRelu activation layer), a convolution layer, an activation layer (such as a tanh activation layer), a multiplication layer (for implementing feature multiplication), and an overlay layer (for implementing feature addition). The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, without limitation. For example, a 1x1 convolution layer can be selected.
[0120] Exemplarily, the attention model in the synthetic transformation network can be a residual non-local attention block (RNAB), that is, RNAB is introduced as an attention model, RNAB is used to obtain non-local information of the image, and uses the non-local information as attention weights to improve decoding performance. An example of RNAB can be shown in Figure 6D, that is, RNAB can include a residual block, a residual block, a downsampling convolution layer (such as a 2x downsampling convolution layer, such as a 3x3 convolution layer), a residual block, a residual block, an upsampling convolution layer (such as a 2x upsampling convolution layer, such as a 3x3 convolution layer), a residual block, a residual block, a convolution layer (such as a 3x3 convolution layer), an S-shaped activation function (Sigmoid), a residual block, a residual block, a residual block, a multiplication layer (for realizing feature multiplication) and an overlay layer (for realizing feature addition).
[0121] For the residual block, the residual block can be shown in Figure 6B, that is, the residual block is similar to the residual layer in Figure 6B, and can also include a convolution layer, a rectified linear unit (Rectified Linear Unit, ReLU) (such as a leaky rectified linear unit LeakyReLU), a convolution layer and an overlay layer (for adding features). In this way, the input features of the residual block are extracted by the convolution layer, the rectified linear unit and the convolution layer, and then added to the input features of the residual block to obtain the output features of the residual block.
[0122] The sub-module consisting of a residual block, a residual block, a downsampling convolution layer, a residual block, a residual block, an upsampling convolution layer, a residual block, a residual block, a convolution layer and a S-shaped activation function can be a non-local attention extraction sub-module, and the sub-module consisting of three residual blocks can be a feature extraction sub-module. The input features of RNAB are multiplied by the output of the feature extraction sub-module after the non-local attention extraction sub-module generates attention weights, and the output features of RNAB are obtained by adding them to the input features of RNAB.
[0123] Example 6: In Examples 1, 2, 3, and 4, a composite transformation network is involved. The implementation of this example is the same as that of Examples 1, 2, 3, or 4, and the repeated details are not repeated here. The composite transformation network involved is described in detail below.
[0124] The synthetic transformation network may include at least an attention module, and the attention module may be a cascaded hybrid attention module. In addition to the attention module, the synthetic transformation network may also include other network layers. In this embodiment, the structure of the synthetic transformation network is not limited, as long as the synthetic transformation network includes the attention module.
[0125] In one possible embodiment, in addition to the attention module, the synthetic transformation network may also include at least one deconvolution layer. Furthermore, in addition to including the attention module and at least one deconvolution layer, the synthetic transformation network may also include other network layers, and there is no limitation on this. When the synthetic transformation network includes at least one deconvolution layer, the attention module may be located behind any deconvolution layer. Alternatively, the attention module may be located behind each deconvolution layer; wherein, the attention modules located behind different deconvolution layers may be exactly the same, or, the attention modules located behind different deconvolution layers may not be exactly the same. Alternatively, the attention module may be located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers); wherein, the attention modules located behind different deconvolution layers may be exactly the same, or, the attention modules located behind different deconvolution layers may not be exactly the same.
[0126] In one possible embodiment, in addition to the attention module, the synthetic transformation network may include at least one of the following network layers: a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer. That is, the synthetic transformation network may include some or all of the above network layers. Of course, in addition to the above network layers, the synthetic transformation network may also include other network layers. This application does not limit this, but only uses the above network layers as an example for illustration.
[0127] 7A , which is an example of a synthetic transformation network, the synthetic transformation network may include, in sequence, a residual layer, a deconvolution layer 1 (i.e., a first deconvolution layer), a cropping layer 1 (i.e., a first cropping layer), a residual activation layer 1 (i.e., a first residual activation layer), a deconvolution layer 2 (i.e., a second deconvolution layer), a cropping layer 2 (i.e., a second cropping layer), a residual activation layer 2 (i.e., a second residual activation layer), a deconvolution layer 3 (i.e., a third deconvolution layer), a cropping layer 3 (i.e., a third cropping layer), a residual activation layer 3 (i.e., a third residual activation layer), a deconvolution layer 4 (i.e., a fourth deconvolution layer), and a cropping layer 4 (i.e., a fourth cropping layer). For example, the input feature of the synthetic transformation network may be a reconstruction feature y_hat, and the output feature of the synthetic transformation network may be a reconstructed image block x_hat. That is, after the reconstruction feature y_hat is input into the synthetic transformation network and processed sequentially by the above-mentioned network layers, the synthetic transformation network may obtain a reconstructed image block x_hat and output the reconstructed image block x_hat.
[0128] In FIG7A , three attention model positions are shown, namely, attention model position 1, attention model position 2, and attention model position 3. Exemplarily, if the synthetic transformation network includes only one attention module (i.e., a cascaded hybrid attention module), the attention module can be located behind the deconvolution layer 1, that is, the attention module is located at attention model position 1. Alternatively, the attention module can be located behind the deconvolution layer 2, that is, the attention module is located at attention model position 2. Alternatively, the attention module can be located behind the deconvolution layer 3, that is, the attention module is located at attention model position 3, for example, the attention module is placed behind the deconvolution layer 3. Of course, the above are just a few examples of the positions of the attention modules, and there is no limitation to this. They can be located at any position in the synthetic transformation network.
[0129] Exemplarily, the synthetic transformation network may include attention module 1 located behind deconvolution layer 1, attention module 2 located behind deconvolution layer 2, and attention module 3 located behind deconvolution layer 3, that is, attention module 1 is located at attention model position 1, attention module 2 is located at attention model position 2, and attention module 3 is located at attention model position 3. Of course, the attention modules can also be located at other positions in the synthetic transformation network, and there is no limitation on this.
[0130] Among them, attention module 1, attention module 2 and attention module 3 can be completely identical, that is, attention module 1, attention module 2 and attention module 3 can adopt the same network structure. Alternatively, attention module 1, attention module 2 and attention module 3 may not be completely identical, for example, the network structure of attention module 1 may be the same as the network structure of attention module 2, and the network structure of attention module 1 may be different from the network structure of attention module 3; or, the network structure of attention module 1 may be the same as the network structure of attention module 3, and the network structure of attention module 1 may be different from the network structure of attention module 2; or, the network structure of attention module 2 may be the same as the network structure of attention module 3, and the network structure of attention module 2 may be different from the network structure of attention module 1; or, the network structure of attention module 1 may be different from the network structure of attention module 2, and the network structure of attention module 1 may be different from the network structure of attention module 3, and the network structure of attention module 2 may be different from the network structure of attention module 3.
[0131] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 1, and an attention module 2 located behind the deconvolution layer 2, that is, the attention module 1 is located at the attention model position 1, and the attention module 2 is located at the attention model position 2. Of course, the attention module can also be located at other positions in the synthetic transformation network, and there is no limitation on this.
[0132] The attention module 1 and the attention module 2 may be identical, for example, the attention module 1 and the attention module 2 may use the same network structure. Alternatively, the attention module 1 and the attention module 2 may not be identical, for example, the network structure of the attention module 1 may be different from the network structure of the attention module 2.
[0133] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 1 and an attention module 2 located behind the deconvolution layer 3, that is, the attention module 1 is located at the attention model position 1, and the attention module 2 is located at the attention model position 3. Of course, the attention module can also be located at other positions in the synthetic transformation network, and there is no limitation on this.
[0134] The attention module 1 and the attention module 2 may be identical, for example, the attention module 1 and the attention module 2 may use the same network structure. Alternatively, the attention module 1 and the attention module 2 may not be identical, for example, the network structure of the attention module 1 may be different from the network structure of the attention module 2.
[0135] Exemplarily, the synthetic transformation network may include an attention module 1 located behind the deconvolution layer 2, and an attention module 2 located behind the deconvolution layer 3, that is, the attention module 1 is located at the attention model position 2, and the attention module 2 is located at the attention model position 3. Of course, the attention module can also be located at other positions in the synthetic transformation network, and there is no limitation on this.
[0136] The attention module 1 and the attention module 2 may be identical, for example, the attention module 1 and the attention module 2 may use the same network structure. Alternatively, the attention module 1 and the attention module 2 may not be identical, for example, the network structure of the attention module 1 may be different from the network structure of the attention module 2.
[0137] An example of the residual layer in the synthetic transformation network can be seen in Figure 6B, that is, the residual layer can include a convolution layer, a rectified linear unit (such as a leaky rectified linear unit LeakyReLU), a convolution layer and a superposition layer in sequence, that is, the input features are extracted by the convolution layer, the rectified linear unit and the convolution layer, and then added to the input features to obtain the output features of the residual layer. Among them, the first convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer can be selected. In addition, the second convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer can be selected.
[0138] An example of a residual activation layer in a synthetic transformation network can be seen in Figure 6C , where the residual activation layer can sequentially include an activation layer (e.g., a LeakyRelu activation layer), a convolutional layer, an activation layer (e.g., a tanh activation layer), a multiplication layer, and an overlay layer. There is no restriction on the structure of the residual activation layer. The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, and there is no restriction on this. For example, a 1x1 convolutional layer can be selected.
[0139] Example 7: This example further describes the attention module involved in Example 6. In Example 6, the attention module in the synthetic transformation network is involved. The attention module can be a cascaded hybrid attention module. The cascaded hybrid attention module replaces RNAB to greatly reduce the computational complexity of decoding while ensuring decoding performance as much as possible. Exemplarily, the cascaded hybrid attention module can extract the attention weight of the input feature and determine the output feature of the cascaded hybrid attention module based on the attention weight and the input feature.
[0140] Exemplarily, the cascaded hybrid attention module may include a first attention submodule (also referred to as a transformer attention submodule, and the transformer attention submodule is denoted as Transformer-based Attention Module) and a second attention submodule (also referred to as an upgraded convolution submodule, and denoted as ConvNext Block). The first attention submodule and the second attention submodule may be two serial submodules, that is, the output features of the first attention submodule are the input features of the second attention submodule. For example, referring to FIG7B , which is a schematic diagram of a cascaded hybrid attention module, the input features (or input images) of the cascaded hybrid attention module serve as the input features of the first attention submodule (the transformer attention submodule is taken as an example in FIG7B ). The input features of the first attention submodule pass through the first attention submodule, and the first attention submodule processes the input features based on the input features to obtain the output features of the first attention submodule. The output features of the first attention submodule are used as the input features of the second attention submodule (taking the upgraded convolution submodule as an example in Figure 7B). The input features of the second attention submodule pass through the second attention submodule, and the second attention submodule processes the input features to obtain the output features of the second attention submodule. The output features of the second attention submodule are used as the output features of the cascaded hybrid attention module.
[0141] Exemplarily, after input features of the first attention submodule are input into the first attention submodule, the first attention submodule performs a first processing on the input features to obtain output features of the first attention submodule. The first processing may include, but is not limited to, at least one of layer normalization, convolution, and dimensionality transformation. This first processing is not limited and is related to the network structure of the first attention submodule. The network structure of the first attention submodule can be arbitrarily configured.
[0142] Exemplarily, after the input features of the second attention submodule are input into the second attention submodule, the second attention submodule performs a second processing on the input features to obtain the output features of the second attention submodule. The second processing may include, but is not limited to, at least one of a depthwise separable convolution operation, layer normalization, a multi-layer perception operation, a linear operation, an activation operation, a downsampling operation, a residual convolution operation, and an upsampling operation. There is no limitation on this second processing, and the second processing is related to the network structure of the second attention submodule. The network structure of the second attention submodule can be arbitrarily configured.
[0143] For example, the second processing may include at least one of a depthwise separable convolution operation, layer normalization, and a multi-layer perception operation. Alternatively, the second processing may include at least one of a layer normalization, a linear operation, and an activation operation. Alternatively, the second processing may include at least one of a downsampling operation, a residual convolution operation, an upsampling operation, and an activation operation.
[0144] Example 8: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Example 6 / Example 7. In Example 6 and Example 7, the cascaded hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; the query vector, the key vector, and the value vector are respectively dimensionally transformed to obtain a dimensionally transformed query vector, a dimensionally transformed key vector, and a dimensionally transformed value vector; the attention weight is determined based on the dimensionally transformed query vector and the dimensionally transformed key vector, and the correction feature corresponding to the input feature is determined based on the attention weight and the dimensionally transformed value vector; the output feature of the first attention submodule is determined based on the input feature and the correction feature.
[0145] Refer to Figure 8A, which is a structural diagram of the first attention sub-module. This is just an example of the first attention sub-module and there is no limitation on the structure of the first attention sub-module.
[0146] Now take the first attention submodule of Figure 8A as an example for explanation. The first attention submodule may include a layer normalization layer, which can perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer normalized features. Layer normalization is a neural network regularization technology used to normalize the input features of each layer of the neural network to improve training and generalization performance. The layer normalization operation may include calculating the mean and variance, standardization, scaling, and translation, so that the output of each hidden unit is within a relatively small range, and each hidden unit has a similar distribution in the entire data set.
[0147] After obtaining the layer-normalized features, a three-way convolution operation can be performed on the layer-normalized features to obtain a query vector Q, a key vector K, and a value vector V. For example, the query vector Q can be obtained by convolving the layer-normalized features through two convolutional layers. The first convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, without limitation. For example, a 1x1 convolutional layer can be selected. The second convolutional layer can be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without limitation. For example, a 3x3 convolutional layer can be selected. The key vector K can be obtained by convolving the layer-normalized features through two convolutional layers. The first convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, with a 1x1 convolutional layer. The second convolutional layer can be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer. For example, a 3x3 convolutional layer can be selected. The layer-normalized features can be convolved through two convolutional layers to obtain a value vector V. The first convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected. The second convolutional layer can be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer. For example, a 3x3 convolutional layer can be selected.
[0148] A dimension transformation operation (i.e., R operation) is performed on the query vector Q, key vector K, and value vector V, respectively, to obtain the dimensionally transformed query vector Q, the dimensionally transformed key vector K, and the dimensionally transformed value vector V. The purpose of the dimension transformation operation is to transform the transformed tensor into the dimensional form required by the multi-head self-attention mechanism. The multi-head self-attention mechanism is a technology widely used in natural language processing (NLP) tasks. This technology uses the self-attention mechanism to build connections between context words, thereby reconstructing the representation of the target word based on the context. The multi-head attention mechanism is a combination of multiple groups of self-attention components, which can learn a variety of different types of contextual influences. By passing the outputs of multiple self-attentions through the parameter matrix to obtain a new output, the multi-head attention mechanism can capture a wider range of correlation features and increase the expressive power of the model.
[0149] The attention weights are determined based on the transformed query vector Q and the transformed key vector K. For example, the attention weights are obtained by performing matrix multiplication on the transformed query vector Q and the transformed key vector K, and applying the result of the matrix multiplication to a softmax function. There are no restrictions on the Softmax process.
[0150] The modified features corresponding to the input features are determined based on the attention weights and the dimensionally transformed value vector V. For example, matrix multiplication is performed on the dimensionally transformed value vector V and the attention weights, a dimension transformation operation (i.e., R operation) is performed on the result of the matrix multiplication, and a convolution operation is performed on the features after the dimension transformation operation to obtain the modified features corresponding to the input features. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer. For example, a 1x1 convolution layer can be selected.
[0151] The output features of the first attention submodule are determined based on the input features and the correction features of the first attention submodule. For example, matrix addition is performed on the input features and the correction features to obtain the output features of the first attention submodule.
[0152] At this point, the processing process of the first attention sub-module is completed, and the output features of the first attention sub-module are obtained.
[0153] Example 9: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Example 6 / Example 7. In Example 6 and Example 7, the cascaded hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are windowed to obtain multiple small-size features; for each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature and small-size value feature corresponding to the small-size feature; the small-size query features corresponding to multiple small-size features are spliced to obtain a query vector; the small-size key features corresponding to multiple small-size features are spliced to obtain a key vector; the small-size value features corresponding to multiple small-size features are spliced to obtain a value vector. Perform dimension transformation operations on the query vector, key vector and value vector respectively to obtain a dimensionally transformed query vector, a dimensionally transformed key vector and a dimensionally transformed value vector; determine an attention weight based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determine a correction feature based on the attention weight and the dimensionally transformed value vector; determine the output feature of the first attention submodule based on the input feature and the correction feature.
[0154] For example, based on the first attention submodule in FIG8A , a window partitioning layer can be added after the layer normalization layer. There is no limitation on the structure of the first attention submodule. For example, an example of the structure of the first attention submodule can be seen in FIG8A .
[0155] Exemplarily, the first attention submodule may include a layer normalization layer, which may be used to perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer-normalized features.
[0156] The layer-normalized features can be window-partitioned through the window partitioning layer to obtain multiple small-size features. For example, the layer-normalized features are divided into multiple small-size features, the width of each small-size feature is wi, and the height of each small-size feature is hi. The widths of different small-size features can be the same or different, and the heights of different small-size features can be the same or different. By dividing the features into multiple small-size features for subsequent operations, the complexity can be reduced.
[0157] For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature, and small-size value feature corresponding to the small-size feature. For example, the small-size feature is convolved through two convolutional layers to obtain the small-size query feature. The first convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, such as a 1x1 convolutional layer. The second convolutional layer can be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, such as a 3x3 convolutional layer. The small-size key feature is convolved through two convolutional layers to obtain the small-size key feature. The first convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, such as a 1x1 convolutional layer. The second convolutional layer can be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer. For example, a 3x3 convolutional layer is selected. The small-sized features are convolved through two convolutional layers to obtain small-sized value features. The first convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer is selected. The second convolutional layer can be a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer. For example, a 3x3 convolutional layer is selected.
[0158] The small-size query features corresponding to all small-size features can be spliced to obtain a query vector Q, and there is no restriction on this feature splicing process. In addition, the small-size key features corresponding to all small-size features can be spliced to obtain a key vector K, and the small-size value features corresponding to all small-size features can be spliced to obtain a value vector V.
[0159] The query vector Q, the key vector K and the value vector V can be respectively subjected to a dimension transformation operation (ie, R operation) to obtain the dimension-transformed query vector Q, the dimension-transformed key vector K and the dimension-transformed value vector V.
[0160] The attention weights can be determined based on the transformed query vector Q and the transformed key vector K. For example, the attention weights can be obtained by performing matrix multiplication on the transformed query vector Q and the transformed key vector K, and passing the result of the matrix multiplication through a softmax function (Softmax). There are no restrictions on the Softmax processing process.
[0161] The modified features corresponding to the input features can be determined based on the attention weights and the dimensionally transformed value vector V. For example, matrix multiplication is performed on the dimensionally transformed value vector V and the attention weights, a dimension transformation operation (i.e., R operation) is performed on the result of the matrix multiplication, and a convolution operation is performed on the features after the dimension transformation operation to obtain the modified features corresponding to the input features. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer. For example, a 1x1 convolution layer can be selected.
[0162] The output features of the first attention submodule can be determined based on the input features and the correction features of the first attention submodule. For example, the input features and the correction features are matrix added to obtain the output features of the first attention submodule.
[0163] At this point, the processing process of the first attention sub-module is completed, and the output features of the first attention sub-module are obtained.
[0164] Example 10: This example further describes the first attention submodule involved in the cascaded hybrid attention module based on Example 6 / Example 7. In Examples 6 and 7, the cascaded hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are windowed to obtain multiple small-scale features; and each small-scale feature is subjected to a three-way convolution operation to obtain a small-scale query feature, a small-scale key feature, and a small-scale value feature corresponding to the small-scale feature. The small-scale query features corresponding to the multiple small-scale features are concatenated to obtain a query vector; the small-scale key features corresponding to the multiple small-scale features are concatenated to obtain a key vector; and the small-scale value features corresponding to the multiple small-scale features are concatenated to obtain a value vector. An attention weight is determined based on the query vector and the key vector, and a correction feature corresponding to the input feature is determined based on the attention weight and the value vector. An output feature of the first attention submodule is determined based on the input feature and the modified feature.
[0165] 8B is a schematic diagram of the structure of the first attention submodule, which is only an example of the first attention submodule. The structure of the first attention submodule is not limited, and the following description is based on the first attention submodule of FIG8B as an example.
[0166] Exemplarily, the first attention submodule may include a layer normalization layer, which may be used to perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer-normalized features.
[0167] The first attention sub-module may include a window division layer, which performs window division on the layer-normalized features to obtain multiple small-size features. For example, the layer-normalized features are divided into multiple small-size features through the window division layer, and the width of each small-size feature is wi, and the height of each small-size feature is hi. The widths of different small-size features may be the same or different, and the heights of different small-size features may be the same or different.
[0168] For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain the small-size query feature, small-size key feature and small-size value feature corresponding to the small-size feature. For example, the small-size query feature is obtained by performing a convolution operation on the small-size feature through a convolution layer (such as a 1x1 convolution layer and a 3x3 convolution layer). The small-size key feature is obtained by performing a convolution operation on the small-size feature through a convolution layer (such as a 1x1 convolution layer and a 3x3 convolution layer). The small-size value feature is obtained by performing a convolution operation on the small-size feature through a convolution layer (such as a 1x1 convolution layer and a 3x3 convolution layer). The small-size query features corresponding to all small-size features are concatenated to obtain a query vector Q, the small-size key features corresponding to all small-size features are concatenated to obtain a key vector K, and the small-size value features corresponding to all small-size features are concatenated to obtain a value vector V.
[0169] The attention weight is determined based on the query vector Q and the key vector K. For example, the query vector Q and the key vector K are matrix multiplied, and the result of the matrix multiplication is passed through the softmax function to obtain the attention weight.
[0170] For the obtained attention weight and value vector V, in one example, a correction feature corresponding to the input feature can be directly obtained based on the attention weight and the value vector V. For example, matrix multiplication is performed on the attention weight and the value vector V, and the result of the matrix multiplication is used as the correction feature corresponding to the input feature. In other words, the second normalization layer in Figure 8B can be removed.
[0171] Furthermore, the output features of the first attention submodule can be determined based on the input features of the first attention submodule and the correction features. For example, matrix addition is performed on the input features and the correction features to obtain the output features of the first attention submodule.
[0172] At this point, the processing process of the first attention sub-module is completed, and the output features of the first attention sub-module are obtained.
[0173] Example 11: This example further describes the first attention submodule involved in the cascaded hybrid attention module on the basis of Example 6 / Example 7. In Example 6 and Example 7, the cascaded hybrid attention module includes a first attention submodule, which is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. For example, the input features of the first attention submodule are layer-normalized to obtain layer-normalized features; the layer-normalized features are subjected to a three-way convolution operation to obtain a query vector, a key vector, and a value vector; the attention weight is determined based on the query vector and the key vector, and the correction feature corresponding to the input feature is determined based on the attention weight and the value vector; the output feature of the first attention submodule is determined based on the input features and the correction features of the first attention submodule.
[0174] For example, based on the first attention submodule in FIG8B , the window partitioning layer after the layer normalization layer can be removed. There is no limitation on the structure of the first attention submodule. For example, an example of the structure of the first attention submodule can be shown in FIG8B .
[0175] Exemplarily, the first attention submodule may include a layer normalization layer, which may be used to perform layer normalization on the input features (or input images) of the first attention submodule to obtain layer-normalized features.
[0176] The first attention submodule may include a convolutional layer that performs a three-way convolution operation on the layer-normalized features to obtain a query vector Q, a key vector K, and a value vector V. For example, the query vector Q can be obtained by convolving the layer-normalized features through a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer). The key vector K can be obtained by convolving the layer-normalized features through a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer). The value vector V can be obtained by convolving the layer-normalized features through a convolutional layer (e.g., a 1x1 convolutional layer and a 3x3 convolutional layer).
[0177] The attention weight is determined based on the query vector Q and the key vector K. For example, the query vector Q and the key vector K are matrix multiplied, and the result of the matrix multiplication is passed through the softmax function to obtain the attention weight.
[0178] In one example, the obtained attention weight and value vector V can be used to directly obtain a modified feature corresponding to the input feature based on the attention weight and value vector V. For example, matrix multiplication can be performed on the attention weight and value vector V, and the result of the matrix multiplication can be used as the modified feature corresponding to the input feature. In other words, the second normalization layer in Figure 8B can be removed.
[0179] Furthermore, the output features of the first attention submodule can be determined based on the input features of the first attention submodule and the correction features. For example, matrix addition is performed on the input features and the correction features to obtain the output features of the first attention submodule.
[0180] At this point, the processing process of the first attention sub-module is completed, and the output features of the first attention sub-module are obtained.
[0181] In Examples 8 to 11, several examples of the first attention submodule are given. This application does not limit the structure of the first attention submodule, and the input features can be first processed to obtain the output features.
[0182] Example 12: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on Example 6 / Example 7. In Example 6 and Example 7, the cascaded hybrid attention module includes a second attention submodule, and the second attention submodule is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a depthwise separable convolution operation is performed on the input features of the second attention submodule to obtain the convolved features (i.e., the features after the depthwise separable convolution operation); the convolved features are layer-normalized to obtain the layer-normalized features; the layer-normalized features are multi-layered perceived to obtain the multi-layer perceived features; and the output features of the second attention submodule are determined based on the input features and the multi-layer perceived features.
[0183] 8C is a schematic diagram of the structure of the second attention submodule, which is only an example of the second attention submodule. The structure of the second attention submodule is not limited, and the second attention submodule of FIG8C is used as an example for explanation.
[0184] Exemplarily, the second attention sub-module may include a depthwise separable convolution layer, which may be a 1x1 convolution layer, a 3x3 convolution layer, a 5x5 convolution layer, or a 7x7 convolution layer, without limitation. For example, a 7x7 convolution layer may be selected, and a depthwise separable convolution operation may be performed on the input features of the second attention sub-module through the depthwise separable convolution layer to obtain the convolved features (i.e., the features after the depthwise separable convolution operation), without limitation to the convolution operation.
[0185] The second attention submodule can include a layer normalization layer, which can be used to normalize the convolutional features to obtain layer-normalized features. The layer normalization operations include calculating the mean and variance, standardization, scaling, and translation, so that the output of each hidden unit is within a relatively small range and each hidden unit has a similar distribution in the dataset.
[0186] The second attention submodule may include a multi-layer perceptron, which may perform multi-layer perception operations on the layer-normalized features to obtain multi-layer perceived features. Referring to FIG8D , which is a schematic diagram of the structure of a multi-layer perceptron, the multi-layer perceptron may include a linear layer, an activation layer, a random dropout layer, a linear layer, and a random dropout layer. Based on this, the layer-normalized features pass through a linear layer, an activation layer, a random dropout layer, a linear layer, and a random dropout layer in sequence to obtain multi-layer perceived features. Among them, the activation layer may be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, and there is no restriction on the type of the activation layer. The random dropout layer may be a DropOut layer, and there is no restriction on the type of the random dropout layer. It should be noted that FIG8D is only an example of a multi-layer perceptron, and there is no restriction on the structure of the multi-layer perceptron.
[0187] The output features of the second attention submodule are determined based on the input features of the second attention submodule and the features after multi-layer perception. For example, the output features are obtained by performing matrix addition on the input features and the features after multi-layer perception.
[0188] At this point, the processing process of the second attention sub-module is completed, and the output features of the second attention sub-module are obtained.
[0189] Example 13: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on Example 6 / Example 7. In Example 6 and Example 7, the cascaded hybrid attention module includes a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, the input features of the second attention submodule can be layer-normalized to obtain layer-normalized features; the layer-normalized features can be subjected to a first linear operation to obtain features after the first linear operation; the features after the first linear operation can be activated to obtain activated features; the activated features can be subjected to a second linear operation to obtain features after the second linear operation; the output features of the second attention submodule can be determined based on the input features of the second attention submodule and the features after the second linear operation.
[0190] 8E is a schematic diagram of the structure of the second attention submodule, which is only an example of the second attention submodule. The structure of the second attention submodule is not limited, and the second attention submodule of FIG. 8E is used as an example for explanation.
[0191] Exemplarily, the second attention submodule may include a layer normalization layer, which may perform layer normalization on the input features of the second attention submodule to obtain layer-normalized features. The second attention submodule may include a linear layer 1, which may perform a first linear operation on the layer-normalized features to obtain features after the first linear operation.
[0192] The second attention submodule may include an activation layer, which may be used to perform an activation operation on the features after the first linear operation to obtain the activated features. The activation layer may be a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and there is no restriction on the type of activation layer. For example, a Relu layer may be selected as the activation layer. The second attention submodule may include a linear layer 2, which may be used to perform a second linear operation on the activated features to obtain the features after the second linear operation.
[0193] The output features of the second attention submodule are determined based on the input features of the second attention submodule and the features after the second linear operation. For example, the output features are obtained by performing matrix addition on the input features and the features after the second linear operation.
[0194] At this point, the processing process of the second attention sub-module is completed, and the output features of the second attention sub-module are obtained.
[0195] Example 14: This example further describes the second attention submodule involved in the cascaded hybrid attention module based on Example 6 / Example 7. In Example 6 and Example 7, the cascaded hybrid attention module includes a second attention submodule, which is used to perform a second processing on the input features of the second attention submodule to obtain the output features of the second attention submodule. For example, a downsampling operation is performed on the input features of the second attention submodule to obtain downsampled features; a residual convolution operation is performed on the downsampled features to obtain residual convolution features; an upsampling operation is performed on the residual convolution features to obtain upsampled features; an activation operation is performed on the upsampled features to obtain activated features; a residual convolution operation is performed on the input features to obtain residual convolution features. The output features of the second attention submodule are determined based on the input features, the activated features, and the convolution features.
[0196] 8F is a schematic diagram of the structure of the second attention submodule, which is only an example of the second attention submodule. The structure of the second attention submodule is not limited, and the second attention submodule of FIG. 8F is used as an example for explanation.
[0197] Exemplarily, the second attention submodule may include a downsampling layer, which may be a convolutional layer with a kernel size of 3x3 and a stride of 2, or a convolutional layer with a kernel size of 3x3 and a stride of 4, or a convolutional layer with a kernel size of 4x4 and a stride of 2, or a convolutional layer with a kernel size of 4x4 and a stride of 4, or a convolutional layer with a kernel size of 5x5 and a stride of 2, or a convolutional layer with a kernel size of 5x5 and a stride of 4, or the downsampling layer may also be an inverse pixel shuffle layer. There is no restriction on the structure of this downsampling layer, such as a convolutional layer with a kernel size of 3x3 and a stride of 2 may be selected as the downsampling layer. On this basis, the input features of the second attention submodule may be downsampled by the downsampling layer to obtain the downsampled features, and there is no restriction on the implementation of this downsampling operation.
[0198] The second attention submodule can include M residual convolution layers, where M can be a positive integer, such as 1, 2, 3, 4, etc., and can perform residual convolution operations on the downsampled features through M residual convolution layers to obtain residual convolution features. For example, the downsampled features are subjected to residual convolution operations through the first residual convolution layer, and the results are subjected to residual convolution operations through the second residual convolution layer, and so on, until the last residual convolution layer outputs the residual convolution features.
[0199] Referring to FIG8G , which is a schematic diagram of the structure of a residual convolution layer, the residual convolution layer may include a convolution layer 1, an activation layer, and a convolution layer 2. The input features of the residual convolution layer may be subjected to a first convolution process by the convolution layer 1 to obtain features after the first convolution process. The features after the first convolution process may be activated by the activation layer to obtain features after the activation process. The features after the activation process may be subjected to a second convolution process by the convolution layer 2 to obtain features after the second convolution process. Then, the input features of the residual convolution layer and the features after the second convolution process are matrix-added to obtain the output features of the residual convolution layer.
[0200] Convolutional layer 1 can be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without any restrictions. For example, a 3x3 convolutional layer is selected. Convolutional layer 2 can be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without any restrictions. For example, a 3x3 convolutional layer is selected. The activation layer can be a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, without any restrictions. For example, a Relu layer or a sigmoid layer is selected.
[0201] Referring to FIG8H , another schematic diagram of the structure of the residual convolution layer is shown. The residual convolution layer may include a convolution layer and an activation layer. The convolution layer may convolve the input features of the residual convolution layer to obtain convolution-processed features. The activation layer may activate the convolution-processed features to obtain activation-processed features. Then, the input features of the residual convolution layer and the activation-processed features are matrix-added to obtain the output features of the residual convolution layer.
[0202] The convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, a 5x5 convolutional layer, or a 7x7 convolutional layer, without any restrictions. For example, a 3x3 convolutional layer can be selected. The activation layer can be a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, without any restrictions. For example, a Relu layer or a sigmoid layer can be selected.
[0203] The second attention submodule may include an upsampling layer, which may be a deconvolution layer with a kernel size of 3x3 and a stride of 2, or a deconvolution layer with a kernel size of 3x3 and a stride of 4, or a deconvolution layer with a kernel size of 4x4 and a stride of 2, or a deconvolution layer with a kernel size of 4x4 and a stride of 4, or a deconvolution layer with a kernel size of 5x5 and a stride of 2, or a deconvolution layer with a kernel size of 5x5 and a stride of 4, or the upsampling layer may also be a pixel reassembly layer. There is no restriction on the structure of the upsampling layer, such as a deconvolution layer with a kernel size of 4x4 and a stride of 2 may be selected as the upsampling layer. On this basis, the features after residual convolution (i.e., the features obtained after residual convolution operation by M residual convolution layers) may be upsampled by the upsampling layer to obtain the upsampled features, and there is no restriction on the implementation of the upsampling operation.
[0204] The second attention submodule may include an activation layer, which may be a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. The structure of the activation layer is not limited, and a Relu layer may be selected. Based on this, the activation layer may be used to activate the upsampled features to obtain activated features.
[0205] The second attention submodule can include N residual convolution layers, where N can be a positive integer, such as 1, 2, 3, 4, etc. N can be the same as or different from M. The input features of the second attention submodule can be subjected to residual convolution operations through N residual convolution layers to obtain convolution features after residual convolution. For example, the input features of the second attention submodule are subjected to residual convolution operations through the first residual convolution layer, and the obtained results are subjected to residual convolution operations through the second residual convolution layer, and so on, until the last residual convolution layer outputs the convolution features after residual convolution.
[0206] Referring to FIG8G , a schematic diagram of the structure of a residual convolution layer is shown, which may include a convolution layer 1, an activation layer, and a convolution layer 2. Referring to FIG8H , another schematic diagram of the structure of a residual convolution layer is shown, which may include a convolution layer and an activation layer. Any of the above residual convolution layers may be used to perform the residual convolution operation, which will not be described in detail here.
[0207] As shown in FIG8F , after obtaining the convolution features after residual convolution (i.e., the features output after residual convolution operation through N residual convolution layers) and the features after activation (i.e., the features output after activation operation through the activation layer), the output features of the second attention submodule can be determined based on the input features of the second attention submodule, the features after activation, and the convolution features after residual convolution. For example, the activated features and the convolution features after residual convolution can be matrix multiplied, and the matrix addition of the multiplication result and the input features of the second attention submodule can be performed to obtain the output features of the second attention submodule.
[0208] At this point, the processing process of the second attention sub-module is completed, and the output features of the second attention sub-module are obtained.
[0209] In Examples 12 to 14, several examples of the second attention sub-module are given. This application does not limit the structure of the second attention sub-module, and the input features can be subjected to a second processing to obtain the output features.
[0210] Example 15: The implementation of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the repeated contents are not repeated here. The synthesis transformation network involved is described in detail below.
[0211] In Example 1, Example 2, Example 3 and Example 4, a synthetic transformation network is involved, and the synthetic transformation network may include at least an attention module, and the attention module may be a first attention submodule. Of course, in addition to the first attention submodule, the synthetic transformation network may also include other network layers. In this embodiment, there is no restriction on the structure of the synthetic transformation network, as long as the synthetic transformation network includes the first attention submodule. For example, the synthetic transformation network may also include at least one deconvolution layer, and the first attention submodule may be located behind any deconvolution layer. Or, the first attention submodule may be located behind each deconvolution layer. Or, the first attention submodule may be located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers).
[0212] Exemplarily, the first attention submodule is used to perform a first processing on the input features of the first attention submodule to obtain the output features of the first attention submodule. In one possible embodiment, the structure of the first attention submodule can be shown in Example 8. In another possible embodiment, the structure of the first attention submodule can be shown in Example 9. In another possible embodiment, the structure of the first attention submodule can be shown in Example 10. In another possible embodiment, the structure of the first attention submodule can be shown in Example 11. Of course, the above are just a few examples of the first attention submodule, and the structure of the first attention submodule is not limited.
[0213] Example 16: The implementation of this example is the same as that of Example 1, Example 2, Example 3 or Example 4, and the repeated contents are not repeated here. The synthesis transformation network involved is described in detail below.
[0214] In Example 1, Example 2, Example 3 and Example 4, a synthetic transformation network is involved, and the synthetic transformation network may include at least an attention module, and the attention module may be a second attention submodule. Of course, in addition to the second attention submodule, the synthetic transformation network may also include other network layers. In this embodiment, there is no restriction on the structure of the synthetic transformation network, as long as the synthetic transformation network includes the second attention submodule. For example, the synthetic transformation network may also include at least one deconvolution layer, and the second attention submodule may be located behind any deconvolution layer. Or, the second attention submodule is located behind each deconvolution layer. Or, the second attention submodule is located behind some deconvolution layers (such as K deconvolution layers among all deconvolution layers, K is greater than 1, and K is less than the total number of deconvolution layers).
[0215] Exemplarily, the second attention submodule is configured to perform a second processing on the input features of the second attention submodule to obtain output features of the second attention submodule. In one possible implementation, the structure of the second attention submodule can be as shown in Example 12. In another possible implementation, the structure of the second attention submodule can be as shown in Example 13. In another possible implementation, the structure of the second attention submodule can be as shown in Example 14. Of course, the above are only a few examples of the second attention submodule, and the structure of the second attention submodule is not limited.
[0216] It can be seen from the above technical solutions that in the embodiment of the present application, for the neural network-based coding and decoding technology, a synthetic transformation network with an attention mechanism is proposed. The synthetic transformation network includes an attention module, and the attention module is a cascaded hybrid attention module. When the synthetic transformation network is realized by the cascaded hybrid attention module, while ensuring the quality of the synthetic image, the network complexity and computational complexity are effectively reduced, and the decoding performance is improved. While reducing the complexity, the quality of the reconstructed image block is effectively guaranteed, and the encoding performance and decoding performance are improved. By replacing the residual non-local attention module with a hybrid attention module (cascaded hybrid attention module), the computational complexity of the decoder is greatly reduced. For example, the second attention submodule in the hybrid attention module only occupies the computational complexity of 21K floating-point multiplication and addition operations / pixel, and the complexity of the first attention submodule is also much lower than that of the residual non-local attention module. Furthermore, the second attention submodule serves as the attention mechanism in the residual non-local attention module. Thanks to the introduction of a high-performance transformer structure, it achieves excellent performance. The 7x7 convolution operation in the second attention submodule introduces non-local information into the residual non-local attention module, and the random dropout layer in the second attention submodule increases network robustness. These features allow the cascaded hybrid attention module to reduce computational complexity without excessive performance loss.
[0217] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 17 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 16 can be implemented in combination.
[0218] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.
[0219] Based on the same concept as the above method, a decoding device is also proposed in an embodiment of the present application, which is applied to the decoding end. The decoding device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding methods in the above embodiments 1 to 16, that is, the processing flow of the decoding end.
[0220] For example, in one possible embodiment, a decoder is configured to: decode a code stream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode another code stream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; input the reconstruction features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.
[0221] Based on the same concept as the above method, an encoding device is also proposed in an embodiment of the present application, which is applied to the encoding end, and the encoding device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 16, that is, the processing flow of the encoding end.
[0222] For example, in one possible embodiment, the encoder is configured to: decode a code stream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode another code stream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; input the reconstruction features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.
[0223] Based on the same concept as the above method, the decoding end device (also called a video decoder) provided in the embodiments of the present application, from a hardware perspective, has a hardware architecture diagram as shown in Figure 9A . The decoding end device includes a processor 901 and a machine-readable storage medium 902 , wherein the machine-readable storage medium 902 stores machine-executable instructions that can be executed by the processor 901 ; the processor 901 is configured to execute the machine-executable instructions to implement the decoding methods of the above embodiments 1-17 of the present application.
[0224] The machine-readable storage medium 902 can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be random access memory (RAM), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.
[0225] Based on the same concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiments of the present application, from a hardware perspective, has a hardware architecture diagram as shown in FIG9B . The encoding end device includes a processor 911 and a machine-readable storage medium 912 , wherein the machine-readable storage medium 912 stores machine-executable instructions that can be executed by the processor 911 ; the processor 911 is configured to execute the machine-executable instructions to implement the encoding methods of the above embodiments 1-17 of the present application.
[0226] The machine-readable storage medium 912 can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be RAM, volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard drive), a solid-state drive, any type of storage disk (such as a CD-ROM, DVD, etc.), or similar storage media, or a combination thereof.
[0227] Based on the same concept as the above method, an embodiment of the present application provides an electronic device, comprising: a processor and a machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the decoding method or encoding method of the above embodiments 1-16 of the present application.
[0228] Based on the same concept as the above method, an embodiment of the present application further provides a machine-readable storage medium, wherein the machine-readable storage medium stores a plurality of computer instructions. When the computer instructions are executed by a processor, the decoding method or encoding method of the above embodiments 1-16 of the present application are implemented, such as the decoding method or encoding method in each of the above embodiments. For example, the machine-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0229] Based on the same concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.
[0230] Based on the same concept as the above method, a decoding device is also proposed in an embodiment of the present application, which can be applied to the decoding end. The decoding device includes: a decoding module, which is used to decode a code stream corresponding to the current image block to obtain the coefficient hyperparameter feature corresponding to the current image block; determine the probability distribution parameters based on the coefficient hyperparameter feature, and decode another code stream corresponding to the current image block based on the probability distribution parameters to obtain the reconstruction feature corresponding to the current image block; a processing module, which is used to input the reconstruction feature into a synthetic transformation network to obtain the reconstructed image block corresponding to the current image block; wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module or; wherein, the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
[0231] Exemplarily, if the attention module is a cascaded hybrid attention module, the processing module is further used to perform a first processing on the input features of the first attention sub-module through the first attention sub-module to obtain the output features of the first attention sub-module, and the output features of the first attention sub-module are the input features of the second attention sub-module; wherein the first processing is at least one of layer normalization, convolution operation and dimensionality transformation operation; and the input features of the second attention sub-module are second processed through the second attention sub-module to obtain the output features of the second attention sub-module; wherein the second processing is at least one of depthwise separable convolution operation, layer normalization, multi-layer perception operation, linear operation, activation operation, downsampling operation, residual convolution operation and upsampling operation.
[0232] Exemplarily, the processing module performs a first processing on the input features of the first attention sub-module through the first attention sub-module, and when obtaining the output features of the first attention sub-module, it is used to: perform layer normalization on the input features of the first attention sub-module to obtain layer-normalized features; perform a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector; perform a dimension transformation operation on the query vector, the key vector and the value vector respectively to obtain a dimensionally transformed query vector, a dimensionally transformed key vector and a dimensionally transformed value vector; determine an attention weight based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determine a corrected feature corresponding to the input feature based on the attention weight and the dimensionally transformed value vector; determine the output feature of the first attention sub-module based on the input feature and the corrected feature.
[0233] Exemplarily, the processing module performs a first processing on the input features of the first attention submodule through the first attention submodule to obtain the output features of the first attention submodule, and is specifically used to: perform layer normalization on the input features of the first attention submodule to obtain layer-normalized features; perform window division on the layer-normalized features to obtain multiple small-size features; for each small-size feature, perform a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; and splice the small-size query features corresponding to the multiple small-size features to obtain a query vector; The small-size key features corresponding to the multiple small-size features are spliced to obtain a key vector; the small-size value features corresponding to the multiple small-size features are spliced to obtain a value vector; the query vector, the key vector and the value vector are respectively subjected to dimension transformation operations to obtain a dimension-transformed query vector, a dimension-transformed key vector and a dimension-transformed value vector; the attention weight is determined based on the dimension-transformed query vector and the dimension-transformed key vector, and the correction feature corresponding to the input feature is determined based on the attention weight and the dimension-transformed value vector; the output feature of the first attention submodule is determined based on the input feature and the correction feature.
[0234] Exemplarily, the processing module performs a first processing on the input features of the first attention sub-module through the first attention sub-module, and when obtaining the output features of the first attention sub-module, is specifically used to: perform layer normalization on the input features of the first attention sub-module to obtain layer-normalized features; perform a three-way convolution operation on the layer-normalized features to obtain a query vector, a key vector and a value vector; determine an attention weight based on the query vector and the key vector, and determine a correction feature corresponding to the input feature based on the attention weight and the value vector; and determine the output feature of the first attention sub-module based on the input feature and the correction feature.
[0235] Exemplarily, the processing module performs a first processing on the input features of the first attention sub-module through the first attention sub-module, and when obtaining the output features of the first attention sub-module, is specifically used for: performing layer normalization on the input features of the first attention sub-module to obtain layer-normalized features; performing window division on the layer-normalized features to obtain multiple small-size features; for each small-size feature, performing a three-way convolution operation on the small-size feature to obtain a small-size query feature, a small-size key feature and a small-size value feature corresponding to the small-size feature; splicing the small-size query features corresponding to the multiple small-size features to obtain a query vector; splicing the small-size key features corresponding to the multiple small-size features to obtain a key vector; splicing the small-size value features corresponding to the multiple small-size features to obtain a value vector; determining the attention weight based on the query vector and the key vector, and determining the correction feature corresponding to the input feature based on the attention weight and the value vector; and determining the output feature of the first attention sub-module based on the input feature and the correction feature.
[0236] Exemplarily, the processing module performs a second processing on the input features of the second attention sub-module through the second attention sub-module to obtain the output features of the second attention sub-module, which is specifically used for: performing layer normalization on the input features of the second attention sub-module to obtain layer-normalized features; performing a first linear operation on the layer-normalized features to obtain features after the first linear operation; performing an activation operation on the features after the first linear operation to obtain activated features; performing a second linear operation on the activated features to obtain features after the second linear operation; and determining the output features of the second attention sub-module based on the input features and the features after the second linear operation.
[0237] Exemplarily, the processing module performs a second processing on the input features of the second attention sub-module through the second attention sub-module to obtain the output features of the second attention sub-module, which is specifically used for: downsampling the input features of the second attention sub-module to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolution features; upsampling the residual convolution features to obtain upsampled features; activating the upsampled features to obtain activated features; performing a residual convolution operation on the input features to obtain residual convolution features, and determining the output features of the second attention sub-module based on the input features of the second attention sub-module, the activated features and the convolution features.
[0238] Exemplarily, the synthetic transformation network further includes at least one deconvolution layer; wherein: the attention module is located behind one of the deconvolution layers.
[0239] Exemplarily, the synthetic transformation network further includes a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer; the synthetic transformation network includes at least one attention module, one of which is located behind the first deconvolution layer.
[0240] Based on the same concept as the above method, an encoding device is also proposed in an embodiment of the present application, which is applied to the encoding end, and the device includes: a decoding module, which is used to decode a code stream corresponding to the current image block to obtain the coefficient hyperparameter feature corresponding to the current image block; determine the probability distribution parameters based on the coefficient hyperparameter feature, and decode another code stream corresponding to the current image block based on the probability distribution parameters to obtain the reconstruction feature corresponding to the current image block; a processing module, which is used to input the reconstruction feature into a synthetic transformation network to obtain the reconstructed image block corresponding to the current image block; wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein, the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.
[0241] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The foregoing is merely an embodiment of the present application and is not intended to limit the present application.
[0242] For those skilled in the art, various modifications and variations are possible in this application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of the claims of this application.
Claims
1. A decoding method, It is characterized in that Applied to a decoding end, the method comprises: Decoding a bitstream corresponding to the current image block to obtain coefficient hyperparameter features corresponding to the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, and decoding another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; Wherein, the synthetic transformation network comprises at least an attention module, and the attention module is a cascaded hybrid attention module; Among them, the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.
2. The method according to claim 1, It is characterized in that If the attention module is a cascaded hybrid attention module, the method further includes: Performing a first processing on the input features of the first attention submodule through the first attention submodule to obtain output features of the first attention submodule, and the output features of the first attention submodule are input features of the second attention submodule; wherein the first processing is at least one of layer normalization, convolution operation and dimensionality transformation operation; The second attention sub-module performs a second processing on the input features of the second attention sub-module to obtain the output features of the second attention sub-module; wherein the second processing is at least one of a depth-separable convolution operation, a layer normalization, a multi-layer perception operation, a linear operation, an activation operation, a downsampling operation, a residual convolution operation, and an upsampling operation.
3. The method according to claim 2, It is characterized in that Performing a first processing on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule includes: Performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; Performing a three-way convolution operation on the normalized features of the layer to obtain a query vector, a key vector, and a value vector; Performing dimension transformation operations on the query vector, the key vector, and the value vector respectively to obtain a dimension-transformed query vector, a dimension-transformed key vector, and a dimension-transformed value vector; Determine an attention weight based on the query vector after dimension transformation and the key vector after dimension transformation, and determine a correction feature corresponding to the input feature based on the attention weight and the value vector after dimension transformation; The output feature of the first attention submodule is determined based on the input feature and the modified feature.
4. The method according to claim 2, It is characterized in that Performing a first processing on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule includes: Performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; Performing window division on the normalized features of the layer to obtain multiple small-size features; For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector; Performing dimension transformation operations on the query vector, the key vector, and the value vector respectively to obtain a dimension-transformed query vector, a dimension-transformed key vector, and a dimension-transformed value vector; Determine an attention weight based on the dimensionally transformed query vector and the dimensionally transformed key vector, and determine a correction feature corresponding to the input feature based on the attention weight and the dimensionally transformed value vector; The output feature of the first attention submodule is determined based on the input feature and the modified feature.
5. The method according to claim 2, It is characterized in that Performing a first processing on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule includes: Performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; Performing a three-way convolution operation on the normalized features of the layer to obtain a query vector, a key vector, and a value vector; determining an attention weight based on the query vector and the key vector, and determining a correction feature corresponding to the input feature based on the attention weight and the value vector; The output feature of the first attention submodule is determined based on the input feature and the modified feature.
6. The method according to claim 2, It is characterized in that Performing a first processing on the input features of the first attention submodule by the first attention submodule to obtain output features of the first attention submodule includes: Performing layer normalization on the input features of the first attention submodule to obtain layer normalized features; Performing window division on the normalized features of the layer to obtain multiple small-size features; For each small-size feature, a three-way convolution operation is performed on the small-size feature to obtain a small-size query feature, a small-size key feature, and a small-size value feature corresponding to the small-size feature; the small-size query features corresponding to the multiple small-size features are concatenated to obtain a query vector; the small-size key features corresponding to the multiple small-size features are concatenated to obtain a key vector; the small-size value features corresponding to the multiple small-size features are concatenated to obtain a value vector; determining an attention weight based on the query vector and the key vector, and determining a correction feature corresponding to the input feature based on the attention weight and the value vector; The output feature of the first attention submodule is determined based on the input feature and the modified feature.
7. The method according to claim 2, It is characterized in that Performing a second processing on the input features of the second attention submodule by the second attention submodule to obtain output features of the second attention submodule includes: Performing layer normalization on the input features of the second attention submodule to obtain layer normalized features; Performing a first linear operation on the normalized features of the layer to obtain features after the first linear operation; Performing an activation operation on the features after the first linear operation to obtain activated features; Performing a second linear operation on the activated features to obtain features after the second linear operation; Determine the output feature of the second attention submodule based on the input feature and the feature after the second linear operation.
8. The method according to claim 2, It is characterized in that Performing a second processing on the input features of the second attention submodule by the second attention submodule to obtain output features of the second attention submodule includes: Performing a downsampling operation on the input features of the second attention submodule to obtain downsampled features; performing a residual convolution operation on the downsampled features to obtain residual convolution features; performing an upsampling operation on the residual convolution features to obtain upsampled features; performing an activation operation on the upsampled features to obtain activated features; Perform a residual convolution operation on the input feature to obtain a convolution feature after residual convolution; The output feature of the second attention submodule is determined based on the input feature, the activated feature and the convolution feature.
9. The method according to any one of claims 1 to 8, It is characterized in that The synthetic transformation network also includes at least one deconvolution layer; wherein: The attention module is placed after one of the deconvolution layers.
10. The method according to claim 9, It is characterized in that The synthetic transformation network also includes a residual layer, a first deconvolution layer, a first cropping layer, a first residual activation layer, a second deconvolution layer, a second cropping layer, a second residual activation layer, a third deconvolution layer, a third cropping layer, a third residual activation layer, a fourth deconvolution layer, and a fourth cropping layer; The synthetic transformation network includes at least one attention module, wherein one attention module is located after the first deconvolution layer.
11. A coding method, It is characterized in that Applied to the encoding end, the method includes: Decoding a code stream corresponding to the current image block to obtain coefficient hyper-parameter features corresponding to the current image block; Determine a probability distribution parameter based on the coefficient hyperparameter feature, and decode another bitstream corresponding to the current image block based on the probability distribution parameter to obtain a reconstruction feature corresponding to the current image block; Inputting the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; Wherein, the synthetic transformation network includes at least an attention module, and the attention module is a cascaded hybrid attention module; wherein, the cascaded hybrid attention module includes a first attention sub-module and a second attention sub-module, and the first attention sub-module and the second attention sub-module are two serial sub-modules.
12. A decoding device, It is characterized in that Applied to a decoding end, the device comprises: A decoding module, configured to decode a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; A processing module is used to input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
13. An encoding device, It is characterized in that Applied to the encoding end, the device includes: A decoding module, configured to decode a bitstream corresponding to a current image block to obtain coefficient hyperparameter features corresponding to the current image block; determine probability distribution parameters based on the coefficient hyperparameter features, and decode another bitstream corresponding to the current image block based on the probability distribution parameters to obtain reconstruction features corresponding to the current image block; A processing module is used to input the reconstructed features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; wherein the synthetic transformation network at least includes an attention module, and the attention module is a cascaded hybrid attention module; wherein the cascaded hybrid attention module includes a first attention submodule and a second attention submodule, and the first attention submodule and the second attention submodule are two serial submodules.
14. A decoding terminal device, It is characterized in that include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 10.
15. A coding terminal device, It is characterized in that include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 11.
16. An electronic device, It is characterized in that include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-11.
17. A machine-readable storage medium, It is characterized in that The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by a processor, the method described in any one of claims 1 to 11 is implemented.
18. A computer application, wherein when the computer application is executed by a processor, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Patent Citations
Video image coding and decoding method and related equipment
CN115118972A
Image compression method and device based on deep learning
CN115484459A
Image encoding method, image decoding method, image encoding device, image decoding device and computer storage medium
CN116016954A
Image compression method and apparatus
US20220286696A1
