A decoding, encoding method, apparatus and device thereof

Abstract

This application provides a decoding and encoding method, apparatus, and device. The decoding method includes: decoding a first bitstream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, determining probability distribution parameters based on the coefficient hyperparameter features of that stage sub-block; decoding a second bitstream of the current image block based on the probability distribution parameters to obtain residual features of that stage sub-block; determining reconstruction features of that stage sub-block based on the residual features and the mean features of that stage sub-block; and determining the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block. The technical solution of this application can improve encoding and decoding performance.

Description

Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a decoding and encoding method, apparatus and device thereof. Background Technology

[0002] To save space, video images are encoded before transmission. Complete video encoding includes processes such as prediction, transform, quantization, entropy coding, and filtering. The prediction process can be divided into intra-frame prediction and inter-frame prediction. Inter-frame prediction utilizes temporal correlation to predict the current pixel using pixels from neighboring encoded images, effectively removing temporal redundancy. Intra-frame prediction utilizes spatial correlation to predict the current pixel using pixels from the encoded blocks of the current frame, removing spatial redundancy.

[0003] With the rapid development of deep learning, it has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning is also gradually being applied in the field of encoding and decoding, where neural networks can be used to encode and decode images. Although neural network-based encoding and decoding methods have shown great performance potential, they still suffer from problems such as poor encoding performance, poor decoding performance, and high complexity. Summary of the Invention

[0004] In view of this, this application provides a decoding and encoding method, apparatus and device, to improve encoding and decoding performance.

[0005] This application provides a decoding method applied at a decoding end, the method comprising:

[0006] Decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block;

[0007] For each stage sub-block, probability distribution parameters are determined based on the coefficient hyperparameter features of that stage sub-block. The second bitstream of the current image block is then decoded based on the probability distribution parameters to obtain the residual features of that stage sub-block.

[0008] The reconstruction characteristics of the sub-block in this stage are determined based on the residual characteristics and the mean characteristics of the sub-block in this stage.

[0009] The reconstructed image block corresponding to the current image block is determined based on the reconstruction features of each stage sub-block.

[0010] This application provides an encoding method applied at an encoding end, the method comprising:

[0011] The current image block is input into the analysis and transformation network to obtain the feature block corresponding to the current image block;

[0012] The feature block is divided into multiple stage sub-blocks of features to be encoded;

[0013] For each stage sub-block corresponding to the current image block, the coefficient hyperparameter features of that stage sub-block are obtained, and the coefficient hyperparameter features of that stage sub-block are encoded into the first bitstream of the current image block;

[0014] The residual characteristics of the sub-block in this stage are determined based on the features to be encoded and the mean features of the sub-block in this stage.

[0015] The probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block in this stage, and the residual features of the sub-block in this stage are encoded into the second bitstream of the current image block based on the probability distribution parameters.

[0016] This application provides a decoding method applied at a decoding end, the method comprising:

[0017] Decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of the current image block;

[0018] Based on the coefficient hyperparameter features, probability distribution parameters are determined, and the second bitstream of the current image block is decoded based on the probability distribution parameters to obtain the residual features of the current image block. Based on the residual features, the reconstruction features of the current image block are determined.

[0019] The auxiliary bitstream corresponding to the current image block is decoded to obtain the bitrate control parameters corresponding to the current image block. The reconstructed features and the bitrate control parameters are then input into the synthesis transform network to obtain the reconstructed image block corresponding to the current image block.

[0020] This application provides a decoding device applied at a decoding end, the device comprising:

[0021] The decoding module is used to decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, the probability distribution parameters are determined based on the coefficient hyperparameter features of the stage sub-block, and the second bitstream of the current image block is decoded based on the probability distribution parameters to obtain the residual features of the stage sub-block.

[0022] The determination module is used to determine the reconstruction features of the sub-block at the current stage based on the residual features and the mean features of the sub-block at the current stage; and to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each sub-block at the current stage.

[0023] This application provides an encoding device applied at an encoding end, the device comprising:

[0024] The acquisition module is used to input the current image block into the analysis and transformation network to obtain the feature block corresponding to the current image block; and to split the feature block into multiple stage sub-blocks of features to be encoded.

[0025] The encoding module is used to obtain the coefficient hyperparameter features of each stage sub-block corresponding to the current image block, and encode the coefficient hyperparameter features of the stage sub-block into the first bitstream of the current image block;

[0026] The determination module is used to determine the residual features of the sub-block in the current stage based on the features to be encoded and the mean features of the sub-block in the current stage.

[0027] The encoding module is further configured to determine the probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block, and to encode the residual features of the stage sub-block into the second bitstream of the current image block based on the probability distribution parameters.

[0028] This application provides a decoding end device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0029] The processor is used to execute machine-executable instructions to implement the above-described decoding method.

[0030] This application provides an encoding terminal device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0031] The processor is used to execute machine-executable instructions to implement the above-described encoding method.

[0032] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the above-described decoding method; or, the processor is configured to execute the machine-executable instructions to implement the above-described encoding method.

[0033] This application provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, implement the above-described decoding method; or, implement the above-described encoding method.

[0034] As can be seen from the above technical solutions, this application proposes a variable-rate encoding and decoding scheme for neural network-based encoding and decoding technology. This improves parallelism, effectively saves feature storage cache, achieves higher bitrate control precision, and minimizes encoding performance loss, resulting in better encoding performance and bitrate control precision. By employing block encoding and decoding, peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is achieved. This allows the neural network to maintain low complexity while effectively ensuring the quality of reconstructed image blocks, improving encoding and decoding performance, and reducing complexity. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a three-dimensional feature matrix in one embodiment of this application;

[0036] Figure 2 This is a flowchart of a decoding method in one embodiment of this application;

[0037] Figure 3 This is a flowchart of an encoding method in one embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the processing procedure of the encoding end in one embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the processing procedure at the decoding end in one embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the processing procedure of the encoding end in one embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the processing procedure of the encoding end in one embodiment of this application;

[0042] Figures 8A-8H This is a schematic diagram of the mean prediction network in one embodiment of this application;

[0043] Figures 9A-9F This is a schematic diagram of aggregation and segmentation in one embodiment of this application;

[0044] Figures 10A-10C This is a schematic diagram of the structure of the synthetic transformation network in one embodiment of this application;

[0045] Figure 11A and Figure 11B This is a schematic diagram of the processing of a convolutional layer in one embodiment of this application;

[0046] Figure 12 This is a schematic diagram illustrating the decision-making process for the λ parameter in one embodiment of this application;

[0047] Figure 13A This is a hardware structure diagram of the decoding end device in one embodiment of this application;

[0048] Figure 13B This is a hardware structure diagram of the encoding end device in one embodiment of this application. Detailed Implementation

[0049] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments and claims of this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any or all possible combinations including one or more of the associated listed items. It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this application, and similarly, second information may also be referred to as first information, depending on the context. Furthermore, the word “if” as used can be interpreted as “when,” “in response to a determination,” or “when…”.

[0050] This application proposes a decoding method and an encoding method, which may involve the following concepts:

[0051] Entropy coding: Entropy coding is a coding process that follows the principle of entropy without losing any information. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods can include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0052] Neural Networks (NNs): Neural networks are artificial neural networks, a computational model composed of numerous interconnected nodes (or neurons). In a neural network, neurons can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units; the representation and processing of information are reflected in the connections between processing units. Neural networks are a non-programmed, brain-like information processing method. The essence of a neural network is to obtain a parallel and distributed information processing function through the transformations and dynamic behavior of the network, mimicking the information processing function of the human brain's nervous system to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and fully connected networks.

[0053] Convolutional Neural Networks (CNNs) are a type of feedforward neural network and one of the most representative network structures in deep learning. The artificial neurons in a CNN can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. The basic structure of a CNN consists of two layers: a feature extraction layer (also called a convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, extracting features from that local area. Once these local features are extracted, their positional relationship with other features is determined. The second layer is a feature mapping layer (also called an activation layer). Each computational layer of the neural network consists of multiple feature maps, each a plane where all neurons have equal weights. Feature mapping structures can use functions such as the Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, and GDN function as activation functions for the convolutional network. Furthermore, because neurons on a single feature map share weights, the number of free parameters in the network is reduced.

[0054] For example, one advantage of convolutional neural networks (CNNs) over image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. Another advantage of CNNs over ordinary neural networks is that ordinary neural networks use fully connected layers, meaning all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, however, avoid this difficulty through local connectivity and weight sharing.

[0055] Deconvolution: Also known as transposed convolution, deconvolution layers work similarly to convolutional layers. The main difference is that deconvolution layers use padding to make the output larger than the input (though they can also remain the same). If the stride is 1, the output size equals the input size; if the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0056] Generalization ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The goal of learning is to learn the patterns hidden behind data pairs. The trained network can also give appropriate outputs for data outside the learning set that have the same pattern. This ability can be called generalization ability.

[0057] Features: The features discussed in this application are three-dimensional feature matrices or tensors of size C*W*H. See [link to relevant documentation]. Figure 1 The diagram shows a schematic of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be either the input or the output of a neural network.

[0058] Rate-Distortion Optimized (RDBEM) principle: Two main metrics for evaluating coding efficiency are bitrate and PSNR (Peak Signal-to-Noise Ratio). A smaller bitrate results in a higher compression ratio, and a higher PSNR leads to better reconstructed image quality. In mode selection, the decision formula essentially evaluates both factors. For example, the cost of a mode is: J(mode) = D + λ*R, where D represents Distortion, typically measured using the SSE metric (Sum of Mean Squares of Differences between the Reconstructed Image Patch and the Source Image). Alternatively, the SAD metric (Sum of Absolute Differences between the Reconstructed Image Patch and the Source Image) can be used to consider the cost. λ is a Lagrange multiplier, and R is the actual number of bits required to encode the image patch in that mode, including the total number of bits needed for encoding mode information, motion information, residuals, etc. Using the RDBEM principle to compare and decide on coding modes during mode selection usually ensures optimal coding performance.

[0059] Numerous encoding tools have been proposed for various modules at the encoding end, and each tool often has multiple modes. The optimal encoding tool for different video sequences often differs. Therefore, during encoding, Rate-Distortion Optimization (RDO) is typically used to compare the encoding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information is transmitted by encoding marker information in the bitstream. Although this method introduces higher encoding complexity, it can adaptively select the optimal mode combination for different content to achieve the best encoding performance. The decoding end can obtain the relevant mode information by directly parsing the marker information, with minimal impact from complexity.

[0060] The decoding and encoding methods in the embodiments of this application will be described in detail below with reference to several examples.

[0061] Example 1: This application proposes a decoding method, see [link to example]. Figure 2 The diagram shown illustrates the flowchart of this decoding method, which can be applied to the decoding end (also known as a video decoder). This method may include:

[0062] Step 201: Decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block. The current image block is divided into multiple stage sub-blocks, meaning the current image block includes multiple stage sub-blocks.

[0063] Step 202: For each stage sub-block, determine the probability distribution parameters based on the coefficient hyperparameter features of that stage sub-block, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain the residual features of that stage sub-block.

[0064] Step 203: Determine the reconstruction features of the stage sub-block based on its residual features and mean features. The mean features of the stage sub-block are obtained based on its coefficient hyperparameter features and / or its reference features, such as based on the coefficient hyperparameter features, or based on both the coefficient hyperparameter features and the reference features.

[0065] Step 204: Determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block.

[0066] For example, for the first stage sub-block, the mean feature of the stage sub-block is obtained through the mean prediction network based on the coefficient hyperparameter features of the stage sub-block, or the configured default reference feature is obtained, and the mean feature of the stage sub-block is obtained through the mean prediction network based on the coefficient hyperparameter features and the default reference feature of the stage sub-block.

[0067] For the i-th stage sub-block (i > 1), obtain the reference features of the i-th stage sub-block based on the reconstruction features of the i-1 preceding stage sub-blocks; obtain the mean features of the stage sub-block through the mean prediction network based on the coefficient hyperparameter features and the reference features of the stage sub-block; wherein, the reference features include all reconstruction features of the preceding i-1 stage sub-blocks; or, some reconstruction features of the preceding i-1 stage sub-blocks; or, the reconstruction features of the i-1 stage sub-block.

[0068] For example, the mean prediction network may include a first prediction network, a second prediction network, and a prediction fusion network. Based on the coefficient hyperparameter features and reference features of the sub-block, the mean prediction network obtains the mean features of the sub-block. This may include: obtaining the first prediction feature corresponding to the coefficient hyperparameter features through the first prediction network; obtaining the second prediction feature corresponding to the reference features through the second prediction network; concatenating the first prediction feature and the second prediction feature and inputting them into the prediction fusion network; and processing the concatenated features through the prediction fusion network to obtain the mean features of the sub-block.

[0069] For example, obtaining the first predicted feature corresponding to the coefficient hyperparameter feature through the first prediction network may include: performing feature enhancement and upsampling operations on the coefficient hyperparameter feature through the first prediction network to obtain the first predicted feature. The feature enhancement operation may include: a convolution operation, or a convolution operation and an activation operation; the upsampling operation may include: a deconvolution operation, a cropping operation and an activation operation, or a deconvolution operation, a cropping operation, an activation operation and a convolution operation.

[0070] For example, obtaining the second predicted feature corresponding to the reference feature through the second prediction network may include: concatenating all reconstructed features in the reference feature according to the channel dimension; performing a convolution operation on the concatenated features to obtain the second predicted feature; or, summing all reconstructed features in the reference feature and performing a convolution operation on the summed features to obtain the second predicted feature. The reference feature includes all reconstructed features of the preceding i-1 stage sub-blocks; or, some reconstructed features of the preceding i-1 stage sub-blocks; or, the reconstructed features of the i-1th stage sub-block.

[0071] For example, the mean feature of the sub-block at this stage can be obtained by processing the spliced ​​features through a predictive fusion network, which may include, but is not limited to: performing convolution operations and at least one fusion operation on the spliced ​​features through the predictive fusion network to obtain the mean feature of the sub-block at this stage; wherein, the fusion operation includes activation operations and convolution operations.

[0072] For example, after determining the reconstruction features of a stage sub-block based on its residual features and mean features, feature enhancement can be performed on the reconstruction features of the stage sub-block to obtain enhanced reconstruction features; wherein, the enhanced reconstruction features are used to determine the mean features of the stage sub-block and the enhanced reconstruction features are used to determine the reconstructed image block.

[0073] For example, determining the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block may include, but is not limited to: performing feature aggregation on the reconstruction features of each stage sub-block to obtain aggregated features, and inputting these aggregated features into a synthetic transform network to obtain the reconstructed image block corresponding to the current image block. Alternatively, performing feature aggregation on the reconstruction features of each stage sub-block to obtain aggregated features, and then dividing these aggregated features into blocks to obtain multiple block features; inputting each block feature into a synthetic transform network to obtain the block reconstructed image block corresponding to that block feature, and then merging the block reconstructed image blocks corresponding to multiple block features to obtain the reconstructed image block corresponding to the current image block. Alternatively, inputting the reconstruction features of each stage sub-block into a synthetic transform network to obtain the block reconstructed image block corresponding to that stage sub-block, and then merging the block reconstructed image blocks corresponding to all stage sub-blocks to obtain the reconstructed image block corresponding to the current image block.

[0074] For example, feature aggregation can be performed on the reconstruction features of each stage sub-block to obtain aggregated features. This may include, but is not limited to: aggregating the reconstruction features of each stage sub-block according to phase to obtain aggregated features; or aggregating the reconstruction features of each stage sub-block according to phase to obtain features aggregated by multiple phases, and then concatenating the features aggregated by multiple phases according to channels to obtain aggregated features.

[0075] For example, dividing the aggregated features into multiple block features may include: determining the target size of the block features; dividing the aggregated features into multiple block features in a top-to-bottom and left-to-right order based on the target size, with each block feature having the target size; or, determining the actual block size and overlap size of the block features, where the actual block size is the block size of the block features excluding the overlap portion, and the overlap size is the size of the overlap portion of adjacent blocks; dividing the aggregated features into multiple block features based on the actual block size and overlap size, with each block feature having the target size; wherein, if the actual block size is tile_c*tile_w*tile_h, and the overlap size is tile_c*padding_w*padding_h, then the target size is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h).

[0076] For example, merging multiple reconstructed image blocks corresponding to segmented features to obtain the reconstructed image block corresponding to the current image block may include, but is not limited to: sorting the multiple reconstructed image blocks corresponding to segmented features in a top-to-bottom and left-to-right order, and then sequentially stitching the sorted reconstructed image blocks to obtain the reconstructed image block corresponding to the current image block. Alternatively, determining the actual segment size and overlap size of each reconstructed image block; sorting the multiple reconstructed image blocks corresponding to segmented features in a top-to-bottom and left-to-right order; and sequentially stitching the sorted reconstructed image blocks corresponding to segmented features based on the actual segment size and overlap size to obtain the reconstructed image block corresponding to the current image block. Here, the reconstructed image block includes multiple reconstructed image blocks, the size of the non-overlapping portion of a reconstructed image block is the actual segment size of that reconstructed image block, and the size of the overlapping portion between a reconstructed image block and an adjacent reconstructed image block is the overlap size of that reconstructed image block. Specifically, for the overlapping portion of the left and right reconstructed image blocks, the value of the overlapping portion is the value of the left reconstructed image block, or the value of the overlapping portion is the average value of the two reconstructed image blocks; for the overlapping portion of the upper and lower reconstructed image blocks, the value of the overlapping portion is the value of the upper reconstructed image block, or the value of the overlapping portion is the average value of the two reconstructed image blocks.

[0077] For example, inputting the aggregated features into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block may include: decoding the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters of the current image block, and inputting the bitrate control parameters and the aggregated features into the synthetic transformation network to obtain the reconstructed image block corresponding to the current image block.

[0078] Each block feature is input into the synthesis transform network to obtain the block reconstructed image block corresponding to that block feature. This can include: decoding the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameter of each block feature, and inputting the block feature and the bitrate control parameter of that block feature into the synthesis transform network to obtain the block reconstructed image block corresponding to that block feature.

[0079] The reconstruction features of each stage sub-block are input into the synthesis transform network to obtain the block-reconstructed image block corresponding to that stage sub-block. This may include: decoding the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters of each stage sub-block, and inputting the reconstruction features of the stage sub-block and the bitrate control parameters of that stage sub-block into the synthesis transform network to obtain the block-reconstructed image block corresponding to that stage sub-block.

[0080] For example, inputting the bitrate control parameters and aggregated features into the synthesis transform network to obtain the reconstructed image block corresponding to the current image block may include: processing the aggregated features through the synthesis transform network to obtain a first feature; processing the bitrate control parameters through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0081] The block feature and its bit rate control parameter are input into the synthesis transform network to obtain the block reconstructed image block corresponding to the block feature. This can include: processing the block feature through the synthesis transform network to obtain a first feature; processing the bit rate control parameter through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the block reconstructed image block corresponding to the block feature based on the third feature.

[0082] The reconstruction features of the stage sub-block and the bit rate control parameters of the stage sub-block are input into the synthesis transform network to obtain the block-based reconstructed image block corresponding to the stage sub-block. This can include: processing the reconstruction features of the stage sub-block through the synthesis transform network to obtain a first feature; processing the bit rate control parameters through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the block-based reconstructed image block corresponding to the stage sub-block based on the third feature.

[0083] For example, the above execution order is merely an example for ease of description. In practical applications, the execution order between steps can be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the method may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments.

[0084] As can be seen from the above technical solutions, this application proposes a variable-rate encoding and decoding scheme for neural network-based encoding and decoding technology. This improves parallelism, effectively saves feature storage cache, achieves higher bitrate control precision, and minimizes encoding performance loss, resulting in better encoding performance and bitrate control precision. By employing block encoding and decoding, peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is achieved. This allows the neural network to maintain low complexity while effectively ensuring the quality of reconstructed image blocks, improving encoding and decoding performance, and reducing complexity.

[0085] Example 2: An encoding method is proposed in this application embodiment, see [link to example]. Figure 3 The diagram shown illustrates the flowchart of this encoding method, which can be applied to the encoding end (also known as a video encoder). This method may include:

[0086] Step 301: Input the current image block into the analysis and transformation network to obtain the feature block corresponding to the current image block.

[0087] Step 302: Divide the feature block into multiple stage sub-blocks of features to be encoded.

[0088] Step 303: For each stage sub-block corresponding to the current image block, obtain the coefficient hyperparameter features of that stage sub-block, and encode the coefficient hyperparameter features of that stage sub-block into the first bitstream of the current image block.

[0089] Step 304: Determine the residual features of the stage sub-block based on the features to be encoded and the mean features of the stage sub-block. The mean features of the stage sub-block are obtained based on the coefficient hyperparameter features and / or the reference features of the stage sub-block, such as based on the coefficient hyperparameter features, or based on both the coefficient hyperparameter features and the reference features.

[0090] Step 305: Determine the probability distribution parameters based on the coefficient hyperparameter features of the sub-block in this stage, and encode the residual features of the sub-block in this stage into the second bitstream of the current image block based on the probability distribution parameters.

[0091] For example, for the first stage sub-block, the mean feature of the stage sub-block is obtained through the mean prediction network based on the coefficient hyperparameter features of the stage sub-block, or the configured default reference feature is obtained, and the mean feature of the stage sub-block is obtained through the mean prediction network based on the coefficient hyperparameter features and the default reference feature of the stage sub-block.

[0092] For the i-th stage sub-block (i > 1), obtain the reference features of the i-th stage sub-block based on the reconstruction features of the i-1 preceding stage sub-blocks; obtain the mean features of the stage sub-block through the mean prediction network based on the coefficient hyperparameter features and the reference features of the stage sub-block; wherein, the reference features include all reconstruction features of the preceding i-1 stage sub-blocks; or, some reconstruction features of the preceding i-1 stage sub-blocks; or, the reconstruction features of the i-1 stage sub-block.

[0093] For example, the processing at the encoding end is similar to that at the decoding end, and the similarities will not be repeated here.

[0094] For example, the above execution order is merely an example for ease of description. In practical applications, the execution order between steps can be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the method may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; multiple steps described in this specification may also be combined into a single step in other embodiments.

[0095] As can be seen from the above technical solutions, this application proposes a variable-rate encoding and decoding scheme for neural network-based encoding and decoding technology. This improves parallelism, effectively saves feature storage cache, achieves higher bitrate control precision, and minimizes encoding performance loss, resulting in better encoding performance and bitrate control precision. By employing block encoding and decoding, peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is achieved. This allows the neural network to maintain low complexity while effectively ensuring the quality of reconstructed image blocks, improving encoding and decoding performance, and reducing complexity.

[0096] Example 3: For the processing procedures at the encoding end in Examples 1 and 2, please refer to... Figure 4 As shown, of course, Figure 4 This is just one example of the processing procedure at the encoding end, and no restrictions are imposed on this processing procedure.

[0097] After obtaining the current image block x (which can be the original image block x, i.e., the input image block), the encoding end can perform analysis and transformation on the current image block x through an analysis and transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Specifically, performing feature transformation on the current image block x through the analysis and transformation network means transforming the current image block x to image features y in the latent domain, thereby facilitating all subsequent processes to be performed in the latent domain.

[0098] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the encoding and decoding process for image blocks can also be directly applied to the image.

[0099] After obtaining image features y, the encoder performs a coefficient hyperparameter feature transformation on y to obtain coefficient hyperparameter features z. For example, image features y can be input into a hyperparameter coding network (i.e., a neural network), which then performs the coefficient hyperparameter feature transformation on y to obtain coefficient hyperparameter features z. The hyperparameter coding network can be a trained neural network, and its training process is not restricted, as long as it can perform the coefficient hyperparameter feature transformation on image features y. The latent domain image features y, after passing through the hyperparameter coding network, yield the hyper-prior latent information z.

[0100] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the quantized hyperparameter feature corresponding to the coefficient hyperparameter feature z, i.e. Figure 4 The Q-operation in the code represents the quantization process. After obtaining the hyperparameter quantization features corresponding to the coefficient hyperparameter features z, these features are encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image patch. Figure 4 The AE operation in the code represents the encoding process, such as entropy encoding. Alternatively, the encoder can directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image patch. The hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.

[0101] After obtaining the Bitstream#1 corresponding to the current image block, the encoding end can send the Bitstream#1 corresponding to the current image block to the decoding end. For the processing of the Bitstream#1 corresponding to the current image block by the decoding end, please refer to the following embodiments.

[0102] After obtaining Bitstream#1 corresponding to the current image block, the encoding end can also decode Bitstream#1 to obtain the hyperparameter quantization features, i.e. Figure 4 In the diagram, AD represents the decoding process. Then, the hyperparameter quantization features are dequantized to obtain the coefficient hyperparameter features z_hat. The coefficient hyperparameter features z_hat and the coefficient hyperparameter features z can be the same or different. Figure 4 The IQ operation in the code is an inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder can decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0103] For the encoding process of Bitstream#1, a fixed probability density model encoding method can be used; for the decoding process of Bitstream#1, a fixed probability density model decoding method can be used. There are no restrictions on the encoding and decoding processes.

[0104] After obtaining the hyperparameter coefficient feature z_hat, the encoder can perform context-based prediction based on the hyperparameter coefficient feature z_hat of the current image patch and the image feature y_hat of the previous image patch (the determination process of image feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image patch. For example, the hyperparameter coefficient feature z_hat and the image feature y_hat can be input into the mean prediction network, which determines the predicted value mu based on the hyperparameter coefficient feature z_hat and the image feature y_hat. This prediction process is not restricted. Specifically, for the context-based prediction process, the input includes the hyperparameter coefficient feature z_hat and the decoded image feature y_hat. The two are jointly input to obtain a more accurate predicted value mu. The predicted value mu is used to subtract the original feature to obtain the residual and add it to the decoded residual to obtain the reconstructed y.

[0105] It is important to note that the mean prediction network is an optional neural network; that is, it is not necessary to use a mean prediction network to determine the predicted value mu. Figure 4 The dashed box in the image indicates that the mean prediction network is optional.

[0106] After obtaining image features y, the encoder can determine residual features r based on image features y and predicted values ​​mu, such as using the difference between image features y and predicted values ​​mu as residual features r. Then, feature processing is performed on the residual features r to obtain image features s. This feature processing method is not restricted and can be any method. In this case, a mean prediction network needs to be deployed to provide the predicted values ​​mu. Alternatively, after obtaining image features y, the encoder can perform feature processing on image features y to obtain image features s. This feature processing method is not restricted and can be any method. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0107] After obtaining image features s, the encoding end can quantize image features s to obtain the quantized image features corresponding to image features s, i.e. Figure 4 The Q operation in the code represents the quantization process. After obtaining the quantized image features corresponding to image features s, the encoder can encode these quantized features to obtain Bitstream #2 (i.e., the second bitstream) corresponding to the current image block. Figure 4The AE operation in the code represents the encoding process, such as entropy encoding. Alternatively, the encoding end can directly encode the image features s to obtain the Bitstream#2 corresponding to the current image block, without involving the quantization process of image features s.

[0108] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can send the Bitstream#2 corresponding to the current image block to the decoding end. For the processing of the Bitstream#2 corresponding to the current image block by the decoding end, please refer to the following embodiments.

[0109] After obtaining Bitstream#2 corresponding to the current image block, the encoding end can also decode Bitstream#2 to obtain the image quantization features, i.e. Figure 4 In this context, AD represents the decoding process. Then, the encoding end can perform inverse quantization on the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. Figure 4 The IQ operation in the code is the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder can decode Bitstream#2 to obtain image features s' without involving the dequantization process of image quantization features.

[0110] After obtaining image features s', the encoder can perform feature recovery (the inverse process of feature processing) on ​​s'. This feature recovery process is not restricted and can be any method, resulting in residual features r_hat. Residual features r_hat and r can be the same or different. After obtaining residual features r_hat, the encoder determines image features y_hat based on residual features r_hat and predicted values ​​mu. Image features y_hat and y can be the same or different; for example, the sum of residual features r_hat and predicted values ​​mu can be used as image features y_hat. In this case, a mean prediction network needs to be deployed to provide the predicted values ​​mu. Alternatively, after obtaining image features s', the encoder can perform feature recovery (the inverse process of feature processing) on ​​s' to obtain image features y_hat. Image features y_hat and y can be the same or different. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as optional.

[0111] After obtaining the image feature y_hat, the encoder can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat can be input into the synthetic transformation network, which will perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0112] In one possible implementation, when the encoding end encodes the image quantization features or image features s to obtain Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization features or image features s based on the probability distribution model. Furthermore, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0113] To obtain the probability distribution model, please refer to [link / reference]. Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat can be input into a probabilistic hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probabilistic hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform the inverse hyperparameter feature transformation on z_hat.

[0114] In one possible implementation, the above-mentioned encoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without any restrictions on the encoding process.

[0115] Example 4: For the processing procedures at the decoding end in Examples 1 and 2, please refer to... Figure 5 As shown, of course, Figure 5 This is just one example of the processing procedure at the decoding end, and no restrictions are imposed on the processing procedure at the decoding end.

[0116] After obtaining Bitstream#1 corresponding to the current image block, the decoding end can further decode Bitstream#1 to obtain the hyperparameter quantization features, i.e. Figure 5 In the diagram, AD represents the decoding process. Then, the hyperparameter quantization features are dequantized to obtain the coefficient hyperparameter features z_hat. The coefficient hyperparameter features z_hat and the coefficient hyperparameter features z can be the same or different. Figure 5 The IQ operation in the code is an inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat, without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0117] For the decoding process of Bitstream#1, a decoding method with a fixed probability density model can be used, and there are no restrictions on this.

[0118] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the decoding process for the image block can also be directly applied to the image.

[0119] After obtaining the coefficient hyperparameter feature z_hat, the decoder can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image patch and the image feature y_hat of the previous image patch (the determination process of image feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., the mean mu) corresponding to the current image patch. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat can be input into the mean prediction network, which determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. This prediction process is not restricted. Specifically, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat; the combined input of these two features yields a more accurate predicted value mu.

[0120] It is important to note that the mean prediction network is an optional neural network; that is, it is not necessary to use a mean prediction network to determine the predicted value mu. Figure 5 The dashed box in the image indicates that the mean prediction network is optional.

[0121] After obtaining Bitstream#2 corresponding to the current image block, the decoding end can further decode Bitstream#2 to obtain the image quantization features, i.e. Figure 5 In this context, AD represents the decoding process. Then, the decoding end can perform inverse quantization on the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. Figure 5 The IQ operation in the code is an inverse quantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain image features s' without involving the inverse quantization process of image quantization features.

[0122] After obtaining image features s', the decoder can perform feature recovery (the inverse process of feature processing) on ​​image features s' to obtain residual features r_hat. Residual features r_hat may be the same as or different from residual features r. After obtaining residual features r_hat, the decoder determines image features y_hat based on residual features r_hat and predicted values ​​mu. Image features y_hat may be the same as or different from image features y. For example, the sum of residual features r_hat and predicted values ​​mu can be used as image features y_hat. In this case, a mean prediction network needs to be deployed to provide the predicted values ​​mu. Alternatively, after obtaining image features s', the decoder can perform feature recovery on image features s' to obtain image features y_hat. Image features y_hat may be the same as or different from image features y. In this case, a mean prediction network is not needed, and the residual process is indicated by a dashed box as an optional process.

[0123] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat can be input into the synthetic transformation network, and the synthetic transformation network can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0124] In one possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on that probability distribution model. To obtain the probability distribution model, see [link to documentation]. Figure 5 As shown, after obtaining the hyperparameter feature z_hat, the decoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the hyperparameter feature z_hat can be input into a probabilistic hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probabilistic hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p.

[0125] In one possible implementation, the above-mentioned decoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without limiting the decoding process.

[0126] Example 5: For Examples 1, 2, 3, and 4, a mean prediction network can be deployed or not. To improve feature encoding performance, we will take the deployment of a mean prediction network as an example. When deploying a mean prediction network, in order to obtain accurate predicted values ​​of image features, it is necessary to use the reconstructed values ​​of adjacent features (i.e., based on the hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block, to perform context-based prediction and obtain the predicted value mu corresponding to the current image block). Therefore, the feature value on the right (using the reconstructed value of the feature on the left or upper left as a reference) can only be encoded after the feature value on the left or upper left has been encoded and reconstructed. This dependency makes the encoding of features can only be performed serially and cannot be performed in parallel, resulting in high complexity.

[0127] In response to the above findings, this embodiment employs block encoding and decoding, which improves parallelism, effectively saves feature storage cache, achieves higher bitrate control precision, minimizes encoding performance loss, and provides better encoding performance and bitrate control precision. This results in lower peak memory usage, shorter decoding time for a single block, and high-speed parallel decoding capability.

[0128] For details on the encoding process, please refer to [link / reference]. Figure 6 As shown, there are no restrictions on the processing of this encoding end.

[0129] 1. After obtaining the original image x, the segmentation unit determines whether to segment the original image x based on the image resolution. For example, if the image resolution is greater than a threshold, the original image x is segmented; otherwise, it is not segmented. If so, the original image x is divided into several image blocks x_p, with overlapping portions between adjacent image blocks. If not, image block x_p is the original image x, meaning the original image x has only one image block, and each image block x_p can be recorded as the current image block. For ease of description, the processing of a current image block x_p will be used as an example in the following explanation.

[0130] 2. After obtaining the current image block x_p, the current image block x_p is analyzed and transformed by the analysis and transformation network to obtain the image feature y_p (feature block y_p) corresponding to the current image block x_p. For example, the current image block x_p is transformed into the image feature y_p in the latent domain by the analysis and transformation network, so that all subsequent processes can be performed in the latent domain.

[0131] 3. The merging and classification unit merges and classifies the image features y_p to obtain the features to be encoded in N stage sub-blocks y_i (i = 1…N), that is, the image features y_p are split into the features to be encoded in N stage sub-blocks y_i.

[0132] 4. For each stage sub-block corresponding to the current image patch, obtain the coefficient hyperparameter features corresponding to that stage sub-block. For example, perform coefficient hyperparameter feature transformation on the feature y_i to be encoded in stage sub-block i to obtain the coefficient hyperparameter features z_i corresponding to stage sub-block i. For example, the feature y_i to be encoded can be input into the hyperparameter coding network (i.e., neural network), and the hyperparameter coding network can perform coefficient hyperparameter feature transformation on the feature y_i to be encoded to obtain the coefficient hyperparameter features z_i.

[0133] 5. For each stage sub-block corresponding to the current image block, encode the hyperparameter features of that stage sub-block into the first bitstream (Bitstream#1) of the current image block. For example, quantize the hyperparameter features z_i corresponding to stage sub-block i to obtain the hyperparameter quantized features corresponding to z_i, and encode these hyperparameter quantized features to obtain the first bitstream corresponding to the current image block. Alternatively, directly encode the hyperparameter features z_i corresponding to stage sub-block i to obtain the first bitstream corresponding to the current image block. After obtaining the first bitstream corresponding to the current image block, send it to the decoding end. For the processing procedure of the first bitstream corresponding to the current image block at the decoding end, please refer to the following embodiments.

[0134] 6. After obtaining the first bitstream corresponding to the current image block, the first bitstream can be decoded to obtain the coefficient hyperparameter features of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_i corresponding to stage sub-block i (i=1…N).

[0135] For example, the first bitstream can be decoded to obtain the hyperparameter quantization features of stage sub-block i. Then, the hyperparameter quantization features can be dequantized to obtain the coefficient hyperparameter features z_hat_i corresponding to stage sub-block i. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter features z_hat_i corresponding to stage sub-block i without involving the dequantization process.

[0136] For the encoding process of the first bitstream, a fixed probability density model encoding method can be used; for the decoding process of the first bitstream, a fixed probability density model decoding method can be used. There are no restrictions on the encoding and decoding processes.

[0137] 7. For each stage sub-block corresponding to the current image block, determine the probability distribution parameters based on the coefficient hyperparameter features of that stage sub-block. For example, the coefficient hyperparameter features z_hat_i corresponding to stage sub-block i can be subjected to inverse coefficient hyperparameter feature transformation to obtain the probability distribution parameters p_i corresponding to stage sub-block i. Alternatively, the coefficient hyperparameter features z_hat_i can be input into the probability hyperparameter decoding network, which can then perform inverse coefficient hyperparameter feature transformation on z_hat_i to obtain the probability distribution parameters p_i. After obtaining the probability distribution parameters p_i, a probability distribution model can be generated based on the probability distribution parameters p_i.

[0138] 8. After obtaining the features to be encoded for N stage sub-blocks, for the feature to be encoded y_1 of the first stage sub-block, input the coefficient hyperparameter feature z_hat_1 of that stage sub-block into the mean prediction network to obtain the mean feature (i.e., predicted value mu_1, mean mu_1) of that stage sub-block. Alternatively, obtain the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 of that stage sub-block and the default reference feature into the mean prediction network to obtain the mean feature of that stage sub-block.

[0139] Based on the feature to be encoded y_1 and the mean feature mu_1 of the first stage sub-block, the residual feature r_1 is determined, such as using the difference between the feature to be encoded y_1 and the mean feature mu_1 as the residual feature r_1. Then, feature processing is performed on the residual feature r_1 to obtain the feature-processed residual feature r_1. This feature processing process is not restricted and can be any feature processing method. Of course, feature processing is an optional step, and feature processing of the residual feature r_1 can also be omitted.

[0140] The residual feature r_1 (or the residual feature r_1 after feature processing) is quantized to obtain the image quantization feature, and this image quantization feature is encoded to obtain the second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r_1 (or the residual feature r_1 after feature processing) can also be directly encoded to obtain the second bitstream corresponding to the current image block. After obtaining the second bitstream corresponding to the current image block, it is sent to the decoding end. For the processing procedure of the second bitstream corresponding to the current image block at the decoding end, please refer to the following embodiments. When encoding the image quantization feature or residual feature r_1, the probability distribution model corresponding to the probability distribution parameter p_1 of the first stage sub-block can be used to encode the image quantization feature or residual feature r_1 to obtain the second bitstream.

[0141] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameter p_1 of the first stage sub-block can be used to decode the second bitstream to obtain image quantization features. These features can then be dequantized, and feature recovery can be performed on the dequantized features to obtain the residual feature r_hat_1. Alternatively, the residual feature r_hat_1 can be directly obtained after dequantizing the image quantization features without involving the feature recovery process. Alternatively, after decoding the second bitstream, feature recovery can be performed on the decoded features to obtain the residual feature r_hat_1, or the residual feature r_hat_1 can be directly obtained after decoding the second bitstream.

[0142] After obtaining the residual feature r_hat_1, the reconstruction feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, such as using the sum of the residual feature r_hat_1 and the mean feature mu_1 as the reconstruction feature y_hat_1.

[0143] 9. After obtaining the features to be encoded for N stage sub-blocks, for the feature to be encoded y_i of the i-th (i>1, and i< or equal to N) stage sub-block, input the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., the predicted value mu_i and the mean mu_i).

[0144] Specifically, the reference features for stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the preceding (i-1) stage sub-blocks, i.e., based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of stage sub-block i can be obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, meaning the reference features include y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i can be obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, meaning the reference features include some of the reconstructed features from y_hat_1 to y_hat_i-1. Or, the reference features of stage sub-block i can be obtained based on the reconstructed features of the (i-1)-th stage sub-block, meaning the reference feature of stage sub-block i is y_hat_i-1.

[0145] The residual feature r_i is determined based on the feature y_i to be encoded and the mean feature mu_i of the i-th stage sub-block. For example, the difference between the feature y_i to be encoded and the mean feature mu_i can be used as the residual feature r_i. Feature processing is then performed on the residual feature r_i to obtain the feature-processed residual feature r_i. Of course, feature processing on the residual feature r_i can also be omitted.

[0146] The residual feature r_i (or the residual feature r_i after feature processing) is quantized to obtain the image quantization feature. This image quantization feature is then encoded to obtain the second bitstream corresponding to the current image block. Alternatively, the residual feature r_i (or the residual feature r_i after feature processing) can be directly encoded to obtain the second bitstream corresponding to the current image block. The second bitstream corresponding to the current image block is sent to the decoding end. For the processing procedure of the second bitstream corresponding to the current image block at the decoding end, please refer to the following embodiments. When encoding the image quantization feature or the residual feature r_i, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block can be used to encode the image quantization feature or the residual feature r_i to obtain the second bitstream.

[0147] After obtaining the second bitstream, the image quantization features can be obtained by decoding the second bitstream using the probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block. The image quantization features can then be dequantized, and feature recovery can be performed on the dequantized features to obtain the residual features r_hat_i. Alternatively, the residual features r_hat_i can be directly obtained after dequantizing the image quantization features. Alternatively, after decoding the second bitstream, feature recovery can be performed on the decoded features to obtain the residual features r_hat_i.

[0148] After obtaining the residual feature r_hat_i, the reconstruction feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, such as using the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstruction feature y_hat_i.

[0149] 10. The clustering block unit can obtain the reconstructed features y_hat_i (i=1…N) of N stage sub-blocks, and perform feature aggregation on the reconstructed features y_hat_i of the N stage sub-blocks to obtain aggregated features. Then, the clustering block unit divides the aggregated features into blocks to obtain at least one block feature y_hat_p, i.e., feature block y_hat_p.

[0150] 11. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p into the synthesis transform network, and the synthesis transform network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0151] 12. The merging unit can obtain multiple segmented reconstructed image blocks x_hat_p corresponding to the segmented features y_hat_p, and merge these segmented reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Clearly, if the original image x is divided into multiple current image blocks x_p, then all the reconstructed image blocks x_hat corresponding to the current image blocks x_p can form a reconstructed image. If the original image x corresponds to one current image block x_p, then the reconstructed image block x_hat is the reconstructed image.

[0152] In the above process, the encoding end can also assign different bitrate control parameters to different stage sub-blocks of different image blocks based on the region of interest or bitrate information, so that different stage sub-blocks of different image blocks consume different bitrates.

[0153] For example, if special processing is needed for the region of interest (such as non-uniform processing to ensure higher quality in the region of interest) to avoid uniform quantization distortion in each image block, the bitrate control parameter can be the quantization parameter (the bitrate control parameter needs to be transmitted to the decoder via the bitstream). See [link to relevant documentation]. Figure 6 As shown, the bitrate control parameter affects the quantization (Q) and dequantization (IQ) processes of the first bitstream, and also affects the quantization (Q) and dequantization (IQ) processes of the second bitstream.

[0154] For example, when quantizing the hyperparameter feature z_i corresponding to stage sub-block i, if stage sub-block i is located within the region of interest, this bitrate control parameter is used to quantize the hyperparameter feature z_i corresponding to stage sub-block i. This bitrate control parameter is used to ensure higher image quality for stage sub-block i. Similarly, when dequantizing the hyperparameter quantized feature corresponding to stage sub-block i, this bitrate control parameter is also required.

[0155] When quantizing the residual feature r_i (or the processed residual feature r_i) corresponding to stage sub-block i, if stage sub-block i is located within the region of interest, this bitrate control parameter is used to quantize the residual feature r_i corresponding to stage sub-block i. This bitrate control parameter is used to ensure higher image quality corresponding to stage sub-block i. When inverse-quantizing the image quantization features corresponding to stage sub-block i, this bitrate control parameter is also required.

[0156] For example, to achieve precise bitrate control, the bitrate control parameters can be the control parameters of the synthesis transform network (the bitrate control parameters need to be transmitted to the decoder via the bitstream), see [link to documentation]. Figure 6As shown, the bitrate control parameter affects both the analysis transform network and the synthesis transform network. For example, when performing analysis transform on the current image block x_p through the analysis transform network, the bitrate control parameter and the current image block x_p are input to the analysis transform network, and the analysis transform network outputs the image feature y_p corresponding to the current image block x_p. When the block feature y_hat_p is input to the synthesis transform network, the block feature y_hat_p and the bitrate control parameter are input to the synthesis transform network, and the synthesis transform network outputs the reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0157] Based on the configured target bitrate (one target bitrate per image block, or one target bitrate per image, or one target bitrate for multiple images), the encoder can try different bitrate control parameters. After adopting a certain bitrate control parameter, if the bitrate corresponding to the current block's bitstream (such as the first bitstream and the second bitstream) is the target bitrate or close to the target bitrate (i.e., the difference between the two is less than a threshold), then this bitrate control parameter is the final bitrate control parameter adopted, and the encoder can send this bitrate control parameter to the decoder.

[0158] Example 6: The processing procedure at the decoding end can be found in [reference needed]. Figure 7 As shown, there are no restrictions on the processing procedures at this decoding end.

[0159] 1. After obtaining the first bitstream (Bitstream#1) corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter features of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_i corresponding to stage sub-block i (i=1…N).

[0160] For example, the first bitstream can be decoded to obtain the hyperparameter quantization features of stage sub-block i. Then, the hyperparameter quantization features can be dequantized to obtain the coefficient hyperparameter features z_hat_i corresponding to stage sub-block i. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter features z_hat_i corresponding to stage sub-block i without involving the dequantization process. For the decoding process of the first bitstream, a decoding method with a fixed probability density model can be used, and there are no restrictions on this decoding process.

[0161] 2. For each stage sub-block corresponding to the current image block, determine the probability distribution parameters based on the coefficient hyperparameter features of that stage sub-block. For example, the coefficient hyperparameter features z_hat_i corresponding to stage sub-block i can be subjected to inverse coefficient hyperparameter feature transformation to obtain the probability distribution parameters p_i corresponding to stage sub-block i. Alternatively, the coefficient hyperparameter features z_hat_i can be input into the probability hyperparameter decoding network, which will then perform inverse coefficient hyperparameter feature transformation on z_hat_i to obtain the probability distribution parameters p_i. After obtaining the probability distribution parameters p_i, a probability distribution model can be generated based on the probability distribution parameters p_i.

[0162] 3. For each stage sub-block corresponding to the current image block, decode the second bitstream (Bitstream#2) of the current image block based on the probability distribution parameters (probability distribution model) of that stage sub-block to obtain the residual features of that stage sub-block.

[0163] For example, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th (i = 1…N) stage sub-block can be used to decode the second bitstream to obtain image quantization features. These features can then be dequantized, and feature recovery can be performed on the dequantized features to obtain residual features r_hat_i. Alternatively, the residual features r_hat_i can be directly obtained after dequantizing the image quantization features. Or, after decoding the second bitstream, feature recovery can be performed on the decoded features to obtain residual features r_hat_i.

[0164] 4. After obtaining the residual features of N stage sub-blocks, for the residual feature r_hat_1 of the first stage sub-block, input the coefficient hyperparameter feature z_hat_1 of that stage sub-block into the mean prediction network to obtain the mean feature of that stage sub-block (i.e., predicted value mu_1, mean mu_1). Alternatively, obtain the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 of that stage sub-block and the default reference feature into the mean prediction network to obtain the mean feature of that stage sub-block.

[0165] Then, the reconstruction feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1. For example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be used as the reconstruction feature y_hat_1.

[0166] 5. After obtaining the residual features of N stage sub-blocks, for the residual feature r_hat_i of the i-th (i>1, and i< or equal to N) stage sub-block, the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i are input into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., predicted value mu_i, mean mu_i).

[0167] Specifically, the reference features for stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the preceding (i-1) stage sub-blocks, i.e., based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of stage sub-block i can be obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, meaning the reference features include y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i can be obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, meaning the reference features include some of the reconstructed features from y_hat_1 to y_hat_i-1. Or, the reference features of stage sub-block i can be obtained based on the reconstructed features of the (i-1)-th stage sub-block, meaning the reference feature of stage sub-block i is y_hat_i-1.

[0168] To control the buffer size, the number of reconstructed features included in the reference features cannot exceed a preset number, assuming the preset number is M. If the number of reconstructed features is less than or equal to M, then the reference features include all reconstructed features from y_hat_1 to y_hat_i-1. If the number of reconstructed features is greater than M, then the reference features include M reconstructed features from y_hat_1 to y_hat_i-1 (i.e., some reconstructed features). To select M reconstructed features from y_hat_1 to y_hat_i-1, a default strategy can be used, such as selecting the M reconstructed features closest to the current stage sub-block in the decoding order, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, etc., until M reconstructed features are selected. Alternatively, other methods can be used to select M reconstructed features without restriction. Alternatively, the index values ​​of the M reconstructed features can be obtained by decoding the bitstream, and the M reconstructed features can be selected based on these decoded index values.

[0169] Then, the reconstruction feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstruction feature y_hat_i.

[0170] 6. The clustering block unit can obtain the reconstructed features y_hat_i (i=1…N) of N stage sub-blocks, and perform feature aggregation on the reconstructed features y_hat_i of the N stage sub-blocks to obtain aggregated features. Then, the clustering block unit divides the aggregated features into blocks to obtain at least one block feature y_hat_p, that is, feature block y_hat_p.

[0171] 7. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p into the synthesis transform network, and the synthesis transform network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0172] 8. The merging unit can obtain multiple segmented reconstructed image blocks x_hat_p corresponding to segmented features y_hat_p, and merge these segmented reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Clearly, if the original image x is divided into multiple current image blocks x_p, then all the reconstructed image blocks x_hat corresponding to the current image blocks x_p can form a reconstructed image. If the original image x corresponds to one current image block x_p, then the reconstructed image block x_hat is the reconstructed image.

[0173] In the above process, different bitrate control parameters can be assigned to different stage sub-blocks of different image blocks based on region of interest or bitrate information, so that different stage sub-blocks of different image blocks consume different bitrates.

[0174] For example, if special processing is required for the region of interest (such as non-uniform processing to ensure higher quality in that region of interest), the bitrate control parameter can be a quantization parameter (the bitrate control parameter needs to be transmitted to the decoder via the bitstream), see [link to relevant documentation]. Figure 7 As shown, the bitrate control parameter affects the inverse quantization (IQ) process of the first bitstream and the inverse quantization (IQ) process of the second bitstream.

[0175] For example, when dequantizing the hyperparameter quantization feature corresponding to stage sub-block i, if the bitrate control parameter corresponding to stage sub-block i has been decoded from the bitstream, and this bitrate control parameter is used to ensure that the image quality corresponding to stage sub-block i is higher, then this bitrate control parameter can be used to dequantize the hyperparameter quantization feature corresponding to stage sub-block i.

[0176] For example, when dequantizing the image quantization features corresponding to stage sub-block i, if the bitrate control parameter corresponding to stage sub-block i has been decoded from the bitstream, and this bitrate control parameter is used to ensure that the image quality corresponding to stage sub-block i is higher, then this bitrate control parameter can be used to dequantize the image quantization features corresponding to stage sub-block i.

[0177] For example, to perform precise bitrate control, the bitrate control parameters can be the control parameters of the synthesis transform network (the bitrate control parameters need to be transmitted to the decoder via the bitstream; the control parameters of the synthesis transform network refer to the parameters that can be used to determine the synthesis transform network). See [link to documentation]. Figure 7 As shown, the bitrate control parameter affects the synthesis transform network. For example, when the block feature y_hat_p is input into the synthesis transform network, both the block feature y_hat_p and the bitrate control parameter are input into the synthesis transform network, and the synthesis transform network outputs the reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0178] Example 7: In Examples 5 and 6, for the i-th (i = 1…N) stage sub-block, after obtaining the reconstruction feature y_hat_i (i.e., the sum of the residual feature r_hat_i and the mean feature mu_i) of the i-th stage sub-block, the reconstruction feature y_hat_i is used to determine the mean feature of subsequent stage sub-blocks, and / or, the reconstruction feature y_hat_i is used to determine the reconstructed image block. Wherein, the reconstruction feature y_hat_i is used to determine the mean feature of subsequent stage sub-blocks: the reconstruction feature y_hat_i serves as a reference feature for subsequent stage sub-blocks, thereby determining the mean feature of subsequent stage sub-blocks. The reconstruction feature y_hat_i is used to determine the reconstructed image block: the reconstruction feature y_hat_i is input to the clustering block unit, thereby determining the reconstructed image block corresponding to the current image block.

[0179] For the i-th (i = 1...N) stage sub-block, after obtaining the reconstruction feature y_hat_i of the i-th stage sub-block, feature enhancement can be performed on the reconstruction feature y_hat_i to obtain the enhanced reconstruction feature y_hat_i'. The enhanced reconstruction feature y_hat_i' is used to determine the mean feature of subsequent stage sub-blocks, and / or, the enhanced reconstruction feature y_hat_i' is used to determine the reconstructed image block. Wherein, the enhanced reconstruction feature y_hat_i' is used to determine the mean feature of subsequent stage sub-blocks: the enhanced reconstruction feature y_hat_i' serves as a reference feature for subsequent stage sub-blocks, thereby determining the mean feature of subsequent stage sub-blocks. The enhanced reconstruction feature y_hat_i' is used to determine the reconstructed image block: the enhanced reconstruction feature y_hat_i' is input to the clustering block unit to determine the reconstructed image block corresponding to the current image block. For example, the bitstream can be decoded to obtain enhanced information, and the reconstructed feature y_hat_i can be enhanced based on the enhanced information to obtain a new reconstructed feature y_hat_i'. The new reconstructed feature y_hat_i' can be used for subsequent predictions from y_hat_i+1 to y_hat_N, or it can be used for subsequent clustering and segmentation processes.

[0180] In one possible implementation, the enhancement process for the reconstructed feature y_hat can be carried out in the following manner:

[0181] Method 1: The input consists of the unenhanced reconstructed features y_hat, Threshold[idx], GreaterFlag[idx], Scale1[idx], and Scale2[idx], and the output is the enhanced reconstructed features y_hat_en. Here, Threshold represents the threshold value; different idx values ​​may correspond to the same or different threshold values. idx represents the index, such as 0, 1, 2, 3, etc. GreaterFlag represents the flag bit; different idx values ​​may correspond to the same or different flag bits. Scale1 represents the scaling factor; different idx values ​​may correspond to the same or different Scale1 values. Scale2 represents the scaling factor; different idx values ​​may correspond to the same or different Scale2 values.

[0182] Step 1: Determine the mask[c, i, j] for each filter (index idx). For example, the following expression can be used to determine mask[c, i, j]. Of course, the following expression is just an example and is not a limitation.

[0183]

[0184] σ[c, i, j] represents the probability distribution parameter value of [c, i, j], where c represents the channel, i represents the x-axis, and j represents the y-axis. If the probability distribution parameter value is greater than Threshold[idx], GreaterFlag[idx] is true and mask[idx, c, i, j] is 1. If the probability distribution parameter value is less than Threshold[idx], GreaterFlag[idx] is false and mask[idx, c, i, j] is 1. Otherwise, mask[idx, c, i, j] is 0.

[0185] Step 2: Execute each filter (index idx) sequentially. If mask[idx, c, i, j] is 1, then y_hat_en[c, i, j] = y_hat[c, i, j] + mean_hat[c, i, j] * Scale2[idx] + residual_hat[c, i, j] * Scale1[idx]; y_hat[c, i, j] = y_hat_en[c, i, j]. If there are multiple filters, y_hat is compensated multiple times.

[0186] Method 2: In Method 1, each channel c has the same scaling factor. In Method 2, different channels c are allowed to have different scaling factors, namely Scale1[c, idx] and Scale2[c, idx]. Whether the current feature is scaled can be controlled by a switch, or scaling can be turned off by setting the channel's scaling factor to 0. The difference between Method 2 and Method 1 lies in the enhancement method: y_hat_en[c, i, j] = y_hat[c, i, j] + mean_hat[c, i, j] * Scale2[c, idx] + residual_hat[c, i, j] * Scale1[c, idx].

[0187] Method 3 is similar to Method 1 in its implementation, but the enhancement method is as follows: y_hat_en[c, i, j] = y_hat[c, i, j] * Scale1[idx] + mean_hat[c, i, j] * Scale2[idx].

[0188] Method 4 is similar to Method 1 in its implementation, but the enhancement method is as follows: y_hat_en[c, i, j] = y_hat[c, i, j] * Scale1[idx] + residual_hat[c, i, j] * Scale2[idx]. In Method 4, Scale1[idx] can also be set to 1 by default, and the adjustment method is only affected by changing the value of Scale2[idx].

[0189] Example 8: In Examples 5 and 6, the hyperparameter feature z_hat_1 of the coefficients of the first stage sub-block can be input into the mean prediction network, which outputs the mean feature of the first stage sub-block. The prediction process of this mean prediction network is not restricted. For the i-th (i = 1…N) stage sub-block, the hyperparameter feature z_hat_i of the i-th stage sub-block and the reference feature (possibly a default reference feature) of the i-th stage sub-block can be input into the mean prediction network, which outputs the mean feature of the i-th stage sub-block. The prediction process of the mean prediction network is explained below.

[0190] See Figure 8A The diagram shown is a schematic of the mean prediction network. The mean prediction network may include a first prediction network, a second prediction network, and a prediction fusion network. Of course, this is just an example, and there are no restrictions on the structure of this mean prediction network.

[0191] The hyperparameter features z_hat_i of the i-th stage sub-block can be input into the first prediction network to obtain the first predicted features corresponding to z_hat_i. The reference features of the i-th stage sub-block (such as the reconstructed features y_hat_1 to y_hat_i-1) can be input into the second prediction network to obtain the second predicted features corresponding to the reference features. Then, the first and second predicted features are concatenated and input into the prediction fusion network. The prediction fusion network processes the concatenated features to obtain the mean feature mu_i of the i-th stage sub-block.

[0192] For the first prediction network: Feature enhancement and upsampling operations are performed on the hyperparameter feature z_hat_i using the first prediction network to obtain the first predicted feature. Feature enhancement operations may include, but are not limited to, convolution operations, or convolution operations and activation operations. These are merely examples of feature enhancement operations and are not limited in scope. Upsampling operations may include, but are not limited to, deconvolution operations, cropping operations, and activation operations, or deconvolution operations, cropping operations, activation operations, and convolution operations. These are merely examples of upsampling operations and are not limited in scope.

[0193] For example, see Figure 8B The diagram shows the structure of the first prediction network, which consists of three augmentation networks and two 2x upsampling networks. Of course, the number of augmentation networks can be more or less, the number of 2x upsampling networks can be more or less, and the 2x upsampling networks can be replaced with upsampling networks of other multiples.

[0194] The hyperparameter feature z_hat_i is processed by enhancement network 1 and 2x upsampling network 1 to obtain enhanced large-size feature 1, whose spatial dimensions (width and height) are twice that of the hyperparameter feature z_hat_i. Enhanced large-size feature 1 is then processed by enhancement network 2 and 2x upsampling network 2 to obtain enhanced large-size feature 2, whose spatial dimensions (width and height) are twice that of enhanced large-size feature 1. Enhanced large-size feature 2 is then processed by enhancement network 3 to obtain the first predicted feature.

[0195] For example, augmented network 1 and augmented network 2 can be the same or different, augmented network 1 and augmented network 3 can be the same or different, and augmented network 2 and augmented network 3 can be the same or different.

[0196] For example, one implementation of an augmentation network (such as augmentation network 1, and / or augmentation network 2, and / or augmentation network 3) can be found in [reference needed]. Figure 8CAs shown, the augmentation network consists of a single convolutional layer, which can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. There are no restrictions on this, such as choosing a 3x3 convolutional layer.

[0197] For example, one implementation of an augmentation network (such as augmentation network 1, and / or augmentation network 2, and / or augmentation network 3) can be found in [reference needed]. Figure 8D As shown, the enhanced network can consist of convolutional layer 1, activation layer 1, convolutional layer 2, activation layer 2, and convolutional layer 3. The convolutional layers (such as convolutional layer 1, and / or convolutional layer 2, and / or convolutional layer 3) can be 1x1, 3x3, or 5x5 convolutional layers; there are no restrictions, such as choosing a 3x3 convolutional layer.

[0198] The activation layer (such as activation layer 1 and / or activation layer 2) can be a ReLU layer, or a leaky ReLU layer, or a sigmoid layer, or a tanh layer, or a Gelu layer. There are no restrictions on the type of activation layer, such as choosing a ReLU layer.

[0199] Of course, the above are just two examples of network enhancement, and there are no restrictions on them; as long as the enhancement function can be achieved, it is acceptable.

[0200] For example, 2x upsampling network 1 and 2x upsampling network 2 can be the same or different.

[0201] For example, one implementation of a 2x upsampling network (such as 2x upsampling network 1 and / or 2x upsampling network 2) can be found in [reference needed]. Figure 8E As shown, a 2x upsampling network can consist of a 2x deconvolution layer, a Crop layer, and an activation layer. The deconvolution layer can be a 2x2, 3x3, 4x4, or 5x5 deconvolution layer; there are no restrictions, such as choosing a 4x4 deconvolution layer. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; there are no restrictions on the type of activation layer, such as choosing a ReLU layer. Furthermore, the Crop layer is used to crop features, reducing the spatial resolution of the features while maintaining the same number of channels.

[0202] For example, one implementation of a 2x upsampling network (such as 2x upsampling network 1 and / or 2x upsampling network 2) can be found in [reference needed]. Figure 8FAs shown, a 2x upsampling network can consist of a 2x deconvolution layer, a Crop layer, an activation layer, and a convolutional layer. The deconvolution layer can be a 2x2, 3x3, 4x4, or 5x5 deconvolution layer. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a Gelu layer. The Crop layer is used to crop features, reducing the spatial resolution of the features while maintaining the same number of channels. The convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer.

[0203] Of course, the above are just two examples of 2x upsampling networks, and there are no restrictions on this; as long as the upsampling function can be achieved, it is acceptable.

[0204] For example, the convolutional layer in the first prediction network can be a regular 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation of the grouped convolution can be found in the following embodiments, and will not be repeated here.

[0205] For the second prediction network: The second prediction network concatenates all reconstructed features from the reference features of the i-th stage sub-block along the channel dimension, and then performs a convolution operation on the concatenated features to obtain the second prediction feature. Alternatively, the second prediction network sums all reconstructed features from the reference features of the i-th stage sub-block, and then performs a convolution operation on the summed features to obtain the second prediction feature. Here, the reference features include all reconstructed features from the preceding i-1 stage sub-blocks; or, some reconstructed features from the preceding i-1 stage sub-blocks; or, the reconstructed features of the i-1-th stage sub-block.

[0206] For example, see Figure 8G The diagram shows the structure of the second prediction network. The second prediction network can consist of a feature concatenation layer and convolutional layers; there are no restrictions on its structure. If the reference feature includes all reconstructed features from the preceding (i-1) stage sub-blocks (e.g., y_hat_1 to y_hat_i-1), the feature concatenation layer can concatenate these reconstructed features along the channel dimension to obtain the concatenated feature. Alternatively, if the reference feature includes only some reconstructed features from the preceding (i-1) stage sub-blocks (e.g., a portion from y_hat_1 to y_hat_i-1), the feature concatenation layer can concatenate these reconstructed features along the channel dimension to obtain the concatenated feature. Alternatively, if the reference feature includes the reconstructed features from the (i-1)th stage sub-block, the feature concatenation layer can directly use the reconstructed features from the (i-1)th stage sub-block as the concatenated feature.

[0207] The feature concatenation layer inputs the concatenated features into the convolutional layer, which then performs a convolution operation on the concatenated features to obtain the second predicted feature. This convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 3x3 convolutional layer. The convolutional layers in the second prediction network can be ordinary 2D convolutions or grouped convolutions, such as 2x grouped convolutions. The implementation of grouped convolutions is described in subsequent embodiments.

[0208] For example, the second prediction network can consist of a feature summing layer and a convolutional layer. If the reference feature includes all the reconstructed features of the preceding i-1 stage sub-blocks, the feature summing layer sums these reconstructed features to obtain the summed feature. Alternatively, if the reference feature includes some of the reconstructed features of the preceding i-1 stage sub-blocks, the feature summing layer sums these reconstructed features to obtain the summed feature. Or, if the reference feature includes the reconstructed features of the i-1th stage sub-block, the feature summing layer can directly use the reconstructed features of the i-1th stage sub-block as the summed feature. The feature summing layer inputs the summed feature to the convolutional layer, which performs a convolution operation on the summed feature to obtain the second prediction feature.

[0209] To control the cache size, the number of reconstructed features included in the reference features cannot exceed a preset number, assuming the preset number is M. Then, if the number of reconstructed features is less than or equal to M, the reference features include all reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, if the number of reconstructed features is greater than M, the reference features include M reconstructed features from y_hat_1 to y_hat_i-1. For example, M reconstructed features can be selected based on a default strategy, such as selecting the M reconstructed features whose decoding order is closest to the current stage sub-block. Alternatively, the index values ​​of the M reconstructed features can be obtained by decoding the bitstream, and the M reconstructed features can be selected based on these decoded index values.

[0210] For the prediction fusion network: the concatenated features (i.e., the concatenated features of the first prediction feature and the second prediction feature) are subjected to convolution and at least one fusion operation through the prediction fusion network to obtain the mean feature mu_i of the i-th stage sub-block; wherein, for each fusion operation, the fusion operation may include an activation operation and a convolution operation.

[0211] For example, see Figure 8HThe diagram shows the structure of a prediction fusion network. This network can consist of convolutional layers and N fusion layers, where N is a positive integer (e.g., 1 or 2). Each fusion layer can consist of activation layers and convolutional layers; there are no restrictions on the network's structure. Clearly, after the input features (i.e., the concatenated features) pass through convolutional layers and N "activation layers + convolutional layers," the output features are obtained, namely the mean feature mu_i of the i-th stage sub-block.

[0212] The convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer; there are no restrictions, such as choosing a 1x1 convolutional layer. For the convolutional layers in the prediction fusion network, they can be ordinary 2D convolutions or grouped convolutions, such as 2x grouped convolutions. The implementation of grouped convolutions is described in subsequent embodiments. The activation layer can be a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer; a ReLU layer can be chosen.

[0213] Example 9: For both the encoding and decoding ends, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block. For example, in Examples 5 and 6, the clustering block unit can perform feature aggregation on the reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks to obtain aggregated features (also called full-size aggregated features y_hat_all), and divide the aggregated features y_hat_all into blocks to obtain at least one block feature y_hat_p, such as block features y_hat_p_1 to block features y_hat_p_K. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p to the synthesis transform network, and the synthesis transform network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p. The merging unit can obtain multiple block reconstructed image blocks x_hat_p corresponding to block features y_hat_p, and merge these block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.

[0214] First, the clustering block unit performs feature aggregation on the reconstructed features of N stage sub-blocks to obtain aggregated features.

[0215] For example, see Figure 9A As shown, the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks can be aggregated according to phase to obtain the aggregated feature y_hat_all. For example, an example of aggregation according to phase can be shown below: after obtaining y_hat_group1, y_hat_group1 can be used as the aggregated feature y_hat_all.

[0216]

[0217] For example, see Figure 9B As shown, the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks can be aggregated according to phase to obtain multiple phase-aggregated features. These phase-aggregated features are then concatenated according to channels to obtain the aggregated feature y_hat_all. For example, see the expression above for an example of aggregation by phase. Aggregation by phase yields y_hat_group1 and y_hat_group2. If there are many reconstructed features, after obtaining y_hat_group1 and y_hat_group2, the aggregated y_hat_groups can be further concatenated according to channels to obtain the aggregated feature y_hat_all.

[0218] For example, before aggregating the reconstructed features of N stage sub-blocks, useless feature values ​​can be removed. For instance, since the height or width of the full-size feature is odd, when the encoding side is divided into four, to ensure that the reconstructed features of each stage sub-block have the same size, the reconstructed features of some stage sub-blocks need to be padded. The padding scheme could be to add a right column or a bottom row to the full-size feature. Based on this, after aggregating the reconstructed features of N stage sub-blocks, useless features need to be removed. See also Figure 9C The example shown illustrates a case where the width is odd. If the height is odd, the bottom row is removed. Of course, the above example assumes a division into four sections. If a division into 16 sections exists, more than one row or column might be removed.

[0219] Second, the clustering block unit divides the aggregated feature y_hat_all into blocks to obtain K block features y_hat_p.

[0220] For example, the aggregated feature y_hat_all can be further divided into blocks to reduce subsequent memory overhead (the synthesis transformation operation requires multiple upsampling of the feature, causing the feature data size to exceed the allowable memory size). Of course, this feature block division process is not necessary. If the current feature size is small, or if it is determined based on the bitstream information that the current feature does not need to be divided, the following block division process will be skipped directly, and y_hat_p will equal y_hat_all, meaning there is only one block feature.

[0221] To divide the aggregated feature y_hat_all into blocks, a feature block division method with no overlapping parts can be used, see [link to relevant documentation]. Figure 9D The image shows an example of dividing the aggregated feature y_hat_all into four block features y_hat_p.

[0222] In a feature segmentation method where there is no overlap, the target size of the segmented features can be determined, and based on the target size, the aggregated feature y_hat_all is divided into multiple segmented features in an order from top to bottom and from left to right, with the size of each segmented feature being the target size, such as segmented features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0223] For example, the target size is the size of each block feature. The target size can be the size read from the bitstream or it can be determined based on the size of the current feature to be divided. There is no restriction on this. If the number of channels of each block feature is tile_c, the width is tile_w, and the height is tile_h, then the aggregated feature y_hat_all will be divided into multiple tile_c*tile_w*tile_h block features in the order from top to bottom and from left to right, such as block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0224] To divide the aggregated feature y_hat_all into blocks, a feature block division method with overlapping parts can be used, see [link to relevant documentation]. Figure 9E The image shows an example of dividing the aggregated feature y_hat_all into four block features y_hat_p.

[0225] In feature segmentation with overlapping parts, the actual segment size and overlap size of the segmented features can be determined. The actual segment size can be the segment size of the segmented features after removing the overlapping parts, and the overlap size can be the size of the overlapping parts of adjacent segments. Based on the actual segment size and overlap size, the aggregated features are divided into multiple segmented features, and the size of each segmented feature is the target size, such as segmented features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0226] For example, determining the actual block size and overlap size. The actual block size is the block size after removing the overlap portion, and the overlap size is the size of the overlapping portion of adjacent blocks. This size information can be read from the bitstream or determined based on the feature size of the current block to be segmented; there is no restriction on this. See also Figure 9EAs shown, if the actual block size is tile_c*tile_w*tile_h and the overlap size is tile_c*padding_w*padding_h, then when overlapping vertically and horizontally, the size of each block feature (i.e., the target size) is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h). Based on this, for the aggregated feature y_hat_all, the features in columns 1 to (tile_w+2*padding_w) and rows 1 to (tile_h+2*padding_h) are assigned to the first block feature. The features in columns 1 to 2*tile_w+3*padding_w and rows 1 to (2*tile_h+3*padding_h) are assigned to the second block feature, and so on.

[0227] See Figure 9E As shown, the aggregated feature y_hat_all of 4x4x4 can be divided into four 4x3x3 block features. In the block features, tile_c is 4, padding_w and padding_h are 1, and tile_w and tile_h are 1.

[0228] For example, if the number of blocks in the last row or column is insufficient, overlapping areas can be added to ensure the final block size meets the requirements. See also Figure 9F As shown, an example is given where the size of y_hat_all is 11x13, padding_w and padding_h are 1, and tile_w and tile_h are 3. The first and second blocks in the first row are 5x5 in size, while the third block, to ensure that its size is also 5x5, is extended to the left by two overlapping columns.

[0229] Third, the clustering block unit inputs each block feature y_hat_p (e.g., K block features y_hat_p) into the synthesis transform network, and the synthesis transform network outputs the block reconstructed image block x_hat_p corresponding to each block feature y_hat_p.

[0230] For each block feature y_hat_p, after inputting the block feature y_hat_p into the synthesis transformation network, the synthesis transformation network performs a synthesis transformation on the block feature y_hat_p to obtain the block reconstructed image block x_hat_p, without any restrictions.

[0231] Fourth, the merging unit merges multiple segmented reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0232] For example, the merging unit can obtain K block reconstructed image blocks x_hat_p corresponding to K block features y_hat_p, and merge the K block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0233] For example, if the aggregated feature y_hat_all is divided into blocks using a feature segmentation method that does not overlap, then the merging unit merges the K reconstructed image blocks x_hat_p using a feature merging method that does not overlap. For instance, the reconstructed image blocks x_hat_p corresponding to the K block features are sorted in a top-to-bottom and left-to-right order, and the sorted K reconstructed image blocks x_hat_p are then sequentially concatenated to obtain the reconstructed image block x_hat corresponding to the current image block. The concatenation process can be... Figure 9D The reverse process will not be elaborated here.

[0234] For example, if the aggregated feature y_hat_all is divided into blocks using a feature segmentation method with overlapping parts, then the merging unit merges the K reconstructed image blocks x_hat_p using a feature merging method with overlapping parts. For instance, the merging unit can determine the actual block size and overlap size of each reconstructed image block x_hat_p. The merging unit can sort the reconstructed image blocks x_hat_p corresponding to the K block features y_hat_p in a top-to-bottom and left-to-right order; based on the actual block size and overlap size, the reconstructed image blocks x_hat_p corresponding to the sorted K block features y_hat_p are sequentially stitched together to obtain the reconstructed image block x_hat corresponding to the current image block. Here, a reconstructed image block can include K reconstructed image blocks. The size of the non-overlapping part of a reconstructed image block can be the actual block size of that reconstructed image block, and the size of the overlapping part between a reconstructed image block and an adjacent reconstructed image block can be the overlap size of that reconstructed image block. The stitching process can be... Figure 9E The reverse process will not be elaborated here.

[0235] For example, determining the actual block size and overlap size: the actual block size is the block size after removing the overlap portion, and the overlap size is the size of the overlapping portion of adjacent blocks. This size information can be read from the bitstream or determined based on the feature size of the current block to be segmented; there are no restrictions on this. Assuming the actual block size is x_c*xtile_w*xtile_h, the overlap width in the overlap size is xpadding_w, and the overlap height in the overlap size is xpadding_h, then, following the receiving order of multiple reconstructed image blocks x_hat_p, the multiple reconstructed image blocks x_hat_p are sequentially concatenated.

[0236] For the overlapping portion of the two reconstructed image blocks, the value of the overlapping portion is the value of the left reconstructed image block, and the value of the right reconstructed image block is discarded. That is, the value of the earliest (left) reconstructed image block is used, and the values ​​of subsequent sub-block overlaps are directly discarded. Alternatively, the value of the overlapping portion is the average value of the two reconstructed image blocks.

[0237] For the overlapping portion of two reconstructed image blocks, the value of the overlapping portion is the value of the upper reconstructed image block, and the value of the lower reconstructed image block is discarded. That is, the value of the earliest (upper) reconstructed image block is used. For subsequent overlapping portions of sub-blocks, the values ​​of the overlapping portions are directly discarded. Alternatively, the value of the overlapping portion is the average value of the two reconstructed image blocks.

[0238] Example 10: For both the encoding and decoding ends, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block. For example, the clustering block unit can perform feature aggregation on the reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks to obtain the aggregated features (also called the full-size aggregated features y_hat_all). The clustering block unit inputs the aggregated features y_hat_all to the synthesis transform network, and the synthesis transform network outputs the reconstructed image block corresponding to the aggregated features y_hat_all. This reconstructed image block is the reconstructed image block x_hat corresponding to the current image block. Compared with Example 9, it does not involve dividing the aggregated features y_hat_all into blocks, nor does it involve merging multiple segmented reconstructed image blocks x_hat_p.

[0239] First, the clustering block unit performs feature aggregation on the reconstructed features of N stage sub-blocks to obtain aggregated features.

[0240] For example, the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks can be aggregated according to phase to obtain the aggregated feature y_hat_all. Alternatively, the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks can be aggregated according to phase to obtain features aggregated by multiple phases. These features aggregated by multiple phases are then concatenated according to channels to obtain the aggregated feature y_hat_all. The feature aggregation process can be found in Example 9 and will not be repeated here.

[0241] Second, the clustering block unit inputs the aggregated feature y_hat_all to the synthesis transformation network. The synthesis transformation network performs a synthesis transformation on the aggregated feature y_hat_all. There are no restrictions on the synthesis transformation method. The synthesis transformation network outputs the reconstructed image block corresponding to the aggregated feature y_hat_all, that is, the reconstructed image block x_hat corresponding to the current image block.

[0242] Example 11: For both the encoder and decoder, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block. For example, after obtaining the reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks, the clustering and segmentation unit does not perform feature aggregation or segmentation of the aggregated features. Instead, it directly inputs the reconstruction features of each stage sub-block into the synthesis transform network, which outputs the segmented reconstructed image block x_hat_p corresponding to each stage sub-block. The merging unit can obtain multiple segmented reconstructed image blocks x_hat_p corresponding to multiple stage sub-blocks and merge these segmented reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.

[0243] First, after obtaining the reconstruction features of N stage sub-blocks, the clustering block unit inputs the reconstruction features of each stage sub-block into the synthesis transformation network. The synthesis transformation network performs a synthesis transformation on the reconstruction features. There are no restrictions on the synthesis transformation method. The synthesis transformation network outputs the block-based reconstructed image block x_hat_p corresponding to each stage sub-block.

[0244] Second, the merging unit merges multiple segmented reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0245] For example, the merging unit can obtain N block reconstructed image blocks x_hat_p corresponding to N stage sub-blocks, and merge the N block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0246] For example, the merging unit merges N reconstructed image blocks x_hat_p using a feature merging method that does not overlap. For instance, the N reconstructed image blocks x_hat_p are sorted from top to bottom and left to right, and the sorted N reconstructed image blocks x_hat_p are sequentially concatenated to obtain the reconstructed image block x_hat corresponding to the current image block. Alternatively, the merging unit merges N reconstructed image blocks x_hat_p using a feature merging method that does overlap. For instance, the actual block size and overlap size of each reconstructed image block x_hat_p are determined, and the N reconstructed image blocks x_hat_p are sorted from top to bottom and left to right; based on the actual block size and overlap size, the sorted N reconstructed image blocks x_hat_p are sequentially concatenated to obtain the reconstructed image block x_hat corresponding to the current image block. A reconstructed image block comprises N sub-blocks. The size of the non-overlapping portion of a sub-block is its actual block size. The size of the overlapping portion between a sub-block and its adjacent sub-block is its overlap size. For the overlap between two left and right sub-blocks, the value of the overlap portion is the value of the left sub-block, and the value of the right sub-block is discarded. Alternatively, the value of the overlap portion is the average of the two sub-blocks. For the overlap between two top and bottom sub-blocks, the value of the overlap portion is the value of the top sub-block, and the value of the bottom sub-block is discarded. Alternatively, the value of the overlap portion is the average of the two sub-blocks.

[0247] Example 12: In Examples 1-11, a synthetic transform network is involved. In order to perform precise bitrate control, the encoder can send bitrate control parameters for the synthetic transform network to the decoder. For example, the encoder can encode the bitrate control parameters in the auxiliary bitstream corresponding to the current image block. The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters, and determine the synthetic transform network based on the bitrate control parameters. That is, the synthetic transform network is determined based on the bitstream information, especially the bitrate-related information. The synthetic transform network can consist of at least one 2x2 deconvolutional layer (e.g., 2x2, 3x3, 4x4, or 5x5 2x2 deconvolutional layers, such as a 4x4 2x2 deconvolutional layer, or other deconvolutional layers), at least one activation layer (e.g., a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer, such as a ReLU layer), and at least one convolutional layer (e.g., a 1x1, 3x3, or 5x5 convolutional layer, such as a 3x3 convolutional layer; the convolutional layers in the synthetic transform network can be ordinary 2D convolutions or grouped convolutions, such as 2x grouped convolutions). The output feature space resolution of the synthetic transform network is a positive integer multiple of the input feature space resolution, such as 16 times (both width and height are 16 times), and the number of output feature channels is 1 or 3. Based on the synthetic transform network, the network can be controlled by a bitrate control parameter, which serves as the input to the synthetic transform network.

[0248] In one possible implementation, for Example 9, the clustering block unit inputs K block features y_hat_p into the synthesis transform network. Therefore, the encoder encodes K rate control parameters corresponding to the K block features y_hat_p in the auxiliary bitstream corresponding to the current image block. The source of these K rate control parameters is not limited. When the encoder uses these K rate control parameters to control the analysis transform network and the synthesis transform network (i.e., the K rate control parameters are used as inputs to the analysis transform network and the synthesis transform network, such as inputting the block feature y_hat_p along with its corresponding rate control parameters to the synthesis transform network as input features), the bitstream corresponding to the current image block can achieve a bitstream rate that is at or close to the target bitstream (i.e., the difference between the two is less than a threshold). The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the K rate control parameters corresponding to the K block features y_hat_p.

[0249] Based on this, the clustering block unit can obtain K block features y_hat_p and K rate control parameters corresponding to the K block features y_hat_p. For each block feature y_hat_p, the block feature y_hat_p and the corresponding rate control parameters are input together into the synthesis transform network. The synthesis transform network generates the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the corresponding rate control parameters.

[0250] For example, the synthetic transform network generates a block-reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the bit rate control parameter corresponding to the block feature y_hat_p. This may include, but is not limited to: processing the block feature y_hat_p through the synthetic transform network to obtain a first feature; processing the bit rate control parameter corresponding to the block feature y_hat_p through the synthetic transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the block-reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the third feature.

[0251] In one possible implementation, for Example 10, the clustering block unit inputs the aggregated feature y_hat_all to the synthesis transform network. Therefore, the encoder encodes a rate control parameter corresponding to the aggregated feature y_hat_all in the auxiliary bitstream corresponding to the current image block, i.e., the rate control parameter corresponding to the current image block. The source of this rate control parameter is not limited. When the encoder uses this rate control parameter to control the analysis transform network and the synthesis transform network (i.e., the rate control parameter is used as input to the analysis transform network and the synthesis transform network; for example, the aggregated feature y_hat_all is input to the synthesis transform network along with the rate control parameter, and the current image block is input to the analysis transform network along with the rate control parameter, as input features), the bitstream corresponding to the current image block can achieve a bitstream rate close to or at the target bitstream. The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the rate control parameter corresponding to the current image block.

[0252] Based on this, the clustering block unit can obtain the aggregated feature y_hat_all and the bit rate control parameter, and input the aggregated feature y_hat_all and the bit rate control parameter together into the synthesis transform network. The synthesis transform network generates the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the bit rate control parameter.

[0253] The synthetic transform network generates a reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the bit rate control parameter. This process can include: processing the aggregated feature y_hat_all through the synthetic transform network to obtain a first feature; processing the bit rate control parameter through the synthetic transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the third feature.

[0254] In one possible implementation, for Example 11, the clustering block unit inputs the reconstructed features of N stage sub-blocks into the synthesis transform network. Therefore, the encoder encodes N rate control parameters corresponding to each of the N stage sub-blocks in the auxiliary bitstream corresponding to the current image block. The source of these N rate control parameters is not limited. When the encoder uses these N rate control parameters to control the analysis transform network and the synthesis transform network (i.e., the N rate control parameters are used as inputs to the analysis transform network and the synthesis transform network, such as the reconstructed features of a stage sub-block being input to the synthesis transform network along with the corresponding rate control parameters), the bitstream of the current image block can achieve a bitstream rate close to or at the target bitstream. The decoder decodes the auxiliary bitstream corresponding to the current image block to obtain the N rate control parameters corresponding to the N stage sub-blocks.

[0255] Based on this, the clustering block unit can obtain the reconstruction features of N stage sub-blocks and the N rate control parameters corresponding to the N stage sub-blocks. For each stage sub-block, the reconstruction features of the stage sub-block and the corresponding rate control parameters are input together into the synthesis transform network. The synthesis transform network generates the block reconstruction image block x_hat_p corresponding to the stage sub-block based on the reconstruction features of the stage sub-block and the corresponding rate control parameters.

[0256] For example, the synthetic transform network generates a block-based reconstructed image block x_hat_p corresponding to the stage sub-block based on the reconstruction features of the stage sub-block and the corresponding bit rate control parameters of the stage sub-block. This may include, but is not limited to: processing the reconstruction features of the stage sub-block through the synthetic transform network to obtain a first feature; processing the bit rate control parameters of the stage sub-block through the synthetic transform network to obtain a second feature; generating a third feature based on the first feature and the second feature; and determining the block-based reconstructed image block x_hat_p corresponding to the stage sub-block based on the third feature.

[0257] Example 13: Based on Example 12, the synthesis transform network can be called a synthesis transform network controlled by a rate control parameter. The rate control parameter can be denoted as the λ parameter, which is obtained by decoding from the auxiliary bitstream corresponding to the current image block. See [link to example]. Figure 10A The diagram shows a schematic of a synthesis transform network controlled by the λ parameter. This synthesis transform network is a variable code rate synthesis transform network, and based on this network, output features with 1 or 3 channels can be obtained. Of course, Figure 10A This is just one example of a synthetic transform network, and there are no restrictions on the structure of this synthetic transform network.

[0258] See Figure 10A As shown, the λ-parameter-controlled synthetic transform network can include at least one Conv (convolutional layer) and at least one λ-RSTB (Residual Swin Transformer Block). Figure 10A Taking four convolutional layers and four λ-RSTBs as an example, the number of convolutional layers and λ-RSTBs can be more or less. Convolutional layers can be 1x1, 3x3, or 5x5; for example, a 3x3 convolutional layer can be chosen. The convolutional layers in the synthetic transform network can be ordinary 2D convolutions or grouped convolutions, such as 2x grouped convolutions. There are no restrictions on the structure of these convolutional layers. λ-RSTB refers to an RSTB network layer controlled by the λ parameter. For the structure of an RSTB network layer controlled by the λ parameter, see [link to documentation]. Figure 10B As shown.

[0259] See Figure 10B As shown, λ-RSTB can include, but is not limited to, network layers such as FE (Feature Embedding), λ-STB, and FU (Feature De-embedding). Of course, this is just an example, and there are no restrictions on the structure of this λ-RSTB. FE is the feature embedding layer, used to implement feature embedding processing, and FU is the feature de-embedding layer, used to implement feature de-embedding processing.

[0260] λ-STB represents an STB network layer controlled by the λ parameter. The structure of an STB network layer controlled by the λ parameter can be found in [link to documentation]. Figure 10B As shown, λ-STB can include, but is not limited to, LN, λ-WA, λ-SWA, MLP (Multilayer Perceptron), etc. Of course, this is just an example, and the structure of this λ-STB is not limited. LN is a linear layer, and λ-STB can include at least one LN; the number of LNs is not limited, but here we take 4 LNs as an example. MLP is a multilayer perceptron, and λ-STB can include at least one MLP; the number of MLPs is not limited, but here we take 2 MLPs as an example.

[0261] The structure of λ-SWA is the same as that of λ-WA. Here, we take λ-WA as an example. λ-WA represents a WA network layer controlled by the λ parameter. The structure of a WA network layer controlled by the λ parameter can be found in [link to documentation]. Figure 10CAs shown, λ-WA can include, but is not limited to, FC (Full Connection) layers, MatMul (matrix multiplication), Scale (scaling), SoftMax (logistic regression, also known as normalization), MatMul, etc. Of course, this is just an example, and there are no restrictions on the structure of this λ-WA.

[0262] The parameter λ can be mapped to a scaling factor through the FC layer. In other words, the rate control parameter is processed by the FC layer of the synthesis transform network to obtain the second feature, which can be the scaling factor.

[0263] The input features (such as block features y_hat_p, aggregated features y_hat_all, or reconstructed features of stage sub-blocks) can be processed by a transformer network to obtain Q features, K features, and V features. The Q features, K features, and V features all originate from the input features themselves and are feature vectors generated by multiplying the input features with the corresponding weights. There are no restrictions on this process. In other words, the synthetic transformation network can include a transformer network. The input features (such as block features, aggregated features, or reconstructed features) are processed by the transformer network of the synthetic transformation network to obtain the first feature.

[0264] After obtaining the Q-feature, K-feature, and V-features, matrix multiplication, scaling, and logistic regression operations can be performed sequentially on the Q-feature and K-feature to obtain intermediate feature 1. The scaling coefficient (obtained by mapping the parameter λ in the FC layer) can be multiplied by the V-feature to obtain intermediate feature 2. Multiplying intermediate feature 1 and intermediate feature 2 matrices yields the output feature of λ-WA. In summary, the scaling coefficient is the first feature, and the Q-feature, K-feature, and V-feature are the second features. Therefore, a third feature can be generated based on the first and second features; this third feature is the output feature of λ-WA.

[0265] After obtaining the output features of λ-WA, combined with Figure 10A and Figure 10B Therefore, the output features of the synthetic transform network can be determined based on the output features of λ-WA (i.e., the third feature), such as the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p, the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all, and the block reconstructed image block x_hat_p corresponding to the stage sub-block.

[0266] See Figure 10C The λ-WA shown below can be calculated as follows. This method allows the output of the image codec to be controlled by the λ parameter, thereby obtaining reconstructed images of different sizes and qualities.

[0267]

[0268] Example 14: In Examples 1-13, a convolutional layer may be involved. This convolutional layer can be a regular 2D convolution or a grouped convolution, such as a 2x grouped convolution. For example, the steps for a regular 2D convolution can be found in [link to example]. Figure 11A As shown, the input features are (H×W×C), and then C′ filters are applied (each filter has a size of (h×w×C), where C is the same as the input feature C), and the input layer is transformed into output features of size (H′×W′×C′).

[0269] The steps for grouped convolution can be found in [link to documentation]. Figure 11B As shown, each filter group can contain C′ / g (e.g., g = 2) filters, with each filter having half the number of channels as a regular 2D convolution filter. Each filter group operates on 1 / g of the original W×H×C channel count, which is W×H×C / g. Thus, each filter group outputs features with C′ / g channels. Finally, the channels can be stacked to obtain the final C′ channels, achieving the same effect as a regular 2D convolution. Clearly, grouped convolution effectively reduces the number of parameters and computational cost.

[0270] Example 15: In Examples 1-14, for precise bitrate control, the encoder needs to send bitrate control parameters to the decoder. These parameters are carried in the auxiliary bitstream corresponding to the current image block. At the encoder, the bitrate control parameters affect the analysis transform network and the synthesis transform network. To ensure that the bitrate corresponding to the current image block's bitstream is at or near the target bitrate (pre-configured bitrate), the encoder can try different bitrate control parameters to find the target bitrate control parameter. Under the target bitrate control parameter, the bitrate corresponding to the current image block's bitstream is at or near the target bitrate, and the encoder encodes the target bitrate control parameter in the auxiliary bitstream corresponding to the current image block.

[0271] If the current image patch corresponds to only one bitrate control parameter, such as the aggregated feature y_hat_all corresponding to one bitrate control parameter, then each bitrate control parameter can be tried sequentially from multiple bitrate control parameters until the bitrate corresponding to the current image patch is the target bitrate or close to the target bitrate, thereby obtaining the target bitrate control parameter.

[0272] If the current image block corresponds to multiple bitrate control parameters, such as K bitrate control parameters corresponding to K block features, or N bitrate control parameters corresponding to N stage sub-blocks, then first set the initial values ​​of the multiple bitrate control parameters. If the bitrate corresponding to the current image block's bitstream does not meet the requirements, then select a candidate bitrate control parameter from the multiple bitrate control parameters and decrease the value of the candidate bitrate control parameter. If the bitrate corresponding to the current image block's bitstream still does not meet the requirements, then continue to select a candidate bitrate control parameter from the multiple bitrate control parameters (the same or different from the previous candidate bitrate control parameter) and decrease the value of the candidate bitrate control parameter. This process continues until the bitrate corresponding to the current image block's bitstream meets the requirements (i.e., the bitrate corresponding to the current image block's bitstream is the target bitrate or close to the target bitrate), thus obtaining the target values ​​of the multiple bitrate control parameters.

[0273] When selecting a candidate bitrate control parameter from multiple bitrate control parameters, one can choose randomly, select the largest bitrate control parameter, or select a candidate bitrate control parameter based on the relationship between bitrate and distortion. For example, if the distortion is A1 when bitrate control parameter A decreases from the current level to the next level, and the distortion is B1 when bitrate control parameter B decreases from the current level to the next level, and if the distortion A1 is less than the distortion B1 (i.e., A1 is the minimum distortion), then bitrate control parameter A is selected as the candidate bitrate control parameter.

[0274] In one possible implementation, how does the encoding end set the λ of each (feature or image) block? i One implementation of the parameter is as follows: for each block x′ i (e.g., block features or stage sub-blocks), using different λ i Encoding is then performed. Before encoding, several discrete values ​​(λ) are first taken for each block, and these discrete λ values ​​are defined as quality levels. The bitrate and distortion for each block and each quality level are then calculated before encoding. This process only needs to be performed once for each frame of the image before the first encoding.

[0275] During the encoding process, λ is selected from the quality levels for each block. First, all blocks are encoded using the same quality level, which is the minimum quality level that ensures the total bitrate is greater than the target bitrate, satisfying the following formula: R t For the target bitrate, This represents the sum of the bitrates of all blocks at quality level q.

[0276]

[0277] Then, the quality level of each block is gradually reduced until the total bitrate is less than the target bitrate. For each block, the gradient between the current quality level and the previous quality level on the rate-distortion curve can be calculated using the following formula:

[0278]

[0279] in, This is the reconstruction loss of block i at quality level q. When using a loss function that decreases with bit rate (e.g., mean squared error) as a metric, the one with the smallest value is chosen. The block i (i.e., the candidate bitrate control parameter with the least distortion) is selected and its quality level is reduced by one. This process is repeated until the total bitrate is less than the target bitrate.

[0280] See Figure 12 The diagram shown illustrates the decision-making process for the λ parameter. Seven quality levels (i.e., seven possible values ​​for the λ parameter) are designed on the rate-distortion curve. At quality level 3, the following can be calculated: In other words, at quality level 3, reducing the quality level of block 2 would result in a greater loss of coding performance; therefore, the quality level of block 1 should be reduced.

[0281] As can be seen from the technical solutions of the above embodiments, this embodiment proposes a variable-rate, adjustable encoding and decoding scheme for neural network-based encoding and decoding technology. This improves parallelism, effectively saves feature storage cache, achieves higher rate control accuracy, and minimizes encoding performance loss, resulting in better encoding performance and rate control accuracy. By employing block encoding and decoding, peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is achieved. This allows the neural network to maintain low complexity while effectively ensuring the quality of reconstructed image blocks, improving encoding and decoding performance, and reducing complexity. For example, the rate control algorithm of this scheme can achieve a rate control accuracy of approximately 99%, while the encoding performance loss is only about 1% compared to no rate control. Taking decoding a 2k resolution image with a block size of 256*256 as an example, by employing block encoding and decoding, peak memory usage is reduced to 1 / 10 of the entire image decoding, while the decoding time of a single block is less than 0.1s, demonstrating high-speed parallel decoding capability. It achieves better encoding and decoding performance with fewer parameters.

[0282] Example 16: This application proposes a decoding method that can be applied to a decoding end (also known as a video decoder). The method may include: decoding a first bitstream of the current image block to obtain the coefficient hyperparameter features of the current image block; determining probability distribution parameters based on the coefficient hyperparameter features; decoding a second bitstream of the current image block based on the probability distribution parameters to obtain the residual features of the current image block; determining the reconstruction features of the current image block based on the residual features; decoding an auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters corresponding to the current image block; and inputting the reconstruction features and bitrate control parameters into a synthesis transform network to obtain the reconstructed image block corresponding to the current image block.

[0283] For example, inputting the reconstruction features and bit rate control parameters into the synthesis transform network to obtain the reconstructed image block corresponding to the current image block may include, but is not limited to: processing the reconstruction features through the synthesis transform network to obtain a first feature; processing the bit rate control parameters through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0284] For example, based on Embodiments 3 and 4, the synthetic transformation network can be adjusted. In Embodiment 16, only the adjustment process of the synthetic transformation network is described. For other processes, please refer to Embodiments 3 and 4.

[0285] For example, precise bitrate control can be achieved through bitrate control parameters, which affect both the analysis transform network and the synthesis transform network. For instance, when the encoder performs analysis transform on the current image patch using the analysis transform network, it inputs the bitrate control parameters and the current image patch to the analysis transform network, which then outputs the image features corresponding to the current image patch. Similarly, when the encoder or decoder inputs the image features y_hat to the synthesis transform network, it inputs both the image features y_hat and the bitrate control parameters, and the synthesis transform network outputs the reconstructed image patch corresponding to the current image patch.

[0286] Based on the configured target bitrate, the encoder can try different bitrate control parameters. After using a certain bitrate control parameter, if the bitrate corresponding to the current block's bitstream is the target bitrate or close to the target bitrate, then this bitrate control parameter is the final bitrate control parameter used, and the encoder can send this bitrate control parameter to the decoder.

[0287] For example, the encoder can send rate control parameters for the synthesis transform network to the decoder. For instance, the encoder can encode the rate control parameters corresponding to the current image block in the auxiliary bitstream corresponding to the current image block, and the decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the rate control parameters corresponding to the current image block, and determine the synthesis transform network based on the rate control parameters. The synthetic transform network can consist of at least one 2x2 deconvolutional layer (e.g., 2x2, 3x3, 4x4, or 5x5 2x2 deconvolutional layers, such as a 4x4 2x2 deconvolutional layer, or other deconvolutional layers), at least one activation layer (e.g., a ReLU layer, a leaky ReLU layer, a sigmoid layer, a tanh layer, or a GeLU layer, such as a ReLU layer), and at least one convolutional layer (e.g., a 1x1, 3x3, or 5x5 convolutional layer, such as a 3x3 convolutional layer; the convolutional layers in the synthetic transform network can be ordinary 2D convolutions or grouped convolutions, such as 2x grouped convolutions). The output feature space resolution of the synthetic transform network is a positive integer multiple of the input feature space resolution, such as 16 times (both width and height are 16 times), and the number of output feature channels is 1 or 3. Based on the synthetic transform network, the network can be controlled by a bitrate control parameter, which serves as the input to the synthetic transform network.

[0288] For example, the encoder encodes the rate control parameter corresponding to the current image patch in the auxiliary bitstream. The source of this rate control parameter is not restricted. When the encoder uses this rate control parameter to control the analysis transform network and the synthesis transform network (i.e., the rate control parameter is used as input to the analysis transform network and the synthesis transform network), it can ensure that the bitstream corresponding to the current image patch has a bitrate at or close to the target bitrate. The decoder can decode the auxiliary bitstream corresponding to the current image patch to obtain the rate control parameter. Based on this, the decoder can input the image feature y_hat and the rate control parameter together into the synthesis transform network. The synthesis transform network then generates a reconstructed image patch corresponding to the current image patch based on the image feature y_hat and the rate control parameter, and outputs the reconstructed image patch.

[0289] The synthetic transform network generates a reconstructed image block corresponding to the current image block based on the image feature y_hat and the bit rate control parameter. This process can include: processing the image feature y_hat (i.e., the reconstructed feature y_hat) through the synthetic transform network to obtain a first feature; processing the bit rate control parameter through the synthetic transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0290] For example, see Figure 10A , Figure 10B and Figure 10C As shown, the synthetic transform network can include at least one Conv (convolutional layer) and at least one λ-RSTB. λ-RSTB can include, but is not limited to, FE, λ-STB, FU, and other network layers. λ-STB can include, but is not limited to, LN, λ-WA, λ-SWA, MLP, etc. The structure of λ-SWA is the same as that of λ-WA. λ-WA can include, but is not limited to, FC layer, MatMul, Scale, SoftMax, MatMul, etc. Based on this, the parameter λ (i.e., the bitrate control parameter) can be mapped to a scaling factor through the FC layer. That is, the bitrate control parameter is processed by the FC layer of the synthetic transform network to obtain the second feature. The image feature y_hat can be processed by a transformer network to obtain Q features, K features, and V features. That is, the image feature y_hat is processed by the transformer network of the synthetic transform network to obtain the first feature. After obtaining the Q-feature, K-feature, and V-feature, matrix multiplication, scaling, and logistic regression operations can be performed sequentially on the Q-feature and K-feature to obtain intermediate feature 1. The scaling coefficient can then be multiplied by the V-feature to obtain intermediate feature 2. Multiplying intermediate feature 1 and intermediate feature 2 gives the output feature of λ-WA, i.e., generating the third feature based on the first and second features; the third feature is the output feature of λ-WA. After obtaining the output feature of λ-WA, combined with... Figure 10A and Figure 10B Therefore, the output features of the synthetic transformation network can be determined based on the output features of λ-WA (i.e., the third feature). For example, the output features can be the reconstructed image block corresponding to the current image block.

[0291] For example, the above embodiments can be implemented individually or in combination. For instance, each of embodiments 1-16 can be implemented individually, and at least two of embodiments 1-16 can be implemented in combination.

[0292] For example, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0293] For example, in the above embodiments, the content of different embodiments can be borrowed from each other. For example, the content of embodiment 3 can be applied to embodiment 16, the content of embodiment 4 can be applied to embodiment 16, the content of embodiments 5-15 can be applied to embodiment 16, and the content of embodiments 5-15 can be borrowed from each other without limitation.

[0294] Based on the same application concept as the above method, this application also proposes a decoding device, which is applied to the decoding end. The device includes: a memory configured to store video data; and a decoder configured to implement the decoding methods in embodiments 1-16 above, i.e., the processing flow of the decoding end.

[0295] For example, in one possible implementation, the decoder is configured to:

[0296] Decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block;

[0297] For each stage sub-block, probability distribution parameters are determined based on the coefficient hyperparameter features of that stage sub-block. The second bitstream of the current image block is then decoded based on the probability distribution parameters to obtain the residual features of that stage sub-block.

[0298] The reconstruction characteristics of the sub-block in this stage are determined based on the residual characteristics and the mean characteristics of the sub-block in this stage.

[0299] The reconstructed image block corresponding to the current image block is determined based on the reconstruction features of each stage sub-block.

[0300] Based on the same application concept as the above method, this application also proposes an encoding device, which is applied to the encoding end. The device includes: a memory configured to store video data; and an encoder configured to implement the encoding methods in embodiments 1-16 above, i.e., the processing flow of the encoding end.

[0301] For example, in one possible implementation, the encoder is configured to:

[0302] The current image block is input into the analysis and transformation network to obtain the feature block corresponding to the current image block;

[0303] The feature block is divided into multiple stage sub-blocks of features to be encoded;

[0304] For each stage sub-block corresponding to the current image block, the coefficient hyperparameter features of that stage sub-block are obtained, and the coefficient hyperparameter features of that stage sub-block are encoded into the first bitstream of the current image block;

[0305] The residual characteristics of the sub-block in this stage are determined based on the features to be encoded and the mean features of the sub-block in this stage.

[0306] The probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block in this stage, and the residual features of the sub-block in this stage are encoded into the second bitstream of the current image block based on the probability distribution parameters.

[0307] Based on the same concept as the above method, the decoding device (also known as a video decoder) provided in this application embodiment, from a hardware perspective, its hardware architecture diagram can be found in [reference needed]. Figure 13A As shown, it includes a processor 1301 and a machine-readable storage medium 1302. The machine-readable storage medium 1302 stores machine-executable instructions that can be executed by the processor 1301. The processor 1301 is used to execute the machine-executable instructions to implement the decoding methods of embodiments 1-16 of this application. For example, in one possible implementation, when the processor 1301 executes the machine-executable instructions, it performs the following steps:

[0308] Decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block;

[0309] For each stage sub-block, probability distribution parameters are determined based on the coefficient hyperparameter features of that stage sub-block. The second bitstream of the current image block is then decoded based on the probability distribution parameters to obtain the residual features of that stage sub-block.

[0310] The reconstruction characteristics of the sub-block in this stage are determined based on the residual characteristics and the mean characteristics of the sub-block in this stage.

[0311] The reconstructed image block corresponding to the current image block is determined based on the reconstruction features of each stage sub-block.

[0312] Based on the same concept as the above method, the encoding end device (also known as a video encoder) provided in this application embodiment, from a hardware perspective, its hardware architecture diagram can be found in [reference needed]. Figure 13B As shown, it includes a processor 1311 and a machine-readable storage medium 1312. The machine-readable storage medium 1312 stores machine-executable instructions that can be executed by the processor 1311. The processor 1311 is used to execute the machine-executable instructions to implement the encoding methods of embodiments 1-16 of this application. For example, in one possible implementation, when the processor 1311 executes the machine-executable instructions, it performs the following steps:

[0313] The current image block is input into the analysis and transformation network to obtain the feature block corresponding to the current image block;

[0314] The feature block is divided into multiple stage sub-blocks of features to be encoded;

[0315] For each stage sub-block corresponding to the current image block, the coefficient hyperparameter features of that stage sub-block are obtained, and the coefficient hyperparameter features of that stage sub-block are encoded into the first bitstream of the current image block;

[0316] The residual characteristics of the sub-block in this stage are determined based on the features to be encoded and the mean features of the sub-block in this stage.

[0317] The probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block in this stage, and the residual features of the sub-block in this stage are encoded into the second bitstream of the current image block based on the probability distribution parameters.

[0318] Based on the same application concept as the methods described above, this application provides an electronic device. It includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor executes the machine-executable instructions to implement the decoding or encoding methods of embodiments 1-16 of this application described above.

[0319] Based on the same application concept as the above methods, embodiments of this application also provide a machine-readable storage medium storing a plurality of computer instructions. When the computer instructions are executed by a processor, they can implement the methods disclosed in the above examples of this application, such as the decoding method or encoding method in the above embodiments.

[0320] Based on the same application concept as the above method, this application embodiment also provides a computer application that, when executed by a processor, can implement the decoding method or encoding method disclosed in the above examples of this application.

[0321] Based on the same concept as the above method, this application also proposes a decoding device that can be applied to a decoding end. The decoding device includes: a decoding module, used to decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, to determine the probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block, and to decode the second bitstream of the current image block based on the probability distribution parameters to obtain the residual features of the stage sub-block; and a determination module, used to determine the reconstruction features of the stage sub-block based on the residual features and the mean features of the stage sub-block; and to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block.

[0322] For example, the determining module is further configured to, for the first stage sub-block, obtain the mean feature of the stage sub-block through a mean prediction network based on the coefficient hyperparameter features of the stage sub-block, or, obtain a configured default reference feature and obtain the mean feature of the stage sub-block through a mean prediction network based on the coefficient hyperparameter features of the stage sub-block and the default reference feature; or, for the i-th stage sub-block, where i is greater than 1, obtain the reference feature of the i-th stage sub-block based on the reconstruction features of the i-1 preceding stage sub-blocks; obtain the mean feature of the stage sub-block through a mean prediction network based on the coefficient hyperparameter features of the stage sub-block and the reference feature; wherein the reference feature includes all reconstruction features of the i-1 preceding stage sub-blocks; or, some reconstruction features of the i-1 preceding stage sub-blocks; or, the reconstruction feature of the i-1 stage sub-block.

[0323] For example, the mean prediction network includes a first prediction network, a second prediction network, and a prediction fusion network. When the determining module obtains the mean feature of the sub-block based on the coefficient hyperparameter features and the reference features of the sub-block, it specifically performs the following steps: obtaining the first prediction feature corresponding to the coefficient hyperparameter features through the first prediction network; obtaining the second prediction feature corresponding to the reference features through the second prediction network; concatenating the first prediction feature and the second prediction feature and inputting them into the prediction fusion network; and processing the concatenated features through the prediction fusion network to obtain the mean feature of the sub-block.

[0324] For example, when the determining module obtains the first predicted feature corresponding to the coefficient hyperparameter feature through the first prediction network, it is specifically used to: perform feature enhancement and upsampling operations on the coefficient hyperparameter feature through the first prediction network to obtain the first predicted feature; the feature enhancement operation includes: convolution operation, or convolution operation and activation operation; the upsampling operation includes: deconvolution operation, cropping operation and activation operation, or deconvolution operation, cropping operation, activation operation and convolution operation. When the determining module obtains the second predicted feature corresponding to the reference feature through the second prediction network, it is specifically used to: concatenate all reconstructed features in the reference feature according to the channel dimension; perform convolution operation on the concatenated features to obtain the second predicted feature; or, add all reconstructed features in the reference feature and perform convolution operation on the added features to obtain the second predicted feature. The determining module processes the spliced ​​features through the predictive fusion network to obtain the mean features of the sub-blocks in this stage. Specifically, this is done by performing convolution and at least one fusion operation on the spliced ​​features through the predictive fusion network to obtain the mean features of the sub-blocks in this stage. The fusion operation includes activation and convolution operations.

[0325] For example, the determining module is further configured to perform feature enhancement on the reconstruction features of the stage sub-block to obtain enhanced reconstruction features; wherein, the enhanced reconstruction features are used to determine the mean features of the stage sub-block, and the enhanced reconstruction features are used to determine the reconstructed image block.

[0326] For example, when the determining module determines the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block, it specifically performs the following: feature aggregation on the reconstruction features of each stage sub-block to obtain aggregated features; inputting the aggregated features into the synthesis transform network to obtain the reconstructed image block corresponding to the current image block; or, feature aggregation on the reconstruction features of each stage sub-block to obtain aggregated features; dividing the aggregated features into blocks to obtain multiple block features; inputting each block feature into the synthesis transform network to obtain block reconstructed image blocks; merging the block reconstructed image blocks corresponding to multiple block features to obtain the reconstructed image block corresponding to the current image block; or, inputting the reconstruction features of each stage sub-block into the synthesis transform network to obtain block reconstructed image blocks; merging the block reconstructed image blocks corresponding to all stage sub-blocks to obtain the reconstructed image block corresponding to the current image block.

[0327] For example, when the determining module performs feature aggregation on the reconstruction features of each stage sub-block to obtain aggregated features, it specifically performs the following: aggregates the reconstruction features of each stage sub-block according to phase to obtain the aggregated features; or, aggregates the reconstruction features of each stage sub-block according to phase to obtain multiple phase-aggregated features, and splices the multiple phase-aggregated features according to channels to obtain the aggregated features.

[0328] For example, when the determining module divides the aggregated feature into multiple block features, it is specifically used to: determine the target size of the block features; divide the aggregated feature into multiple block features in an equal order from top to bottom and from left to right based on the target size, and the size of each block feature is the target size; or, determine the actual block size and overlap size of the block features, where the actual block size is the block size of the block features after removing the overlapping part, and the overlap size is the size of the overlapping part of adjacent blocks; divide the aggregated feature into multiple block features based on the actual block size and the overlap size, and the size of each block feature is the target size.

[0329] For example, when the determining module merges multiple reconstructed image blocks corresponding to segmented features to obtain the reconstructed image block corresponding to the current image block, it specifically performs the following steps: sorting the multiple reconstructed image blocks corresponding to segmented features in a top-to-bottom and left-to-right order, and sequentially stitching the sorted multiple reconstructed image blocks together to obtain the reconstructed image block corresponding to the current image block; or, determining the actual segment size and overlap size of each reconstructed image block; sorting the multiple reconstructed image blocks corresponding to segmented features in a top-to-bottom and left-to-right order; and sequentially stitching the sorted multiple reconstructed image blocks together based on the actual segment size and the overlap size. The images are stitched together to obtain the reconstructed image block corresponding to the current image block. The size of the non-overlapping portion of the reconstructed image block is the actual block size of the reconstructed image block, and the size of the overlapping portion of the reconstructed image block with the adjacent reconstructed image block is the overlap size of the reconstructed image block. For the overlapping portion of the left and right reconstructed image blocks, the value of the overlapping portion is the value of the left reconstructed image block, or the value of the overlapping portion is the average value of the two reconstructed image blocks. For the overlapping portion of the top and bottom reconstructed image blocks, the value of the overlapping portion is the value of the top reconstructed image block, or the value of the overlapping portion is the average value of the two reconstructed image blocks.

[0330] For example, when the determining module inputs the aggregated features to the synthetic transform network to obtain the reconstructed image block corresponding to the current image block, it specifically performs the following steps: decoding the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters of the current image block, and inputting the bitrate control parameters and the aggregated features to the synthetic transform network to obtain the reconstructed image block corresponding to the current image block; or, when the determining module inputs each block feature to the synthetic transform network to obtain the block reconstructed image block, it specifically performs the following steps: decoding the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters of each block feature, and inputting the block feature and the bitrate control parameters of the block feature to the synthetic transform network to obtain the block reconstructed image block corresponding to the block feature; or, when the determining module inputs the reconstructed features of each stage sub-block to the synthetic transform network to obtain the block reconstructed image block, it specifically performs the following steps: decoding the auxiliary bitstream corresponding to the current image block to obtain the bitrate control parameters of each stage sub-block, and inputting the reconstructed features of the stage sub-block and the bitrate control parameters of the stage sub-block to the synthetic transform network to obtain the block reconstructed image block.

[0331] For example, when the determining module inputs the bitrate control parameter and the aggregated features into the synthesis transform network to obtain the reconstructed image block corresponding to the current image block, it specifically performs the following steps: processing the aggregated features through the synthesis transform network to obtain a first feature; processing the bitrate control parameter through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features, and determining the reconstructed image block corresponding to the current image block based on the third feature; or, when the determining module inputs the block features and the bitrate control parameter of the block features into the synthesis transform network to obtain the reconstructed image block, it specifically performs the following steps: processing the block features through the synthesis transform network to obtain a first feature. The process involves: processing the rate control parameters through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the block-based reconstructed image block corresponding to the third feature. Alternatively, when the determining module inputs the reconstruction features of the stage sub-block and the rate control parameters of the stage sub-block to the synthesis transform network to obtain the block-based reconstructed image block, it specifically performs the following steps: processing the reconstruction features of the stage sub-block through the synthesis transform network to obtain a first feature; processing the rate control parameters through the synthesis transform network to obtain a second feature; generating a third feature based on the first and second features; and determining the block-based reconstructed image block corresponding to the stage sub-block based on the third feature.

[0332] Based on the same concept as the above method, this application also proposes an encoding device, which is applied at the encoding end. The device includes: an acquisition module, used to input a current image block into an analysis transform network to obtain a feature block corresponding to the current image block; to split the feature block into multiple stage sub-blocks of features to be encoded; an encoding module, used to acquire the coefficient hyperparameter features of each stage sub-block corresponding to the current image block, and to encode the coefficient hyperparameter features of the stage sub-block into a first bitstream of the current image block; a determination module, used to determine the residual features of the stage sub-block based on the features to be encoded and the mean features of the stage sub-block; the encoding module is further used to determine a probability distribution parameter based on the coefficient hyperparameter features of the stage sub-block, and to encode the residual features of the stage sub-block into a second bitstream of the current image block based on the probability distribution parameter.

[0333] For example, the determining module is further configured to, for the first stage sub-block, obtain the mean feature of the stage sub-block through a mean prediction network based on the coefficient hyperparameter features of the stage sub-block, or, obtain a configured default reference feature and obtain the mean feature of the stage sub-block through a mean prediction network based on the coefficient hyperparameter features of the stage sub-block and the default reference feature; or, for the i-th stage sub-block, where i is greater than 1, obtain the reference feature of the i-th stage sub-block based on the reconstruction features of the i-1 preceding stage sub-blocks; obtain the mean feature of the stage sub-block through a mean prediction network based on the coefficient hyperparameter features of the stage sub-block and the reference feature; wherein the reference feature includes all reconstruction features of the i-1 preceding stage sub-blocks; or, some reconstruction features of the i-1 preceding stage sub-blocks; or, the reconstruction feature of the i-1 stage sub-block.

[0334] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above descriptions are merely embodiments of this application and are not intended to limit this application.

[0335] Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A decoding method, comprising: The method comprises: decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block; for each stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; determining a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and a mean feature of the stage sub-block; determining a reconstruction image block corresponding to the current image block based on the reconstruction feature of each stage sub-block.

2. The method of claim 1, wherein, Before the determining of the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block, the method further comprises: for a first stage sub-block, obtaining a mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block through a mean prediction network, or obtaining a configured default reference feature, and obtaining the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature through the mean prediction network; or for an i-th stage sub-block, i being greater than 1, obtaining a reference feature of the i-th stage sub-block based on reconstruction features of i-1 stage sub-blocks in front of the i-th stage sub-block; and obtaining the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through the mean prediction network; wherein the reference feature comprises all reconstruction features of the i-1 stage sub-blocks in front; or part of the reconstruction features of the i-1 stage sub-blocks in front; or the reconstruction feature of the i-1 stage sub-block; wherein the mean prediction network comprises a first prediction network, a second prediction network, and a prediction fusion network, and the obtaining of the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through the mean prediction network comprises: obtaining a first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network; obtaining a second prediction feature corresponding to the reference feature through the second prediction network; inputting the first prediction feature and the second prediction feature after feature splicing to the prediction fusion network, processing the spliced features through the prediction fusion network, and obtaining the mean feature of the stage sub-block.

3. The method of claim 2, wherein: the obtaining of the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network comprises: performing feature enhancement operation and up-sampling operation on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature; the feature enhancement operation comprises convolution operation, or convolution operation and activation operation; and the up-sampling operation comprises deconvolution operation, cropping operation and activation operation, or deconvolution operation, cropping operation, activation operation and convolution operation.

4. The method of claim 2, wherein: the obtaining of the second prediction feature corresponding to the reference feature through the second prediction network comprises: The features in the reference features are spliced according to the channel dimension; the spliced features are subjected to convolution operation to obtain the second prediction features; or the features in the reference features are added, and the added features are subjected to convolution operation to obtain the second prediction features.

5. The method of claim 2, wherein, the processing of the spliced features by the prediction fusion network to obtain the mean features of the sub-blocks in the stage comprises: the spliced features are subjected to convolution operation and at least one fusion operation by the prediction fusion network to obtain the mean features of the sub-blocks in the stage; wherein the fusion operation comprises activation operation and convolution operation.

6. The method according to any one of claims 1 to 5, characterized in that, after the determination of the reconstructed features of the sub-blocks in the stage based on the residual features of the sub-blocks in the stage and the mean features of the sub-blocks in the stage, the method comprises: the reconstructed features of the sub-blocks in the stage are subjected to feature enhancement to obtain enhanced reconstructed features; wherein the enhanced reconstructed features are used to determine the mean features of the sub-blocks in the stage, and the enhanced reconstructed features are used to determine the reconstructed image block.

7. The method of any one of claims 1-5, wherein, the determination of the reconstructed image block corresponding to the current image block based on the reconstructed features of each sub-block in the stage comprises: the reconstructed features of each sub-block in the stage are subjected to feature aggregation to obtain aggregated features; the aggregated features are input into a synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; wherein the auxiliary code stream corresponding to the current image block is decoded to obtain the rate control parameters of the current image block, the rate control parameters and the aggregated features are input into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; or, the reconstructed features of each sub-block in the stage are subjected to feature aggregation to obtain aggregated features; the aggregated features are subjected to blocking to obtain a plurality of blocked features; each blocked feature is input into a synthesis transformation network to obtain a blocked reconstructed image block; the blocked reconstructed image blocks corresponding to the plurality of blocked features are merged to obtain the reconstructed image block corresponding to the current image block; wherein the auxiliary code stream corresponding to the current image block is decoded to obtain the rate control parameters of each blocked feature, the blocked feature and the rate control parameters of the blocked feature are input into the synthesis transformation network to obtain the blocked reconstructed image block corresponding to the blocked feature; or, the reconstructed features of each sub-block in the stage are input into a synthesis transformation network to obtain a blocked reconstructed image block; the blocked reconstructed image blocks corresponding to all the sub-blocks in the stage are merged to obtain the reconstructed image block corresponding to the current image block; wherein the auxiliary code stream corresponding to the current image block is decoded to obtain the rate control parameters of each sub-block in the stage, the reconstructed features of the sub-block and the rate control parameters of the sub-block are input into the synthesis transformation network to obtain the blocked reconstructed image block.

8. The method of claim 7, wherein, the feature aggregation of the reconstructed features of each sub-block in the stage to obtain aggregated features comprises: the reconstructed features of each sub-block in the stage are aggregated according to the phase to obtain the aggregated features; or, The reconstruction features of each stage sub-block are aggregated according to phases to obtain a plurality of phase-aggregated features, and the plurality of phase-aggregated features are spliced according to channels to obtain the aggregated features.

9. The method of claim 7, wherein, The aggregated features are divided into a plurality of block features, including: determining a target size of the block features; and based on the target size, averaging the aggregated features into the plurality of block features in a top-to-bottom and left-to-right order, and each block feature has a size of the target size; or determining an actual block size and an overlap size of the block features, the actual block size being a block size of the block features excluding the overlap size, and the overlap size being a size of an overlap portion between adjacent block features; and based on the actual block size and the overlap size, dividing the aggregated features into the plurality of block features, and each block feature has a size of the target size.

10. The method of claim 9, wherein, The plurality of block features are merged to obtain a reconstruction image block corresponding to the current image block, including: sorting the plurality of block features in a top-to-bottom and left-to-right order, and splicing the sorted plurality of block features in sequence to obtain the reconstruction image block corresponding to the current image block; or determining an actual block size and an overlap size of each block feature; sorting the plurality of block features in a top-to-bottom and left-to-right order; and based on the actual block size and the overlap size, splicing the sorted plurality of block features in sequence to obtain the reconstruction image block corresponding to the current image block; wherein, a non-overlap portion of a block feature has a size of the actual block size of the block feature, and an overlap portion between the block feature and an adjacent block feature has a size of the overlap size of the block feature; for an overlap portion between a left block feature and a right block feature, a value of the overlap portion is a value of the left block feature, or a value of the overlap portion is an average value of the left block feature and the right block feature; for an overlap portion between an upper block feature and a lower block feature, a value of the overlap portion is a value of the upper block feature, or a value of the overlap portion is an average value of the upper block feature and the lower block feature.

11. An encoding method, characterized by, The method includes: inputting the current image block into an analysis transformation network to obtain a feature block corresponding to the current image block; splitting the feature block into to-be-encoded features of a plurality of stage sub-blocks; for each stage sub-block corresponding to the current image block, obtaining a coefficient hyperparameter feature of the stage sub-block, and encoding the coefficient hyperparameter feature of the stage sub-block into a first code stream of the current image block; determining a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and a mean feature of the stage sub-block; determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encoding the residual feature of the stage sub-block into a second code stream of the current image block based on the probability distribution parameter.

12. The method of claim 11, wherein, Before determining the residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block, the method further comprises: For the first stage sub-block, the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block through a mean prediction network, or a default reference feature configured is obtained, and the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature through the mean prediction network; or for the i-th stage sub-block, i is greater than 1, a reference feature of the i-th stage sub-block is obtained based on the reconstructed features of the previous i-1 stage sub-blocks; the mean feature of the i-th stage sub-block is obtained based on the coefficient hyperparameter feature of the i-th stage sub-block and the reference feature through the mean prediction network; wherein the reference feature comprises all reconstructed features of the previous i-1 stage sub-blocks; or part of the reconstructed features of the previous i-1 stage sub-blocks; or the reconstructed feature of the i-1-th stage sub-block.

13. A decoding apparatus, characterized by comprising: The apparatus comprises: a decoding module configured to decode a first code stream of a current image block to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block; for each stage sub-block, determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decode a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; a determining module configured to determine a reconstructed feature of each stage sub-block based on the residual feature of the stage sub-block and a mean feature of the stage sub-block, and determine a reconstructed image block corresponding to the current image block based on the reconstructed features of the stage sub-blocks.

14. An encoding apparatus, comprising: The apparatus comprises: an obtaining module configured to input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block, and split the feature block into to-be-encoded features of multiple stage sub-blocks; an encoding module configured to, for each stage sub-block corresponding to the current image block, obtain a coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into a first code stream of the current image block; a determining module configured to determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and a mean feature of the stage sub-block; the encoding module is further configured to determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into a second code stream of the current image block based on the probability distribution parameter.

15. A decoding-side device, comprising: comprise: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method in any one of claims 1-10.

16. An encoding end device, comprising: comprise: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method in any one of claims 11-12.

17. A machine-readable storage medium, characterized in that, The machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-10, or the computer instructions are executed by the processor to implement the method in any one of claims 11-12.

Images