Decoding and coding method, device and equipment

Abstract

The invention provides a decoding and coding method, device and equipment, and the decoding method comprises the steps: decoding a first code stream of a current image block, and obtaining the coefficient hyper-parameter characteristics of each stage of sub-block of the current image block; for each stage sub-block, determining a probability distribution parameter based on a coefficient hyper-parameter feature of the stage sub-block, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; determining a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block; and determining a reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks in each stage. Through the technical scheme of the invention, the coding performance and the decoding performance can be improved.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding and coding method, device and equipment thereof. Background Art

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to use the pixels of the adjacent encoded images to predict the current pixel, so as to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to use the pixels of the encoded blocks of the current frame image to predict the current pixel, so as to remove the redundancy in the video space domain.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor encoding performance, poor decoding performance and high complexity. Summary of the invention

[0004] In view of this, the present application provides a decoding and encoding method, device and equipment thereof to improve encoding performance and decoding performance.

[0005] The present application provides a decoding method, which is applied to a decoding end, and the method includes:

[0006] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;

[0007] For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyper-parameter feature of the sub-block at the stage, and a second bitstream of the current image block is decoded based on the probability distribution parameter to obtain a residual feature of the sub-block at the stage;

[0008] Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0009] A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.

[0010] The present application provides an encoding method, which is applied to an encoding end, and the method includes:

[0011] Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0012] Splitting the feature block into features to be encoded of multiple stage sub-blocks;

[0013] For each stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter feature of the stage sub-block, and encode the coefficient hyper-parameter feature of the stage sub-block into the first bitstream of the current image block;

[0014] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0015] Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.

[0016] The present application provides a decoding method, which is applied to a decoding end, and the method includes:

[0017] Decoding the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block;

[0018] Determine probability distribution parameters based on the coefficient hyperparameter features, decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the current image block, and determine reconstruction features of the current image block based on the residual features;

[0019] The auxiliary code stream corresponding to the current image block is decoded to obtain a rate control parameter corresponding to the current image block, and the reconstruction feature and the rate control parameter are input into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.

[0020] The present application provides a decoding device, applied to a decoding end, the device comprising:

[0021] A decoding module is used to decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, determine the probability distribution parameters based on the coefficient hyperparameter features of the sub-block at that stage, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain the residual features of the sub-block at that stage;

[0022] The determination module is used to determine the reconstruction features of the sub-blocks at this stage based on the residual features of the sub-blocks at this stage and the mean features of the sub-blocks at this stage; and determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage.

[0023] The present application provides a coding device, applied to a coding end, the device comprising:

[0024] The acquisition module is used to input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks;

[0025] An encoding module, configured to obtain, for each stage sub-block corresponding to the current image block, coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into a first bitstream of the current image block;

[0026] A determination module, used to determine the residual features of the sub-block at this stage based on the to-be-encoded features of the sub-block at this stage and the mean features of the sub-block at this stage;

[0027] The encoding module is further used to determine probability distribution parameters based on coefficient hyperparameter features of the sub-block at this stage, and encode the residual features of the sub-block at this stage into the second code stream of the current image block based on the probability distribution parameters.

[0028] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0029] The processor is used to execute machine executable instructions to implement the above decoding method.

[0030] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0031] The processor is used to execute machine executable instructions to implement the above encoding method.

[0032] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method; or, the processor is used to execute the machine-executable instructions to implement the above-mentioned encoding method.

[0033] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned decoding method is implemented; or, the above-mentioned encoding method is implemented.

[0034] The present application provides a computer application, which, when executed by a processor, implements the above-mentioned decoding method; or, when executed by a processor, implements the above-mentioned encoding method.

[0035] As can be seen from the above technical solutions, in the embodiment of the present application, a coding and decoding scheme with variable and adjustable bit rate is proposed for the coding and decoding technology based on neural network, which improves parallelism, effectively saves feature storage cache, achieves higher bit rate control accuracy, and has less coding performance loss, and has better coding performance and bit rate control accuracy. By adopting block coding and decoding, the peak memory occupancy is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is provided, so that the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, improve coding performance and decoding performance, and reduce complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;

[0037] Figure 2 is a flowchart of a decoding method in one embodiment of the present application;

[0038] Figure 3 is a flowchart of an encoding method in one embodiment of the present application;

[0039] Figure 4 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;

[0040] Figure 5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0041] Figure 6 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;

[0042] Figure 7 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0043] Figure 8A-8H It is a schematic diagram of the structure of a mean prediction network in one embodiment of the present application;

[0044] Figure 9A-9F is a schematic diagram of aggregation and segmentation in one embodiment of the present application;

[0045] Figure 10A-10C is a schematic diagram of the structure of a synthetic transformation network in one embodiment of the present application;

[0046] Fig.11A and Fig. 11B is a processing schematic diagram of a convolutional layer in one embodiment of the present application;

[0047] Fig.12 is a schematic diagram of decision making of the λ parameter in one embodiment of the present application;

[0048] Fig.13A is a hardware structure diagram of a decoding end device in one embodiment of the present application;

[0049] Fig. 13B It is a hardware structure diagram of an encoding end device in one implementation of the present application. DETAILED DESCRIPTION

[0050] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at ... ", or "when ... ", or "in response to determination".

[0051] In the embodiments of the present application, a decoding method and an encoding method are proposed, which may involve the following concepts:

[0052] Entropy coding: Entropy coding is the coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information of the source (a measure of uncertainty). Entropy coding methods may include but are not limited to: Shannon coding, Huffman coding, and arithmetic coding.

[0053] Neural Network (NN): Neural network refers to artificial neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the outside world; output units realize the output of processing results; hidden units are units between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural network, and imitate the information processing function of human brain nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0054] Convolutional Neural Network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function (S-type function), ReLU function (rectified linear unit function), Leaky-ReLU function, PReLU function (rectified linear unit function with leakage), GDN Generalized Difference Network (generalized split normalization) function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping surface share weights, the number of free parameters of the network is reduced.

[0055] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through local connections, weight sharing and other methods.

[0056] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input through padding (of course, it can also remain the same). If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0057] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to fresh samples. The purpose of learning is to learn the rules hidden behind the data. For data outside the learning set with the same rules, the trained network can also give appropriate output. This ability can be called generalization ability.

[0058] Feature: The feature involved in this application is a three-dimensional feature matrix or tensor of C*W*H, see Figure 1 As shown, it is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of a neural network or the output of a neural network.

[0059] Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J(mode) = D + λ*R, where D represents Distortion, which can usually be measured using the SSE indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to achieve cost considerations, the SAD indicator can also be used. SAD refers to the sum of the absolute values ​​of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make a comparison decision on the encoding mode, the best encoding performance can usually be guaranteed.

[0060] A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best tool or mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal combination of tools or modes for different content to obtain the best coding performance. The decoding end can obtain relevant tool or mode information by directly parsing the flag information, and the complexity is less affected.

[0061] The decoding method and encoding method in the embodiments of the present application are described in detail below in combination with several embodiments.

[0062] Example 1: A decoding method is proposed in the embodiment of the present application, see Figure 2 As shown, it is a flowchart of the decoding method, which can be applied to a decoding end (also called a video decoder), and the method may include:

[0063] Step 201: decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage subblock of the current image block, wherein the current image block is divided into a plurality of stage subblocks, that is, the current image block includes a plurality of stage subblocks.

[0064] Step 202: for each stage sub-block, determine probability distribution parameters based on coefficient hyperparameter features of the stage sub-block, decode the second bitstream of the current image block based on the probability distribution parameters, and obtain residual features of the stage sub-block.

[0065] Step 203: Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage. The mean feature of the sub-block at this stage is obtained based on the coefficient hyper-parameter feature of the sub-block at this stage and / or the reference feature of the sub-block at this stage, such as based on the coefficient hyper-parameter feature, or based on the coefficient hyper-parameter feature and the reference feature.

[0066] Step 204: Determine a reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage.

[0067] Exemplarily, for the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or, the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature and the default reference feature of the sub-block at this stage.

[0068] For the i-th stage sub-block, i is greater than 1, and the reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the i-1 stage sub-blocks before the i-th stage sub-block; the mean feature of the stage sub-block is obtained through a mean prediction network based on the coefficient hyper-parameter feature and the reference feature of the stage sub-block; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction features of the i-1-th stage sub-block.

[0069] Exemplarily, the mean prediction network may include a first prediction network, a second prediction network and a prediction fusion network. The mean feature of the sub-block at this stage is obtained through the mean prediction network based on the coefficient hyper-parameter feature and the reference feature of the sub-block at this stage. It may include: obtaining the first prediction feature corresponding to the coefficient hyper-parameter feature through the first prediction network; obtaining the second prediction feature corresponding to the reference feature through the second prediction network; splicing the first prediction feature and the second prediction feature to obtain a spliced ​​feature, inputting the spliced ​​feature into the prediction fusion network, processing the spliced ​​feature through the prediction fusion network, and obtaining the mean feature of the sub-block at this stage.

[0070] Exemplarily, obtaining the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network may include: performing a feature enhancement operation and an upsampling operation on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature. The feature enhancement operation may include: a convolution operation, or a convolution operation and an activation operation; the upsampling operation may include: a deconvolution operation, a cropping operation, and an activation operation, or a deconvolution operation, a cropping operation, an activation operation, and a convolution operation. Exemplarily, the activation operation is a Relu operation.

[0071] Exemplarily, obtaining the second prediction feature corresponding to the reference feature through the second prediction network may include: performing feature splicing on all the reconstructed features in the reference feature according to the channel dimension; performing a convolution operation on the spliced ​​features to obtain the second prediction feature; or, performing feature addition on all the reconstructed features in the reference feature, and performing a convolution operation on the added features to obtain the second prediction feature. The reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, partial reconstruction features of the previous i-1 stage sub-blocks; or, reconstruction features of the i-1th stage sub-block.

[0072] Exemplarily, processing the spliced ​​features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage may include but is not limited to: performing a convolution operation and at least one fusion operation on the spliced ​​features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage; wherein the fusion operation includes an activation operation and a convolution operation. Exemplarily, the activation operation is a Relu operation.

[0073] Exemplarily, after determining the reconstruction features of the sub-block at this stage based on the residual features of the sub-block at this stage and the mean features of the sub-block at this stage, the reconstruction features of the sub-block at this stage can also be enhanced to obtain enhanced reconstruction features; wherein the enhanced reconstruction features are used to determine the mean features of the sub-block at this stage, and the enhanced reconstruction features are used to determine the reconstructed image block.

[0074] Exemplarily, determining the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage may include, but is not limited to: feature aggregation may be performed on the reconstruction features of the sub-blocks at each stage to obtain aggregated features, and the aggregated features may be input into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block. Alternatively, feature aggregation may be performed on the reconstruction features of the sub-blocks at each stage to obtain aggregated features, and the aggregated features may be divided into blocks to obtain multiple block features; each block feature may be input into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature, and the block-reconstructed image blocks corresponding to the multiple block features may be merged to obtain a reconstructed image block corresponding to the current image block. Alternatively, the reconstruction features of the sub-blocks at each stage may be input into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the sub-blocks at that stage, and the block-reconstructed image blocks corresponding to the sub-blocks at all stages may be merged to obtain a reconstructed image block corresponding to the current image block.

[0075] Exemplarily, the reconstructed features of each stage sub-block are aggregated to obtain aggregated features, which may include but is not limited to: the reconstructed features of each stage sub-block may be aggregated according to phase to obtain aggregated features; or, the reconstructed features of each stage sub-block may be aggregated according to phase to obtain features after multiple phase aggregation, and then, the features after multiple phase aggregation may be spliced ​​according to channels to obtain aggregated features.

[0076] Exemplarily, dividing the aggregated features into multiple block features may include: determining a target size of the block features; dividing the aggregated features into multiple block features in an even order from top to bottom and from left to right based on the target size, and the size of each block feature is the target size; or, determining an actual block size and an overlap size of the block features, the actual block size is the block size of the block features without the overlapped part, and the overlap size is the size of the overlapped part of adjacent blocks; dividing the aggregated features into multiple block features based on the actual block size and the overlap size, and the size of each block feature is the target size; wherein, if the actual block size is tile_c*tile_w*tile_h, and the overlap size is tile_c*padding_w*padding_h, then the target size is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h).

[0077] Exemplarily, merging the block-reconstructed image blocks corresponding to multiple block features to obtain the reconstructed image block corresponding to the current image block may include, but is not limited to: sorting the block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right, and sequentially splicing the block-reconstructed image blocks corresponding to the sorted multiple block features to obtain the reconstructed image block corresponding to the current image block. Alternatively, determining the actual block size and overlap size of each block-reconstructed image block; sorting the block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right; and sequentially splicing the block-reconstructed image blocks corresponding to the sorted multiple block features based on the actual block size and overlap size to obtain the reconstructed image block corresponding to the current image block. Wherein, the reconstructed image block includes multiple block-reconstructed image blocks, the size of the non-overlapping part of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping part of the block-reconstructed image block and the adjacent block-reconstructed image block is the overlap size of the block-reconstructed image block. Among them, for the overlapping part of the left and right reconstructed image blocks, the value of the overlapping part is the value of the left reconstructed image block, or the value of the overlapping part is the average of the two reconstructed image blocks; for the overlapping part of the upper and lower reconstructed image blocks, the value of the overlapping part is the value of the upper reconstructed image block, or the value of the overlapping part is the average of the two reconstructed image blocks.

[0078] Exemplarily, inputting the aggregated features into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block may include: decoding the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of the current image block, inputting the rate control parameter and the aggregated features into the synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.

[0079] Inputting each block feature into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature may include: decoding an auxiliary code stream corresponding to the current image block to obtain a rate control parameter for each block feature, inputting the block feature and the rate control parameter of the block feature into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature.

[0080] Inputting the reconstruction features of each stage sub-block into a synthesis transformation network to obtain a block-by-block reconstructed image block corresponding to the stage sub-block may include: decoding the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of each stage sub-block, inputting the reconstruction features of the stage sub-block and the rate control parameter of the stage sub-block into a synthesis transformation network to obtain a block-by-block reconstructed image block corresponding to the stage sub-block.

[0081] Exemplarily, inputting the rate control parameters and the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block may include: processing the aggregated features through the synthetic transformation network to obtain a first feature; processing the rate control parameters through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0082] Inputting the block feature and the rate control parameter of the block feature into a synthetic transformation network to obtain a block reconstructed image block corresponding to the block feature can include: processing the block feature through a synthetic transformation network to obtain a first feature; processing the rate control parameter of the sub-block at this stage through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block corresponding to the block feature based on the third feature.

[0083] Inputting the reconstruction features of the stage sub-block and the rate control parameters of the stage sub-block into a synthetic transformation network to obtain a block-by-block reconstructed image block corresponding to the stage sub-block can include: processing the reconstruction features of the stage sub-block through the synthetic transformation network to obtain a first feature; processing the rate control parameters of the stage sub-block through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block-by-block reconstructed image block corresponding to the stage sub-block based on the third feature.

[0084] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0085] As can be seen from the above technical solutions, in the embodiment of the present application, a coding and decoding scheme with variable and adjustable bit rate is proposed for the coding and decoding technology based on neural network, which improves parallelism, effectively saves feature storage cache, achieves higher bit rate control accuracy, and has less coding performance loss, and has better coding performance and bit rate control accuracy. By adopting block coding and decoding, the peak memory occupancy is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is provided, so that the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, improve coding performance and decoding performance, and reduce complexity.

[0086] Example 2: This application proposes an encoding method, see Figure 3 As shown, it is a schematic diagram of the process of the encoding method, which can be applied to an encoding end (also called a video encoder), and the method may include:

[0087] Step 301: Input the current image block to the analysis transformation network to obtain a feature block corresponding to the current image block.

[0088] Step 302: split the feature block into features to be encoded in multiple stage sub-blocks.

[0089] Step 303: For each stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into the first code stream of the current image block.

[0090] Step 304: Determine the residual feature of the sub-block at the stage based on the feature to be encoded of the sub-block at the stage and the mean feature of the sub-block at the stage, wherein the mean feature of the sub-block at the stage is obtained based on the coefficient hyper-parameter feature of the sub-block at the stage and / or the reference feature of the sub-block at the stage, such as based on the coefficient hyper-parameter feature, or based on the coefficient hyper-parameter feature and the reference feature.

[0091] Step 305: Determine probability distribution parameters based on the coefficient hyper-parameter features of the sub-block at this stage, and encode the residual features of the sub-block at this stage into a second code stream of the current image block based on the probability distribution parameters.

[0092] Exemplarily, for the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or, the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature and the default reference feature of the sub-block at this stage.

[0093] For the i-th stage sub-block, i is greater than 1, and the reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the i-1 stage sub-blocks before the i-th stage sub-block; the mean feature of the stage sub-block is obtained through a mean prediction network based on the coefficient hyper-parameter feature and the reference feature of the stage sub-block; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction features of the i-1-th stage sub-block.

[0094] Exemplarily, the processing process at the encoding end is similar to the processing process at the decoding end, and the similarities will not be repeated.

[0095] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0096] As can be seen from the above technical solutions, in the embodiment of the present application, a coding and decoding scheme with variable and adjustable bit rate is proposed for the coding and decoding technology based on neural network, which improves parallelism, effectively saves feature storage cache, achieves higher bit rate control accuracy, and has less coding performance loss, and has better coding performance and bit rate control accuracy. By adopting block coding and decoding, the peak memory occupancy is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is provided, so that the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, improve coding performance and decoding performance, and reduce complexity.

[0097] Embodiment 3: For Embodiment 1 and Embodiment 2, the processing process of the encoding end can be seen in Figure 4 As shown, of course, Figure 4 This is just an example of the processing process at the encoding end, and there is no limitation on the processing process at the encoding end.

[0098] After obtaining the current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoder may analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Among them, performing feature transformation on the current image block x through the analysis transformation network means: transforming the current image block x to an image feature y in a latent domain, thereby facilitating all subsequent processes to operate on the latent domain.

[0099] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0100] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network. There is no restriction on the training process of the hyperparameter encoding network, and it only needs to perform coefficient hyperparameter feature transformation on the image feature y. The image feature y in the latent domain obtains the super-prior latent information z (i.e., coefficient hyperparameter feature z) after passing through the hyperparameter encoding network.

[0101] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 4 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, i.e. Figure 4 The AE operation in represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the mean and the parameters of the probability distribution model (i.e., the probability distribution parameters).

[0102] After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to the subsequent embodiments.

[0103] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 4 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0104] For the encoding process of Bitstream#1, an encoding method of a fixed probability density model can be used, and for the decoding process of Bitstream#1, a decoding method of a fixed probability density model can be used. There is no restriction on this encoding and decoding process.

[0105] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain a residual by subtracting from the original feature and to obtain the reconstructed feature y_hat by adding the decoded residual.

[0106] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0107] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature y, the encoder can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0108] After obtaining the image feature s, the encoder can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 4 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoder can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e. Figure 4The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.

[0109] After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing of Bitstream#2 corresponding to the current image block by the decoder, refer to the subsequent embodiments.

[0110] After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the figure represents the decoding process. Then, the encoder can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0111] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the prediction value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0112] After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0113] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0114] To obtain the probability distribution model, continue to refer to Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network, and it is sufficient to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0115] In a possible implementation, the processing process at the encoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without any limitation on the encoding process.

[0116] Embodiment 4: For Embodiment 1 and Embodiment 2, the processing process of the decoding end can be seen in Figure 5 As shown, of course, Figure 5 This is just an example of the processing process at the decoding end, and there is no limitation on the processing process at the decoding end.

[0117] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 5 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 5 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0118] For the decoding process of Bitstream#1, a decoding method of a fixed probability density model may be used, and there is no restriction on this.

[0119] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0120] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0121] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 5 The dashed box in indicates that the mean prediction network is optional.

[0122] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 5 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 5 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0123] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, and the image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the prediction value mu is provided by the mean prediction network. Alternatively, after obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the image feature y_hat, which is the same as or different from the image feature y. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0124] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0125] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to Figure 5 As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0126] In a possible implementation, the processing process at the above decoding end can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without limiting the decoding process.

[0127] Example 5: For Example 1, Example 2, Example 3 and Example 4, a mean prediction network may be deployed or not. In order to improve the feature encoding performance, the deployment of the mean prediction network is taken as an example. When the mean prediction network is deployed, in order to obtain the accurate prediction value of the image feature, it is necessary to use the reconstructed value of the adjacent feature (that is, based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block, a context-based prediction is performed to obtain the prediction value mu corresponding to the current image block). Therefore, the feature value on the right (using the reconstructed value of the feature on the left or upper left side as a reference) can only be encoded after the feature value on the left or upper left side is encoded and reconstructed. This dependency makes the encoding between features only serially executed, and cannot be executed in parallel, resulting in high complexity.

[0128] In view of the above findings, in this embodiment, by adopting block encoding and decoding, the degree of parallelism can be improved, the feature storage cache can be effectively saved, higher bit rate control accuracy can be achieved, the coding performance loss is smaller, it has better coding performance and bit rate control accuracy, the peak memory occupancy is lower, the decoding time of a single block is shorter, and it has high-speed parallel decoding capabilities.

[0129] For the processing of the encoding end, please refer to Figure 6 As shown, there is no restriction on the processing process of the encoding end.

[0130] 1. After obtaining the original image x, the block division unit determines whether to divide the original image x into blocks based on the image resolution. For example, if the image resolution is greater than the threshold, the original image x is divided into blocks, otherwise, the original image x is not divided into blocks. If yes, the original image x is divided into several image blocks x_p, and there are overlapping parts between adjacent image blocks. If not, the image block x_p is the original image x, that is, the original image x has only one image block, and each image block x_p can be recorded as the current image block. For the convenience of description, the processing process of a current image block x_p is used as an example for explanation.

[0131] 2. After obtaining the current image block x_p, the current image block x_p is analyzed and transformed through the analysis transformation network to obtain the image feature y_p (feature block y_p) corresponding to the current image block x_p. For example, the current image block x_p is transformed into the image feature y_p in the latent domain through the analysis transformation network, so that all subsequent processes can be operated on the latent domain.

[0132] 3. The merging and classification unit merges and classifies the image feature y_p to obtain the to-be-encoded features y_i (i=1…N) of the N-stage sub-blocks, that is, the image feature y_p is split into the to-be-encoded features y_i of the N-stage sub-blocks.

[0133] 4. For each stage sub-block corresponding to the current image block, obtain the coefficient hyperparameter feature corresponding to the stage sub-block. For example, perform coefficient hyperparameter feature transformation on the feature to be encoded y_i of stage sub-block i to obtain the coefficient hyperparameter feature z_i corresponding to stage sub-block i. For example, the feature to be encoded y_i can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the feature to be encoded y_i to obtain the coefficient hyperparameter feature z_i.

[0134] 5. For each stage sub-block corresponding to the current image block, the coefficient hyperparameter feature of the stage sub-block is encoded into the first bitstream (Bitstream#1) of the current image block. For example, the coefficient hyperparameter feature z_i corresponding to the stage sub-block i is quantized to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_i, and the hyperparameter quantization feature is encoded to obtain the first bitstream corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z_i corresponding to the stage sub-block i is directly encoded to obtain the first bitstream corresponding to the current image block. After obtaining the first bitstream corresponding to the current image block, the first bitstream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the first bitstream corresponding to the current image block, refer to the subsequent embodiments.

[0135] 6. After obtaining the first code stream corresponding to the current image block, the first code stream can also be decoded to obtain the coefficient hyperparameter features of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i (i=1...N).

[0136] For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature of the stage sub-block i, and then the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i without involving the dequantization process.

[0137] For the encoding process of the first code stream, an encoding method of a fixed probability density model may be adopted, and for the decoding process of the first code stream, a decoding method of a fixed probability density model may be adopted, and there is no restriction on the encoding and decoding process.

[0138] 7. For each stage sub-block corresponding to the current image block, determine the probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block. For example, the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i can be inversely transformed to obtain the probability distribution parameter p_i corresponding to the stage sub-block i. For example, the coefficient hyperparameter feature z_hat_i is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network inversely transforms the coefficient hyperparameter feature z_hat_i to obtain the probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, a probability distribution model can be generated based on the probability distribution parameter p_i.

[0139] 8. After obtaining the features to be encoded of the N stage sub-blocks, for the feature to be encoded y_1 of the first stage sub-block, the coefficient hyperparameter feature z_hat_1 of the stage sub-block is input to the mean prediction network to obtain the mean feature of the stage sub-block (i.e., predicted value mu_1, mean mu_1). Alternatively, obtain the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 and the default reference feature of the stage sub-block to the mean prediction network to obtain the mean feature of the stage sub-block.

[0140] The residual feature r_1 is determined based on the feature to be encoded y_1 and the mean feature mu_1 of the sub-block in the first stage, such as taking the difference between the feature to be encoded y_1 and the mean feature mu_1 as the residual feature r_1. Then, feature processing is performed on the residual feature r_1 to obtain the residual feature r_1 after feature processing. There is no restriction on this feature processing process, and any feature processing method can be used. Of course, feature processing is an optional step, and feature processing may not be performed on the residual feature r_1.

[0141] The residual feature r_1 (or the residual feature r_1 after feature processing) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r_1 (or the residual feature r_1 after feature processing) can also be directly encoded to obtain the second bitstream corresponding to the current image block. After obtaining the second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the second bitstream corresponding to the current image block, refer to the subsequent embodiments. Among them, when encoding the image quantization feature or the residual feature r_1, the probability distribution model corresponding to the probability distribution parameter p_1 of the sub-block in the first stage can be used to encode the image quantization feature or the residual feature r_1 to obtain the second bitstream.

[0142] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameter p_1 of the sub-block in the first stage can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_1, or, after dequantizing the image quantization features, the residual features r_hat_1 are directly obtained without involving a feature recovery process. Alternatively, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain residual features r_hat_1, or, after decoding the second bitstream, the residual features r_hat_1 are directly obtained.

[0143] After the residual feature r_hat_1 is obtained, the reconstructed feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, such as taking the sum of the residual feature r_hat_1 and the mean feature mu_1 as the reconstructed feature y_hat_1.

[0144] 9. After obtaining the features to be encoded of N stage sub-blocks, for the feature to be encoded y_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i are input into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., predicted value mu_i, mean mu_i).

[0145] Among them, the reference feature for stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, it can be obtained based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature of stage sub-block i is obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference feature includes y_hat_1 to y_hat_i-1. Alternatively, the reference feature of stage sub-block i is obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference feature includes some of the reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, the reference feature of stage sub-block i is obtained based on the reconstructed features of the i-1-th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.

[0146] The residual feature r_i is determined based on the feature to be encoded y_i and the mean feature mu_i of the sub-block in the i-th stage, such as taking the difference between the feature to be encoded y_i and the mean feature mu_i as the residual feature r_i. The residual feature r_i is feature processed to obtain the residual feature r_i after feature processing. Of course, the residual feature r_i may not be feature processed.

[0147] The residual feature r_i (or the residual feature r_i after feature processing) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second code stream corresponding to the current image block. Alternatively, the residual feature r_i (or the residual feature r_i after feature processing) is directly encoded to obtain the second code stream corresponding to the current image block. The second code stream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the second code stream corresponding to the current image block, refer to the subsequent embodiments. Among them, when encoding the image quantization feature or the residual feature r_i, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block can be used to encode the image quantization feature or the residual feature r_i to obtain the second code stream.

[0148] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_i, or directly obtain residual features r_hat_i after dequantizing the image quantization features. Alternatively, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain residual features r_hat_i, or the residual features r_hat_i are directly obtained after decoding the second bitstream.

[0149] After the residual feature r_hat_i is obtained, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, such as taking the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0150] 10. The clustering and blocking unit can obtain the reconstruction features y_hat_i (i=1...N) of the N stage sub-blocks, and perform feature aggregation on the reconstruction features y_hat_i of the N stage sub-blocks to obtain aggregated features. Then, the clustering and blocking unit blocks the aggregated features to obtain at least one block feature y_hat_p, that is, feature block y_hat_p.

[0151] 11. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0152] 12. The merging unit can obtain the block-reconstructed image blocks x_hat_p corresponding to multiple block features y_hat_p, and merge these block-reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Obviously, if the original image x is divided into multiple current image blocks x_p, then the reconstructed image blocks x_hat corresponding to all current image blocks x_p can form a reconstructed image. If the original image x corresponds to one current image block x_p, then the reconstructed image block x_hat is the reconstructed image.

[0153] In the above process, the encoder can also allocate different rate control parameters to sub-blocks at different stages of different image blocks based on the region of interest or rate information, so that sub-blocks at different stages of different image blocks consume different rates.

[0154] For example, if it is necessary to perform special processing on the region of interest (such as non-uniform processing to ensure higher quality of the region of interest) to avoid uniform quantization distortion of each image block, the rate control parameter can be a quantization parameter (the rate control parameter needs to be transmitted to the decoding end through the bitstream), see Figure 6 As shown, the rate control parameter affects the quantization process (Q) and the inverse quantization process (IQ) of the first code stream, and affects the quantization process (Q) and the inverse quantization process (IQ) of the second code stream.

[0155] For example, when quantizing the coefficient hyperparameter feature z_i corresponding to the stage sub-block i, if the stage sub-block i is located in the region of interest, the rate control parameter is used to quantize the coefficient hyperparameter feature z_i corresponding to the stage sub-block i, and the rate control parameter is used to ensure that the image quality corresponding to the stage sub-block i is higher. When dequantizing the hyperparameter quantization feature corresponding to the stage sub-block i, the rate control parameter is also needed to dequantize the hyperparameter quantization feature corresponding to the stage sub-block i.

[0156] When quantizing the residual feature r_i (or residual feature r_i after feature processing) corresponding to the stage sub-block i, if the stage sub-block i is located in the region of interest, the rate control parameter is used to quantize the residual feature r_i corresponding to the stage sub-block i, and the rate control parameter is used to ensure that the image quality corresponding to the stage sub-block i is higher. When dequantizing the image quantization feature corresponding to the stage sub-block i, the rate control parameter is also needed to dequantize the image quantization feature corresponding to the stage sub-block i.

[0157] For example, in order to perform precise bit rate control, the bit rate control parameter may be a control parameter of the synthetic transform network (the bit rate control parameter needs to be transmitted to the decoding end through the bit stream), see Figure 6As shown, the rate control parameters affect the analysis transformation network and the synthesis transformation network. For example, when the analysis transformation network is used to perform analysis transformation on the current image block x_p, the rate control parameters and the current image block x_p are input to the analysis transformation network, and the analysis transformation network outputs the image feature y_p corresponding to the current image block x_p. When the block feature y_hat_p is input to the synthesis transformation network, the block feature y_hat_p and the rate control parameters are input to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0158] Based on the configured target bit rate (each image block corresponds to a target bit rate, or each image corresponds to a target bit rate, or multiple images correspond to a target bit rate), the encoder can try to use different rate control parameters. After using a certain rate control parameter, if the bit rate corresponding to the bit stream of the current block (such as the first bit stream and the second bit stream) is the target bit rate or close to the target bit rate (that is, the difference between the two is less than the threshold), then this rate control parameter is the final rate control parameter used, and the encoder can send this rate control parameter to the decoder.

[0159] Example 6: The processing process of the decoding end can be found in Figure 7 As shown, there is no restriction on the processing process of the decoding end.

[0160] 1. After obtaining the first bitstream (Bitstream#1) corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter features of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i (i=1...N).

[0161] For example, the first bitstream can be decoded to obtain the hyperparameter quantization feature of stage sub-block i, and then the hyperparameter quantization feature can be dequantized to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i without involving the dequantization process. For the decoding process of the first bitstream, a decoding method of a fixed probability density model can be used, and there is no restriction on this decoding process.

[0162] 2. For each stage sub-block corresponding to the current image block, determine the probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block. For example, the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i can be inversely transformed to obtain the probability distribution parameter p_i corresponding to the stage sub-block i. For example, the coefficient hyperparameter feature z_hat_i is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network inversely transforms the coefficient hyperparameter feature z_hat_i to obtain the probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, a probability distribution model can be generated based on the probability distribution parameter p_i.

[0163] 3. For each stage sub-block corresponding to the current image block, the second bitstream (Bitstream#2) of the current image block is decoded based on the probability distribution parameters (probability distribution model) of the sub-block at this stage to obtain the residual features of the sub-block at this stage.

[0164] For example, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th (i=1…N) stage sub-block can be used to decode the second bitstream to obtain the image quantization feature, dequantize the image quantization feature, and restore the dequantized feature to obtain the residual feature r_hat_i, or directly obtain the residual feature r_hat_i after dequantizing the image quantization feature. Alternatively, after decoding the second bitstream, restore the decoded feature to obtain the residual feature r_hat_i, or directly obtain the residual feature r_hat_i after decoding the second bitstream.

[0165] 4. After obtaining the residual features of the N-stage sub-blocks, for the residual feature r_hat_1 of the first-stage sub-block, the coefficient hyperparameter feature z_hat_1 of the sub-block is input into the mean prediction network to obtain the mean feature of the sub-block at this stage (i.e., predicted value mu_1, mean mu_1). Alternatively, obtain the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 of the sub-block at this stage and the default reference feature into the mean prediction network to obtain the mean feature of the sub-block at this stage.

[0166] Then, the reconstructed feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1. For example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be used as the reconstructed feature y_hat_1.

[0167] 5. After obtaining the residual features of N stage sub-blocks, for the residual feature r_hat_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i are input into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., predicted value mu_i, mean mu_i).

[0168] Among them, the reference feature for stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, it can be obtained based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature of stage sub-block i is obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference feature includes y_hat_1 to y_hat_i-1. Alternatively, the reference feature of stage sub-block i is obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference feature includes some of the reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, the reference feature of stage sub-block i is obtained based on the reconstructed features of the i-1-th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.

[0169] In order to control the cache size, the number of reconstructed features included in the reference feature cannot exceed the preset number. Assuming the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference feature includes all reconstructed features from y_hat_1 to y_hat_i-1. If the number of reconstructed features is greater than M, the reference feature includes M reconstructed features from y_hat_1 to y_hat_i-1 (i.e., part of the reconstructed features). In order to select M reconstructed features from y_hat_1 to y_hat_i-1, M reconstructed features can be selected based on a default strategy, such as selecting M reconstructed features that are closest to the sub-block in the current stage in the decoding order, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, etc., until M reconstructed features are selected. Of course, other methods can also be used to select M reconstructed features, which are not limited to this. Alternatively, the index values ​​of M reconstructed features can be obtained by decoding the bitstream, and M reconstructed features can be selected based on the index values ​​obtained by decoding.

[0170] Then, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstructed feature y_hat_i.

[0171] 6. The clustering and blocking unit can obtain the reconstruction features y_hat_i (i=1...N) of the N stage sub-blocks, and perform feature aggregation on the reconstruction features y_hat_i of the N stage sub-blocks to obtain aggregated features. Then, the clustering and blocking unit blocks the aggregated features to obtain at least one block feature y_hat_p, that is, feature block y_hat_p.

[0172] 7. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0173] 8. The merging unit can obtain the block-reconstructed image blocks x_hat_p corresponding to multiple block features y_hat_p, and merge these block-reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Obviously, if the original image x is divided into multiple current image blocks x_p, then the reconstructed image blocks x_hat corresponding to all current image blocks x_p can form a reconstructed image. If the original image x corresponds to one current image block x_p, then the reconstructed image block x_hat is the reconstructed image.

[0174] In the above process, different rate control parameters may be allocated to sub-blocks at different stages of different image blocks based on the region of interest or rate information, so that sub-blocks at different stages of different image blocks consume different rates.

[0175] For example, if the region of interest needs to be specially processed (such as non-uniform processing to ensure higher quality of the region of interest), the rate control parameter may be a quantization parameter (the rate control parameter needs to be transmitted to the decoding end through the bit stream), see Figure 7 As shown, the rate control parameter affects the inverse quantization process (IQ) of the first code stream and the inverse quantization process (IQ) of the second code stream.

[0176] For example, when dequantizing the hyperparametric quantization features corresponding to stage sub-block i, if the rate control parameters corresponding to stage sub-block i have been decoded from the bitstream, and the rate control parameters are used to ensure that the image quality corresponding to stage sub-block i is higher, then the rate control parameters can be used to dequantize the hyperparametric quantization features corresponding to stage sub-block i.

[0177] For another example, when dequantizing the image quantization features corresponding to stage sub-block i, if the rate control parameter corresponding to stage sub-block i has been decoded from the bitstream, and the rate control parameter is used to ensure that the image quality corresponding to stage sub-block i is higher, then the rate control parameter can be used to dequantize the image quantization features corresponding to stage sub-block i.

[0178] For example, in order to perform precise bit rate control, the bit rate control parameter may be a control parameter of a synthetic transform network (the bit rate control parameter needs to be transmitted to the decoding end through a bit stream, and the control parameter of the synthetic transform network means that the synthetic transform network can be determined based on the bit rate control parameter), see Figure 7 As shown, the rate control parameter affects the synthesis transformation network. For example, when the block feature y_hat_p is input to the synthesis transformation network, the block feature y_hat_p and the rate control parameter are input to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.

[0179] Embodiment 7: In Embodiment 5 and Embodiment 6, for the i-th (i=1…N) stage sub-block, after obtaining the reconstruction feature y_hat_i (i.e., the sum of the residual feature r_hat_i and the mean feature mu_i) of the i-th stage sub-block, the reconstruction feature y_hat_i is used to determine the mean feature of the subsequent stage sub-block, and / or, the reconstruction feature y_hat_i is used to determine the reconstructed image block. Among them, the reconstruction feature y_hat_i is used to determine the mean feature of the subsequent stage sub-block means: the reconstruction feature y_hat_i is used as a reference feature of the subsequent stage sub-block, thereby determining the mean feature of the subsequent stage sub-block. The reconstruction feature y_hat_i is used to determine the reconstructed image block means: the reconstruction feature y_hat_i is input into the clustering block unit, thereby determining the reconstructed image block corresponding to the current image block.

[0180] For the i-th (i=1…N) stage sub-block, after obtaining the reconstruction feature y_hat_i of the i-th stage sub-block, the reconstruction feature y_hat_i can also be feature enhanced to obtain the enhanced reconstruction feature y_hat_i', and the enhanced reconstruction feature y_hat_i' is used to determine the mean feature of the subsequent stage sub-block, and / or the enhanced reconstruction feature y_hat_i' is used to determine the reconstructed image block. Among them, the enhanced reconstruction feature y_hat_i' is used to determine the mean feature of the subsequent stage sub-block, which means: the enhanced reconstruction feature y_hat_i' is used as a reference feature of the subsequent stage sub-block, so as to determine the mean feature of the subsequent stage sub-block. The enhanced reconstruction feature y_hat_i' is used to determine the reconstructed image block, which means: the enhanced reconstruction feature y_hat_i' is input into the clustering block unit, so as to determine the reconstructed image block corresponding to the current image block. For example, the bitstream can be decoded to obtain enhancement information, and the reconstructed feature y_hat_i can be enhanced based on the enhancement information to obtain a new reconstructed feature y_hat_i'. The new reconstructed feature y_hat_i' can be used for subsequent predictions of y_hat_i+1 to y_hat_N, and can also be used for subsequent clustering and blocking processes.

[0181] In a possible implementation, the enhancement process of reconstructing the feature y_hat may be performed in the following manner:

[0182] Method 1: The input is the reconstructed feature y_hat, Threshold[idx], GreaterFlag[idx], Scale1[idx] and Scale2[idx] before enhancement, and the output is the reconstructed feature y_hat_en after enhancement. Among them, Threshold represents the threshold value, and different idx may correspond to the same or different threshold values. Idx represents the index, such as 0, 1, 2, 3, etc. GreaterFlag represents the flag bit, and different idx may correspond to the same or different flag bits. Scale1 represents the scaling factor, and different idx may correspond to the same or different Scale1. Scale2 represents the scaling factor, and different idx may correspond to the same or different Scale2.

[0183] Step 1: Determine mask[c,i,j] under each filter (index is idx). For example, the following expression can be used to determine mask[c,i,j]. Of course, the following expression is only an example and is not limited to this.

[0184]

[0185] σ[c,i,j] represents the probability distribution parameter value of [c,i,j], c represents the channel, i represents the horizontal coordinate, and j represents the vertical coordinate. If the probability distribution parameter value is greater than Threshold[idx], GreaterFlag[idx] is true and mask[idx,c,i,j] is 1. If the probability distribution parameter value is less than Threshold[idx], GreaterFlag[idx] is false and mask[idx,c,i,j] is 1. Otherwise, mask[idx,c,i,j] is 0.

[0186] Step 2: Execute each set of filters in turn (index is idx). If mask[idx,c,i,j] is 1, then y_hat_en[c,i,j]=y_hat[c,i,j]+mean_hat[c,i,j]*Scale2[idx]+residual_hat[c,i,j]*Scale1[idx]; y_hat[c,i,j]=y_hat_en[c,i,j]. If there are multiple sets of filters, y_hat is compensated multiple times. Among them, mean_hat represents the mean feature, that is, the aforementioned mu_i; residual_hat represents the residual feature, that is, the aforementioned r_hat.

[0187] Method 2: In method 1, the scaling factor of each channel c is the same. In method 2, different channels c allow different scaling factors, that is, the scaling factors are Scale1[c,idx] and Scale2[c,idx]. The current feature can be scaled by controlling the switch, or the scaling factor of the channel can be changed to 0 to turn off the scaling. Compared with method 1, the difference between method 2 and method 1 lies in the enhancement method: y_hat_en[c,i,j]=y_hat[c,i,j]+mean_hat[c,i,j]*Scale2[c,idx]+residual_hat[c,i,j]*Scale1[c,idx].

[0188] Method 3: Compared with Method 1, the implementation methods of Method 3 are similar. The difference is that the enhancement method is as follows: y_hat_en[c,i,j]=y_hat[c,i,j]*Scale1[idx]+mean_hat[c,i,j]*Scale2[idx].

[0189] Method 4: Compared with Method 1, the implementation methods of Method 4 are similar, except that the enhancement method is as follows: y_hat_en[c,i,j]=y_hat[c,i,j]*Scale1[idx]+residual_hat[c,i,j]*Scale2[idx]. In Method 4, Scale1[idx] can also be defaulted to 1, and the adjustment method is only affected by changing the Scale2[idx] value.

[0190] Embodiment 8: In Embodiment 5 and Embodiment 6, the coefficient hyperparameter feature z_hat_1 of the first stage sub-block can be input into the mean prediction network, and the mean prediction network outputs the mean feature of the first stage sub-block, and the prediction process of this mean prediction network is not restricted. For the i-th (i=1…N) stage sub-block, the coefficient hyperparameter feature z_hat_i of the i-th stage sub-block and the reference feature (which may be the default reference feature) of the i-th stage sub-block can be input into the mean prediction network, and the mean prediction network outputs the mean feature of the i-th stage sub-block. The prediction process of the mean prediction network is described below.

[0191] See also Fig. 8A As shown, it is a structural diagram of the mean prediction network. The mean prediction network may include a first prediction network, a second prediction network and a prediction fusion network. Of course, this is just an example and there is no limitation on the structure of the mean prediction network.

[0192] The coefficient hyperparameter feature z_hat_i of the i-th stage sub-block can be input into the first prediction network, and the first prediction feature corresponding to the coefficient hyperparameter feature z_hat_i can be obtained through the first prediction network. The reference feature of the i-th stage sub-block (such as the reconstructed features y_hat_1 to y_hat_i-1) can be input into the second prediction network, and the second prediction feature corresponding to the reference feature can be obtained through the second prediction network. Then, the first prediction feature and the second prediction feature are spliced ​​to obtain the spliced ​​feature, and the spliced ​​feature is input into the prediction fusion network. The spliced ​​feature is processed by the prediction fusion network to obtain the mean feature mu_i of the i-th stage sub-block.

[0193] For the first prediction network: perform feature enhancement and upsampling operations on the coefficient hyperparameter feature z_hat_i through the first prediction network to obtain a first prediction feature. The feature enhancement operation may include but is not limited to: a convolution operation, or a convolution operation and an activation operation. Of course, the above are only examples of feature enhancement operations, and there is no limitation on this feature enhancement operation. The upsampling operation may include but is not limited to: a deconvolution operation, a cropping operation, and an activation operation, or a deconvolution operation, a cropping operation, an activation operation, and a convolution operation. Of course, the above are only examples of upsampling operations, and there is no limitation on this upsampling operation. Exemplarily, the activation operation is a Relu operation.

[0194] For example, see Figure 8B As shown, it is a schematic diagram of the structure of the first prediction network. The first prediction network consists of 3 enhanced networks and 2 2x upsampling networks. Of course, the number of enhanced networks can be more or less, the number of 2x upsampling networks can be more or less, and the 2x upsampling network can also be replaced by upsampling networks of other multiples.

[0195] The coefficient hyperparameter feature z_hat_i is passed through enhanced network 1 and 2x upsampling network 1 to obtain enhanced large-size feature 1, and the spatial size (width and height) of enhanced large-size feature 1 is twice that of the coefficient hyperparameter feature z_hat_i. The enhanced large-size feature 1 is passed through enhanced network 2 and 2x upsampling network 2 to obtain enhanced large-size feature 2, and the spatial size (width and height) of enhanced large-size feature 2 is twice that of enhanced large-size feature 1. The enhanced large-size feature 2 is passed through enhanced network 3 to obtain the first prediction feature.

[0196] Exemplarily, enhanced network 1 and enhanced network 2 may be the same or different, enhanced network 1 and enhanced network 3 may be the same or different, and enhanced network 2 and enhanced network 3 may be the same or different.

[0197] For example, an implementation of an enhanced network (such as enhanced network 1, and / or enhanced network 2, and / or enhanced network 3) can be seen in Figure 8CAs shown, the enhanced network consists of a convolutional layer, which can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. There is no restriction on this. For example, a 3x3 convolutional layer can be selected.

[0198] For another example, an implementation of an enhanced network (such as enhanced network 1, and / or enhanced network 2, and / or enhanced network 3) can be found in Fig.8D As shown, the enhanced network can be composed of convolution layer 1, activation layer 1, convolution layer 2, activation layer 2, and convolution layer 3. Among them, the convolution layer (such as convolution layer 1, and / or convolution layer 2, and / or convolution layer 3) can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, which is not limited to this, such as a 3x3 convolution layer can be selected.

[0199] The activation layer (such as activation layer 1 and / or activation layer 2) can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh (hyperbolic tangent) layer, or a Gelu (Gaussian Error Linear Units) layer. There is no restriction on the type of the activation layer, such as a relu layer can be selected.

[0200] Of course, the above are just two examples of enhancing the network, and there is no limitation to this, as long as the enhanced function can be achieved.

[0201] Exemplarily, the 2x upsampling network 1 and the 2x upsampling network 2 may be the same or different.

[0202] For example, an implementation of a 2x upsampling network (such as 2x upsampling network 1 and / or 2x upsampling network 2) can be found in Fig. 8E As shown, the 2x upsampling network can be composed of a 2x deconvolution layer, a Crop layer, and an activation layer. The deconvolution layer can be a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer, and there is no restriction on this. For example, a 4x4 deconvolution layer can be selected. The activation layer can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. There is no restriction on the type of activation layer, and a relu layer can be selected. In addition, the Crop layer is used to crop features so that the spatial resolution of the features becomes smaller and the number of channels remains unchanged.

[0203] For another example, an implementation of a 2x upsampling network (such as 2x upsampling network 1 and / or 2x upsampling network 2) can be seen in Figure 8FAs shown, the 2x upsampling network can be composed of a 2x deconvolution layer, a Crop layer, an activation layer, and a convolution layer. The deconvolution layer can be a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer. The activation layer can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. The Crop layer is used to crop features so that the spatial resolution of the features becomes smaller and the number of channels remains unchanged. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this, such as selecting a 3x3 convolution layer.

[0204] Of course, the above are just two examples of a 2x upsampling network, and there is no limitation to this, as long as the upsampling function can be achieved.

[0205] Exemplarily, for the convolution layer in the first prediction network, it can be an ordinary 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation method of the grouped convolution can be found in subsequent embodiments and will not be repeated here.

[0206] For the second prediction network: all the reconstructed features in the reference features of the i-th stage sub-block are spliced ​​according to the channel dimension through the second prediction network, and the spliced ​​features are convolved to obtain the second prediction features. Alternatively, all the reconstructed features in the reference features of the i-th stage sub-block are added through the second prediction network, and the added features are convolved to obtain the second prediction features. The reference features include all the reconstructed features of the previous i-1 stage sub-blocks; or, part of the reconstructed features of the previous i-1 stage sub-blocks; or, the reconstructed features of the i-1-th stage sub-block.

[0207] For example, see Figure 8G As shown, it is a schematic diagram of the structure of the second prediction network. The second prediction network can be composed of a feature splicing layer and a convolution layer. There is no restriction on the structure of the second prediction network. If the reference feature includes all the reconstructed features of the previous i-1 stage sub-blocks (such as y_hat_1 to y_hat_i-1), the feature splicing layer can perform feature splicing (concat) on these reconstructed features according to the channel dimension to obtain the spliced ​​features. Alternatively, if the reference feature includes part of the reconstructed features of the previous i-1 stage sub-blocks (such as the part from y_hat_1 to y_hat_i-1), the feature splicing layer can perform feature splicing on these reconstructed features according to the channel dimension to obtain the spliced ​​features. Alternatively, if the reference feature includes the reconstructed features of the i-1th stage sub-block, the feature splicing layer can directly use the reconstructed features of the i-1th stage sub-block as the spliced ​​features.

[0208] The feature splicing layer can input the spliced ​​features to the convolution layer, and the convolution layer performs a convolution operation on the spliced ​​features to obtain the second prediction feature. Among them, the convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no limitation on this. For example, a 3x3 convolution layer can be selected. For the convolution layer in the second prediction network, it can be an ordinary 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation method of the grouped convolution can be referred to the subsequent embodiments.

[0209] For another example, the second prediction network may be composed of a feature addition layer and a convolution layer. If the reference feature includes all the reconstructed features of the previous i-1 stage sub-blocks, the feature addition layer adds (sums) the reconstructed features to obtain the added features. Alternatively, if the reference feature includes some of the reconstructed features of the previous i-1 stage sub-blocks, the feature addition layer adds the reconstructed features to obtain the added features. Alternatively, if the reference feature includes the reconstructed features of the i-1th stage sub-block, the feature addition layer may directly use the reconstructed features of the i-1th stage sub-block as the added features. The feature addition layer inputs the added features to the convolution layer, and the convolution layer performs a convolution operation on the added features to obtain the second prediction feature.

[0210] In order to control the cache size, the number of reconstructed features included in the reference feature cannot exceed the preset number. Assuming the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference feature includes all reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, if the number of reconstructed features is greater than M, the reference feature includes M reconstructed features from y_hat_1 to y_hat_i-1. For example, M reconstructed features can be selected based on the default strategy, for example, M reconstructed features that are closest to the sub-block in the current stage in the decoding order can be selected. Alternatively, the index values ​​of the M reconstructed features can be obtained by decoding the bitstream, and the M reconstructed features can be selected based on the index values ​​obtained by decoding.

[0211] For the prediction fusion network: the concatenated features (i.e., the concatenated features of the first prediction features and the second prediction features) are subjected to a convolution operation and at least one fusion operation through the prediction fusion network to obtain the mean feature mu_i of the i-th stage sub-block; wherein, for each fusion operation, the fusion operation may include an activation operation and a convolution operation. Exemplarily, the activation operation is a Relu operation.

[0212] For example, see Figure 8HAs shown in the figure, it is a schematic diagram of the structure of the prediction fusion network. The prediction fusion network can be composed of a convolution layer and N fusion layers, where N is a positive integer, such as 1 or 2, and each fusion layer can be composed of an activation layer and a convolution layer. There is no restriction on the structure of the prediction fusion network. Obviously, after the input feature (i.e., the concatenated feature) passes through the convolution layer and N "activation layer + convolution layer", the output feature, i.e., the mean feature mu_i of the sub-block at the i-th stage, can be obtained.

[0213] Among them, the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no limitation on this. For example, a 1x1 convolution layer can be selected. For the convolution layer in the prediction fusion network, it can be an ordinary 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation method of the grouped convolution is shown in the subsequent embodiments. The activation layer can be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer. For example, a relu layer can be selected.

[0214] Embodiment 9: For the encoding end and the decoding end, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks in each stage. For example, in Embodiment 5 and Embodiment 6, the clustering and blocking unit can perform feature aggregation on the reconstruction features (y_hat_1 to y_hat_N) of the N stage sub-blocks to obtain aggregated features (also referred to as full-size aggregated features y_hat_all), and block the aggregated features y_hat_all to obtain at least one block feature y_hat_p, such as block features y_hat_p_1 to block features y_hat_p_K. For each block feature y_hat_p, the clustering and blocking unit inputs the block feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p. The merging unit can obtain the block-reconstructed image blocks x_hat_p corresponding to the multiple block features y_hat_p, and merge these block-reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.

[0215] First, the clustering block unit aggregates the reconstructed features of the N-stage sub-blocks to obtain aggregated features.

[0216] For example, see Fig. 9A As shown in , the reconstructed features (y_hat_1 to y_hat_N) of the N-stage sub-blocks can be aggregated according to the phase to obtain the aggregated feature y_hat_all. Fig. 9AN=4 is used as an example for explanation. For example, an example of aggregation according to phase can be shown as follows. After y_hat_group1 is obtained, y_hat_group1 can be used as the aggregated feature y_hat_all.

[0217]

[0218] For example, see Fig. 9B As shown, the reconstructed features (y_hat_1 to y_hat_N) of the N-stage sub-blocks can be aggregated according to the phase to obtain multiple phase-aggregated features, and the multiple phase-aggregated features can be spliced ​​according to the channel to obtain the aggregated feature y_hat_all. For example, the example of aggregation according to the phase can be seen in the above expression. According to the phase aggregation, y_hat_group1 and y_hat_group2 are obtained. If there are many reconstructed features, after obtaining y_hat_group1 and y_hat_group2, based on the aggregated multiple y_hat_groups, they can be spliced ​​according to the channel to obtain the aggregated feature y_hat_all.

[0219] Exemplarily, before the reconstructed features of the N-stage sub-blocks are aggregated, useless feature values ​​can be removed. For example, since the height or width of the full-size feature is an odd number, when the encoding side is divided into four, in order to ensure that the size of the reconstructed features of each stage sub-block is the same, the reconstructed features of some stage sub-blocks need to be padded. The padding solution can be to add a right column or a lower row to the full-size feature. Based on this, after the reconstructed features of the N-stage sub-blocks are aggregated, useless features need to be removed. See Fig. 9C As shown, an example with an odd width is given. If the height is an odd number, the bottom row is removed. Of course, the above is a case of one divided into four. If there is a one-to-16 division, the features removed may be more than one row or one column.

[0220] Second, the clustering block unit blocks the aggregated feature y_hat_all to obtain K block features y_hat_p.

[0221] For example, the aggregated feature y_hat_all can also be divided into blocks to reduce subsequent memory overhead (the synthetic transformation operation requires multiple upsampling of the feature, which makes the feature data size exceed the size allowed by the memory). Of course, the feature block process is not a necessary process. If the size of the current feature is small, or it is determined based on the bitstream information that the current feature does not need to be divided into blocks, the following block process will be skipped directly, and y_hat_p will be equal to y_hat_all, that is, there is only one block feature.

[0222] In order to block the aggregated feature y_hat_all, you can use the feature block method without overlapping parts, see Fig.9D As shown, it is an example of dividing the aggregated feature y_hat_all into four block features y_hat_p.

[0223] In the feature blocking method without overlapping parts, the target size of the block feature can be determined, and based on the target size, the aggregated feature y_hat_all is evenly divided into multiple block features in order from top to bottom and from left to right, and the size of each block feature is the target size, such as divided into block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0224] For example, the target size is the size of each block feature. The target size can be the size read from the bitstream, or it can be determined based on the feature size to be currently blocked. There is no restriction on this. If the number of channels of each block feature is tile_c, the width is tile_w, and the height is tile_h, then the aggregated feature y_hat_all is evenly divided into multiple tile_c*tile_w*tile_h block features in the order from top to bottom and from left to right, such as block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0225] In order to block the aggregated feature y_hat_all, you can use the feature block method with overlapping parts, see Fig.9E As shown, it is an example of dividing the aggregated feature y_hat_all into four block features y_hat_p.

[0226] In the feature blocking method with overlapping parts, the actual blocking size and overlapping size of the blocking features can be determined. The actual blocking size can be the blocking size of the blocking features without the overlapping parts, and the overlapping size can be the size of the overlapping parts of adjacent blocks. Based on the actual blocking size and the overlapping size, the aggregated features are divided into multiple blocking features, and the size of each blocking feature is the target size, such as the blocking features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0227] For example, determine the actual block size and overlap size. The actual block size is the block size without the overlapped part, and the overlap size is the size of the overlapped part of adjacent blocks. These size information can be read from the bitstream or determined based on the characteristic size of the current block to be divided. There is no restriction on this. Fig.9EAs shown in the figure, if the actual block size is tile_c*tile_w*tile_h and the overlap size is tile_c*padding_w*padding_h, then when the top, bottom, left and right overlap, the size of each block feature (i.e., the target size) is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h). Based on this, for the aggregated feature y_hat_all, the features from the 1st to (tile_w+2*padding_w)th columns and the 1st to (tile_h+2*padding_h)th rows are divided into the first block feature. The features from the tile_h+padding_h+1th to (2*tile_h+3*padding_h)th rows in the tile_w+padding_w+1th to 2*tile_w+3*padding_wth columns are divided into the second block feature, and so on.

[0228] See also Fig.9E As shown, the 4x4x4 aggregated feature y_hat_all can be divided into four 4x3x3 block features. In the block features, tile_c is 4, padding_w and padding_h are 1, and tile_w and tile_h are 1.

[0229] For example, if the number of blocks in the last row or column is insufficient, the overlapping area can be increased so that the final block size meets the requirement. Fig.9F As shown, an example is given, the size of y_hat_all is 11x13, padding_w and padding_h are 1, tile_w and tile_h are 3. The size of the first block and the second block in the first row is 5x5, and the third block, in order to ensure that the size of the block is also 5x5, expands the overlapping area of ​​2 columns to the left.

[0230] Third, the clustering block unit inputs each block feature y_hat_p (such as K block features y_hat_p) to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to each block feature y_hat_p.

[0231] For each block feature y_hat_p, after the block feature y_hat_p is input into the synthesis transformation network, the block feature y_hat_p is synthesized and transformed through the synthesis transformation network to obtain the block reconstructed image block x_hat_p, and there is no restriction on this.

[0232] Fourthly, the merging unit merges the multiple segmented reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.

[0233] Exemplarily, the merging unit may obtain K block-reconstructed image blocks x_hat_p corresponding to the K block features y_hat_p, and merge the K block-reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0234] Exemplarily, if the aggregated feature y_hat_all is divided into blocks by a feature dividing method without overlapping parts, then the merging unit merges the K block-reconstructed image blocks x_hat_p by a feature merging method without overlapping parts. For example, the block-reconstructed image blocks x_hat_p corresponding to the K block features are sorted from top to bottom and from left to right, and the block-reconstructed image blocks x_hat_p corresponding to the sorted K block features are spliced ​​in sequence to obtain the reconstructed image block x_hat corresponding to the current image block. The splicing process can be Fig.9D The reverse process will not be described here.

[0235] Exemplarily, if the aggregated feature y_hat_all is divided into blocks by a feature dividing method with overlapping parts, then the merging unit merges the K block-reconstructed image blocks x_hat_p by a feature merging method with overlapping parts. For example, the merging unit can determine the actual block size and overlapping size of each block-reconstructed image block x_hat_p. The merging unit can sort the block-reconstructed image blocks x_hat_p corresponding to the K block features y_hat_p in order from top to bottom and from left to right; based on the actual block size and overlapping size, the block-reconstructed image blocks x_hat_p corresponding to the sorted K block features y_hat_p are spliced ​​in sequence to obtain the reconstructed image block x_hat corresponding to the current image block. The reconstructed image block may include K block-reconstructed image blocks, the size of the non-overlapping part of the block-reconstructed image block may be the actual block size of the block-reconstructed image block, and the size of the overlapping part of the block-reconstructed image block and the adjacent block-reconstructed image block may be the overlapping size of the block-reconstructed image block. The splicing process may be Fig.9E The reverse process will not be described here.

[0236] For example, the actual block size and overlap size are determined. The actual block size is the block size without overlapping parts, and the overlap size is the size of the overlapping parts of adjacent blocks. These size information can be read from the bitstream or determined based on the characteristic size of the current block to be divided. There is no restriction on this. Assuming that the actual block size is x_c*xtile_w*xtile_h, the overlap width in the overlap size is xpadding_w, and the overlap height in the overlap size is xpadding_h, then, according to the order in which the multiple block-reconstructed image blocks x_hat_p are received, the multiple block-reconstructed image blocks x_hat_p are sequentially spliced.

[0237] For the overlapping part of the left and right sub-block reconstructed image blocks, the value of the overlapping part is the value of the left sub-block reconstructed image block, and the value of the right sub-block reconstructed image block is discarded, that is, the value of the earliest (left) sub-block reconstructed image block is used, and the value of the overlapping part of the subsequent sub-blocks is directly discarded. Alternatively, the value of the overlapping part is the average value of the two sub-block reconstructed image blocks.

[0238] For the overlapping part of the upper and lower sub-block reconstructed image blocks, the value of the overlapping part is the value of the upper sub-block reconstructed image block, and the value of the lower sub-block reconstructed image block is discarded, that is, the value of the earliest (upper) sub-block reconstructed image block is used, and the value of the overlapping part of the subsequent sub-blocks is directly discarded. Alternatively, the value of the overlapping part is the average value of the two sub-block reconstructed image blocks.

[0239] Embodiment 10: For the encoding end and the decoding end, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks in each stage. For example, the clustering block unit can perform feature aggregation on the reconstruction features of the sub-blocks of N stages (y_hat_1 to y_hat_N) to obtain aggregated features (also called full-size aggregated features y_hat_all). The clustering block unit inputs the aggregated features y_hat_all to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block corresponding to the aggregated features y_hat_all, which is also the reconstructed image block x_hat corresponding to the current image block. Compared with Embodiment 9, it does not involve the segmentation of the aggregated features y_hat_all, nor does it involve the merging of multiple segmented reconstructed image blocks x_hat_p.

[0240] First, the clustering block unit aggregates the reconstructed features of the N-stage sub-blocks to obtain aggregated features.

[0241] Exemplarily, the reconstructed features (y_hat_1 to y_hat_N) of the N-stage sub-blocks can be aggregated according to the phase to obtain the aggregated feature y_hat_all. Alternatively, the reconstructed features (y_hat_1 to y_hat_N) of the N-stage sub-blocks can be aggregated according to the phase to obtain features after multiple phase aggregations, and the features after multiple phase aggregations are spliced ​​according to the channel to obtain the aggregated feature y_hat_all. Among them, feature aggregation can be referred to in Example 9, which will not be repeated here.

[0242] Second, the clustering block unit inputs the aggregated feature y_hat_all to the synthesis transformation network, and performs synthesis transformation on the aggregated feature y_hat_all through the synthesis transformation network. There is no restriction on the synthesis transformation method. The synthesis transformation network outputs the reconstructed image block corresponding to the aggregated feature y_hat_all, that is, the reconstructed image block x_hat corresponding to the current image block.

[0243] Embodiment 11: For the encoding end and the decoding end, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block. For example, after obtaining the reconstruction features of N stage sub-blocks (y_hat_1 to y_hat_N), the clustering block unit does not aggregate the reconstruction features, nor does it block the aggregated features. Instead, it directly inputs the reconstruction features of each stage sub-block to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to each stage sub-block. The merging unit can obtain the block reconstructed image blocks x_hat_p corresponding to multiple stage sub-blocks, and merge these block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.

[0244] First, after obtaining the reconstruction features of N stage sub-blocks, the clustering block unit inputs the reconstruction features of each stage sub-block into the synthesis transformation network, and performs synthesis transformation on the reconstruction features through the synthesis transformation network. There is no restriction on this synthesis transformation method, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to each stage sub-block.

[0245] Second, the merging unit merges the multiple partitioned reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.

[0246] Exemplarily, the merging unit may obtain N sub-block reconstructed image blocks x_hat_p corresponding to the N stage sub-blocks, and merge the N sub-block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0247] Exemplarily, the merging unit merges the N block-reconstructed image blocks x_hat_p by a feature merging method without overlapping parts. For example, the N block-reconstructed image blocks x_hat_p are sorted in order from top to bottom and from left to right, and the sorted N block-reconstructed image blocks x_hat_p are spliced ​​in sequence to obtain the reconstructed image block x_hat corresponding to the current image block. Alternatively, the merging unit merges the N block-reconstructed image blocks x_hat_p by a feature merging method with overlapping parts. For example, the actual block size and overlap size of each block-reconstructed image block x_hat_p are determined, and the N block-reconstructed image blocks x_hat_p are sorted in order from top to bottom and from left to right; based on the actual block size and overlap size, the sorted N block-reconstructed image blocks x_hat_p are spliced ​​in sequence to obtain the reconstructed image block x_hat corresponding to the current image block. The reconstructed image block includes N block-reconstructed image blocks, the size of the non-overlapping part of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping part of the block-reconstructed image block and the adjacent block-reconstructed image block is the overlapping size of the block-reconstructed image block. For the overlapping part of the left and right block-reconstructed image blocks, the value of the overlapping part is the value of the left block-reconstructed image block, and the value of the right block-reconstructed image block is discarded. Or the value of the overlapping part is the average of the two block-reconstructed image blocks. For the overlapping part of the upper and lower block-reconstructed image blocks, the value of the overlapping part is the value of the upper block-reconstructed image block, and the value of the lower block-reconstructed image block is discarded. Or the value of the overlapping part is the average of the two block-reconstructed image blocks.

[0248] Example 12: In Example 1 to Example 11, a synthetic transformation network is involved. In order to perform precise bit rate control, the encoder can send bit rate control parameters for the synthetic transformation network to the decoder. For example, the encoder can encode the bit rate control parameters in the auxiliary bit stream corresponding to the current image block, and the decoder can decode the auxiliary bit stream corresponding to the current image block to obtain the bit rate control parameters, and determine the synthetic transformation network based on the bit rate control parameters, that is, determine the synthetic transformation network based on the bit stream information, especially the bit rate related information. The synthesis transformation network may be composed of at least one 2x deconvolution layer (such as 2x2, 3x3, 4x4 or 5x5 2x deconvolution layer, such as 4x4 2x deconvolution layer, of course, other multiples of deconvolution layer), at least one activation layer (such as relu layer, leaky relu layer, sigmoid layer, tanh layer, Gelu layer, such as relu layer), at least one convolution layer (such as 1x1, 3x3 or 5x5 convolution layer, such as 3x3 convolution layer, the convolution layer in the synthesis transformation network may be a common 2D convolution or a grouped convolution, such as 2x grouped convolution), the output feature spatial resolution of the synthesis transformation network is a positive integer multiple of the input feature spatial resolution, such as 16 times (length and width are both 16 times), etc., and the number of output feature channels is 1 or 3. Based on the synthesis transformation network, the synthesis transformation network may be controlled based on the rate control parameter, that is, the rate control parameter is used as the input of the synthesis transformation network.

[0249] In a possible implementation, for Example 9, the clustering block unit will input K block features y_hat_p into the synthesis transformation network. Therefore, the encoder encodes K rate control parameters corresponding to the K block features y_hat_p in the auxiliary code stream corresponding to the current image block. There is no restriction on the source of the K rate control parameters. When the encoder uses these K rate control parameters to control the analysis transformation network and the synthesis transformation network (that is, the K rate control parameters are used as inputs of the analysis transformation network and the synthesis transformation network, such as the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p are input to the synthesis transformation network together as input features), the code rate corresponding to the code stream of the current image block can be the target code rate or close to the target code rate (that is, the difference between the two is less than the threshold). The decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the K rate control parameters corresponding to the K block features y_hat_p.

[0250] Based on this, the clustering blocking unit can obtain K block features y_hat_p and K rate control parameters corresponding to the K block features y_hat_p. For each block feature y_hat_p, the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p are input into the synthesis transformation network together, and the synthesis transformation network generates a block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p.

[0251] Exemplarily, the synthetic transformation network generates a block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p, which may include but is not limited to: processing the block feature y_hat_p through the synthetic transformation network to obtain a first feature; processing the rate control parameter corresponding to the block feature y_hat_p through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the third feature.

[0252] In a possible implementation, for Example 10, the clustering and blocking unit will input the aggregated feature y_hat_all into the synthesis transformation network. Therefore, the encoder encodes a rate control parameter corresponding to the aggregated feature y_hat_all in the auxiliary bitstream corresponding to the current image block, that is, the rate control parameter corresponding to the current image block. There is no restriction on the source of the rate control parameter. When the encoder uses the rate control parameter to control the analysis transformation network and the synthesis transformation network (that is, the rate control parameter is used as the input of the analysis transformation network and the synthesis transformation network, such as the aggregated feature y_hat_all and the rate control parameter are input to the synthesis transformation network together, and the current image block and the rate control parameter are input to the analysis transformation network together as input features), the rate corresponding to the bitstream of the current image block can be the target rate or close to the target rate. The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the rate control parameter corresponding to the current image block.

[0253] Based on this, the clustering block unit can obtain the aggregated feature y_hat_all and the bit rate control parameter, and input the aggregated feature y_hat_all and the bit rate control parameter together to the synthesis transformation network, which generates the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the bit rate control parameter.

[0254] The synthetic transformation network generates a reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the bit rate control parameter, which may include: processing the aggregated feature y_hat_all through the synthetic transformation network to obtain a first feature; processing the bit rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the third feature.

[0255] In a possible implementation, for Example 11, the clustering and blocking unit inputs the reconstruction features of the N stage sub-blocks into the synthesis transformation network, so the encoder encodes the N rate control parameters corresponding to the N stage sub-blocks in the auxiliary code stream corresponding to the current image block, and there is no restriction on the source of the N rate control parameters. When the encoder uses these N rate control parameters to control the analysis transformation network and the synthesis transformation network (that is, the N rate control parameters are used as inputs of the analysis transformation network and the synthesis transformation network, such as the reconstruction features of the stage sub-block and the rate control parameters corresponding to the stage sub-block are input into the synthesis transformation network together), the code rate corresponding to the code stream of the current image block can be the target code rate or close to the target code rate. The decoder decodes the auxiliary code stream corresponding to the current image block to obtain the N rate control parameters corresponding to the N stage sub-blocks.

[0256] Based on this, the clustering block unit can obtain the reconstruction features of N stage sub-blocks and the N rate control parameters corresponding to the N stage sub-blocks. For each stage sub-block, the reconstruction features of the stage sub-block and the rate control parameters corresponding to the stage sub-block are input into the synthesis transformation network together, and the synthesis transformation network generates the block reconstructed image block x_hat_p corresponding to the stage sub-block based on the reconstruction features of the stage sub-block and the rate control parameters corresponding to the stage sub-block.

[0257] Exemplarily, the synthetic transformation network generates a block-reconstructed image block x_hat_p corresponding to the sub-block at this stage based on the reconstruction features of the sub-block at this stage and the rate control parameters corresponding to the sub-block at this stage, which may include but is not limited to: processing the reconstruction features of the sub-block at this stage through the synthetic transformation network to obtain a first feature; processing the rate control parameters corresponding to the sub-block at this stage through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block-reconstructed image block x_hat_p corresponding to the sub-block at this stage based on the third feature.

[0258] Embodiment 13: Based on Embodiment 12, the synthesis transformation network can be called a synthesis transformation network controlled by a rate control parameter. The rate control parameter can be recorded as a λ parameter. The λ parameter is obtained by decoding the auxiliary code stream corresponding to the current image block. Fig. 10A As shown, it is a schematic diagram of the structure of the synthesis transformation network controlled by the λ parameter. The synthesis transformation network is a synthesis transformation network with a variable bit rate. Based on the synthesis transformation network, output features with a channel number of 1 or 3 can be obtained. Of course, Fig. 10A This is just an example of a synthetic transformation network, and there is no limitation on the structure of this synthetic transformation network.

[0259] See also Fig. 10A As shown, the synthetic transformation network controlled by the λ parameter may include at least one Conv (convolutional layer) and at least one λ-RSTB (Residual Swin Transformer Block). Fig. 10A In the example, 4 convolutional layers and 4 λ-RSTBs are used. The number of convolutional layers can be more or less, and the number of λ-RSTBs can be more or less. The convolutional layer can be a 1x1, 3x3, or 5x5 convolutional layer. For example, a 3x3 convolutional layer can be selected. The convolutional layer in the synthetic transformation network can be a normal 2D convolution or a grouped convolution, such as a 2x grouped convolution. There is no restriction on the structure of this convolutional layer. λ-RSTB represents the RSTB network layer controlled by the λ parameter. For the structure of the RSTB network layer controlled by the λ parameter, see Fig. 10B shown.

[0260] See also Fig. 10B As shown, λ-RSTB may include but is not limited to FE (Feature Embedding), λ-STB (Swin Transformer Block), FU (Feature Un-embedding) and other network layers. Of course, this is just an example and there is no limitation on the structure of this λ-RSTB. FE is a feature embedding layer used to implement feature embedding processing, and FU is a feature un-embedding layer used to implement feature un-embedding processing.

[0261] λ-STB represents the STB network layer controlled by the λ parameter. The structure of the STB network layer controlled by the λ parameter is shown in Fig. 10BAs shown, λ-STB may include but is not limited to LN (liner), λ-WA (Weight Averaging), λ-SWA (Stochastic Weight Averaging), MLP (Multilayer Perceptron), etc. Of course, this is just an example, and the structure of this λ-STB is not limited. LN is a linear layer, and λ-STB may include at least one LN. There is no limit on the number of LNs. Here, 4 LNs are taken as an example. MLP is a multilayer perceptron, and λ-STB may include at least one MLP. There is no limit on the number of MLPs. Here, 2 MLPs are taken as an example.

[0262] The structure of λ-SWA is the same as that of λ-WA. Here, λ-WA is taken as an example. λ-WA represents the WA network layer controlled by the λ parameter. The structure of the WA network layer controlled by the λ parameter can be found in Fig. 10C As shown, λ-WA may include but is not limited to FC (Full Connection) layer, MatMul (matrix multiplication), Scale, SoftMax (logistic regression, also known as normalization), MatMul, etc. Of course, this is just an example and there is no limitation on the structure of λ-WA.

[0263] The parameter λ may be mapped into a scaling factor through the FC layer, that is, the rate control parameter is processed through the FC layer of the synthetic transform network to obtain a second feature, which may be a scaling factor.

[0264] The input features (such as block features y_hat_p, or aggregated features y_hat_all, or reconstructed features of stage sub-blocks) can be processed through a transformer network to obtain Q features, K features, and V features. The Q features, K features, and V features all originate from the input features themselves, and are feature vectors generated by multiplying the input features with corresponding weights. There is no restriction on this process. In other words, the synthetic transformation network can include a transformer network, and the input features (such as block features, or aggregated features, or reconstructed features) are processed through the transformer network of the synthetic transformation network to obtain the first feature.

[0265] After obtaining the Q feature, K feature, and V feature, matrix multiplication, scaling, and logistic regression operations can be performed based on the Q feature and the K feature in sequence to obtain the intermediate feature 1. The scaling coefficient (obtained by mapping the FC layer to the parameter λ) can be multiplied by the V feature to obtain the intermediate feature 2. The intermediate feature 1 is matrix multiplied by the intermediate feature 2 to obtain the output feature of λ-WA. In summary, the scaling coefficient is the second feature, and the Q feature, K feature, and V feature are the first feature. Therefore, the third feature can be generated based on the first feature and the second feature, and the third feature is the output feature of λ-WA.

[0266] After obtaining the output features of λ-WA, combined Fig. 10A and Fig. 10B , the output features of the synthetic transformation network can be determined based on the output features of λ-WA (i.e., the third feature), such as the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p, the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all, and the block reconstructed image block x_hat_p corresponding to the stage sub-block.

[0267] See also Fig. 10C The calculation method of λ-WA and λ-WA shown in FIG. 1 can be as follows. Through this method, the λ parameter can be used to control the output of the image codec, so as to obtain bit streams of different sizes and reconstructed images of different qualities.

[0268]

[0269] Example 14: In Example 1 to Example 13, a convolution layer may be involved. The convolution layer may be a common 2D convolution or a grouped convolution, such as a 2x grouped convolution. For example, the steps of a common 2D convolution can be referred to Fig.11A As shown, the input feature is (H×W×C), and then, C' filters are applied (the size of each filter is (h×w×C), and the size of the filter C is the same as the size of the input feature C), and the input layer is converted to an output feature of size (H'×W'×C').

[0270] The steps of group convolution can be found in Fig. 11BAs shown, for each filter group, it can contain C' / g (if g is 2) filters, and the number of channels of each filter is half of that of the ordinary 2D convolution filter. Each filter group acts on 1 / g of the number of channels corresponding to the original W×H×C, that is, W×H×C / g. In this way, each filter group corresponds to the output of C' / g channel features. Finally, the channels can be stacked to obtain the final C' channels, thereby achieving the same effect as the above-mentioned ordinary 2D convolution. Obviously, through group convolution, the number of parameters and the amount of calculation can be effectively reduced.

[0271] Example 15: In Example 1 to Example 14, in order to perform precise bit rate control, the encoder needs to send bit rate control parameters to the decoder, such as the bit rate control parameters are carried by the auxiliary bit stream corresponding to the current image block. At the encoder, the bit rate control parameters affect the analysis transformation network and the synthesis transformation network. In order to make the bit rate corresponding to the bit stream of the current image block the target bit rate or close to the target bit rate (pre-configured bit rate), the encoder can try different bit rate control parameters to find the target bit rate control parameters. Under the target bit rate control parameters, the bit rate corresponding to the bit stream of the current image block is the target bit rate or close to the target bit rate, and the encoder encodes the target bit rate control parameters in the auxiliary bit stream corresponding to the current image block.

[0272] If the current image block corresponds to only one rate control parameter, such as the aggregated feature y_hat_all corresponds to one rate control parameter, each rate control parameter can be tried in turn from multiple rate control parameters until the rate corresponding to the bitstream of the current image block is the target rate or close to the target rate, thereby obtaining the target rate control parameter.

[0273] If the current image block corresponds to multiple rate control parameters, such as K rate control parameters corresponding to K block features and N rate control parameters corresponding to N stage sub-blocks, then the initial values ​​of the multiple rate control parameters are set first. If the bit rate corresponding to the code stream of the current image block does not meet the requirements, a candidate rate control parameter is selected from the multiple rate control parameters, and the value of the candidate rate control parameter is reduced. If the bit rate corresponding to the code stream of the current image block does not meet the requirements, then a candidate rate control parameter (the same as or different from the previous candidate rate control parameter) is continuously selected from the multiple rate control parameters, and the value of the candidate rate control parameter is reduced, and so on, until the bit rate corresponding to the code stream of the current image block meets the requirements (that is, the bit rate corresponding to the code stream of the current image block is the target bit rate or close to the target bit rate), and the target values ​​of the multiple rate control parameters are obtained.

[0274] When selecting a candidate rate control parameter from multiple rate control parameters, a rate control parameter may be randomly selected as the candidate rate control parameter, or the largest rate control parameter may be selected as the candidate rate control parameter, or the candidate rate control parameter may be selected based on the relationship between rate and distortion. For example, when rate control parameter A is reduced from the current level to the next level, the distortion is A1, and when rate control parameter B is reduced from the current level to the next level, the distortion is B1. If distortion A1 is less than distortion B1, that is, distortion A1 is the minimum distortion, rate control parameter A is selected as the candidate rate control parameter.

[0275] In a possible implementation, how does the encoder set the λ of each (feature or image) block? i One implementation of the parameters is as follows: For each block x i ′ (such as block features or stage sub-blocks), using different λ i Before encoding, several discrete λ are taken for each block, and these discrete λ are defined as quality levels. The bit rate and distortion of each block and each quality level are counted before encoding. For each frame of image, this process only needs to be performed once before the first encoding.

[0276] During the encoding process, for each block, select λ in the quality level for encoding. During the encoding process, first, all blocks are encoded using the same quality level, which is the minimum quality level that can make the total bit rate greater than the target bit rate, that is, satisfying the following formula, R t is the target bit rate, It represents the sum of the bit rates of all blocks at quality level q.

[0277]

[0278] Then, the quality level of each block is gradually reduced until the total bit rate is less than the target bit rate. For each block, the gradient of the current quality level and the previous quality level on the rate-distortion curve can be calculated, that is, the gradient is calculated using the following formula:

[0279]

[0280] in, is the reconstruction loss of block i at quality level q. When using a loss function that decreases with bit rate (such as mean square error) as a metric, the one with the minimum The block i (i.e., the candidate rate control parameter with the minimum distortion) is selected and its quality level is reduced by one. This process is repeated until the total bit rate is less than the target bit rate.

[0281] See also Fig.12As shown in the figure, it is a schematic diagram of the decision of the λ parameter. Seven quality levels (i.e., seven values ​​of the λ parameter) are designed on the rate-distortion curve. At quality level 3, it can be calculated That is, at quality level 3, reducing the quality level of block 2 will result in a greater coding performance loss, therefore, the quality level of block 1 is reduced.

[0282] It can be seen from the technical solutions of the above embodiments that in this embodiment, for the neural network-based coding and decoding technology, a coding and decoding scheme with variable and adjustable code rate is proposed to improve parallelism, effectively save feature storage cache, achieve higher code rate control accuracy, and have less coding performance loss, with better coding performance and code rate control accuracy. By adopting block coding and decoding, the peak memory occupancy is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capability is provided, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve the coding performance and decoding performance, and reduce complexity. For example, the code rate control algorithm of this scheme can achieve a code rate control accuracy of about 99%, and compared with no code rate control, the coding performance loss is only about 1%. Taking the decoding of a 2k resolution image at a block size of 256*256 as an example, by adopting block coding and decoding, the peak memory occupancy is reduced to 1 / 10 of the decoding of the entire image, and the decoding time of a single block is less than 0.1s, with the ability of high-speed parallel decoding. With less parameter addition, it has better coding and decoding performance.

[0283] Embodiment 16: A decoding method is proposed in the embodiment of the present application, which can be applied to a decoding end (also called a video decoder). The method may include: decoding a first code stream of a current image block to obtain coefficient hyperparameter features of the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, decoding a second code stream of the current image block based on the probability distribution parameters to obtain residual features of the current image block, and determining reconstruction features of the current image block based on the residual features; decoding an auxiliary code stream corresponding to the current image block to obtain a rate control parameter corresponding to the current image block, and inputting the reconstruction features and the rate control parameters into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.

[0284] Exemplarily, the reconstruction features and the rate control parameters are input into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block, which may include but is not limited to: processing the reconstruction features through a synthetic transformation network to obtain a first feature; processing the rate control parameters through a synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0285] Exemplarily, on the basis of Embodiment 3 and Embodiment 4, the synthetic transformation network may be adjusted. In Embodiment 16, only the adjustment process of the synthetic transformation network is described. For other processes, please refer to Embodiment 3 and Embodiment 4.

[0286] Exemplarily, precise bit rate control can be performed through bit rate control parameters, and the bit rate control parameters affect the analysis transformation network and the synthesis transformation network. For example, when the encoder performs analysis transformation on the current image block through the analysis transformation network, the bit rate control parameters and the current image block are input to the analysis transformation network, and the analysis transformation network outputs the image features corresponding to the current image block. When the encoder or decoder inputs the image feature y_hat to the synthesis transformation network, the image feature y_hat and the bit rate control parameters are input to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block corresponding to the current image block.

[0287] Based on the configured target bit rate, the encoder can try to use different rate control parameters. After using a certain rate control parameter, if the bit rate corresponding to the bit stream of the current block is the target bit rate or close to the target bit rate, then this rate control parameter is the final rate control parameter used. The encoder can send this rate control parameter to the decoder.

[0288] Exemplarily, the encoder can send rate control parameters for the synthesis transformation network to the decoder. For example, the encoder can encode the rate control parameters corresponding to the current image block in the auxiliary code stream corresponding to the current image block, and the decoder can decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameters corresponding to the current image block, and determine the synthesis transformation network based on the rate control parameters. The synthesis transformation network may be composed of at least one 2x deconvolution layer (such as 2x2, 3x3, 4x4 or 5x5 2x deconvolution layer, such as 4x4 2x deconvolution layer, of course, other multiples of deconvolution layer), at least one activation layer (such as relu layer, leaky relu layer, sigmoid layer, tanh layer, Gelu layer, such as relu layer), at least one convolution layer (such as 1x1, 3x3 or 5x5 convolution layer, such as 3x3 convolution layer, the convolution layer in the synthesis transformation network may be a common 2D convolution or a grouped convolution, such as 2x grouped convolution), the output feature spatial resolution of the synthesis transformation network is a positive integer multiple of the input feature spatial resolution, such as 16 times (length and width are both 16 times), etc., and the number of output feature channels is 1 or 3. Based on the synthesis transformation network, the synthesis transformation network may be controlled based on the rate control parameter, that is, the rate control parameter is used as the input of the synthesis transformation network.

[0289] Exemplarily, the encoder encodes the rate control parameter corresponding to the current image block in the auxiliary code stream corresponding to the current image block, and does not restrict the source of the rate control parameter. When the encoder uses the rate control parameter to control the analysis transformation network and the synthesis transformation network (that is, the rate control parameter is used as the input of the analysis transformation network and the synthesis transformation network), the rate corresponding to the code stream of the current image block can be the target rate or close to the target rate. The decoder can decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameter corresponding to the current image block. Based on this, the decoder can input the image feature y_hat and the rate control parameter together to the synthesis transformation network, and the synthesis transformation network generates a reconstructed image block corresponding to the current image block based on the image feature y_hat and the rate control parameter, and outputs the reconstructed image block.

[0290] Among them, the synthetic transformation network generates a reconstructed image block corresponding to the current image block based on the image feature y_hat and the bit rate control parameter, which can include: processing the image feature y_hat (i.e., the reconstructed feature y_hat) through the synthetic transformation network to obtain a first feature; processing the bit rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0291] For example, see Fig. 10A , Fig. 10B and Fig. 10CAs shown, the synthetic transformation network may include at least one Conv (convolutional layer) and at least one λ-RSTB, λ-RSTB may include but not limited to FE, λ-STB, FU and other network layers, λ-STB may include but not limited to LN, λ-WA, λ-SWA, MLP, etc., the structure of λ-SWA is the same as that of λ-WA, λ-WA may include but not limited to FC layer, MatMul, Scale, SoftMax, MatMul, etc., on this basis, the parameter λ (i.e., the bit rate control parameter) can be mapped into a scaling factor through the FC layer, that is, the bit rate control parameter is processed through the FC layer of the synthetic transformation network to obtain the second feature. The image feature y_hat can be processed through the transformer network to obtain Q features, K features, and V features, that is, the image feature y_hat is processed through the transformer network of the synthetic transformation network to obtain the first feature. After obtaining the Q feature, K feature, and V feature, matrix multiplication, scaling, and logistic regression operations can be performed based on the Q feature and the K feature to obtain intermediate feature 1; the scaling coefficient can be multiplied by the V feature to obtain intermediate feature 2; intermediate feature 1 and intermediate feature 2 are matrix multiplied to obtain the output feature of λ-WA, that is, the third feature is generated based on the first feature and the second feature. The third feature is the output feature of λ-WA. After obtaining the output feature of λ-WA, combined with Fig. 10A and Fig. 10B , the output feature of the synthetic transformation network can be determined based on the output feature of λ-WA (i.e., the third feature). For example, the output feature can be the reconstructed image block corresponding to the current image block.

[0292] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 16 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 16 can be implemented in combination.

[0293] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0294] Illustratively, in the above-mentioned embodiments, the contents of different embodiments can be borrowed from each other, such as the content of Example 3 can be applied to Example 16, the content of Example 4 can be applied to Example 16, the content of Examples 5-15 can be applied to Example 16, and the contents between Examples 5-15 can be borrowed from each other, and there is no limitation on this.

[0295] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The device is applied to the decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 16, that is, the processing flow of the decoding end.

[0296] For example, in one possible implementation, the decoder is configured to implement:

[0297] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;

[0298] For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyper-parameter feature of the sub-block at the stage, and a second bitstream of the current image block is decoded based on the probability distribution parameter to obtain a residual feature of the sub-block at the stage;

[0299] Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0300] A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.

[0301] Based on the same application concept as the above method, an encoding device is also proposed in the embodiment of the present application. The device is applied to the encoding end, and the device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 16, that is, the processing flow of the encoding end.

[0302] For example, in one possible implementation, the encoder is configured to implement:

[0303] Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0304] Splitting the feature block into features to be encoded of multiple stage sub-blocks;

[0305] For each stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter feature of the stage sub-block, and encode the coefficient hyper-parameter feature of the stage sub-block into the first bitstream of the current image block;

[0306] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0307] Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.

[0308] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig.13A As shown. It includes: a processor 1301 and a machine-readable storage medium 1302, the machine-readable storage medium 1302 stores machine-executable instructions that can be executed by the processor 1301; the processor 1301 is used to execute the machine-executable instructions to implement the decoding method of the above-mentioned embodiments 1-16 of the present application. For example, in a possible implementation, the processor 1301 performs the following steps when executing the machine-executable instructions:

[0309] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;

[0310] For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyper-parameter feature of the sub-block at the stage, and a second bitstream of the current image block is decoded based on the probability distribution parameter to obtain a residual feature of the sub-block at the stage;

[0311] Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0312] A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.

[0313] Based on the same application concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Fig. 13B As shown. It includes: a processor 1311 and a machine-readable storage medium 1312, the machine-readable storage medium 1312 stores machine-executable instructions that can be executed by the processor 1311; the processor 1311 is used to execute the machine-executable instructions to implement the encoding method of the above-mentioned embodiments 1-16 of the present application. For example, in a possible implementation, the processor 1311 performs the following steps when executing the machine-executable instructions:

[0314] Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0315] Splitting the feature block into features to be encoded of multiple stage sub-blocks;

[0316] For each stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter feature of the stage sub-block, and encode the coefficient hyper-parameter feature of the stage sub-block into the first bitstream of the current image block;

[0317] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0318] Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.

[0319] Based on the same application concept as the above method, an embodiment of the present application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method or encoding method of the above embodiments 1-16 of the present application.

[0320] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method or encoding method in the above embodiments.

[0321] Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.

[0322] Based on the same application concept as the above method, a decoding device is also proposed in an embodiment of the present application, which can be applied to a decoding end, and the decoding device includes: a decoding module, used to decode a first code stream of a current image block to obtain coefficient hyper-parameter features of each stage sub-block of the current image block; for each stage sub-block, determine probability distribution parameters based on the coefficient hyper-parameter features of the sub-block of the stage, decode the second code stream of the current image block based on the probability distribution parameters to obtain residual features of the sub-block of the stage; a determination module, used to determine a reconstruction feature of the sub-block of the stage based on the residual feature of the sub-block of the stage and the mean feature of the sub-block of the stage; determine a reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-block of each stage.

[0323] Exemplarily, the determination module is also used to obtain, for the first stage sub-block, the mean feature of the sub-block of this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage, or to obtain the configured default reference feature, and to obtain the mean feature of the sub-block of this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage and the default reference feature; or, for the i-th stage sub-block, i is greater than 1, to obtain the reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; to obtain the mean feature of the sub-block of this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage and the reference feature; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction feature of the i-1th stage sub-block.

[0324] Exemplarily, the mean prediction network includes a first prediction network, a second prediction network and a prediction fusion network. When the determination module obtains the mean feature of the sub-block at this stage through the mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the reference feature, it is specifically used to: obtain the first prediction feature corresponding to the coefficient hyper-parameter feature through the first prediction network; obtain the second prediction feature corresponding to the reference feature through the second prediction network; splice the first prediction feature and the second prediction feature and input them into the prediction fusion network, process the spliced ​​features through the prediction fusion network, and obtain the mean feature of the sub-block at this stage.

[0325] Exemplarily, when the determination module obtains the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network, it is specifically used to: perform feature enhancement and upsampling operations on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature; the feature enhancement operation includes: convolution operation, or convolution operation and activation operation; the upsampling operation includes: deconvolution operation, cropping operation and activation operation, or deconvolution operation, cropping operation, activation operation and convolution operation. When the determination module obtains the second prediction feature corresponding to the reference feature through the second prediction network, it is specifically used to: perform feature splicing on all reconstructed features in the reference feature according to the channel dimension; perform convolution operation on the spliced ​​features to obtain the second prediction feature; or, perform feature addition on all reconstructed features in the reference feature, and perform convolution operation on the added features to obtain the second prediction feature. The determination module processes the spliced ​​features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage, and is specifically used to: perform a convolution operation and at least one fusion operation on the spliced ​​features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage; wherein the fusion operation includes an activation operation and a convolution operation.

[0326] Exemplarily, the determination module is also used to perform feature enhancement on the reconstruction features of the sub-block at the stage to obtain enhanced reconstruction features; wherein the enhanced reconstruction features are used to determine the mean features of the stage sub-block, and the enhanced reconstruction features are used to determine the reconstructed image block.

[0327] Exemplarily, when the determination module determines the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage, it is specifically used to: perform feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain aggregated features; input the aggregated features to a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; or, perform feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain aggregated features; block the aggregated features to obtain multiple block features; input each block feature to a synthetic transformation network to obtain a block-reconstructed image block; merge the block-reconstructed image blocks corresponding to multiple block features to obtain the reconstructed image block corresponding to the current image block; or, input the reconstruction features of the sub-blocks at each stage to a synthetic transformation network to obtain a block-reconstructed image block; merge the block-reconstructed image blocks corresponding to all stage sub-blocks to obtain the reconstructed image block corresponding to the current image block.

[0328] Exemplarily, the determination module performs feature aggregation on the reconstruction features of each stage sub-block to obtain the aggregated features by: aggregating the reconstruction features of each stage sub-block according to phase to obtain the aggregated features; or aggregating the reconstruction features of each stage sub-block according to phase to obtain multiple phase-aggregated features, and splicing the multiple phase-aggregated features according to channels to obtain the aggregated features.

[0329] Exemplarily, when the determination module blocks the aggregated features to obtain multiple block features, it is specifically used to: determine the target size of the block features; based on the target size, evenly divide the aggregated features into multiple block features in order from top to bottom and from left to right, and the size of each block feature is the target size; or, determine the actual block size and overlapping size of the block features, the actual block size is the block size of the block features without the overlapping parts, and the overlapping size is the size of the overlapping parts of adjacent blocks; based on the actual block size and the overlapping size, divide the aggregated features into multiple block features, and the size of each block feature is the target size.

[0330] Exemplarily, the determination module merges the block-reconstructed image blocks corresponding to the multiple block features to obtain the reconstructed image block corresponding to the current image block, and is specifically used to: sort the block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right, and sequentially splice the sorted multiple block-reconstructed image blocks to obtain the reconstructed image block corresponding to the current image block; or determine the actual block size and overlap size of each block-reconstructed image block; sort the block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right; and sequentially splice the sorted multiple block-reconstructed image blocks based on the actual block size and the overlap size. splicing to obtain a reconstructed image block corresponding to the current image block; wherein, the size of the non-overlapping part of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping part of the block-reconstructed image block and the adjacent block-reconstructed image block is the overlapping size of the block-reconstructed image block; for the overlapping part of the left and right block-reconstructed image blocks, the value of the overlapping part is the value of the left block-reconstructed image block, or the value of the overlapping part is the average of the two block-reconstructed image blocks; for the overlapping part of the upper and lower block-reconstructed image blocks, the value of the overlapping part is the value of the upper block-reconstructed image block, or the value of the overlapping part is the average of the two block-reconstructed image blocks.

[0331] Exemplarily, when the determination module inputs the aggregated features into the synthesis transformation network, and obtains the reconstructed image block corresponding to the current image block, it is specifically used to: decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of the current image block, and input the rate control parameter and the aggregated features into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; or, when the determination module inputs each block feature into the synthesis transformation network to obtain the block-reconstructed image block, it is specifically used to: decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of each block feature, and input the block feature and the rate control parameter of the block feature into the synthesis transformation network to obtain the block-reconstructed image block corresponding to the block feature; or, when the determination module inputs the reconstruction feature of each stage sub-block into the synthesis transformation network to obtain the block-reconstructed image block, it is specifically used to: decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of each stage sub-block, and input the reconstruction feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthesis transformation network to obtain the block-reconstructed image block.

[0332] Exemplarily, the determination module inputs the rate control parameter and the aggregated feature into the synthesis transformation network, and when obtaining the reconstructed image block corresponding to the current image block, it is specifically used to: process the aggregated feature through the synthesis transformation network to obtain the first feature; process the rate control parameter through the synthesis transformation network to obtain the second feature; generate a third feature based on the first feature and the second feature, and determine the reconstructed image block corresponding to the current image block based on the third feature; or, the determination module inputs the block feature and the rate control parameter of the block feature into the synthesis transformation network, and when obtaining the block reconstructed image block, it is specifically used to: process the block feature through the synthesis transformation network to obtain the first feature ; Processing the rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block-reconstructed image block corresponding to the block feature based on the third feature; or, the determination module inputs the reconstruction feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthetic transformation network to obtain the block-reconstructed image block, which is specifically used for: processing the reconstruction feature of the stage sub-block through the synthetic transformation network to obtain the first feature; processing the rate control parameter through the synthetic transformation network to obtain the second feature; generating a third feature based on the first feature and the second feature, and determining the block-reconstructed image block corresponding to the stage sub-block based on the third feature.

[0333] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application, and the device is applied to the encoding end, and the device includes: an acquisition module, which is used to input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks; an encoding module, which is used to obtain the coefficient hyper-parameter features of the stage sub-block for each stage sub-block corresponding to the current image block, and encode the coefficient hyper-parameter features of the stage sub-block into the first code stream of the current image block; a determination module, which is used to determine the residual features of the stage sub-block based on the features to be encoded of the stage sub-block and the mean features of the stage sub-block; the encoding module is also used to determine the probability distribution parameters based on the coefficient hyper-parameter features of the stage sub-block, and encode the residual features of the stage sub-block into the second code stream of the current image block based on the probability distribution parameters.

[0334] Exemplarily, the determination module is also used to obtain, for the first stage sub-block, the mean feature of the sub-block of this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage, or to obtain the configured default reference feature, and to obtain the mean feature of the sub-block of this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage and the default reference feature; or, for the i-th stage sub-block, i is greater than 1, to obtain the reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; to obtain the mean feature of the sub-block of this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage and the reference feature; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction feature of the i-1th stage sub-block.

[0335] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

[0336] For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. An image decoding method, characterized in that: The method comprises: Decode the code stream of the current image block to obtain the coefficient hyper-parameter features of the sub-blocks of each stage of the current image block; For any stage sub-block, a probability distribution parameter is determined based on the coefficient hyper-parameter feature of the stage sub-block, and a code stream of the current image block is decoded based on the probability distribution parameter to obtain the residual feature of the stage sub-block; For the first stage sub-block, based on the coefficient hyper-parameter feature of the sub-block at this stage, the mean feature of the sub-block at this stage is obtained through a mean prediction network; and / or, for the i-th stage sub-block, i is greater than 1, based on the reconstruction features of the i-1 stage sub-blocks before the i-th stage sub-block, the reference feature of the i-th stage sub-block is obtained; based on the coefficient hyper-parameter feature of the sub-block at this stage and the reference feature, the mean feature of the sub-block at this stage is obtained through a mean prediction network; For any stage sub-block, the reconstruction feature of the stage sub-block is determined based on the residual feature of the stage sub-block and the mean feature of the stage sub-block; and the reconstructed image block corresponding to the current image block is determined based on the reconstruction features of multiple stage sub-blocks.

2. The method according to claim 1, characterized in that: The reference features include all the reconstruction features of the previous i-1 stage sub-blocks; or, Partially reconstructed features of the previous i-1 stage sub-blocks; or, Reconstructed features of the i-1th stage sub-block.

3. The method according to claim 1, characterized in that The mean prediction network includes a second prediction network, and the coefficient hyperparameter feature of the sub-block at this stage and the reference feature are used to obtain the mean feature of the sub-block at this stage through the mean prediction network, including: Acquire a second prediction feature corresponding to the reference feature through a second prediction network; The mean feature of the sub-block at this stage is determined based on the second prediction feature and the coefficient hyper-parameter feature of the sub-block at this stage.

4. The method according to claim 3, characterized in that The obtaining, through a second prediction network, a second prediction feature corresponding to the reference feature includes: If the reference feature includes at least two reconstruction features, then all the reconstruction features in the reference feature are spliced ​​according to the channel dimension; a convolution operation is performed on the spliced ​​features to obtain the second prediction feature; or, If the reference feature includes a reconstruction feature, a convolution operation is performed on the reconstruction feature to obtain the second prediction feature.

5. The method according to claim 3, characterized in that: The mean prediction network includes a first prediction network, and the determining the mean feature of the sub-block at this stage based on the second prediction feature and the coefficient hyper-parameter feature of the sub-block at this stage includes: Performing feature splicing on the second prediction feature and the coefficient hyperparameter feature of the sub-block at this stage to obtain a spliced ​​feature, and inputting the spliced ​​feature to the first prediction network; Acquire a first prediction feature corresponding to the concatenated feature through a first prediction network; The mean feature of the sub-blocks at this stage is determined based on the first prediction feature.

6. The method according to claim 5, characterized in that Obtaining the coefficient hyperparameter feature or the first prediction feature corresponding to the concatenated feature through the first prediction network includes: Performing a feature enhancement operation on the coefficient hyperparameter feature or the concatenated feature through a first prediction network to obtain the first prediction feature; wherein the feature enhancement operation includes: a convolution operation and an activation operation; Wherein, the activation operation includes a Relu activation operation.

7. The method according to claim 5, characterized in that Obtaining the coefficient hyperparameter feature or the first prediction feature corresponding to the concatenated feature through the first prediction network includes: Performing convolution operation, activation operation, convolution operation, activation operation, and convolution operation on the coefficient hyperparameter feature or the concatenated feature in sequence through a first prediction network to obtain the first prediction feature; Wherein, the activation operation includes a Relu activation operation.

8. The method according to any one of claims 1 to 7, characterized in that: After determining the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage, the method includes: The reconstruction features of the sub-blocks at this stage are enhanced to obtain enhanced reconstruction features; wherein the enhanced reconstruction features are used to determine the mean features of the sub-blocks at the stage, and the enhanced reconstruction features are used to determine the reconstructed image blocks.

9. An image encoding method, characterized in that: The method comprises: Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block; Splitting the feature block into features to be encoded of multiple stage sub-blocks; For any stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter feature of the stage sub-block, and encode the coefficient hyper-parameter feature of the stage sub-block into the code stream of the current image block; For the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the default reference feature; and / or, for the i-th stage sub-block, i is greater than 1, the reference feature of the i-th stage sub-block is obtained based on the reconstruction feature of the i-1 stage sub-blocks before the i-th stage sub-block; the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the reference feature; For any stage sub-block, the residual features of the stage sub-block are determined based on the to-be-encoded features of the stage sub-block and the mean features of the stage sub-block; probability distribution parameters are determined based on the coefficient hyper-parameter features of the stage sub-block, and the residual features of the stage sub-block are encoded into the code stream of the current image block based on the probability distribution parameters.

10. An image decoding device, characterized in that: The device comprises: A decoding module is used to decode the bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block; for any stage sub-block, determine the probability distribution parameters based on the coefficient hyperparameter features of the sub-block at that stage, and decode the bitstream of the current image block based on the probability distribution parameters to obtain the residual features of the sub-block at that stage; A determination module is used to obtain, for a first stage sub-block, a mean feature of the sub-block at this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or to obtain a configured default reference feature, and obtain the mean feature of the sub-block at this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the default reference feature; and / or, for an i-th stage sub-block, i is greater than 1, obtain a reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; and obtain the mean feature of the sub-block at this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the reference feature; The determination module is also used to determine, for any stage sub-block, the reconstruction features of the stage sub-block based on the residual features of the stage sub-block and the mean features of the stage sub-block; and determine the reconstructed image block corresponding to the current image block based on the reconstruction features of multiple stage sub-blocks.

11. An image encoding device, characterized in that: The device comprises: The acquisition module is used to input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks; An encoding module, configured to obtain, for any stage sub-block corresponding to the current image block, a coefficient hyper-parameter feature of the stage sub-block, and encode the coefficient hyper-parameter feature of the stage sub-block into a bitstream of the current image block; A determination module is used to obtain, for a first stage sub-block, a mean feature of the sub-block at this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or to obtain a configured default reference feature, and obtain the mean feature of the sub-block at this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the default reference feature; and / or, for an i-th stage sub-block, i is greater than 1, obtain a reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; and obtain the mean feature of the sub-block at this stage through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the reference feature; The determination module is further used to determine, for any stage sub-block, the residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block; The encoding module is also used to determine probability distribution parameters based on the coefficient hyperparameter characteristics of the sub-block at this stage, and encode the residual characteristics of the sub-block at this stage into the code stream of the current image block based on the probability distribution parameters.

12. An image decoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 8.

13. An image encoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 9.

14. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by a processor, the method described in any one of claims 1 to 8 is implemented; or, when the computer instructions are executed by a processor, the method described in claim 9 is implemented.

15. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented; or when the computer program is executed by a processor, the method according to claim 9 is implemented.

Images