A decoding, encoding method, apparatus and device thereof

Abstract

The application provides a decoding method, an encoding method, a device and equipment thereof. The decoding method comprises: decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block; for each stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; determining a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and a mean value feature of the stage sub-block; and determining a reconstruction image block corresponding to the current image block based on the reconstruction feature of each stage sub-block. Through the technical scheme, the encoding performance and the decoding performance can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coding and decoding, in particular to a decoding method and an encoding method, a decoding device and an encoding device. BACKGROUND

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy encoding, filtering and the like. For the prediction process, the prediction process can include intra prediction and inter prediction. Inter prediction refers to using the correlation in the time domain of a video to predict the current pixel using the pixels of the adjacent encoded image, so as to effectively remove the temporal redundancy of the video. Intra prediction refers to using the correlation in the spatial domain of a video to predict the current pixel using the pixels of the encoded block of the current frame, so as to remove the spatial redundancy of the video.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems such as image classification and object detection. Deep learning has also gradually begun to be applied in the field of coding and decoding, that is, a neural network can be used to encode and decode images. Although the neural network-based coding and decoding method has shown great performance potential, the neural network-based coding and decoding method still has problems such as poor coding performance, poor decoding performance and high complexity. SUMMARY

[0004] Therefore, the present application provides a decoding method and an encoding method, a decoding device and an encoding device, which improve the coding performance and the decoding performance.

[0005] The present application provides a decoding method applied to a decoding end, the method comprising:

[0006] decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block;

[0007] For each stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block;

[0008] determining a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and a mean feature of the stage sub-block;

[0009] determining a reconstructed image block corresponding to the current image block based on the reconstruction feature of each stage sub-block.

[0010] The present application provides an encoding method applied to an encoding end, the method comprising:

[0011] input the current image block into an analysis transform network to obtain a feature block corresponding to the current image block;

[0012] split the feature block into to-be-encoded features of a plurality of stage sub-blocks;

[0013] For each stage sub-block corresponding to the current image block, obtain a coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into a first code stream of the current image block;

[0014] determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block;

[0015] determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into a second code stream of the current image block based on the probability distribution parameter.

[0016] The application provides a decoding method applied to a decoding end, and the method comprises the following steps:

[0017] decode a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block;

[0018] determine a probability distribution parameter based on the coefficient hyperparameter feature, decode a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the current image block, and determine a reconstructed feature of the current image block based on the residual feature;

[0019] decode an auxiliary code stream corresponding to the current image block to obtain a code rate control parameter corresponding to the current image block, and input the reconstructed feature and the code rate control parameter into a synthesis transform network to obtain a reconstructed image block corresponding to the current image block.

[0020] The application provides a decoding device applied to a decoding end, and the device comprises the following:

[0021] A decoding module is configured to decode a first code stream of a current image block to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block, and for each stage sub-block, determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decode a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block;

[0022] A determining module is configured to determine a reconstructed feature of each stage sub-block based on the residual feature of the stage sub-block and a mean feature of the stage sub-block, and determine a reconstructed image block corresponding to the current image block based on the reconstructed features of the stage sub-blocks.

[0023] The application provides an encoding device applied to an encoding end, and the device comprises the following:

[0024] an acquisition module, configured to input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block, and split the feature block into to-be-encoded features of a plurality of stage sub-blocks;

[0025] an encoding module, configured to, for each stage sub-block corresponding to the current image block, acquire a coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into a first code stream of the current image block;

[0026] a determination module, configured to determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block;

[0027] The encoding module is further configured to determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into a second code stream of the current image block based on the probability distribution parameter.

[0028] The present application provides a decoding end device, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor;

[0029] The processor is used to execute the machine executable instructions to realize the decoding method.

[0030] The present application provides an encoding end device, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor;

[0031] The processor is used to execute the machine executable instructions to realize the encoding method.

[0032] The present application provides an electronic device, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor; the processor is used to execute the machine executable instructions to realize the decoding method; or, the processor is used to execute the machine executable instructions to realize the encoding method.

[0033] The present application provides a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by the processor to realize the decoding method; or, the computer instructions are executed by the processor to realize the encoding method.

[0034] The present application provides a computer application program, the computer application program is executed by the processor to realize the decoding method; or, the computer application program is executed by the processor to realize the encoding method.

[0035] From the above technical solutions, in the embodiment of the application, for the neural network-based coding and decoding technology, a coding and decoding scheme with variable and adjustable code rate is proposed, the parallelism is improved, the feature storage cache is effectively saved, higher code rate control accuracy is achieved, the coding performance loss is smaller, and better coding performance and code rate control accuracy are achieved. By using block coding, the peak memory occupation is lower, the decoding time of a single block is smaller, and high-speed parallel decoding capability is achieved, so that the neural network can maintain low complexity while effectively ensuring the quality of the reconstructed image block, improving the coding performance and decoding performance, and reducing the complexity. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a schematic diagram of a three-dimensional feature matrix in an embodiment of the application;

[0037] Figure 2 is a flowchart of a decoding method in an embodiment of the application;

[0038] Figure 3 is a flowchart of an encoding method in an embodiment of the application;

[0039] Figure 4 is a schematic diagram of the processing process of the encoding end in an embodiment of the application;

[0040] Figure 5 is a schematic diagram of the processing process of the decoding end in an embodiment of the application;

[0041] Figure 6 is a schematic diagram of the processing process of the encoding end in an embodiment of the application;

[0042] Figure 7 is a schematic diagram of the processing process of the decoding end in an embodiment of the application;

[0043] Figures 8A-8H is a structural schematic diagram of a mean prediction network in an embodiment of the application;

[0044] Figures 9A-9F is a schematic diagram of aggregation and blocking in an embodiment of the application;

[0045] Figures 10A-10C is a structural schematic diagram of a synthesis transformation network in an embodiment of the application;

[0046] Figure 11A and Figure 11B is a processing schematic diagram of a convolutional layer in an embodiment of the application;

[0047] Figure 12 is a decision schematic diagram of a lambda parameter in an embodiment of the application;

[0048] Figure 13A is a hardware structure diagram of a decoding end device in an embodiment of the present application;

[0049] Figure 13B is a hardware structure diagram of an encoding end device in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The terminology used in the present application merely for the purpose of describing particular embodiments, and is not intended to limit the present application. The singular forms "a", "an", and "the" used in the embodiments and claims of the present application are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or", as used herein, refer to any or all possible combinations of one or more of the associated listed items. It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal sequence. These terms are used only to distinguish one piece of information from another. For example, a first item can be termed a second item, and, similarly, a second item can be termed a first item, depending on the context, without departing from the scope of the embodiments of the present application. In addition, the word "if" can be interpreted as meaning "when" or "in response to determining".

[0051] The embodiments of the present application propose a decoding method and an encoding method, which can involve the following concepts:

[0052] Entropy Encoding: Entropy encoding is an encoding process that does not lose any information according to the entropy principle. The information entropy is the average amount of information of a source (a measure of uncertainty). The encoding mode of entropy encoding can include, but is not limited to, Shannon encoding, Huffman encoding and arithmetic coding.

[0053] Neural Network (NN): Neural network refers to an artificial neural network. The neural network is an operation model composed of a large number of nodes (or neurons) connected with each other. In the neural network, the neuron processing unit can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in the neural network can be divided into three categories: input units, output units and hidden units. The input unit accepts the signals and data of the external world; the output unit realizes the output of the processing result; the hidden unit is the unit between the input and the output unit, which cannot be observed from the outside of the system. The connection weight between neurons reflects the connection strength between units, and the information representation and processing are embodied in the connection relationship of processing units. The neural network is a non-programmed, brain-like information processing method. The essence of the neural network is to obtain a parallel distributed information processing function through the transformation and dynamics of the neural network, and to simulate the information processing function of the human brain neural system at different levels and levels. In the field of video processing, the commonly used neural network can include but is not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0054] Convolutional Neural Network (CNN): Convolutional neural network is a kind of feedforward neural network, which is one of the most representative network structures in deep learning technology. The artificial neuron of the convolutional neural network can respond to a part of the surrounding units in the coverage range, and has excellent performance for large image processing. The basic structure of the convolutional neural network includes two layers. One is the feature extraction layer (also called the convolution layer), and the input of each neuron is connected with the local receptive field of the previous layer, and the local feature is extracted. Once the local feature is extracted, the positional relationship between the features is also determined. The second is the feature mapping layer (also called the activation layer). Each calculation layer of the neural network is composed of multiple feature mappings, and each feature mapping is a plane. The weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function (S function), ReLU function (rectified linear unit function), Leaky-ReLU function, PReLU function (leaky rectified linear unit function), GDN Generalized Difference Network, generalized split standardization) function, etc. as the activation function of the convolutional network. In addition, since the neurons on a mapping plane share the weights, the number of free parameters of the network is reduced.

[0055] For example, one advantage of convolutional neural networks (CNNs) over image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. Another advantage of CNNs over ordinary neural networks is that ordinary neural networks use fully connected layers, meaning all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, however, avoid this difficulty through local connectivity and weight sharing.

[0056] Deconvolution: Also known as transposed convolution, deconvolution layers work similarly to convolutional layers. The main difference is that deconvolution layers use padding to make the output larger than the input (though they can also remain the same). If the stride is 1, the output size equals the input size; if the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0057] Generalization ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns hidden behind the data. The trained network can also give appropriate outputs for data outside the learning set that have the same pattern. This ability can be called generalization ability.

[0058] Features: The features discussed in this application are three-dimensional feature matrices or tensors of size C*W*H. See [link to relevant documentation]. Figure 1 The diagram shows a schematic of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be either the input or the output of a neural network.

[0059] Rate-Distortion Optimized: There are two indicators to evaluate the coding efficiency: code rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream is, the greater the compression rate is, and the greater the PSNR is, the better the reconstructed image quality is. In mode selection, the decision formula is essentially a comprehensive evaluation of the two. For example, the cost of a mode: J (mode) = D + λ * R, where D represents Distortion, which can usually be measured using the SSE indicator, which is the sum of the squares of the differences between the reconstructed image block and the source image. In order to achieve cost consideration, the SAD indicator can also be used, which is the sum of the absolute values of the differences between the reconstructed image block and the source image. λ is the Lagrange multiplier, and R is the actual number of bits required for image block coding under the mode, including the total number of bits required for coding mode information, motion information, and residual error. In mode selection, if the rate-distortion principle is used to compare and decide the coding mode, the best coding performance can usually be guaranteed.

[0060] A large number of encoding tools are proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the encoding tool that can obtain the optimal coding performance is often different. Therefore, in the encoding process, RDO (Rate-Distortion Opitimize) is usually used to compare the coding performance of different tools or modes to select the best tool or mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bit stream. Although this method brings higher encoding complexity, it can adaptively select the optimal tool or mode combination for different contents to obtain the optimal coding performance. The decoding end can obtain the relevant tool or mode information by directly parsing the marker information, and the complexity is less affected.

[0061] The decoding method and the encoding method in the embodiments of the present application will be described in detail in combination with several embodiments.

[0062] Embodiment 1: A decoding method is proposed in the embodiments of the present application, as shown in FIG. 1, which is a flowchart of the decoding method. The method can be applied to the decoding end (also referred to as a video decoder). The method can include the following steps. Figure 2

[0063] Step 201: Decoding the first code stream of the current image block to obtain the coefficient hyperparameter feature of each stage sub-block of the current image block. Wherein the current image block is divided into multiple stage sub-blocks, that is, the current image block includes multiple stage sub-blocks.

[0064] ​Step 202, for each stage sub-block, determining a probability distribution parameter based on the coefficient hyper-parameter feature of the stage sub-block, decoding the second code stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block.

[0065] Step 203, determining the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block. The mean feature of the stage sub-block is obtained based on the coefficient hyper-parameter feature of the stage sub-block and / or the reference feature of the stage sub-block, such as being obtained based on the coefficient hyper-parameter feature, or being obtained based on the coefficient hyper-parameter feature and the reference feature.

[0066] Step 204, determining the reconstructed image block corresponding to the current image block based on the reconstruction feature of each stage sub-block.

[0067] For example, for the first stage sub-block, the mean feature of the stage sub-block is obtained based on the coefficient hyper-parameter feature of the stage sub-block through the mean prediction network, or the default reference feature configured is obtained, and the mean feature of the stage sub-block is obtained based on the coefficient hyper-parameter feature of the stage sub-block and the default reference feature through the mean prediction network.

[0068] For the i-th stage sub-block, i is greater than 1, the reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the previous i-1 stage sub-blocks; the mean feature of the stage sub-block is obtained based on the coefficient hyper-parameter feature of the stage sub-block and the reference feature through the mean prediction network; wherein the reference feature includes all reconstruction features of the previous i-1 stage sub-blocks; or part of the reconstruction features of the previous i-1 stage sub-blocks; or the reconstruction feature of the i-1 stage sub-block.

[0069] For example, the mean prediction network can include a first prediction network, a second prediction network and a prediction fusion network, and the mean feature of the stage sub-block is obtained based on the coefficient hyper-parameter feature of the stage sub-block and the reference feature through the mean prediction network, which can include: obtaining the first prediction feature corresponding to the coefficient hyper-parameter feature through the first prediction network; obtaining the second prediction feature corresponding to the reference feature through the second prediction network; splicing the first prediction feature and the second prediction feature to obtain the spliced feature, inputting the spliced feature into the prediction fusion network, and processing the spliced feature through the prediction fusion network to obtain the mean feature of the stage sub-block.

[0070] Exemplarily, obtaining the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network can include: performing feature enhancement operation and up-sampling operation on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature. The feature enhancement operation can include convolution operation, or convolution operation and activation operation. The up-sampling operation can include deconvolution operation, cropping operation and activation operation, or deconvolution operation, cropping operation, activation operation and convolution operation. Exemplarily, the activation operation is Relu operation.

[0071] Exemplarily, obtaining the second prediction feature corresponding to the reference feature through the second prediction network can include: performing feature concatenation on all the reconstructed features in the reference feature in the channel dimension; performing convolution operation on the concatenated feature to obtain the second prediction feature; or performing feature addition on all the reconstructed features in the reference feature, and performing convolution operation on the added feature to obtain the second prediction feature. The reference feature includes all the reconstructed features of the i-1 stage sub-blocks; or part of the reconstructed features of the i-1 stage sub-blocks; or the reconstructed features of the i-1 stage sub-block.

[0072] Exemplarily, processing the concatenated feature through the prediction fusion network to obtain the mean feature of the stage sub-block can include but is not limited to: performing convolution operation and at least one fusion operation on the concatenated feature through the prediction fusion network to obtain the mean feature of the stage sub-block; wherein the fusion operation includes activation operation and convolution operation. Exemplarily, the activation operation is Relu operation.

[0073] Exemplarily, after determining the reconstructed feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block, the reconstructed feature of the stage sub-block can be further enhanced to obtain an enhanced reconstructed feature; wherein the enhanced reconstructed feature is used to determine the mean feature of the stage sub-block, and the enhanced reconstructed feature is used to determine the reconstructed image block.

[0074] For example, determining the reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block can include, but is not limited to: performing feature aggregation on the reconstructed features of each stage sub-block to obtain aggregated features, and inputting the aggregated features into a synthesis transformation network to obtain the reconstructed image block corresponding to the current image block. Alternatively, performing feature aggregation on the reconstructed features of each stage sub-block to obtain aggregated features, and performing block division on the aggregated features to obtain a plurality of block features; inputting each block feature into the synthesis transformation network to obtain a block reconstructed image block corresponding to the block feature, and merging the block reconstructed image blocks corresponding to the plurality of block features to obtain the reconstructed image block corresponding to the current image block. Alternatively, inputting the reconstructed features of each stage sub-block into the synthesis transformation network to obtain a block reconstructed image block corresponding to the stage sub-block, and merging the block reconstructed image blocks corresponding to all stage sub-blocks to obtain the reconstructed image block corresponding to the current image block.

[0075] For example, performing feature aggregation on the reconstructed features of each stage sub-block to obtain aggregated features can include, but is not limited to: performing phase aggregation on the reconstructed features of each stage sub-block to obtain aggregated features; or performing phase aggregation on the reconstructed features of each stage sub-block to obtain a plurality of phase-aggregated features, and then performing channel splicing on the plurality of phase-aggregated features to obtain aggregated features.

[0076] For example, performing block division on the aggregated features to obtain a plurality of block features can include: determining a target size of the block features; based on the target size, averaging the aggregated features into a plurality of block features in a top-to-bottom and left-to-right order, and the size of each block feature is the target size; or determining an actual block size and an overlap size of the block features, the actual block size being a block size of the block features excluding the overlap part, and the overlap size being a size of the overlapping part of adjacent blocks; based on the actual block size and the overlap size, dividing the aggregated features into a plurality of block features, and the size of each block feature is the target size; wherein, if the actual block size is tile_c*tile_w*tile_h, and the overlap size is tile_c*padding_w*padding_h, the target size is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h).

[0077] The merging of the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features to obtain the reconstructed image block corresponding to the current image block can include, but is not limited to: sorting the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features in a top-to-bottom and left-to-right order, and sequentially splicing the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features after sorting to obtain the reconstructed image block corresponding to the current image block. Alternatively, the actual sub-block size and the overlap size of each sub-block reconstructed image block are determined, the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features are sorted in a top-to-bottom and left-to-right order, and the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features after sorting are sequentially spliced based on the actual sub-block size and the overlap size to obtain the reconstructed image block corresponding to the current image block. The reconstructed image block includes a plurality of sub-block reconstructed image blocks, the size of the non-overlapping part of the sub-block reconstructed image block is the actual sub-block size of the sub-block reconstructed image block, and the size of the overlapping part of the sub-block reconstructed image block and the adjacent sub-block reconstructed image block is the overlap size of the sub-block reconstructed image block. For the overlapping part of the left and right two sub-block reconstructed image blocks, the value of the overlapping part is the value of the left sub-block reconstructed image block, or the value of the overlapping part is the average value of the two sub-block reconstructed image blocks. For the overlapping part of the upper and lower two sub-block reconstructed image blocks, the value of the overlapping part is the value of the upper sub-block reconstructed image block, or the value of the overlapping part is the average value of the two sub-block reconstructed image blocks.

[0078] The input of the aggregated feature to the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block can include: decoding the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of the current image block, and inputting the rate control parameter and the aggregated feature to the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block.

[0079] The input of each sub-block feature to the synthesis transformation network to obtain the sub-block reconstructed image block corresponding to the sub-block feature can include: decoding the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of each sub-block feature, and inputting the sub-block feature and the rate control parameter of the sub-block feature to the synthesis transformation network to obtain the sub-block reconstructed image block corresponding to the sub-block feature.

[0080] The input of the reconstructed feature of each stage sub-block to the synthesis transformation network to obtain the sub-block reconstructed image block corresponding to the stage sub-block can include: decoding the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of each stage sub-block, and inputting the reconstructed feature of the stage sub-block and the rate control parameter of the stage sub-block to the synthesis transformation network to obtain the sub-block reconstructed image block corresponding to the stage sub-block.

[0081] The inputting the code rate control parameter and the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block can include: processing the aggregated feature by the synthesis transformation network to obtain a first feature; processing the code rate control parameter by the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0082] The inputting the code rate control parameter and the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block can include: processing the aggregated feature by the synthesis transformation network to obtain a first feature; processing the code rate control parameter by the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0083] The inputting the code rate control parameter and the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block can include: processing the aggregated feature by the synthesis transformation network to obtain a first feature; processing the code rate control parameter by the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0084] The above execution order is only an example given for the convenience of description, and in actual application, the execution order between steps can also be changed, and the execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method can not be executed in the order shown and described in the specification, and the steps included in the method can be more or less than those described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; multiple steps described in the specification can also be combined into a single step for description in other embodiments.

[0085] As can be seen from the above technical solutions, in the embodiments of the present application, for the neural network-based coding and decoding technology, a coding and decoding scheme with variable and adjustable code rate is proposed, the parallelism is improved, the feature storage cache is effectively saved, higher code rate control accuracy is achieved, the coding performance loss is smaller, and better coding performance and code rate control accuracy are achieved. By using block-based coding, the peak memory occupation is lower, the decoding time of a single block is smaller, and high-speed parallel decoding capability is achieved, so that the neural network can effectively guarantee the quality of the reconstructed image block while maintaining low complexity, improve the coding performance and decoding performance, and reduce the complexity.

[0086] Embodiment 2: An encoding method is proposed in the embodiment of the present application, see Figure 3 As shown in FIG. 2, it is a flowchart of the encoding method, which can be applied to an encoding end (also referred to as a video encoder). The method can include the following steps:

[0087] Step 301: input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block.

[0088] Step 302: split the feature block into a plurality of stage sub-blocks to obtain the to-be-encoded features of the stage sub-blocks.

[0089] Step 303: for each stage sub-block corresponding to the current image block, obtain the coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into the first code stream of the current image block.

[0090] Step 304: determine the residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block. The mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block and / or the reference feature of the stage sub-block, such as being obtained based on the coefficient hyperparameter feature, or being obtained based on the coefficient hyperparameter feature and the reference feature.

[0091] Step 305: determine the probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into the second code stream of the current image block based on the probability distribution parameter.

[0092] For example, for the first stage sub-block, the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block through the mean prediction network, or the default reference feature configured is obtained, and the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature through the mean prediction network.

[0093] For the i-th stage sub-block, i is greater than 1, the reference feature of the i-th stage sub-block is obtained based on the reconstructed features of the previous i-1 stage sub-blocks; the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through the mean prediction network; wherein the reference feature includes all reconstructed features of the previous i-1 stage sub-blocks; or part of the reconstructed features of the previous i-1 stage sub-blocks; or the reconstructed feature of the i-1 stage sub-block.

[0094] For example, the processing process of the encoding end is similar to that of the decoding end, and the same parts are not repeated.

[0095] Exemplarily, the execution sequence described above is only an example given for the convenience of description, and in actual application, the execution sequence between steps can also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the corresponding method can not necessarily be executed in the order shown and described in the specification, and the steps included in the method can be more or less than those described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; multiple steps described in the specification can also be combined into a single step for description in other embodiments.

[0096] From the above technical solutions, in the embodiments of the present application, for the neural network-based coding and decoding technology, a coding and decoding scheme with variable and adjustable code rate is proposed, the parallelism is improved, the feature storage cache is effectively saved, higher code rate control accuracy is achieved, the coding performance loss is smaller, and better coding performance and code rate control accuracy are achieved. By using block coding, the peak memory occupation is lower, the decoding time of a single block is smaller, and high-speed parallel decoding capability is achieved, so that the neural network effectively guarantees the quality of the reconstructed image block while maintaining low complexity, improves the coding performance and decoding performance, and reduces the complexity.

[0097] Embodiment 3: For embodiments 1 and 2, the processing process at the encoding end can be seen from Figure 4 of course, Figure 4 which is only an example of the processing process at the encoding end, and the processing process at the encoding end is not limited.

[0098] After obtaining the current image block x (the current image block x can be the original image block x, that is, the input image block), the encoding end can analyze and transform the current image block x through the analysis transformation network (that is, the neural network) to obtain the image feature y corresponding to the current image block x. Wherein, the feature transformation of the current image block x through the analysis transformation network is to transform the current image block x into the image feature y in the latent domain, so as to facilitate the operation of all subsequent processes in the latent domain.

[0099] The image can be divided into one image block or multiple image blocks, if the image is divided into one image block, the current image block x can also be the image, that is, the coding and decoding process for the image block can also be directly used for the image.

[0100] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. For example, the image feature y can be input to a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and the training process of the hyperparameter encoding network is not limited. The hyperparameter encoding network can perform coefficient hyperparameter feature transformation on the image feature y to obtain the hyperprior latent information z (i.e., the coefficient hyperparameter feature z).

[0101] After obtaining the coefficient hyperparameter feature z, the encoding end can quantize the coefficient hyperparameter feature z to obtain a hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, i.e., Figure 4 The Q operation in the above formula is a quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain a Bitstream #1 (i.e., a first code stream) corresponding to the current image block, i.e., Figure 4 The AE operation in the above formula represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the coefficient hyperparameter feature z to obtain the Bitstream #1 corresponding to the current image block. The hyperparameter quantization feature or the coefficient hyperparameter feature z carried in the Bitstream #1 is mainly used to obtain parameters of the mean and the probability distribution model (i.e., probability distribution parameters).

[0102] After obtaining the Bitstream #1 corresponding to the current image block, the encoding end can send the Bitstream #1 corresponding to the current image block to the decoding end. The processing process of the decoding end for the Bitstream #1 corresponding to the current image block is described in subsequent embodiments.

[0103] After obtaining the Bitstream #1 corresponding to the current image block, the encoding end can also decode the Bitstream #1 to obtain a hyperparameter quantization feature, i.e., Figure 4 The AD in the above formula represents a decoding process, and then the hyperparameter quantization feature is dequantized to obtain a coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z, Figure 4 The IQ operation in the above formula is a dequantization process. Alternatively, after obtaining the Bitstream #1 corresponding to the current image block, the encoding end can also decode the Bitstream #1 to obtain the coefficient hyperparameter feature z_hat, without involving the dequantization process of the coefficient hyperparameter feature z_hat.

[0104] For the encoding process of Bitstream#1, a fixed probability density model encoding method can be used, and for the decoding process of Bitstream#1, a fixed probability density model decoding method can be used, and the encoding and decoding processes are not limited.

[0105] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in subsequent embodiments) to obtain the prediction value mu (i.e., the mean value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input into the mean value prediction network, and the mean value prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. The prediction process is not limited. For the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, which are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain the residual by subtracting the original feature, and the reconstructed feature y_hat is obtained by adding the decoded residual.

[0106] It should be noted that the mean value prediction network is an optional neural network, i.e., there can be no mean value prediction network, i.e., the prediction value mu does not need to be determined by the mean value prediction network. Figure 4 The dashed box in the above formula indicates that the mean value prediction network is optional.

[0107] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the prediction value mu, such as the difference between the image feature y and the prediction value mu as the residual feature r. Then, the image feature s is obtained by performing feature processing on the residual feature r, and the feature processing process is not limited and can be any feature processing manner. In this case, the mean value prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature y, the encoding end can perform feature processing on the image feature y to obtain the image feature s, and the feature processing process is not limited and can be any feature processing manner. In this case, the mean value prediction network is not needed, and the residual process is optional, which is indicated by the dashed box.

[0108] After obtaining the image feature s, the encoding end can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, i.e., Q(s) in the above formula. Figure 4 After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain the Bitstream#2 (i.e., the second code stream) corresponding to the current image block, i.e., Bitstream#2 in the above formula. Figure 4The AE operation in the above formula represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the image feature s to obtain Bitstream #2 corresponding to the current image block without involving the quantization process of the image feature s.

[0109] After obtaining Bitstream #2 corresponding to the current image block, the encoding end can send Bitstream #2 corresponding to the current image block to the decoding end. For the processing process of the decoding end for Bitstream #2 corresponding to the current image block, see the subsequent embodiments.

[0110] After obtaining Bitstream #2 corresponding to the current image block, the encoding end can also decode Bitstream #2 to obtain the image quantized feature, that is, Figure 4 AD in the above formula represents a decoding process. Then, the encoding end can dequantize the image quantized feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 4 The IQ operation in the above formula is a dequantization process. Alternatively, after obtaining Bitstream #2 corresponding to the current image block, the encoding end can also decode Bitstream #2 to obtain the image feature s' without involving the dequantization process of the image quantized feature.

[0111] After obtaining the image feature s', the encoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' without limitation. Any feature restoration manner can be used to obtain the residual feature r_hat, which can be the same as or different from the residual feature r. After obtaining the residual feature r_hat, the encoding end determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, which can be the same as or different from the image feature y. For example, the sum of the residual feature r_hat and the prediction value mu is taken as the image feature y_hat. In this case, a mean prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature s', the encoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' to obtain the image feature y_hat, which can be the same as or different from the image feature y. In this case, the mean prediction network does not need to be deployed, and the residual process is optional, as indicated by the dashed box.

[0112] After obtaining the image feature y_hat, the encoding end can perform synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x, such as inputting the image feature y_hat to a synthesis transformation network to perform synthesis transformation on the image feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0113] In one possible implementation, when the encoding end encodes the image quantization features or image features s to obtain Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization features or image features s based on the probability distribution model. Furthermore, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0114] To obtain the probability distribution model, please refer to [link / reference]. Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoder can perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat can be input into a probabilistic hyperparameter decoding network, which will then perform an inverse hyperparameter feature transformation on z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on p. The probabilistic hyperparameter decoding network can be a trained neural network; the training process of this network is not restricted, as long as it can perform the inverse hyperparameter feature transformation on z_hat.

[0115] In one possible implementation, the above-mentioned encoding process can be executed by a deep learning model or a neural network model to achieve end-to-end image compression and encoding, without any restrictions on the encoding process.

[0116] Example 4: For the processing procedures at the decoding end in Examples 1 and 2, please refer to... Figure 5 As shown, of course, Figure 5 This is just one example of the processing procedure at the decoding end, and no restrictions are imposed on the processing procedure at the decoding end.

[0117] After obtaining Bitstream#1 corresponding to the current image block, the decoding end can further decode Bitstream#1 to obtain the hyperparameter quantization features, i.e. Figure 5 In the diagram, AD represents the decoding process. Then, the hyperparameter quantization features are dequantized to obtain the coefficient hyperparameter features z_hat. The coefficient hyperparameter features z_hat and the coefficient hyperparameter features z can be the same or different. Figure 5 The IQ operation in the code is an inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat, without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0118] For the decoding process of Bitstream#1, a decoding method of fixed probability density model can be adopted, and no limitation is made.

[0119] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the image, that is, the decoding process of the image block can also be directly used for the image.

[0120] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in subsequent embodiments), to obtain the prediction value mu (i.e., the mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input into the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. The prediction process is not limited. For the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0121] It should be noted that the mean prediction network is an optional neural network, that is, there can be no mean prediction network, that is, the prediction value mu does not need to be determined by the mean prediction network. Figure 5 The dashed box in the above formula indicates that the mean prediction network is optional.

[0122] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode the Bitstream#2 to obtain the image quantization feature, that is, Figure 5 AD in the above formula indicates the decoding process. Then, the decoding end can perform inverse quantization on the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 5 The IQ operation in the above formula is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode the Bitstream#2 to obtain the image feature s', without involving the inverse quantization process of the image quantization feature.

[0123] After obtaining the image feature s', the decoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' to obtain a residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoding end determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, which is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, a mean prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature s', the decoding end can perform feature restoration on the image feature s' to obtain the image feature y_hat, which can be the same as or different from the image feature y. In this case, a mean prediction network does not need to be deployed, and the residual process is optional, as indicated by the dashed box.

[0124] After obtaining the image feature y_hat, the decoding end can perform synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x, such as inputting the image feature y_hat to a synthesis transformation network to perform synthesis transformation on the image feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat, thereby completing the image reconstruction process.

[0125] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine a probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to FIG. 8. Figure 5 As shown in FIG. 8, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain a probability distribution parameter p, such as inputting the coefficient hyperparameter feature z_hat to a probability hyperparameter decoding network to perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat by the probability hyperparameter decoding network to obtain the probability distribution parameter p, and after obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network, and the training process of the probability hyperparameter decoding network is not limited. The probability hyperparameter decoding network can only perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0126] In a possible implementation, the processing process of the decoding end described above can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, and the decoding process is not limited.

[0127] Embodiment 5: For Embodiment 1, Embodiment 2, Embodiment 3 and Embodiment 4, a mean prediction network can be deployed or not deployed, and in order to improve the feature encoding performance, the mean prediction network is taken as an example. In the deployment of the mean prediction network, in order to obtain the accurate prediction value of the image feature, the reconstructed value of the adjacent feature (i.e., based on the current image block coefficient parameter feature z_hat and the image feature y_hat of the previous image block, the context-based prediction is performed to obtain the prediction value mu corresponding to the current image block) is required, therefore, the feature value on the right side (using the reconstructed value of the feature on the left side or the upper left side as a reference) can only be subjected to the encoding process of the current feature after the encoding and reconstruction of the feature on the left side or the upper left side are completed, and this dependence relationship makes the encoding between the features only serially executable but not in parallel, thereby resulting in a higher complexity.

[0128] For the above finding, in this embodiment, by adopting the block-based coding and decoding, the parallelism can be improved, the feature storage cache can be effectively saved, the higher rate control accuracy can be achieved, the encoding performance loss is smaller, the better encoding performance and rate control accuracy are obtained, the peak memory occupation is lower, the decoding time of a single block is smaller, and the high-speed parallel decoding capability is possessed.

[0129] For the processing process of the encoding end, refer to Figure 6 The processing process of the encoding end is not limited.

[0130] 1. After obtaining the original image x, the block unit determines whether to block the original image x based on the image resolution, for example, if the image resolution is greater than a threshold, the original image x is blocked, otherwise, the original image x is not blocked. If yes, the original image x is divided into a plurality of image blocks x_p, and there is an overlapping part between adjacent image blocks, if no, the image block x_p is the original image x, that is, the original image x has only one image block, and each image block x_p can be recorded as a current image block. For convenience of description, the processing process of one current image block x_p is taken as an example for description.

[0131] 2. After obtaining the current image block x_p, the current image block x_p is analyzed and transformed by the analysis transformation network to obtain the image feature y_p (feature block y_p) corresponding to the current image block x_p. For example, the current image block x_p is transformed into the image feature y_p in the latent domain by the analysis transformation network, so that all subsequent processes are operated in the latent domain.

[0132] 3. The merging and classifying unit merges and classifies the image feature y_p to obtain N stage sub-blocks of the to-be-encoded feature y_i (i=1…N), that is, the image feature y_p is split into N stage sub-blocks of the to-be-encoded feature y_i.

[0133] 4. Obtain the coefficient hyper-parameter feature corresponding to each stage sub-block of the current image block. For example, perform coefficient hyper-parameter feature transformation on the to-be-encoded feature y_i of the stage sub-block i to obtain the coefficient hyper-parameter feature z_i corresponding to the stage sub-block i. For example, the to-be-encoded feature y_i can be input to a hyper-parameter encoding network (i.e., a neural network), and the hyper-parameter encoding network can perform coefficient hyper-parameter feature transformation on the to-be-encoded feature y_i to obtain the coefficient hyper-parameter feature z_i.

[0134] 5. Encode the coefficient hyper-parameter feature of each stage sub-block of the current image block into the first code stream (Bitstream#1) of the current image block. For example, perform quantization on the coefficient hyper-parameter feature z_i corresponding to the stage sub-block i to obtain a hyper-parameter quantized feature corresponding to the coefficient hyper-parameter feature z_i, and encode the hyper-parameter quantized feature to obtain the first code stream corresponding to the current image block. Alternatively, directly encode the coefficient hyper-parameter feature z_i corresponding to the stage sub-block i to obtain the first code stream corresponding to the current image block. After obtaining the first code stream corresponding to the current image block, the first code stream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end with respect to the first code stream corresponding to the current image block, see the subsequent embodiments.

[0135] 6. After obtaining the first code stream corresponding to the current image block, the first code stream can also be decoded to obtain the coefficient hyper-parameter feature of each stage sub-block of the current image block, such as the coefficient hyper-parameter feature z_hat_i corresponding to the stage sub-block i (i = 1…N).

[0136] For example, the first code stream can be decoded to obtain the hyper-parameter quantized feature of the stage sub-block i, and then the hyper-parameter quantized feature is dequantized to obtain the coefficient hyper-parameter feature z_hat_i corresponding to the stage sub-block i. Alternatively, the first code stream can be decoded to obtain the coefficient hyper-parameter feature z_hat_i corresponding to the stage sub-block i without involving the dequantization process.

[0137] The encoding process of the first code stream can adopt a fixed probability density model encoding method, and the decoding process of the first code stream can adopt a fixed probability density model decoding method, and the encoding and decoding processes are not limited.

[0138] 7. For each stage sub-block corresponding to the current image block, determine the probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block. For example, the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i can be inverse transformed to obtain the probability distribution parameter p_i corresponding to stage sub-block i, for example, the coefficient hyperparameter feature z_hat_i is input into the probability hyperparameter decoding network, and the coefficient hyperparameter feature z_hat_i is inverse transformed by the probability hyperparameter decoding network to obtain the probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, the probability distribution model can be generated based on the probability distribution parameter p_i.

[0139] 8. After obtaining the to-be-encoded features of the N stage sub-blocks, for the to-be-encoded feature y_1 of the first stage sub-block, the coefficient hyperparameter feature z_hat_1 of the stage sub-block is input into the mean prediction network to obtain the mean feature (i.e. the predicted value mu_1, the mean mu_1) of the stage sub-block. Alternatively, a configured default reference feature is obtained, and the coefficient hyperparameter feature z_hat_1 of the stage sub-block and the default reference feature are input into the mean prediction network to obtain the mean feature of the stage sub-block.

[0140] The residual feature r_1 is determined based on the to-be-encoded feature y_1 and the mean feature mu_1 of the first stage sub-block, for example, the difference between the to-be-encoded feature y_1 and the mean feature mu_1 is taken as the residual feature r_1. Then, the residual feature r_1 is processed to obtain the feature-processed residual feature r_1, and the feature processing process is not limited and can be any feature processing mode. Of course, the feature processing is an optional step, and the residual feature r_1 can also not be processed.

[0141] The residual feature r_1 (or the feature-processed residual feature r_1) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second code stream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r_1 (or the feature-processed residual feature r_1) can also be directly encoded to obtain the second code stream corresponding to the current image block. After obtaining the second code stream corresponding to the current image block, the second code stream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the second code stream corresponding to the current image block, see the subsequent embodiments. When the image quantization feature or the residual feature r_1 is encoded, the probability distribution model corresponding to the probability distribution parameter p_1 of the first stage sub-block can be used to encode the image quantization feature or the residual feature r_1 to obtain the second code stream.

[0142] After obtaining the second code stream, the image quantized feature can be decoded from the second code stream by using the probability distribution model corresponding to the probability distribution parameter p_1 of the first stage sub-block to obtain the image quantized feature, the image quantized feature is dequantized, and the feature recovery is performed on the dequantized feature to obtain the residual feature r_hat_1, or the image quantized feature is dequantized to directly obtain the residual feature r_hat_1 without involving the feature recovery process. Alternatively, after decoding the second code stream, the feature recovery is performed on the decoded feature to obtain the residual feature r_hat_1, or the residual feature r_hat_1 is directly obtained after decoding the second code stream.

[0143] After obtaining the residual feature r_hat_1, the reconstructed feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, such as the sum of the residual feature r_hat_1 and the mean feature mu_1 as the reconstructed feature y_hat_1.

[0144] 9、After obtaining the to-be-encoded features of the N stage sub-blocks, for the to-be-encoded feature y_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature z_hat_i of the stage sub-block i and the reference feature of the stage sub-block i are input into the mean prediction network to obtain the mean feature (i.e., the predicted value mu_i, the mean mu_i) of the stage sub-block i.

[0145] Among them, the reference feature of the stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the i-1 stage sub-blocks before the stage sub-block i, that is, based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature of the stage sub-block i is obtained based on all the reconstructed features y_hat_1 to y_hat_i-1, that is, the reference feature includes y_hat_1 to y_hat_i-1. Alternatively, the reference feature of the stage sub-block i is obtained based on part of the reconstructed features y_hat_1 to y_hat_i-1, that is, the reference feature includes part of the reconstructed features y_hat_1 to y_hat_i-1. Alternatively, the reference feature of the stage sub-block i is obtained based on the reconstructed feature of the i-1 stage sub-block, that is, the reference feature of the stage sub-block i is y_hat_i-1.

[0146] The residual feature r_i is determined based on the to-be-encoded feature y_i and the mean feature mu_i of the i-th stage sub-block, such as the difference between the to-be-encoded feature y_i and the mean feature mu_i as the residual feature r_i. The residual feature r_i is processed to obtain the processed residual feature r_i. Of course, the residual feature r_i can also not be processed.

[0147] The residual feature r_i (or the residual feature r_i processed by the feature) is quantized to obtain an image quantized feature, and the image quantized feature is encoded to obtain a second code stream corresponding to the current image block. Alternatively, the residual feature r_i (or the residual feature r_i processed by the feature) is directly encoded to obtain a second code stream corresponding to the current image block. The second code stream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end with respect to the second code stream corresponding to the current image block, see the subsequent embodiments. When the image quantized feature or the residual feature r_i is encoded, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block can be used to encode the image quantized feature or the residual feature r_i to obtain the second code stream.

[0148] After obtaining the second code stream, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block can be used to decode the second code stream to obtain the image quantized feature, and the image quantized feature is dequantized, and the feature is recovered after the dequantization to obtain the residual feature r_hat_i, or the image quantized feature is dequantized to directly obtain the residual feature r_hat_i. Alternatively, after decoding the second code stream, the feature is recovered to obtain the residual feature r_hat_i, or the residual feature r_hat_i is directly obtained after decoding the second code stream.

[0149] After obtaining the residual feature r_hat_i, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0150] 10、The clustering and blocking unit can obtain the reconstructed features y_hat_i of the N stage sub-blocks (i=1…N), and aggregate the reconstructed features y_hat_i of the N stage sub-blocks to obtain an aggregated feature. Then, the clustering and blocking unit blocks the aggregated feature to obtain at least one blocked feature y_hat_p, that is, the feature block y_hat_p.

[0151] 11、For each blocked feature y_hat_p, the clustering and blocking unit inputs the blocked feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the blocked reconstructed image block x_hat_p corresponding to the blocked feature y_hat_p.

[0152] 12、The merging unit can obtain the sub-block reconstructed image blocks x_hat_p corresponding to the plurality of sub-block features y_hat_p, and merge the sub-block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Obviously, if the original image x is divided into a plurality of current image blocks x_p, the reconstructed image blocks x_hat corresponding to all the current image blocks x_p can constitute a reconstructed image. If the original image x corresponds to one current image block x_p, the reconstructed image block x_hat is the reconstructed image.

[0153] In the above process, the encoding end can also assign different rate control parameters to different stage sub-blocks of different image blocks based on the region of interest or the code rate information, so that different stage sub-blocks of different image blocks consume different code rates.

[0154] For example, if special processing (such as non-uniform processing to ensure higher quality of the region of interest) is required for the region of interest, and the quantization distortion of each image block is not uniform, the rate control parameter can be a quantization parameter (the rate control parameter needs to be transmitted to the decoding end through the code stream), as shown in Figure 6 As shown, the rate control parameter affects the quantization process (Q) and the inverse quantization process (IQ) of the first code stream, and affects the quantization process (Q) and the inverse quantization process (IQ) of the second code stream.

[0155] For example, when quantizing the coefficient hyperparameter feature z_i corresponding to the stage sub-block i, if the stage sub-block i is located in the region of interest, the coefficient hyperparameter feature z_i corresponding to the stage sub-block i is quantized using the rate control parameter, which is used to ensure higher image quality of the stage sub-block i. When inverse quantizing the hyperparameter quantized feature corresponding to the stage sub-block i, the hyperparameter quantized feature corresponding to the stage sub-block i also needs to be inverse quantized using the rate control parameter.

[0156] For example, when quantizing the coefficient hyperparameter feature z_i corresponding to the stage sub-block i, if the stage sub-block i is located in the region of interest, the coefficient hyperparameter feature z_i corresponding to the stage sub-block i is quantized using the rate control parameter, which is used to ensure higher image quality of the stage sub-block i. When inverse quantizing the hyperparameter quantized feature corresponding to the stage sub-block i, the hyperparameter quantized feature corresponding to the stage sub-block i also needs to be inverse quantized using the rate control parameter.

[0157] For example, in order to accurately control the code rate, the rate control parameter can be a control parameter of the synthesis transformation network (the rate control parameter needs to be transmitted to the decoding end through the code stream), as shown in Figure 6As shown, the rate control parameter affects the analysis transform network and the synthesis transform network. For example, when the analysis transform network analyzes the current image block x_p, the rate control parameter and the current image block x_p are input into the analysis transform network, and the analysis transform network outputs the image feature y_p corresponding to the current image block x_p. When the patch feature y_hat_p is input into the synthesis transform network, the patch feature y_hat_p and the rate control parameter are input into the synthesis transform network, and the synthesis transform network outputs the patch reconstructed image block x_hat_p corresponding to the patch feature y_hat_p.

[0158] Based on the configured target rate (each image block corresponds to a target rate, or each image corresponds to a target rate, or multiple images correspond to a target rate), the encoding end can try to use different rate control parameters. After using a certain rate control parameter, if the code rate corresponding to the code stream (such as the first code stream and the second code stream) for the current block is the target rate or close to the target rate (i.e., the difference between the two is less than a threshold value), then this rate control parameter is the final rate control parameter, and the encoding end can send this rate control parameter to the decoding end.

[0159] Embodiment 6: The processing procedure of the decoding end can refer to Figure 7 As shown, the processing procedure of the decoding end is not limited.

[0160] 1. After obtaining the first code stream (Bitstream#1) corresponding to the current image block, the first code stream is decoded to obtain the coefficient hyperparameter feature of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_hat_i of stage sub-block i (i = 1…N).

[0161] For example, the first code stream can be decoded to obtain the hyperparameter quantization feature of stage sub-block i, and then the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i. Alternatively, the first code stream can be decoded to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i without involving the dequantization process. The decoding process of the first code stream can use a fixed probability density model decoding method, and the decoding process is not limited.

[0162] 2、For each stage sub-block corresponding to the current image block, determine the probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block. For example, the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i can be inverse transformed to obtain the probability distribution parameter p_i corresponding to the stage sub-block i, for example, the coefficient hyperparameter feature z_hat_i is input into the probability hyperparameter decoding network, and the coefficient hyperparameter feature z_hat_i is inverse transformed by the probability hyperparameter decoding network to obtain the probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, the probability distribution model can be generated based on the probability distribution parameter p_i.

[0163] 3、For each stage sub-block corresponding to the current image block, the second code stream (Bitstream#2) of the current image block is decoded based on the probability distribution parameter (probability distribution model) of the stage sub-block to obtain the residual feature of the stage sub-block.

[0164] For example, the probability distribution model corresponding to the probability distribution parameter p_i of the i-th (i=1…N) stage sub-block can be used to decode the second code stream to obtain the image quantization feature, and the image quantization feature is inverse quantized, and the inverse quantized feature is recovered to obtain the residual feature r_hat_i, or the image quantization feature is inverse quantized to directly obtain the residual feature r_hat_i. Or, after decoding the second code stream, the decoded feature is recovered to obtain the residual feature r_hat_i, or the residual feature r_hat_i is directly obtained after decoding the second code stream.

[0165] 4、After obtaining the residual features of the N stage sub-blocks, for the residual feature r_hat_1 of the first stage sub-block, the coefficient hyperparameter feature z_hat_1 of the stage sub-block is input into the mean prediction network to obtain the mean feature (i.e. predicted value mu_1, mean mu_1) of the stage sub-block. Or, get the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 of the stage sub-block and the default reference feature into the mean prediction network to obtain the mean feature of the stage sub-block.

[0166] Then, the reconstruction feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, for example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be taken as the reconstruction feature y_hat_1.

[0167] 5、After obtaining the residual features of N stage sub-blocks, for the residual features r_hat_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient parameter feature z_hat_i of the stage sub-block i and the reference feature of the stage sub-block i are input into the mean prediction network to obtain the mean feature (i.e., the predicted value mu_i, the mean mu_i) of the stage sub-block i.

[0168] The reference feature of the stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the i-1 stage sub-blocks before the stage sub-block i, i.e., based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature of the stage sub-block i is obtained based on all the reconstructed features y_hat_1 to y_hat_i-1, i.e., the reference feature includes y_hat_1 to y_hat_i-1. Alternatively, the reference feature of the stage sub-block i is obtained based on part of the reconstructed features y_hat_1 to y_hat_i-1, i.e., the reference feature includes part of the reconstructed features y_hat_1 to y_hat_i-1. Alternatively, the reference feature of the stage sub-block i is obtained based on the reconstructed feature of the i-1 stage sub-block, i.e., the reference feature of the stage sub-block i is y_hat_i-1.

[0169] In order to control the size of the cache, the number of reconstructed features included in the reference feature cannot exceed a preset number, assuming that the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference feature includes all the reconstructed features y_hat_1 to y_hat_i-1, and if the number of reconstructed features is greater than M, the reference feature includes M reconstructed features (i.e., part of the reconstructed features) y_hat_1 to y_hat_i-1. In order to select M reconstructed features from y_hat_1 to y_hat_i-1, M reconstructed features can be selected based on a default strategy, such as selecting M reconstructed features closest to the current stage sub-block in the decoding order, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, and so on until M reconstructed features are selected, of course, other ways of selecting M reconstructed features can also be used, which are not limited. Alternatively, the index values of M reconstructed features can be obtained by decoding the code stream, and M reconstructed features are selected based on the decoded index values.

[0170] Then, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, for example, the sum of the residual feature r_hat_i and the mean feature mu_i can be taken as the reconstructed feature y_hat_i.

[0171] 6、The clustering and partition unit can obtain the reconstructed features y_hat_i (i = 1…N) of the N-stage sub-blocks, and aggregate the reconstructed features y_hat_i of the N-stage sub-blocks to obtain aggregated features. Then, the clustering and partition unit partitions the aggregated features to obtain at least one partitioned feature y_hat_p, i.e., a feature block y_hat_p.

[0172] 7、For each partitioned feature y_hat_p, the clustering and partition unit inputs the partitioned feature y_hat_p to the synthesis transform network, and the synthesis transform network outputs a partitioned reconstructed image block x_hat_p corresponding to the partitioned feature y_hat_p.

[0173] 8、The merging unit can obtain the partitioned reconstructed image blocks x_hat_p corresponding to the plurality of partitioned features y_hat_p, and merge the partitioned reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat corresponding to the current image block. Obviously, if the original image x is divided into a plurality of current image blocks x_p, the reconstructed image blocks x_hat corresponding to all the current image blocks x_p can constitute a reconstructed image. If the original image x corresponds to one current image block x_p, the reconstructed image block x_hat is the reconstructed image.

[0174] In the above process, different code rate control parameters can be assigned to different stage sub-blocks of different image blocks based on the region of interest or the code rate information, so that different stage sub-blocks of different image blocks consume different code rates.

[0175] For example, if the region of interest needs to be processed specially (e.g., non-uniform processing to ensure that the quality of the region of interest is higher), the code rate control parameter can be a quantization parameter (the code rate control parameter needs to be transmitted to the decoding end through a code stream), as shown in FIG. 6. Figure 7 As shown, the code rate control parameter affects the inverse quantization process (IQ) of the first code stream and the inverse quantization process (IQ) of the second code stream.

[0176] For example, when inverse quantizing the hyper-parameterized quantized feature corresponding to the stage sub-block i, if the code rate control parameter corresponding to the stage sub-block i has been decoded from the code stream, the code rate control parameter is used to ensure that the image quality corresponding to the stage sub-block i is higher, and the code rate control parameter can be used to inverse quantize the hyper-parameterized quantized feature corresponding to the stage sub-block i.

[0177] For another example, when inverse quantizing the image quantized feature corresponding to the stage sub-block i, if the code rate control parameter corresponding to the stage sub-block i has been decoded from the code stream, the code rate control parameter is used to ensure that the image quality corresponding to the stage sub-block i is higher, and the code rate control parameter can be used to inverse quantize the image quantized feature corresponding to the stage sub-block i.

[0178] For example, in order to perform accurate rate control, the rate control parameter can be a control parameter of the synthesis transform network (the rate control parameter needs to be transmitted to the decoding end through the bitstream, and the control parameter of the synthesis transform network refers to that the synthesis transform network can be determined based on the rate control parameter), see Figure 7 As shown in FIG. 9, the rate control parameter affects the synthesis transform network. For example, when the patch feature y_hat_p is input to the synthesis transform network, the patch feature y_hat_p and the rate control parameter are input to the synthesis transform network, and the synthesis transform network outputs the patch reconstructed image block x_hat_p corresponding to the patch feature y_hat_p.

[0179] In embodiments 5 and 6, for the i-th (i = 1…N) stage subblock, after obtaining the reconstructed feature y_hat_i (i.e., the sum of the residual feature r_hat_i and the mean feature mu_i) of the i-th stage subblock, the reconstructed feature y_hat_i is used to determine the mean feature of the subsequent stage subblock, and / or the reconstructed feature y_hat_i is used to determine the reconstructed image block. Wherein, the reconstructed feature y_hat_i is used to determine the mean feature of the subsequent stage subblock refers to that the reconstructed feature y_hat_i is used as the reference feature of the subsequent stage subblock, so as to determine the mean feature of the subsequent stage subblock. The reconstructed feature y_hat_i is used to determine the reconstructed image block refers to that the reconstructed feature y_hat_i is input to the clustering patch unit, so as to determine the reconstructed image block corresponding to the current image block.

[0180] For the i-th (i = 1…N) stage subblock, after obtaining the reconstructed feature y_hat_i of the i-th stage subblock, the reconstructed feature y_hat_i can also be enhanced to obtain an enhanced reconstructed feature y_hat_i', and the enhanced reconstructed feature y_hat_i' is used to determine the mean feature of the subsequent stage subblock, and / or the enhanced reconstructed feature y_hat_i' is used to determine the reconstructed image block. Wherein, the enhanced reconstructed feature y_hat_i' is used to determine the mean feature of the subsequent stage subblock refers to that the enhanced reconstructed feature y_hat_i' is used as the reference feature of the subsequent stage subblock, so as to determine the mean feature of the subsequent stage subblock. The enhanced reconstructed feature y_hat_i' is used to determine the reconstructed image block refers to that the enhanced reconstructed feature y_hat_i' is input to the clustering patch unit, so as to determine the reconstructed image block corresponding to the current image block. For example, the bitstream can be decoded to obtain enhancement information, and the reconstructed feature y_hat_i is enhanced based on the enhancement information to obtain a new reconstructed feature y_hat_i', which can be used for prediction of subsequent y_hat_i+1 to y_hat_N, or can be used for subsequent clustering patch process.

[0181] In one possible implementation, for the enhancement process of the reconstructed feature y_hat, the following approach can be adopted:

[0182] Method 1, the input is the reconstructed feature y_hat before enhancement, Threshold[idx], GreaterFlag[idx], Scale1[idx] and Scale2[idx], and the output is the enhanced reconstructed feature y_hat_en. Among them, Threshold represents the threshold, different idx can correspond to the same or different threshold, idx represents the index, such as 0, 1, 2, 3, etc. GreaterFlag represents the flag bit, different idx can correspond to the same or different flag bit. Scale1 represents the scaling factor, different idx can correspond to the same or different Scale1. Scale2 represents the scaling factor, different idx can correspond to the same or different Scale2.

[0183] First step: determine the mask[c,i,j] under each set of filter (index is idx). For example, the mask[c,i,j] can be determined by the following expression, of course, the following expression is only an example, which is not limited.

[0184]

[0185] σ[c,i,j] represents the probability distribution parameter value of [c,i,j], c represents the channel, i represents the horizontal coordinate, and j represents the vertical coordinate. If the probability distribution parameter value is greater than Threshold[idx], GreaterFlag[idx] is true, and mask[idx,c,i,j] is 1. If the probability distribution parameter value is less than Threshold[idx], GreaterFlag[idx] is false, and mask[idx,c,i,j] is 1. Otherwise, mask[idx,c,i,j] is 0.

[0186] Second step: execute each set of filter (index is idx) in turn. If mask[idx,c,i,j] is 1, then y_hat_en[c,i,j] = y_hat[c,i,j] + mean_hat[c,i,j]*Scale2[idx] + residual_hat[c,i,j]*Scale1[idx]; y_hat[c,i,j] = y_hat_en[c,i,j]. If there are multiple sets of filter, y_hat is compensated multiple times. Among them, mean_hat represents the mean feature, that is, the aforementioned mu_i; residual_hat represents the residual feature, that is, the aforementioned r_hat.

[0187] Method 2, in method 1, the scaling factor of each channel c is the same, in method 2, different channels c allow different scaling factors, that is, the scaling factor is Scale1[c, idx] and Scale2[c, idx], the current feature can be scaled by controlling the switch, or the scaling factor of the channel can be changed to 0 to turn off scaling. Method 2 is different from method 1 in that the enhancement method is as follows: y_hat_en[c, i, j] = y_hat[c, i, j] + mean_hat[c, i, j] * Scale2[c, idx] + residual_hat[c, i, j] * Scale1[c, idx].

[0188] Method 3, method 3 is similar to method 1 in implementation, the difference is that the enhancement method is as follows: y_hat_en[c, i, j] = y_hat[c, i, j] * Scale1[idx] + mean_hat[c, i, j] * Scale2[idx].

[0189] Method 4, method 4 is similar to method 1 in implementation, the difference is that the enhancement method is as follows: y_hat_en[c, i, j] = y_hat[c, i, j] * Scale1[idx] + residual_hat[c, i, j] * Scale2[idx]. In method 4, Scale1[idx] can also be defaulted to 1, and only Scale2[idx] value is changed to affect the adjustment method.

[0190] Embodiment 8: In embodiments 5 and 6, the coefficient hyperparameter feature z_hat_1 of the first stage subblock can be input to the mean prediction network, and the mean feature of the first stage subblock is output by the mean prediction network. The prediction process of the mean prediction network is not limited. For the i-th (i = 1…N) stage subblock, the coefficient hyperparameter feature z_hat_i of the i-th stage subblock and the reference feature (which can be a default reference feature) of the i-th stage subblock can be input to the mean prediction network, and the mean feature of the i-th stage subblock is output by the mean prediction network. The prediction process of the mean prediction network is described below.

[0191] Referring to Figure 8A , it is a structural schematic diagram of the mean prediction network, which can include a first prediction network, a second prediction network and a prediction fusion network. Of course, this is just an example, and the structure of the mean prediction network is not limited.

[0192] The hyperparameter features z_hat_i of the i-th stage sub-block can be input into the first prediction network to obtain the first predicted features corresponding to z_hat_i. The reference features of the i-th stage sub-block (such as the reconstructed features y_hat_1 to y_hat_i-1) can be input into the second prediction network to obtain the second predicted features corresponding to the reference features. Then, the first and second predicted features are concatenated to obtain the concatenated features. These concatenated features are then input into the prediction fusion network, which processes them to obtain the mean feature mu_i of the i-th stage sub-block.

[0193] For the first prediction network: feature enhancement and upsampling operations are performed on the hyperparameter feature z_hat_i through the first prediction network to obtain the first predicted feature. The feature enhancement operation may include, but is not limited to, convolution operations, or convolution operations and activation operations. Of course, the above are just examples of feature enhancement operations and are not limited to any particular operation. The upsampling operation may include, but is not limited to, deconvolution operations, pruning operations, and activation operations, or deconvolution operations, pruning operations, activation operations, and convolution operations. Of course, the above are just examples of upsampling operations and are not limited to any particular operation. For example, the activation operation is a ReLU operation.

[0194] For example, see Figure 8B The diagram shows the structure of the first prediction network, which consists of three enhancement networks and two 2x upsampling networks. Of course, the number of enhancement networks can be more or less, the number of 2x upsampling networks can be more or less, and the 2x upsampling networks can be replaced with upsampling networks of other multiples.

[0195] The hyperparameter feature z_hat_i is processed by enhancement network 1 and 2x upsampling network 1 to obtain enhanced large-size feature 1, whose spatial dimensions (width and height) are twice that of the hyperparameter feature z_hat_i. Enhanced large-size feature 1 is then processed by enhancement network 2 and 2x upsampling network 2 to obtain enhanced large-size feature 2, whose spatial dimensions (width and height) are twice that of enhanced large-size feature 1. Enhanced large-size feature 2 is then processed by enhancement network 3 to obtain the first predicted feature.

[0196] For example, augmented network 1 and augmented network 2 can be the same or different, augmented network 1 and augmented network 3 can be the same or different, and augmented network 2 and augmented network 3 can be the same or different.

[0197] For example, one implementation of an augmentation network (such as augmentation network 1, and / or augmentation network 2, and / or augmentation network 3) can be found in [reference needed]. Figure 8CAs shown, the enhancement network can be composed of a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer. The convolutional layer can be a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, without limitation, such as a 3x3 convolutional layer.

[0198] For example, one embodiment of the enhancement network (such as enhancement network 1, and / or enhancement network 2, and / or enhancement network 3) can refer to Figure 8D As shown, the enhancement network can be composed of a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer. The convolutional layer can be a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, without limitation, such as a 3x3 convolutional layer.

[0199] The activation layer can be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, without limitation to the type of activation layer, such as a relu layer.

[0200] Of course, the above are only two examples of the enhancement network, without limitation, as long as the enhancement function can be achieved.

[0201] For example, the 2x upsampling network 1 and the 2x upsampling network 2 can be the same or different.

[0202] For example, one embodiment of the 2x upsampling network (such as 2x upsampling network 1 and / or 2x upsampling network 2) can refer to Figure 8E As shown, the 2x upsampling network can be composed of a 2x deconvolutional layer, a Crop layer, and an activation layer. The deconvolutional layer can be a 2x2 deconvolutional layer, or a 3x3 deconvolutional layer, or a 4x4 deconvolutional layer, or a 5x5 deconvolutional layer, without limitation, such as a 4x4 deconvolutional layer. The activation layer can be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, without limitation to the type of activation layer, such as a relu layer. In addition, the Crop layer is used to crop the features, so that the spatial resolution of the features becomes smaller, and the number of channels remains unchanged.

[0203] For example, one embodiment of the 2x upsampling network (such as 2x upsampling network 1 and / or 2x upsampling network 2) can refer to Figure 8FAs shown, the 2x up-sampling network can be composed of a 2x de-convolution layer, a Crop layer, an activation layer, and a convolution layer. The de-convolution layer can be a 2x2 de-convolution layer, or a 3x3 de-convolution layer, or a 4x4 de-convolution layer, or a 5x5 de-convolution layer. The activation layer can be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer. The Crop layer is used to crop the features so that the spatial resolution of the features is reduced while the number of channels remains unchanged. The convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, without limitation, such as a 3x3 convolution layer.

[0204] Of course, the above are only two examples of the 2x up-sampling network, without limitation, as long as the up-sampling function can be implemented.

[0205] For example, the convolution layer in the first prediction network can be a normal 2D convolution, or a grouped convolution, such as a 2x grouped convolution. The implementation of the grouped convolution can be referred to in subsequent embodiments, which will not be described here.

[0206] For the second prediction network: all reconstructed features in the reference features of the i-th stage sub-block are concatenated in the channel dimension through the second prediction network, and the convolution operation is performed on the concatenated features to obtain the second prediction feature. Alternatively, all reconstructed features in the reference features of the i-th stage sub-block are added through the second prediction network, and the convolution operation is performed on the added features to obtain the second prediction feature. The reference features include all reconstructed features of the previous i-1 stage sub-blocks, or part of the reconstructed features of the previous i-1 stage sub-blocks, or the reconstructed features of the i-1 stage sub-block.

[0207] For example, referring to Figure 8G As shown, it is a structure diagram of the second prediction network, which can be composed of a feature concatenation layer and a convolution layer, without limitation to the structure of the second prediction network. If the reference features include all reconstructed features of the previous i-1 stage sub-blocks (such as y_hat_1 to y_hat_i-1), the feature concatenation layer can concatenate the reconstructed features in the channel dimension to obtain the concatenated features. Alternatively, if the reference features include part of the reconstructed features of the previous i-1 stage sub-blocks (such as part of y_hat_1 to y_hat_i-1), the feature concatenation layer can concatenate the reconstructed features in the channel dimension to obtain the concatenated features. Alternatively, if the reference features include the reconstructed features of the i-1 stage sub-block, the feature concatenation layer can directly take the reconstructed features of the i-1 stage sub-block as the concatenated features.

[0208] The feature concatenation layer can input the concatenated features to a convolution layer, and the convolution layer can perform convolution operation on the concatenated features to obtain the second prediction feature. The convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and the selection is not limited, for example, a 3x3 convolution layer can be selected. For the convolution layer in the second prediction network, it can be a normal 2D convolution, or a grouped convolution, for example, a 2-grouped convolution, and the implementation of the grouped convolution can refer to subsequent embodiments.

[0209] For example, the second prediction network can be composed of a feature addition layer and a convolution layer. If the reference features include all reconstructed features of the previous i-1 stage subblocks, the feature addition layer can perform feature addition on the reconstructed features to obtain the added features. Alternatively, if the reference features include part of the reconstructed features of the previous i-1 stage subblocks, the feature addition layer can perform feature addition on the reconstructed features to obtain the added features. Alternatively, if the reference features include the reconstructed features of the i-1 stage subblock, the feature addition layer can directly take the reconstructed features of the i-1 stage subblock as the added features. The feature addition layer inputs the added features to the convolution layer, and the convolution layer performs convolution operation on the added features to obtain the second prediction feature.

[0210] In order to control the cache size, the number of reconstructed features included in the reference features cannot exceed a preset number, assuming that the preset number is M, if the number of reconstructed features is less than or equal to M, the reference features include all reconstructed features in y_hat_1 to y_hat_i-1. Alternatively, if the number of reconstructed features is greater than M, the reference features include M reconstructed features in y_hat_1 to y_hat_i-1. For example, M reconstructed features can be selected based on a default strategy, for example, M reconstructed features closest to the current stage subblock in decoding order can be selected. Alternatively, the index values of M reconstructed features can be obtained by decoding the code stream, and M reconstructed features can be selected based on the decoded index values.

[0211] For the prediction fusion network: the concatenated features (i.e., the concatenated features of the first prediction feature and the second prediction feature) are subjected to convolution operation and at least one fusion operation by the prediction fusion network to obtain the mean feature mu_i of the i-th stage subblock. For each fusion operation, the fusion operation can include an activation operation and a convolution operation. For example, the activation operation is a Relu operation.

[0212] For example, referring to Figure 8HAs shown in FIG. 9, the structure diagram of the prediction fusion network is shown, the prediction fusion network can be composed of a convolutional layer and N fusion layers, N is a positive integer, such as N is 1 or 2, and each fusion layer can be composed of an activation layer and a convolutional layer, and the structure of the prediction fusion network is not limited. Obviously, after the input features (i.e. the features after splicing) pass through the convolutional layer and the N "activation layer + convolutional layer", the output features, i.e. the mean features mu_i of the i-th stage sub-block, can be obtained.

[0213] The convolutional layer can be a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer, which is not limited, such as a 1x1 convolutional layer can be selected. For the convolutional layer in the prediction fusion network, it can be a normal 2D convolution, or a grouped convolution, such as a 2-fold grouped convolution, and the implementation of the grouped convolution is described in the subsequent embodiments. The activation layer can be a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, such as a relu layer can be selected.

[0214] Embodiment 9: For the encoding end and the decoding end, the reconstructed image block corresponding to the current image block needs to be determined based on the reconstructed features of each stage sub-block. For example, in embodiments 5 and 6, the clustering and blocking unit can perform feature aggregation on the reconstructed features (y_hat_1 to y_hat_N) of the N stage sub-blocks to obtain the aggregated features (which can also be referred to as the full-size aggregated features y_hat_all), and perform blocking on the aggregated features y_hat_all to obtain at least one blocking feature y_hat_p, such as the blocking feature y_hat_p_1 to the blocking feature y_hat_p_K. For each blocking feature y_hat_p, the clustering and blocking unit inputs the blocking feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the blocking reconstructed image block x_hat_p corresponding to the blocking feature y_hat_p. The merging unit can obtain the blocking reconstructed image blocks x_hat_p corresponding to the plurality of blocking features y_hat_p, and merge the blocking reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.

[0215] First, the clustering and blocking unit performs feature aggregation on the reconstructed features of the N stage sub-blocks to obtain the aggregated features.

[0216] For example, as shown in FIG. 9, Figure 9A As shown in FIG. 9, the reconstructed features (y_hat_1 to y_hat_N) of the N stage sub-blocks can be aggregated according to the phase to obtain the aggregated features y_hat_all. In Figure 9ATake N = 4 as an example. For example, an example of aggregation according to the phase can be as follows: after obtaining y_hat_group1, y_hat_group1 can be taken as the aggregated feature y_hat_all.

[0217]

[0218] For example, as shown in Figure 9B As shown, the reconstructed features (y_hat_1 to y_hat_N) of the N stage subblocks can be aggregated according to the phase to obtain a plurality of phase aggregated features, and the plurality of phase aggregated features can be spliced according to the channel to obtain the aggregated feature y_hat_all. For example, an example of aggregation according to the phase is shown in the above expression, and y_hat_group1 and y_hat_group2 are obtained by aggregating according to the phase. If there are more reconstructed features, after obtaining y_hat_group1 and y_hat_group2, based on the aggregated plurality of y_hat_group, the aggregated feature y_hat_all can be obtained by splicing according to the channel.

[0219] For example, before the aggregation of the reconstructed features of the N stage subblocks, useless feature values can also be removed. For example, because the height or width of the full-size feature is odd, when the encoding side is divided into four, in order to ensure that the size of the reconstructed feature of each stage subblock is the same, some of the reconstructed features of the stage subblocks need to be padded. The padding scheme can add a column to the right or a row to the bottom of the full-size feature. Based on this, after the aggregation of the reconstructed features of the N stage subblocks, the useless features need to be removed. For example, as shown in Figure 9C As shown, an example of an odd width is given. If the height is odd, the bottom row is removed. Of course, the above is the case of being divided into four. If it is divided into 16, the removed features can be more than one row or one column.

[0220] Second, the clustering and blocking unit blocks the aggregated feature y_hat_all to obtain K block features y_hat_p.

[0221] For example, the aggregated feature y_hat_all can also be blocked to reduce subsequent memory overhead (the synthesis transformation operation needs to up-sample the feature multiple times, so that the feature data size exceeds the allowable size of the memory). Of course, this feature blocking process is not necessary. If the size of the current feature is small, or based on the bitstream information, it is determined that the current feature does not need to be blocked, the following blocking process will be directly skipped, y_hat_p is equal to y_hat_all, that is, there is only one block feature.

[0222] In order to block the post-aggregated feature y_hat_all, a feature blocking manner without overlapping part can be adopted, as shown in FIG. 6, which is an example of dividing the post-aggregated feature y_hat_all into four block features y_hat_p. Figure 9D

[0223] In the feature blocking manner without overlapping part, the target size of the block feature can be determined, and the post-aggregated feature y_hat_all can be evenly divided into a plurality of block features in the order from top to bottom and from left to right based on the target size, and the size of each block feature is the target size, such as block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0224] For example, the target size is the size of each block feature, which can be read from the code stream or determined based on the size of the current feature to be blocked, and no limitation is made in this regard. If the number of channels of each block feature is tile_c, the width is tile_w, and the height is tile_h, the post-aggregated feature y_hat_all can be evenly divided into a plurality of tile_c*tile_w*tile_h block features in the order from top to bottom and from left to right, such as block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0225] In order to block the post-aggregated feature y_hat_all, a feature blocking manner with overlapping part can be adopted, as shown in FIG. 7, which is an example of dividing the post-aggregated feature y_hat_all into four block features y_hat_p. Figure 9E

[0226] In the feature blocking manner with overlapping part, the actual blocking size and the overlapping size of the block feature can be determined. The actual blocking size can be the blocking size without the overlapping part, and the overlapping size can be the size of the overlapping part of adjacent blocks. The post-aggregated feature y_hat_all can be divided into a plurality of block features based on the actual blocking size and the overlapping size, and the size of each block feature is the target size, such as block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0227] For example, the actual blocking size and the overlapping size are determined. The actual blocking size is the blocking size without the overlapping part, and the overlapping size is the size of the overlapping part of adjacent blocks. These size information can be read from the code stream or determined based on the size of the current feature to be blocked, and no limitation is made in this regard. See FIG. 8. Figure 9E ​​As shown, if the actual tile size is tile_c*tile_w*tile_h, and the overlap size is tile_c*padding_w*padding_h, then the size of each tile feature (i.e., the target size) is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h) after up and down and left and right overlaps. Based on this, for the aggregated feature y_hat_all, the first to (tile_h+2*padding_h) row features in the first to (tile_w+2*padding_w) columns are divided into the first tile feature. The tile_h+padding_h+1 to (2*tile_h+3*padding_h) row features in the tile_w+padding_w+1 to 2*tile_w+3*padding_w columns are divided into the second tile feature, and so on.

[0228] Referring to Figure 9E As shown, the aggregated feature y_hat_all of 4x4x4 can be divided into 4 tile features of 4x3x3, where tile_c is 4, padding_w and padding_h are 1, and tile_w and tile_h are 1.

[0229] For example, for the last row or column of tiles, if the number is not enough, the overlap area can be increased so that the final tile size meets the requirements. Referring to Figure 9F As shown, an example is given, where the size of y_hat_all is 11x13, padding_w and padding_h are 1, and tile_w and tile_h are 3. The size of the first tile and the second tile in the first row is 5x5, and the third tile, in order to ensure that the size of the tile is also 5x5, expands 2 columns of overlap area to the left.

[0230] Thirdly, the clustering tile unit inputs each tile feature y_hat_p (such as K tile features y_hat_p) to the synthesis transformation network, and outputs a tile reconstruction image block x_hat_p corresponding to each tile feature y_hat_p by the synthesis transformation network.

[0231] For each tile feature y_hat_p, after the tile feature y_hat_p is input to the synthesis transformation network, the synthesis transformation network is used to perform synthesis transformation on the tile feature y_hat_p to obtain a tile reconstruction image block x_hat_p, and no limitation is made in this regard.

[0232] Fourthly, the merging unit merges a plurality of tile reconstruction image blocks x_hat_p to obtain a reconstruction image block x_hat.

[0233] The merging unit can obtain K patch reconstructed image blocks x_hat_p corresponding to the K patch features y_hat_p, and merge the K patch reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.

[0234] For example, if the feature patching manner without overlapping parts is used to patch the aggregated features y_hat_all, the merging unit merges the K patch reconstructed image blocks x_hat_p using the feature merging manner without overlapping parts. For example, the K patch reconstructed image blocks x_hat_p corresponding to the K patch features y_hat_p are sorted in the order from top to bottom and from left to right, and the K patch reconstructed image blocks x_hat_p after sorting are spliced in turn to obtain a reconstructed image block x_hat corresponding to the current image block. The splicing process can be the inverse process of Figure 9D , which will not be described herein.

[0235] For example, if the feature patching manner with overlapping parts is used to patch the aggregated features y_hat_all, the merging unit merges the K patch reconstructed image blocks x_hat_p using the feature merging manner with overlapping parts. For example, the merging unit can determine the actual patch size and the overlapping size of each patch reconstructed image block x_hat_p. The merging unit can sort the K patch reconstructed image blocks x_hat_p corresponding to the K patch features y_hat_p in the order from top to bottom and from left to right, and splice the K patch reconstructed image blocks x_hat_p after sorting in turn based on the actual patch size and the overlapping size to obtain a reconstructed image block x_hat corresponding to the current image block. The reconstructed image block can include K patch reconstructed image blocks, the size of the non-overlapping part of a patch reconstructed image block can be the actual patch size of the patch reconstructed image block, and the size of the overlapping part between the patch reconstructed image block and the adjacent patch reconstructed image block can be the overlapping size of the patch reconstructed image block. The splicing process can be the inverse process of Figure 9E , which will not be described herein.

[0236] For example, the actual tile size and the overlap size can be read from the bitstream or determined based on the feature size of the current tile to be partitioned, without limitation. Assuming that the actual tile size is x_c*xtile_w*xtile_h, the overlap width in the overlap size is xpadding_w, and the overlap height in the overlap size is xpadding_h, the plurality of tile reconstructed image blocks x_hat_p are sequentially spliced in the receiving order of the plurality of tile reconstructed image blocks x_hat_p.

[0237] For the overlap part of the left and right two tile reconstructed image blocks, the value of the overlap part is the value of the left tile reconstructed image block, and the value of the right tile reconstructed image block is discarded, that is, the value of the earliest (left) tile reconstructed image block is used, and the value of the overlap part of the subsequent sub-block is directly discarded. Alternatively, the value of the overlap part is the average value of the two tile reconstructed image blocks.

[0238] For the overlap part of the upper and lower two tile reconstructed image blocks, the value of the overlap part is the value of the upper tile reconstructed image block, and the value of the lower tile reconstructed image block is discarded, that is, the value of the earliest (upper) tile reconstructed image block is used, and the value of the overlap part of the subsequent sub-block is directly discarded. Alternatively, the value of the overlap part is the average value of the two tile reconstructed image blocks.

[0239] Embodiment 10: For the encoding end and the decoding end, the reconstructed image block corresponding to the current image block needs to be determined based on the reconstructed features of each stage sub-block. For example, the clustering and partitioning unit can perform feature aggregation on the reconstructed features (y_hat_1 to y_hat_N) of the N stage sub-blocks to obtain the aggregated features (which can also be referred to as full-size aggregated features y_hat_all). The clustering and partitioning unit inputs the aggregated features y_hat_all to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block corresponding to the aggregated features y_hat_all, which is the reconstructed image block x_hat corresponding to the current image block. Compared with Embodiment 9, the aggregated features y_hat_all are not partitioned, and the plurality of tile reconstructed image blocks x_hat_p are not merged.

[0240] First, the clustering and partitioning unit aggregates the reconstructed features of the N stage sub-blocks to obtain the aggregated features.

[0241] For example, the reconstruction features (y_hat_1 to y_hat_N) of the N stage sub-blocks can be aggregated according to the phase to obtain the aggregated feature y_hat_all. Or, the reconstruction features (y_hat_1 to y_hat_N) of the N stage sub-blocks can be aggregated according to the phase to obtain a plurality of phase aggregated features, and the plurality of phase aggregated features are spliced according to the channel to obtain the aggregated feature y_hat_all. For the feature aggregation, please refer to Embodiment 9, which will not be repeated here.

[0242] Secondly, the clustering and blocking unit inputs the aggregated feature y_hat_all to the synthesis transformation network, and performs synthesis transformation on the aggregated feature y_hat_all through the synthesis transformation network. The synthesis transformation manner is not limited, and the synthesis transformation network outputs the reconstructed image block corresponding to the aggregated feature y_hat_all, that is, the reconstructed image block x_hat corresponding to the current image block.

[0243] Embodiment 11: For the encoding end and the decoding end, the reconstructed image block corresponding to the current image block needs to be determined based on the reconstruction feature of each stage sub-block. For example, after the clustering and blocking unit obtains the reconstruction features (y_hat_1 to y_hat_N) of the N stage sub-blocks, the reconstruction features are not aggregated, nor are the aggregated features blocked, but each stage sub-block reconstruction feature is directly input to the synthesis transformation network, and the synthesis transformation network outputs the block reconstruction image block x_hat_p corresponding to each stage sub-block. The merging unit can obtain a plurality of block reconstruction image blocks x_hat_p corresponding to the stage sub-blocks, and merge the block reconstruction image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.

[0244] First, after the clustering and blocking unit obtains the reconstruction features of the N stage sub-blocks, the reconstruction features of each stage sub-block are input to the synthesis transformation network, and the synthesis transformation network performs synthesis transformation on the reconstruction features. The synthesis transformation manner is not limited, and the synthesis transformation network outputs the block reconstruction image block x_hat_p corresponding to each stage sub-block.

[0245] Secondly, the merging unit merges a plurality of block reconstruction image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0246] For example, the merging unit can obtain N block reconstruction image blocks x_hat_p corresponding to the N stage sub-blocks, and merge the N block reconstruction image blocks x_hat_p to obtain the reconstructed image block x_hat.

[0247] For example, the merging unit merges the N sub-reconstructed image blocks x_hat_p in a manner of feature merging without overlapping part. For example, the N sub-reconstructed image blocks x_hat_p are sorted in an order from top to bottom and from left to right, and then the N sub-reconstructed image blocks x_hat_p are spliced in sequence to obtain the reconstructed image block x_hat corresponding to the current image block. Alternatively, the merging unit merges the N sub-reconstructed image blocks x_hat_p in a manner of feature merging with overlapping part. For example, the actual sub-block size and the overlapping size of each sub-reconstructed image block x_hat_p are determined, the N sub-reconstructed image blocks x_hat_p are sorted in an order from top to bottom and from left to right, and then the N sub-reconstructed image blocks x_hat_p are spliced in sequence based on the actual sub-block size and the overlapping size to obtain the reconstructed image block x_hat corresponding to the current image block. The reconstructed image block includes N sub-reconstructed image blocks, the size of the non-overlapping part of a sub-reconstructed image block is the actual sub-block size of the sub-reconstructed image block, and the size of the overlapping part between the sub-reconstructed image block and the adjacent sub-reconstructed image block is the overlapping size of the sub-reconstructed image block. For the overlapping part between the left and right sub-reconstructed image blocks, the value of the overlapping part is the value of the left sub-reconstructed image block, and the value of the right sub-reconstructed image block is discarded. Alternatively, the value of the overlapping part is the average value of the two sub-reconstructed image blocks. For the overlapping part between the top and bottom sub-reconstructed image blocks, the value of the overlapping part is the value of the top sub-reconstructed image block, and the value of the bottom sub-reconstructed image block is discarded. Alternatively, the value of the overlapping part is the average value of the two sub-reconstructed image blocks.

[0248] In Embodiment 12, in Embodiments 1-11, the encoding end can send the code rate control parameters for the synthesis transformation network to the decoding end in order to perform accurate code rate control. For example, the encoding end can encode the code rate control parameters in the auxiliary code stream corresponding to the current image block, the decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the code rate control parameters, and determine the synthesis transformation network based on the code rate control parameters, that is, determine the synthesis transformation network based on the code stream information, especially the code rate related information. The synthesis transformation network can be composed of at least one 2 times deconvolution layer (such as 2 times deconvolution layer of 2x2, 3x3, 4x4 or 5x5, such as 2 times deconvolution layer of 4x4, of course, other multiple deconvolution layers can also be selected), at least one activation layer (such as relu layer, or leaky relu layer, or sigmoid layer, or tanh layer, or Gelu layer, such as relu layer can be selected), at least one convolution layer (such as 1x1, 3x3 or 5x5 convolution layer, such as 3x3 convolution layer can be selected, for the convolution layer in the synthesis transformation network, it can be a normal 2D convolution or a grouped convolution, such as 2 times grouped convolution), the output feature space resolution of the synthesis transformation network is a positive integer multiple of the input feature space resolution, such as 16 times (16 times in length and width), and the output feature channel number is 1 or 3. Based on the synthesis transformation network, the code rate control parameters can be used to control the synthesis transformation network, that is, the code rate control parameters are used as the input of the synthesis transformation network.

[0249] In a possible implementation, for Embodiment 9, the clustering and blocking unit inputs K block features y_hat_p to the synthesis transformation network, therefore, the encoding end encodes K code rate control parameters corresponding to K block features y_hat_p in the auxiliary code stream corresponding to the current image block, and the source of the K code rate control parameters is not limited. When the encoding end uses the K code rate control parameters to control the analysis transformation network and the synthesis transformation network (that is, the K code rate control parameters are used as the input of the analysis transformation network and the synthesis transformation network, such as the code rate control parameters corresponding to the block feature y_hat_p and the block feature y_hat_p are input to the synthesis transformation network as the input feature), the code rate corresponding to the code stream of the current image block can be the target code rate or close to the target code rate (that is, the difference between the two is less than a threshold). The decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the K code rate control parameters corresponding to the K block features y_hat_p.

[0250] Based on this, the clustering and blocking unit can obtain K block features y_hat_p and K rate control parameters corresponding to the K block features y_hat_p, for each block feature y_hat_p, input the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p into the synthesis transformation network, and generate the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p by the synthesis transformation network.

[0251] For example, the synthesis transformation network generates the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p, which can include but is not limited to: processing the block feature y_hat_p by the synthesis transformation network to obtain a first feature; processing the rate control parameter corresponding to the block feature y_hat_p by the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the third feature.

[0252] In a possible implementation, for example 10, the clustering and blocking unit inputs the aggregated feature y_hat_all into the synthesis transformation network, therefore, the encoding end encodes a rate control parameter corresponding to the aggregated feature y_hat_all in the auxiliary code stream corresponding to the current image block, that is, the rate control parameter corresponding to the current image block, and the source of the rate control parameter is not limited. When the encoding end uses the rate control parameter to control the analysis transformation network and the synthesis transformation network (that is, the rate control parameter is input into the analysis transformation network and the synthesis transformation network as an input feature, such as the aggregated feature y_hat_all and the rate control parameter are input into the synthesis transformation network, and the current image block and the rate control parameter are input into the analysis transformation network), the code stream corresponding to the current image block can be at the target code rate or close to the target code rate. The decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameter corresponding to the current image block.

[0253] Based on this, the clustering and blocking unit can obtain the aggregated feature y_hat_all and the rate control parameter, and input the aggregated feature y_hat_all and the rate control parameter into the synthesis transformation network, and generate the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the rate control parameter by the synthesis transformation network.

[0254] The synthesis transformation network generates the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the rate control parameter, which can include: processing the aggregated feature y_hat_all by the synthesis transformation network to obtain a first feature; processing the rate control parameter by the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the third feature.

[0255] In a possible implementation, for example 11, the clustering and partition unit inputs the reconstructed features of the N-stage subblocks to the synthesis transformation network, and thus the encoding end encodes N rate control parameters corresponding to the N-stage subblocks in the auxiliary code stream corresponding to the current image block. The source of the N rate control parameters is not limited. When the encoding end uses the N rate control parameters to control the analysis transformation network and the synthesis transformation network (i.e., the N rate control parameters are used as inputs of the analysis transformation network and the synthesis transformation network, such as the reconstructed features of the stage subblock and the rate control parameter corresponding to the stage subblock are input to the synthesis transformation network), the code stream corresponding to the current image block can have a target code rate or a code rate close to the target code rate. The decoding end decodes the auxiliary code stream corresponding to the current image block to obtain N rate control parameters corresponding to the N-stage subblocks.

[0256] Based on this, the clustering and partition unit can obtain the reconstructed features of the N-stage subblocks and the N rate control parameters corresponding to the N-stage subblocks. For each stage subblock, the reconstructed feature of the stage subblock and the rate control parameter corresponding to the stage subblock are input to the synthesis transformation network together, and the synthesis transformation network generates the partitioned reconstructed image block x_hat_p corresponding to the stage subblock based on the reconstructed feature of the stage subblock and the rate control parameter corresponding to the stage subblock.

[0257] For example, the synthesis transformation network generates the partitioned reconstructed image block x_hat_p corresponding to the stage subblock based on the reconstructed feature of the stage subblock and the rate control parameter corresponding to the stage subblock, which can include but is not limited to: processing the reconstructed feature of the stage subblock by the synthesis transformation network to obtain a first feature; processing the rate control parameter corresponding to the stage subblock by the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the partitioned reconstructed image block x_hat_p corresponding to the stage subblock based on the third feature.

[0258] In the embodiment 13, the synthesis transformation network can be referred to as a synthesis transformation network controlled by a rate control parameter, and the rate control parameter can be denoted as a parameter λ. The parameter λ is decoded from the auxiliary code stream corresponding to the current image block, as shown inFigure 10A The diagram shows a schematic of a synthesis transform network controlled by the λ parameter. This synthesis transform network is a variable code rate synthesis transform network, and based on this network, output features with 1 or 3 channels can be obtained. Of course, Figure 10A This is just one example of a synthetic transform network, and there are no restrictions on the structure of this synthetic transform network.

[0259] See Figure 10A As shown, the λ-parameter-controlled synthetic transform network can include at least one Conv (convolutional layer) and at least one λ-RSTB (Residual Swin Transformer Block). Figure 10A Taking four convolutional layers and four λ-RSTBs as an example, the number of convolutional layers and λ-RSTBs can be more or less. Convolutional layers can be 1x1, 3x3, or 5x5; for example, a 3x3 convolutional layer can be chosen. The convolutional layers in the synthetic transform network can be ordinary 2D convolutions or grouped convolutions, such as 2x grouped convolutions. There are no restrictions on the structure of these convolutional layers. λ-RSTB refers to an RSTB network layer controlled by the λ parameter. For the structure of an RSTB network layer controlled by the λ parameter, see [link to documentation]. Figure 10B As shown.

[0260] See Figure 10B As shown, λ-RSTB can include, but is not limited to, network layers such as FE (Feature Embedding), λ-STB (Swin Transformer Block), and FU (Feature Un-embedding). Of course, this is just an example, and the structure of this λ-RSTB is not limited. FE is the feature embedding layer, used to implement feature embedding processing, and FU is the feature un-embedding layer, used to implement feature un-embedding processing.

[0261] λ-STB represents an STB network layer controlled by the λ parameter. The structure of an STB network layer controlled by the λ parameter can be found in [link to documentation]. Figure 10BAs shown, the λ-STB can include but is not limited to a LN (liner), a λ-WA (Weight Averaging), a λ-SWA (Stochastic Weight Averaging), an MLP (Multilayer Perceptron), etc., of course, this is just an example, and the structure of the λ-STB is not limited here. The LN is a linear layer, and the λ-STB can include at least one LN, and the number of LNs is not limited here, and four LNs are taken as an example here. The MLP is a multilayer perceptron, and the λ-STB can include at least one MLP, and the number of MLPs is not limited here, and two MLPs are taken as an example here.

[0262] The structure of the λ-SWA is the same as that of the λ-WA, and the λ-WA is taken as an example here, and the λ-WA represents a WA network layer controlled by a λ parameter. The structure of the WA network layer controlled by the λ parameter can be referred to Figure 10C As shown, the λ-WA can include but is not limited to a FC (FullConnection) layer, a MatMul (matrix multiplication), a Scale (scaling), a SoftMax (logistic regression, also known as normalization), a MatMul, etc., of course, this is just an example, and the structure of the λ-WA is not limited here.

[0263] The parameter λ can be mapped into a scaling coefficient through the FC layer, that is, the rate control parameter is processed through the FC layer of the synthetic transformation network to obtain a second feature, and the second feature can be a scaling coefficient.

[0264] The input feature (such as the block feature y_hat_p, or the aggregated feature y_hat_all, or the reconstructed feature of the stage sub-block) can be processed through the transformer network to obtain Q features, K features, and V features. The Q features, K features, and V features all originate from the input feature itself and are characteristic vectors generated by multiplying the input feature with corresponding weights, and the process is not limited here, that is, the synthetic transformation network can include a transformer network, and the input feature (such as the block feature, or the aggregated feature, or the reconstructed feature) is processed through the transformer network of the synthetic transformation network to obtain a first feature.

[0265] After obtaining the Q feature, the K feature, and the V feature, a matrix multiplication operation, a scaling operation, and a logistic regression operation can be sequentially performed based on the Q feature and the K feature to obtain an intermediate feature 1. A scaling coefficient (mapped by the FC layer from the parameter λ) can be multiplied with the V feature to obtain an intermediate feature 2. The intermediate feature 1 and the intermediate feature 2 are matrix multiplied to obtain the output feature of the λ-WA. In summary, the scaling coefficient is a second feature, and the Q feature, the K feature, and the V feature are first features. Therefore, a third feature, which is the output feature of the λ-WA, can be generated based on the first features and the second feature.

[0266] After obtaining the output feature of the λ-WA, the output feature of the λ-WA can be combined with Figure 10A and Figure 10B to determine the output feature of the synthesis transformation network based on the output feature of the λ-WA (i.e., the third feature), such as the patch reconstruction image block x_hat_p corresponding to the patch feature y_hat_p, the reconstruction image block x_hat corresponding to the aggregated feature y_hat_all, and the patch reconstruction image block x_hat_p corresponding to the stage sub-block.

[0267] Referring to the λ-WA shown in Figure 10C , the λ-WA can be calculated as shown below. In this way, the output of the image encoder-decoder can be controlled by using the λ parameter, so as to obtain code streams of different sizes and reconstructed images of different qualities.

[0268]

[0269] In embodiments 1-13, a convolutional layer can be involved, which can be a normal 2D convolution or a grouped convolution, such as a 2-grouped convolution. For example, the steps of the normal 2D convolution can be seen in Figure 11A , where the input feature is (H x W x C), and then C' filters (each filter has a size of (h x w x C), and the C of the size of the filter is the same as the C of the input feature) are applied, and the input layer is converted into an output feature with a size of (H' x W' x C').

[0270] The steps of the grouped convolution can be seen in Figure 11BAs shown, for each filters group, there can be C' / g (e.g. g is 2) filters, each of which has half the number of channels of a normal 2D convolution filter. Each filters group operates on 1 / g of the original WxHxC corresponding number of channels, i.e. WxHxC / g. In this way, each filters group outputs a feature with C' / g channels. Finally, the channels can be stacked to obtain the final C' channels, thereby achieving the same effect as the normal 2D convolution described above. Obviously, by grouping the convolution, the parameter quantity and the calculation quantity can be effectively reduced.

[0271] In embodiments 1-14, in order to perform accurate rate control, the encoding end needs to send a rate control parameter to the decoding end, such as a rate control parameter carried in the auxiliary code stream corresponding to the current image block, and the rate control parameter affects the analysis transform network and the synthesis transform network at the encoding end. In order to make the code rate of the code stream of the current image block be the target code rate or close to the target code rate (pre-configured code rate), the encoding end can try different rate control parameters to find the target rate control parameter, under which the code rate of the code stream of the current image block is the target code rate or close to the target code rate, and the encoding end encodes the target rate control parameter in the auxiliary code stream corresponding to the current image block.

[0272] If the current image block corresponds to only one rate control parameter, such as the aggregated feature y_hat_all corresponding to one rate control parameter, each rate control parameter can be tried in turn from the multiple rate control parameters until the code rate of the code stream of the current image block is the target code rate or close to the target code rate, thereby obtaining the target rate control parameter.

[0273] If the current image block corresponds to multiple rate control parameters, such as K block features corresponding to K rate control parameters, N stage sub-blocks corresponding to N rate control parameters, the initial values of the multiple rate control parameters are first set, if the code rate of the code stream of the current image block does not meet the requirements, a candidate rate control parameter is selected from the multiple rate control parameters, the value of the candidate rate control parameter is reduced, if the code rate of the code stream of the current image block does not meet the requirements, a candidate rate control parameter (the same as or different from the last candidate rate control parameter) is continuously selected from the multiple rate control parameters, the value of the candidate rate control parameter is reduced, and so on, until the code rate of the code stream of the current image block meets the requirements (i.e. the code rate of the code stream of the current image block is the target code rate or close to the target code rate), thereby obtaining the target values of the multiple rate control parameters.

[0274] In selecting a candidate rate control parameter from the plurality of rate control parameters, one rate control parameter can be randomly selected as the candidate rate control parameter, or the largest rate control parameter can be selected as the candidate rate control parameter, or the candidate rate control parameter can be selected based on the rate-distortion relationship. For example, when the rate control parameter A is reduced from the current level to the next level, the distortion is A1, when the rate control parameter B is reduced from the current level to the next level, the distortion is B1, if the distortion A1 is less than the distortion B1, i.e., the distortion A1 is the minimum distortion, the rate control parameter A is selected as the candidate rate control parameter.

[0275] In one possible implementation, how the encoder sets λ for each (feature or image) block i One implementation of the parameter is as follows: for each block x i ′ (like a block feature or a stage sub-block), a different λ i is used for encoding. Before encoding, first, a number of discrete λs are taken for each block, and these discrete λs are defined as quality levels, and the rate and distortion of each block, each quality level are counted before encoding. For each frame image, this process only needs to be performed once before the first encoding.

[0276] In the encoding process, λ in the quality level is selected for each block to encode. In the encoding process, first, all blocks are encoded using the same quality level, which is the minimum quality level that can make the total rate greater than the target rate, i.e., satisfies the following formula, R t is the target rate, represents the sum of the rates of all blocks under the quality level q.

[0277]

[0278] Subsequently, the quality level of each block is gradually reduced until the total rate is less than the target rate. For each block, the gradient of the current quality level and the previous quality level on the rate-distortion curve can be calculated, i.e., the gradient is calculated using the following formula:

[0279]

[0280] wherein, is the reconstruction loss of block i under the quality level q, when a loss function that decreases with rate (such as mean square error) is used as the metric, the block i with the smallest is selected (i.e., the candidate rate control parameter with the minimum distortion), and it is lowered by one quality level. In this way, until the total rate is less than the target rate.

[0281] See Figure 12As shown, it is a decision diagram of the parameter λ, 7 quality levels (i.e. 7 values of the parameter λ) are designed on the rate-distortion curve, at quality level 3, the quality level of block 2 can be calculated That is, at quality level 3, reducing the quality level of block 2 will lead to greater loss of coding performance, and therefore, the quality level of block 1 is reduced.

[0282] As can be seen from the technical solutions of the above embodiments, in this embodiment, for the neural network-based coding and decoding technology, a coding and decoding scheme with variable and adjustable code rate is proposed, the parallelism is improved, the feature storage cache is effectively saved, higher code rate control accuracy is achieved, the coding performance loss is smaller, and better coding performance and code rate control accuracy are achieved. By using block-based coding and decoding, the peak memory occupancy is lower, the decoding time of a single block is smaller, and high-speed parallel decoding capability is achieved, so that the neural network can maintain low complexity while effectively ensuring the quality of the reconstructed image block, improving the coding and decoding performance, and reducing the complexity. For example, the code rate control algorithm of the present scheme can achieve a code rate control accuracy of about 99%, and compared with the code rate control, the coding performance loss is only about 1%. Taking decoding an image of 2k resolution in a block size of 256*256 as an example, by using block-based coding and decoding, the peak memory occupancy is reduced to 1 / 10 of that of the whole image decoding, and the decoding time of a single block is less than 0.1s, and high-speed parallel decoding capability is achieved. With less increase in the number of parameters, better coding and decoding performance is achieved.

[0283] In an embodiment of the present application, a decoding method is provided, which can be applied to a decoding end (also referred to as a video decoder), and can include: decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block; determining a probability distribution parameter based on the coefficient hyperparameter feature, decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the current image block, and determining a reconstructed feature of the current image block based on the residual feature; decoding an auxiliary code stream corresponding to the current image block to obtain a code rate control parameter corresponding to the current image block, and inputting the reconstructed feature and the code rate control parameter into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.

[0284] For example, inputting the reconstructed feature and the code rate control parameter into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block can include but is not limited to: processing the reconstructed feature through the synthesis transformation network to obtain a first feature; processing the code rate control parameter through the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0285] For example, based on the embodiments 3 and 4, the synthesis transformation network can be adjusted, and only the adjustment process of the synthesis transformation network is described in the embodiment 16, and the other processes can be referred to the embodiments 3 and 4.

[0286] For example, the accurate rate control can be performed by the rate control parameter, and the rate control parameter affects the analysis transformation network and the synthesis transformation network. For example, when the current image block is analyzed and transformed by the analysis transformation network at the encoding end, the rate control parameter and the current image block are input into the analysis transformation network, and the analysis transformation network outputs the image features corresponding to the current image block. When the image features y_hat are input into the synthesis transformation network at the encoding end or the decoding end, the image features y_hat and the rate control parameter are input into the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block corresponding to the current image block.

[0287] Based on the configured target rate, the encoding end can try to use different rate control parameters. After using a certain rate control parameter, if the rate corresponding to the code stream of the current block is the target rate or close to the target rate, this rate control parameter is the finally used rate control parameter, and the encoding end can send the rate control parameter to the decoding end.

[0288] For example, the encoding end can send the rate control parameter for the synthesis transformation network to the decoding end. For example, the encoding end can encode the rate control parameter corresponding to the current image block in the auxiliary code stream corresponding to the current image block, the decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameter corresponding to the current image block, and determine the synthesis transformation network based on the rate control parameter. The synthesis transformation network can be composed of at least one 2 times deconvolution layer (such as 2 times deconvolution layer of 2x2, 3x3, 4x4 or 5x5, such as 4x4 2 times deconvolution layer, of course, other multiple deconvolution layers can also be selected), at least one activation layer (such as relu layer, or leaky relu layer, or sigmoid layer, or tanh layer, or Gelu layer, such as relu layer can be selected), at least one convolution layer (such as 1x1, 3x3 or 5x5 convolution layer, such as 3x3 convolution layer can be selected, the convolution layer in the synthesis transformation network can be ordinary 2D convolution or grouped convolution, such as 2 times grouped convolution), the output feature space resolution of the synthesis transformation network is a positive integer multiple of the input feature space resolution, such as 16 times (16 times long and wide), and the output feature channel number is 1 or 3. Based on the synthesis transformation network, the synthesis transformation network can be controlled based on the rate control parameter, that is, the rate control parameter is input into the synthesis transformation network.

[0289] For example, the encoding end encodes the code rate control parameter corresponding to the current image block in the auxiliary code stream corresponding to the current image block, and the source of the code rate control parameter is not limited. When the encoding end uses the code rate control parameter to control the analysis transform network and the synthesis transform network (i.e., the code rate control parameter is used as the input of the analysis transform network and the synthesis transform network), the code stream corresponding to the current image block can have a code rate close to the target code rate. The decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the code rate control parameter corresponding to the current image block. Based on this, the decoding end can input the image feature y_hat and the code rate control parameter into the synthesis transform network, and the synthesis transform network generates the reconstructed image block corresponding to the current image block based on the image feature y_hat and the code rate control parameter, and outputs the reconstructed image block.

[0290] In the method, the synthesis transform network generates the reconstructed image block corresponding to the current image block based on the image feature y_hat and the code rate control parameter, which can include: processing the image feature y_hat (i.e., the reconstructed feature y_hat) through the synthesis transform network to obtain a first feature; processing the code rate control parameter through the synthesis transform network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.

[0291] For example, referring to Figure 10A 、 Figure 10B and Figure 10CAs shown, the synthetic transformation network can include at least one Conv (convolution layer) and at least one λ-RSTB, the λ-RSTB can include but is not limited to FE, λ-STB, FU, etc. network layer, the λ-STB can include but is not limited to LN, λ-WA, λ-SWA, MLP, etc., the structure of λ-SWA is the same as that of λ-WA, the λ-WA can include but is not limited to FC layer, MatMul, Scale, SoftMax, MatMul, etc., on this basis, the parameter λ (i.e. rate control parameter) can be mapped into a scaling coefficient through the FC layer, that is, the rate control parameter is processed through the FC layer of the synthetic transformation network to obtain the second feature. The image feature y_hat can be processed through the transformer network to obtain the Q feature, K feature and V feature, that is, the image feature y_hat is processed through the transformer network of the synthetic transformation network to obtain the first feature. After obtaining the Q feature, K feature and V feature, the matrix multiplication operation, scaling operation and logistic regression operation can be sequentially performed based on the Q feature and K feature to obtain the intermediate feature 1; the scaling coefficient can be multiplied with the V feature to obtain the intermediate feature 2; the intermediate feature 1 and the intermediate feature 2 are matrix multiplied to obtain the output feature of the λ-WA, that is, the third feature is generated based on the first feature and the second feature, and the third feature is the output feature of the λ-WA. After obtaining the output feature of the λ-WA, the output feature of the synthetic transformation network can be determined based on the output feature of the λ-WA (i.e. the third feature), such as the output feature can be the reconstructed image block corresponding to the current image block. Figure 10A and Figure 10B Based on the output feature of the λ-WA (i.e. the third feature), the output feature of the synthetic transformation network can be determined, such as the output feature can be the reconstructed image block corresponding to the current image block.

[0292] For example, each of the above embodiments can be implemented alone or in combination, for example, each of embodiments 1-16 can be implemented alone, and at least two of embodiments 1-16 can be implemented in combination.

[0293] For example, in each of the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0294] For example, in each of the above embodiments, the content of different embodiments can be borrowed from each other, such as the content of embodiment 3 can be applied to embodiment 16, the content of embodiment 4 can be applied to embodiment 16, the content of embodiments 5-15 can be applied to embodiment 16, and the content of embodiments 5-15 can be borrowed from each other, which is not limited.

[0295] Based on the same application concept as the above method, the embodiment of the present application also proposes a decoding device, which is applied to a decoding end, and the device comprises: a memory configured to store video data; and a decoder configured to implement the decoding method in the above embodiments 1-16, i.e., the processing flow of the decoding end.

[0296] For example, in a possible implementation, the decoder is configured to implement:

[0297] decode the first code stream of the current image block to obtain the coefficient hyperparameter feature of each stage sub-block of the current image block;

[0298] For each stage sub-block, determine the probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decode the second code stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block;

[0299] determine the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean value feature of the stage sub-block;

[0300] determine the reconstructed image block corresponding to the current image block based on the reconstruction feature of each stage sub-block.

[0301] Based on the same application concept as the above method, the embodiment of the present application also proposes a decoding device, which is applied to a decoding end, and the device comprises: a memory configured to store video data; and a decoder configured to implement the decoding method in the above embodiments 1-16, i.e., the processing flow of the decoding end.

[0302] For example, in a possible implementation, the decoder is configured to implement:

[0303] input the current image block into an analysis transformation network to obtain a feature block corresponding to the current image block;

[0304] split the feature block into to-be-encoded features of multiple stage sub-blocks;

[0305] For each stage sub-block corresponding to the current image block, obtain the coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into the first code stream of the current image block;

[0306] determine the residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean value feature of the stage sub-block;

[0307] determine the probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into the second code stream of the current image block based on the probability distribution parameter.

[0308] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided by the embodiments of the present application can be specifically referred to from the hardware level as shown in the hardware architecture diagram of the decoding end device Figure 13A The decoding end device includes a processor 1301 and a machine readable storage medium 1302, and the machine readable storage medium 1302 stores machine executable instructions that can be executed by the processor 1301. The processor 1301 is configured to execute the machine executable instructions to implement the decoding method of the above embodiments 1-16 of the present application. For example, in a possible implementation, when the processor 1301 executes the machine executable instructions, the following steps are performed:

[0309] Decoding the first code stream of the current image block to obtain the coefficient hyperparameter feature of each stage sub-block of the current image block;

[0310] For each stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decoding the second code stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block;

[0311] Determining the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block;

[0312] Determining the reconstructed image block corresponding to the current image block based on the reconstruction feature of each stage sub-block.

[0313] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided by the embodiments of the present application can be specifically referred to from the hardware level as shown in the hardware architecture diagram of the decoding end device Figure 13B The decoding end device includes a processor 1301 and a machine readable storage medium 1302, and the machine readable storage medium 1302 stores machine executable instructions that can be executed by the processor 1301. The processor 1301 is configured to execute the machine executable instructions to implement the decoding method of the above embodiments 1-16 of the present application. For example, in a possible implementation, when the processor 1301 executes the machine executable instructions, the following steps are performed:

[0314] Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0315] Splitting the feature block into to-be-encoded features of multiple stage sub-blocks;

[0316] For each stage sub-block corresponding to the current image block, obtaining the coefficient hyperparameter feature of the stage sub-block, and encoding the coefficient hyperparameter feature of the stage sub-block into the first code stream of the current image block;

[0317] determine the residual feature of the stage sub-block based on the mean feature of the stage sub-block and the to-be-encoded feature of the stage sub-block;

[0318] determine the probability distribution parameter based on the coefficient hyper-parameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into the second code stream of the current image block based on the probability distribution parameter.

[0319] Based on the same application concept as the above method, the embodiments of the present application provide an electronic device. It includes a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions that can be executed by the processor; the processor is used to execute the machine executable instructions to realize the decoding method or the encoding method of the above embodiments 1-16 of the present application.

[0320] Based on the same application concept as the above method, the embodiments of the present application also provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by the processor, and the decoding method or the encoding method disclosed in the above examples of the present application can be realized.

[0321] Based on the same application concept as the above method, the embodiments of the present application also provide a computer application program, which can realize the decoding method or the encoding method disclosed in the above examples of the present application when executed by the processor.

[0322] Based on the same application concept as the above method, the embodiments of the present application also provide a decoding device, which can be applied to a decoding end, and the decoding device includes: a decoding module, which is used to decode the first code stream of the current image block to obtain the coefficient hyper-parameter feature of each stage sub-block of the current image block; for each stage sub-block, the probability distribution parameter is determined based on the coefficient hyper-parameter feature of the stage sub-block, and the second code stream of the current image block is decoded based on the probability distribution parameter to obtain the residual feature of the stage sub-block; a determination module, which is used to determine the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block; and the reconstruction image block corresponding to the current image block is determined based on the reconstruction feature of each stage sub-block.

[0323] The determination module is further configured to obtain, for the first stage sub-block, a mean feature of the first stage sub-block based on the coefficient hyperparameter feature of the first stage sub-block and a default reference feature configured in advance, or obtain, for the i-th stage sub-block, i > 1, a reference feature of the i-th stage sub-block based on reconstructed features of i-1 stage sub-blocks in front of the i-th stage sub-block, and obtain the mean feature of the i-th stage sub-block based on the coefficient hyperparameter feature of the i-th stage sub-block and the reference feature; wherein the reference feature comprises all of the reconstructed features of the i-1 stage sub-blocks in front, or part of the reconstructed features of the i-1 stage sub-blocks in front, or the reconstructed feature of the i-1 stage sub-block.

[0324] The determination module is further configured to obtain, for the first stage sub-block, a mean feature of the first stage sub-block based on the coefficient hyperparameter feature of the first stage sub-block and a default reference feature configured in advance, or obtain, for the i-th stage sub-block, i > 1, a reference feature of the i-th stage sub-block based on reconstructed features of i-1 stage sub-blocks in front of the i-th stage sub-block, and obtain the mean feature of the i-th stage sub-block based on the coefficient hyperparameter feature of the i-th stage sub-block and the reference feature; wherein the reference feature comprises all of the reconstructed features of the i-1 stage sub-blocks in front, or part of the reconstructed features of the i-1 stage sub-blocks in front, or the reconstructed feature of the i-1 stage sub-block.

[0325] The determination module is further configured to obtain, for the first stage sub-block, a mean feature of the first stage sub-block based on the coefficient hyperparameter feature of the first stage sub-block and a default reference feature configured in advance, or obtain, for the i-th stage sub-block, i > 1, a reference feature of the i-th stage sub-block based on reconstructed features of i-1 stage sub-blocks in front of the i-th stage sub-block, and obtain the mean feature of the i-th stage sub-block based on the coefficient hyperparameter feature of the i-th stage sub-block and the reference feature; wherein the reference feature comprises all of the reconstructed features of the i-1 stage sub-blocks in front, or part of the reconstructed features of the i-1 stage sub-blocks in front, or the reconstructed feature of the i-1 stage sub-block.

[0326] Illustratively, the determining module is further configured to perform feature enhancement on the reconstructed features of the stage sub-blocks to obtain enhanced reconstructed features; and wherein the enhanced reconstructed features are used to determine the mean features of the stage sub-blocks, and the enhanced reconstructed features are used to determine the reconstructed image block.

[0327] Illustratively, when determining the reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block, the determining module is specifically configured to: perform feature aggregation on the reconstructed features of each stage sub-block to obtain aggregated features; input the aggregated features into a synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; or perform feature aggregation on the reconstructed features of each stage sub-block to obtain aggregated features; perform block division on the aggregated features to obtain a plurality of divided features; input each divided feature into the synthesis transformation network to obtain a divided reconstructed image block; and combine the divided reconstructed image blocks corresponding to the plurality of divided features to obtain the reconstructed image block corresponding to the current image block; or input the reconstructed features of each stage sub-block into the synthesis transformation network to obtain a divided reconstructed image block; and combine the divided reconstructed image blocks corresponding to all stage sub-blocks to obtain the reconstructed image block corresponding to the current image block.

[0328] Illustratively, when performing feature aggregation on the reconstructed features of each stage sub-block to obtain aggregated features, the determining module is specifically configured to: perform phase aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features; or perform phase aggregation on the reconstructed features of each stage sub-block to obtain a plurality of phase-aggregated features, and perform channel splicing on the plurality of phase-aggregated features to obtain the aggregated features.

[0329] Illustratively, when performing block division on the aggregated features to obtain a plurality of divided features, the determining module is specifically configured to: determine a target size of a divided feature; average the aggregated features into a plurality of divided features in a top-to-bottom and left-to-right order based on the target size, and the size of each divided feature is the target size; or determine an actual division size and an overlap size of a divided feature, the actual division size being a division size of a divided feature excluding an overlapping part, and the overlap size being a size of an overlapping part of adjacent divided blocks; and divide the aggregated features into a plurality of divided features based on the actual division size and the overlap size, and the size of each divided feature is the target size.

[0330] Illustratively, when the determining module merges the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features to obtain the reconstructed image block corresponding to the current image block, it is specifically configured to: sort the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features in a top-to-bottom and left-to-right order, and sequentially splice the sorted plurality of sub-block reconstructed image blocks to obtain the reconstructed image block corresponding to the current image block; or determine an actual sub-block size and an overlap size of each sub-block reconstructed image block, sort the plurality of sub-block reconstructed image blocks corresponding to the plurality of sub-block features in a top-to-bottom and left-to-right order, and based on the actual sub-block size and the overlap size, sequentially splice the sorted plurality of sub-block reconstructed image blocks to obtain the reconstructed image block corresponding to the current image block; wherein the size of a non-overlapping part of a sub-block reconstructed image block is the actual sub-block size of the sub-block reconstructed image block, and the size of an overlapping part of a sub-block reconstructed image block and an adjacent sub-block reconstructed image block is the overlap size of the sub-block reconstructed image block; for an overlapping part of two sub-block reconstructed image blocks on the left and right, the value of the overlapping part is the value of the left sub-block reconstructed image block, or the value of the overlapping part is the average of the two sub-block reconstructed image blocks; for an overlapping part of two sub-block reconstructed image blocks on the top and bottom, the value of the overlapping part is the value of the top sub-block reconstructed image block, or the value of the overlapping part is the average of the two sub-block reconstructed image blocks.

[0331] Illustratively, when the determining module inputs the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block, it is specifically configured to: decode the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of the current image block, and input the rate control parameter and the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; or when the determining module inputs each sub-block feature into the synthesis transformation network to obtain a sub-block reconstructed image block, it is specifically configured to: decode the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of each sub-block feature, and input the sub-block feature and the rate control parameter of the sub-block feature into the synthesis transformation network to obtain the sub-block reconstructed image block corresponding to the sub-block feature; or when the determining module inputs the reconstructed feature of each stage sub-block into the synthesis transformation network to obtain a sub-block reconstructed image block, it is specifically configured to: decode the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of each stage sub-block, and input the reconstructed feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthesis transformation network to obtain the sub-block reconstructed image block.

[0332] For example, when the determining module inputs the rate control parameter and the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block, the determining module specifically: processes the aggregated feature through the synthesis transformation network to obtain a first feature; processes the rate control parameter through the synthesis transformation network to obtain a second feature; generates a third feature based on the first feature and the second feature, and determines the reconstructed image block corresponding to the current image block based on the third feature; or when the determining module inputs the sub-block feature and the rate control parameter of the sub-block feature into the synthesis transformation network to obtain the sub-block reconstructed image block, the determining module specifically: processes the sub-block feature through the synthesis transformation network to obtain a first feature; processes the rate control parameter through the synthesis transformation network to obtain a second feature; generates a third feature based on the first feature and the second feature, and determines the sub-block reconstructed image block corresponding to the sub-block feature based on the third feature; or when the determining module inputs the reconstructed feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthesis transformation network to obtain the sub-block reconstructed image block, the determining module specifically: processes the reconstructed feature of the stage sub-block through the synthesis transformation network to obtain a first feature; processes the rate control parameter through the synthesis transformation network to obtain a second feature; generates a third feature based on the first feature and the second feature, and determines the sub-block reconstructed image block corresponding to the stage sub-block based on the third feature.

[0333] Based on the same application concept as the above method, an encoding device is also proposed in the embodiments of the present application. The device is applied to an encoding end, and the device comprises: an acquisition module, configured to input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block; and split the feature block into a plurality of stage sub-blocks to be encoded; an encoding module, configured to, for each stage sub-block corresponding to the current image block, acquire a coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into a first code stream of the current image block; and a determining module, configured to determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and a mean feature of the stage sub-block; and the encoding module is further configured to determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into a second code stream of the current image block based on the probability distribution parameter.

[0334] For example, the determining module is further configured to, for the first stage sub-block, obtain the mean feature of the first stage sub-block based on the coefficient hyper-parameter feature of the first stage sub-block through the mean prediction network, or obtain a configured default reference feature, and obtain the mean feature of the first stage sub-block based on the coefficient hyper-parameter feature of the first stage sub-block and the default reference feature through the mean prediction network; or, for the i-th stage sub-block, i is greater than 1, obtain the reference feature of the i-th stage sub-block based on the reconstructed features of the previous i-1 stage sub-blocks; obtain the mean feature of the i-th stage sub-block based on the coefficient hyper-parameter feature of the i-th stage sub-block and the reference feature through the mean prediction network; wherein the reference feature comprises all of the reconstructed features of the previous i-1 stage sub-blocks; or part of the reconstructed features of the previous i-1 stage sub-blocks; or the reconstructed feature of the i-1-th stage sub-block.

[0335] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. The present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. The embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

[0336] The present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. An image decoding method characterized by, The method comprises: decoding the code stream of the current image block to obtain the coefficient hyperparameter feature of each stage sub-block of the current image block; for any one stage sub-block, determining the probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, decoding the code stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block; determining the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block; decoding the code stream corresponding to the current image block to obtain the code rate control parameter of the current image block; and determining the reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the code rate control parameter.

2. The method of claim 1, wherein, The determination of the reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the code rate control parameter comprises: performing feature aggregation on the reconstruction features of the plurality of stage sub-blocks to obtain aggregated features; inputting the code rate control parameter and the aggregated features into a synthesis transformation network, processing the aggregated features through the synthesis transformation network to obtain first features, processing the code rate control parameter through the synthesis transformation network to obtain second features, and generating third features based on the first features and the second features; determining the reconstructed image block corresponding to the current image block based on the third features.

3. The method of claim 1, wherein, The determination of the reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the code rate control parameter comprises: performing feature aggregation on the reconstruction features of the plurality of stage sub-blocks to obtain aggregated features; and performing block division on the aggregated features to obtain a plurality of block features, wherein the code rate control parameter of the current image block comprises code rate control parameters of the plurality of block features; for any one block feature, inputting the block feature and the code rate control parameter of the block feature into a synthesis transformation network, processing the block feature through the synthesis transformation network to obtain first features, processing the code rate control parameter of the block feature through the synthesis transformation network to obtain second features, generating third features based on the first features and the second features, and determining a block reconstructed image block corresponding to the block feature based on the third features; merging the block reconstructed image blocks corresponding to the plurality of block features to obtain the reconstructed image block corresponding to the current image block.

4. The method of claim 1, wherein, The determination of the reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the code rate control parameter comprises: the code rate control parameter of the current image block comprises code rate control parameters of the plurality of stage sub-blocks, for any one stage sub-block, inputting the reconstruction feature of the stage sub-block and the code rate control parameter of the stage sub-block into a synthesis transformation network, processing the reconstruction feature of the stage sub-block through the synthesis transformation network to obtain first features, processing the code rate control parameter of the stage sub-block through the synthesis transformation network to obtain second features, generating third features based on the first features and the second features, and determining a block reconstructed image block corresponding to the stage sub-block based on the third features; Merge the sub-blocked reconstructed image blocks corresponding to all the stages to obtain a reconstructed image block corresponding to the current image block.

5. The method of claim 1, wherein, The determining of the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block comprises: For the first stage sub-block, the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block through a mean prediction network, or a default reference feature configured is obtained, and the mean feature of the stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature through the mean prediction network.

6. The method of claim 1, wherein, The determining of the reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block comprises: For the i-th stage sub-block, i is greater than 1, a reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the previous i-1 stage sub-blocks, the mean feature of the i-th stage sub-block is obtained based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through the mean prediction network, and for any one stage sub-block, the reconstruction feature of the stage sub-block is determined based on the residual feature of the stage sub-block and the mean feature of the stage sub-block.

7. The method of claim 6, wherein the reference feature comprises all the reconstruction features of the previous i-1 stage sub-blocks; or part of the reconstruction features of the previous i-1 stage sub-blocks; or the reconstruction feature of the i-1-th stage sub-block.

8. The method of claim 6, wherein, The mean prediction network comprises a second prediction network, and the obtaining of the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through the mean prediction network comprises: obtaining a second prediction feature corresponding to the reference feature through the second prediction network; determining the mean feature of the stage sub-block based on the second prediction feature and the coefficient hyperparameter feature of the stage sub-block.

9. An image coding method characterized by, The method comprises: inputting a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block; splitting the feature block into to-be-encoded features of multiple stage sub-blocks; for any one stage sub-block corresponding to the current image block, obtaining a coefficient hyperparameter feature of the stage sub-block and encoding the coefficient hyperparameter feature of the stage sub-block into a code stream of the current image block; for any one stage sub-block, determining a residual feature of the stage sub-block based on a to-be-encoded feature of the stage sub-block and a mean feature of the stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encoding the residual feature of the stage sub-block into the code stream of the current image block based on the probability distribution parameter; determining a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block, obtaining a code rate control parameter of the current image block, determining a reconstructed image block corresponding to the current image block based on the reconstruction features of the multiple stage sub-blocks and the code rate control parameter, and encoding the code rate control parameter in a code stream corresponding to the current image block.

10. An image decoding apparatus characterized by comprising: The apparatus comprises: The decoding module is configured to decode a code stream of the current image block to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block; for any one stage sub-block, determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and decode the code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; The determining module is configured to determine a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block; The decoding module is further configured to decode a code stream corresponding to the current image block to obtain a code rate control parameter of the current image block; The determining module is further configured to determine a reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the code rate control parameter.

11. An image coding apparatus characterized by comprising: The apparatus comprises: The obtaining module is configured to input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block, and split the feature block into to-be-encoded features of a plurality of stage sub-blocks; The encoding module is configured to, for any one stage sub-block corresponding to the current image block, obtain a coefficient hyperparameter feature of the stage sub-block, and encode the coefficient hyperparameter feature of the stage sub-block into a code stream of the current image block; The determining module is configured to, for any one stage sub-block, determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block; The encoding module is further configured to determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block, and encode the residual feature of the stage sub-block into the code stream of the current image block based on the probability distribution parameter; The determining module is further configured to determine a reconstruction feature of the stage sub-block based on the residual feature of the stage sub-block and the mean feature of the stage sub-block, obtain a code rate control parameter of the current image block, and determine a reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the code rate control parameter; The encoding module is further configured to encode the code rate control parameter into the code stream corresponding to the current image block.

12. An image decoding apparatus comprising: Comprise: A processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; The processor is used to execute the machine executable instructions to realize the method in any one of claims 1-8.

13. An image coding apparatus characterized by comprising: Comprise: A processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; The processor is used to execute the machine executable instructions to realize the method in claim 9.

14. A machine-readable storage medium, characterized in that, The machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by the processor to realize the method in any one of claims 1-8, or the computer instructions are executed by the processor to realize the method in claim 9.

15. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to realize the method in any one of claims 1-8, or the computer program is executed by the processor to realize the method in claim 9.

Images