Decoding method and apparatus, encoding method and apparatus, device and medium
Patent Information
- Application Number
- MYPI2025006372
- Authority / Receiving Office
- MY · MY
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-14
- Filing Date
- 2024-04-12
- Publication Date
- 2024-10-14
AI Technical Summary
The coding and decoding method based on neural networks has the problems of poor coding performance, poor decoding performance and high complexity, and it is difficult to effectively improve the coding performance and decoding performance of video images.
A coding and decoding scheme with a variable code rate is proposed. Through the block coding and decoding method, the parallelism is improved, the feature storage cache is saved, the code rate control accuracy is higher, the complexity is reduced, and the quality of the reconstructed image blocks is ensured.
It achieves higher encoding and decoding performance, reduces complexity, has high-speed parallel decoding capabilities, lower peak memory usage, shorter decoding time for a single block, code rate control accuracy reaches 99%, and encoding performance loss is only 1 %.
Smart Images

Figure PI2025006372_DRAWING_001 
Figure PI2025006372_DRAWING_002 
Figure PI2025006372_DRAWING_003
Abstract
Description
A decoding and encoding method, device, equipment and medium Technical Field
[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding and encoding method, apparatus, device and medium thereof. Background Art
[0002] To save space, video images are encoded before transmission. Complete video encoding involves prediction, transformation, quantization, entropy coding, and filtering. Prediction can include intra-frame prediction and inter-frame prediction. Inter-frame prediction leverages temporal correlations in the video to predict the current pixel using pixels from adjacent coded images, effectively removing temporal redundancy. Intra-frame prediction leverages spatial correlations in the video to predict the current pixel using pixels from coded blocks in the current frame, effectively removing spatial redundancy.
[0003] With the rapid development of deep learning, it has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning is also gradually being applied to codecs, enabling the use of neural networks to encode and decode images. Although neural network-based codecs demonstrate significant performance potential, they still suffer from issues such as poor encoding and decoding performance and high complexity.
[0004] Summary of the Invention
[0005] In view of this, the present application provides a decoding and encoding method, apparatus, device and medium thereof to improve encoding and decoding performance.
[0006] The present application provides a decoding method, applied to a decoding end, the method comprising:
[0007] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;
[0008] For each stage sub-block, determine probability distribution parameters based on coefficient hyperparameter features of the sub-block at that stage, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the sub-block at that stage;
[0009] Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage;
[0010] A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.
[0011] The present application provides an encoding method, applied to an encoding end, the method comprising:
[0012] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;
[0013] Splitting the feature block into features to be encoded of multiple stage sub-blocks;
[0014] For each stage sub-block corresponding to the current image block, obtain coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into a first bitstream of the current image block;
[0015] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;
[0016] Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.
[0017] The present application provides a decoding method, applied to a decoding end, the method comprising:
[0018] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block;
[0019] Determining probability distribution parameters based on the coefficient hyperparameter features, decoding the second code stream of the current image block based on the probability distribution parameters to obtain residual features of the current image block, and determining reconstruction features of the current image block based on the residual features;
[0020] The auxiliary code stream corresponding to the current image block is decoded to obtain a rate control parameter corresponding to the current image block, and the reconstruction feature and the rate control parameter are input into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.
[0021] The present application provides a decoding device, applied to a decoding end, comprising:
[0022] a decoding module configured to decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; determine probability distribution parameters for each stage sub-block based on the coefficient hyperparameter features of the stage sub-block, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the stage sub-block;
[0023] The determination module is used to determine the reconstruction features of the sub-blocks at this stage based on the residual features of the sub-blocks at this stage and the mean features of the sub-blocks at this stage; and determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage.
[0024] The present application provides an encoding device, applied to an encoding end, comprising:
[0025] An acquisition module is used to input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks;
[0026] an encoding module, configured to obtain, for each stage sub-block corresponding to the current image block, coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into a first bitstream of the current image block;
[0027] A determination module, configured to determine the residual features of the sub-block at this stage based on the to-be-encoded features of the sub-block at this stage and the mean features of the sub-block at this stage;
[0028] The encoding module is further configured to determine probability distribution parameters based on coefficient hyperparameter features of the sub-block at this stage, and encode the residual features of the sub-block at this stage into the second code stream of the current image block based on the probability distribution parameters.
[0029] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;
[0030] The processor is used to execute machine-executable instructions to implement the above decoding method.
[0031] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;
[0032] The processor is used to execute machine executable instructions to implement the above encoding method.
[0033] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method; or, the processor is used to execute the machine-executable instructions to implement the above-mentioned encoding method.
[0034] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned decoding method is implemented; or, the above-mentioned encoding method is implemented.
[0035] The present application provides a computer application, which, when executed by a processor, implements the above-mentioned decoding method; or, when executed by a processor, implements the above-mentioned encoding method.
[0036] As can be seen from the above technical solutions, in the embodiments of the present application, a coding and decoding scheme with variable and adjustable bit rates is proposed for neural network-based coding and decoding technology, which improves parallelism, effectively saves feature storage cache, achieves higher bit rate control accuracy, and has less coding performance loss, with better coding performance and bit rate control accuracy. By adopting block coding and decoding, the peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capabilities are provided, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve coding and decoding performance, and reduce complexity. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] FIG1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;
[0038] FIG2 is a flowchart of a decoding method in one embodiment of the present application;
[0039] FIG3 is a flow chart of an encoding method in one embodiment of the present application;
[0040] FIG4 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;
[0041] FIG5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;
[0042] FIG6 is a schematic diagram of a processing process of an encoding end in one embodiment of the present application;
[0043] FIG7 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;
[0044] 8A-8H are schematic diagrams of the structure of a mean prediction network in one embodiment of the present application;
[0045] 9A-9F are schematic diagrams of aggregation and segmentation in one embodiment of the present application;
[0046] 10A-10C are schematic diagrams of the structure of a synthesis transformation network in one embodiment of the present application;
[0047] 11A and 11B are schematic diagrams of processing of a convolutional layer in one embodiment of the present application;
[0048] FIG12 is a schematic diagram of a decision-making process for the λ parameter in one embodiment of the present application;
[0049] FIG13A is a hardware structure diagram of a decoding end device in one embodiment of the present application;
[0050] FIG13B is a hardware structure diagram of an encoding end device in one embodiment of the present application. DETAILED DESCRIPTION
[0051] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at the time of...", or "when...", or "in response to determination".
[0052] In the embodiments of the present application, a decoding method and an encoding method are proposed, which may involve the following concepts:
[0053] Entropy encoding: Entropy encoding is a method of encoding that does not lose any information during the encoding process according to the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.
[0054] Neural Network (NN): A neural network refers to an artificial neural network. A NN is a computational model composed of a large number of interconnected nodes (or neurons). In a NN, neuronal processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. Processing units in a NN can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; and hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information is reflected in the connections between the processing units. A NN is a non-programmed, brain-like information processing method. Its essence is to achieve parallel and distributed information processing capabilities through the transformations and dynamics of the neural network, mimicking the information processing capabilities of the human brain to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.
[0055] Convolutional Neural Network (CNN): A convolutional neural network (CNN) is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to surrounding cells within a certain coverage area, making it an excellent choice for processing large images. The basic structure of a CNN consists of two layers: the feature extraction layer (also known as the convolution layer). Each neuron's input is connected to the local receptive field of the previous layer, extracting local features. Once a local feature is extracted, its positional relationship with other features is determined. The second layer is the feature mapping layer (also known as the activation layer). Each computational layer of the neural network consists of multiple feature maps. Each feature map is a plane where all neurons have equal weights. Feature mapping structures can use sigmoid functions, ReLU functions (rectified linear units), leaky-ReLU functions, PReLU functions (leaky rectified linear units), and GDN (generalized difference network) functions as activation functions. In addition, since neurons on a mapping surface share weights, the number of free parameters of the network is reduced.
[0056] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid complex pre-processing of images (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected. This will result in a huge number of parameters, making network training time-consuming or even difficult to train. Convolutional neural networks reduce this difficulty through methods such as local connections and weight sharing.
[0057] Deconvolution: Deconvolution is also called transposed convolution. It works similarly to convolution, but uses padding to make the output larger than the input (though they can remain the same). A stride of 1 equals the input size. A stride of N means the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.
[0058] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns hidden behind the data. For data outside the learning set with the same patterns, the trained network can also give appropriate output. This ability can be called generalization ability.
[0059] Feature: The features referred to in this application are three-dimensional feature matrices or tensors of C*W*H. See Figure 1, which shows a schematic diagram of a three-dimensional feature matrix. In this matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. A three-dimensional feature matrix can be either the input or the output of a neural network.
[0060] Rate-Distortion Optimized (RDO): Coding efficiency is evaluated using two key metrics: bit rate and Peak Signal to Noise Ratio (PSNR). A smaller bitrate results in a higher compression ratio, while a higher PSNR indicates better reconstructed image quality. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of these two metrics. For example, the cost of a mode is: J(mode) = D + λ*R, where D represents distortion. This is typically measured using the SSE metric (mean square sum of squared differences between the reconstructed image block and the source image). For cost considerations, the SAD metric (the sum of absolute differences between the reconstructed image block and the source image) can also be used. λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in that mode, including the bits required for coding mode information, motion information, and residual information. Using the rate-distortion principle to compare coding modes during mode selection generally ensures optimal coding performance.
[0061] Numerous coding tools have been proposed for various modules on the encoding side, and each tool often has multiple modes. Different coding tools often achieve optimal coding performance for different video sequences. Therefore, during the encoding process, RDO (Rate-Distortion Optimize) is often used to compare the coding performance of different tools or modes to select the optimal one. After determining the optimal tool or mode, the decision-making information for the tool or mode is transmitted by encoding marker information in the bitstream. Although this approach introduces higher coding complexity, it can adaptively select the optimal tool or mode combination for different content, achieving optimal coding performance. The decoding side can obtain relevant tool or mode information by directly parsing the marker information, with minimal complexity impact.
[0062] The following describes in detail the decoding method and encoding method in the embodiments of the present application in combination with several embodiments.
[0063] Embodiment 1: In the embodiment of the present application, a decoding method is proposed. FIG2 is a flow chart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method may include:
[0064] Step 201: Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block, wherein the current image block is divided into multiple stage sub-blocks, that is, the current image block includes multiple stage sub-blocks.
[0065] Step 202: For each stage sub-block, determine probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block, decode the second code stream of the current image block based on the probability distribution parameters, and obtain residual features of the stage sub-block.
[0066] Step 203: Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage. The mean feature of the sub-block at this stage is obtained based on the coefficient hyperparameter feature of the sub-block at this stage and / or the reference feature of the sub-block at this stage, such as based on the coefficient hyperparameter feature, or based on the coefficient hyperparameter feature and the reference feature.
[0067] Step 204: Determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks in each stage.
[0068] Exemplarily, for the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyperparameter feature of the sub-block at this stage, or, the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyperparameter feature and the default reference feature of the sub-block at this stage.
[0069] For the i-th stage sub-block, where i is greater than 1, the reference features of the i-th stage sub-block are obtained based on the reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block; the mean features of the stage sub-block are obtained through a mean prediction network based on the coefficient hyperparameter features and the reference features of the stage sub-block; wherein the reference features include all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction features of the i-1-th stage sub-block.
[0070] Exemplarily, the mean prediction network may include a first prediction network, a second prediction network and a prediction fusion network. The mean feature of the sub-block at this stage is obtained through the mean prediction network based on the coefficient hyperparameter feature and the reference feature of the sub-block at this stage. It may include: obtaining the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network; obtaining the second prediction feature corresponding to the reference feature through the second prediction network; splicing the first prediction feature and the second prediction feature to obtain a spliced feature, inputting the spliced feature into the prediction fusion network, processing the spliced feature through the prediction fusion network, and obtaining the mean feature of the sub-block at this stage.
[0071] Exemplarily, obtaining a first prediction feature corresponding to a coefficient hyperparameter feature through a first prediction network may include: performing a feature enhancement operation and an upsampling operation on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature. The feature enhancement operation may include: a convolution operation, or a convolution operation and an activation operation; the upsampling operation may include: a deconvolution operation, a cropping operation, and an activation operation, or a deconvolution operation, a cropping operation, an activation operation, and a convolution operation. Exemplarily, the activation operation is a Relu operation.
[0072] Exemplarily, obtaining the second prediction feature corresponding to the reference feature through the second prediction network may include: concatenating all reconstructed features in the reference feature according to the channel dimension; performing a convolution operation on the concatenated features to obtain the second prediction feature; or adding all reconstructed features in the reference feature and performing a convolution operation on the added features to obtain the second prediction feature. The reference feature includes all the reconstructed features of the previous i-1 stage sub-blocks; or, partial reconstructed features of the previous i-1 stage sub-blocks; or, the reconstructed features of the i-1th stage sub-block.
[0073] Exemplarily, processing the concatenated features through the prediction fusion network to obtain the mean features of the sub-blocks at that stage may include, but is not limited to: performing a convolution operation and at least one fusion operation on the concatenated features through the prediction fusion network to obtain the mean features of the sub-blocks at that stage; wherein the fusion operation includes an activation operation and a convolution operation. Exemplarily, the activation operation is a ReLU operation.
[0074] Exemplarily, after determining the reconstruction features of the sub-block at this stage based on the residual features of the sub-block at this stage and the mean features of the sub-block at this stage, the reconstruction features of the sub-block at this stage can also be feature enhanced to obtain enhanced reconstruction features; wherein the enhanced reconstruction features are used to determine the mean features of the sub-block at this stage, and the enhanced reconstruction features are used to determine the reconstructed image block.
[0075] Exemplarily, determining the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage may include, but is not limited to: performing feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain aggregated features, and inputting the aggregated features into a synthesis transformation network to obtain the reconstructed image block corresponding to the current image block. Alternatively, the reconstruction features of the sub-blocks at each stage may be aggregated to obtain aggregated features, and the aggregated features may be divided into blocks to obtain multiple block features; each block feature may be input into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature, and the block-reconstructed image blocks corresponding to the multiple block features may be merged to obtain the reconstructed image block corresponding to the current image block. Alternatively, the reconstruction features of the sub-blocks at each stage may be input into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the sub-block at that stage, and the block-reconstructed image blocks corresponding to all the sub-blocks at all stages may be merged to obtain the reconstructed image block corresponding to the current image block.
[0076] Exemplarily, the reconstructed features of each stage sub-block are aggregated to obtain aggregated features, which may include but is not limited to: the reconstructed features of each stage sub-block can be aggregated according to phase to obtain aggregated features; or the reconstructed features of each stage sub-block can be aggregated according to phase to obtain multiple phase-aggregated features, and then the multiple phase-aggregated features can be spliced according to channels to obtain aggregated features.
[0077] Exemplarily, dividing the aggregated features into multiple block features may include: determining a target size of the block features; dividing the aggregated features into multiple block features in an even order from top to bottom and from left to right based on the target size, and the size of each block feature is the target size; or, determining an actual block size and an overlap size of the block features, the actual block size being the block size of the block features excluding the overlapping parts, and the overlap size being the size of the overlapping parts of adjacent blocks; dividing the aggregated features into multiple block features based on the actual block size and the overlap size, and the size of each block feature being the target size; wherein, if the actual block size is tile_c*tile_w*tile_h, and the overlap size is tile_c*padding_w*padding_h, then the target size is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h).
[0078] Exemplarily, merging the block-reconstructed image blocks corresponding to multiple block features to obtain a reconstructed image block corresponding to the current image block may include, but is not limited to: sorting the block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right, and sequentially splicing the block-reconstructed image blocks corresponding to the sorted multiple block features to obtain a reconstructed image block corresponding to the current image block. Alternatively, determining the actual block size and overlap size of each block-reconstructed image block; sorting the block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right; and sequentially splicing the block-reconstructed image blocks corresponding to the sorted multiple block features based on the actual block size and overlap size to obtain a reconstructed image block corresponding to the current image block. Wherein, the reconstructed image block includes multiple block-reconstructed image blocks, the size of the non-overlapping portion of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping portion of the block-reconstructed image block with the adjacent block-reconstructed image block is the overlap size of the block-reconstructed image block. Among them, for the overlapping part of the left and right reconstructed image blocks, the value of the overlapping part is the value of the left reconstructed image block, or the value of the overlapping part is the average of the two reconstructed image blocks; for the overlapping part of the upper and lower reconstructed image blocks, the value of the overlapping part is the value of the upper reconstructed image block, or the value of the overlapping part is the average of the two reconstructed image blocks.
[0079] Exemplarily, inputting the aggregated features into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block may include: decoding the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of the current image block, inputting the rate control parameter and the aggregated features into the synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.
[0080] Inputting each block feature into a synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature may include: decoding an auxiliary code stream corresponding to the current image block to obtain a rate control parameter for each block feature, inputting the block feature and the rate control parameter for the block feature into the synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature.
[0081] Inputting the reconstruction features of the sub-block at each stage into a synthesis transformation network to obtain a block-by-block reconstructed image block corresponding to the sub-block at that stage may include: decoding the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of the sub-block at each stage, inputting the reconstruction features of the sub-block at that stage and the rate control parameter of the sub-block at that stage into the synthesis transformation network to obtain a block-by-block reconstructed image block corresponding to the sub-block at that stage.
[0082] Exemplarily, inputting the bit rate control parameters and the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block may include: processing the aggregated features through the synthetic transformation network to obtain a first feature; processing the bit rate control parameters through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.
[0083] Inputting the block feature and the rate control parameter of the block feature into a synthetic transformation network to obtain a block reconstructed image block corresponding to the block feature can include: processing the block feature through the synthetic transformation network to obtain a first feature; processing the rate control parameter of the block feature through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block corresponding to the block feature based on the third feature.
[0084] Inputting the reconstruction features of the stage sub-block and the rate control parameters of the stage sub-block into a synthetic transformation network to obtain a block-by-block reconstructed image block corresponding to the stage sub-block can include: processing the reconstruction features of the stage sub-block through the synthetic transformation network to obtain a first feature; processing the rate control parameters of the stage sub-block through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block-by-block reconstructed image block corresponding to the stage sub-block based on the third feature.
[0085] For example, the above execution order is only for the convenience of describing the examples given. In actual applications, the execution order between the steps can also be changed, and this execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than the steps described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.
[0086] As can be seen from the above technical solutions, in the embodiments of the present application, a coding and decoding scheme with variable and adjustable bit rates is proposed for neural network-based coding and decoding technology, which improves parallelism, effectively saves feature storage cache, achieves higher bit rate control accuracy, and has less coding performance loss, with better coding performance and bit rate control accuracy. By adopting block coding and decoding, the peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capabilities are provided, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve coding and decoding performance, and reduce complexity.
[0087] Embodiment 2: In the embodiment of the present application, an encoding method is proposed. FIG3 is a flow chart of the encoding method. The method can be applied to an encoding end (also called a video encoder). The method may include:
[0088] Step 301: Input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block.
[0089] Step 302: Split the feature block into features to be encoded in multiple stage sub-blocks.
[0090] Step 303: For each stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into the first code stream of the current image block.
[0091] Step 304: Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage. The mean feature of the sub-block at this stage is obtained based on the coefficient hyperparameter feature of the sub-block at this stage and / or the reference feature of the sub-block at this stage, such as the coefficient hyperparameter feature, or the coefficient hyperparameter feature and the reference feature.
[0092] Step 305: Determine probability distribution parameters based on the coefficient hyperparameter features of the sub-block at this stage, and encode the residual features of the sub-block at this stage into the second code stream of the current image block based on the probability distribution parameters.
[0093] Exemplarily, for the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyperparameter feature of the sub-block at this stage, or, the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyperparameter feature and the default reference feature of the sub-block at this stage.
[0094] For the i-th stage sub-block, where i is greater than 1, the reference features of the i-th stage sub-block are obtained based on the reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block; the mean features of the stage sub-block are obtained through a mean prediction network based on the coefficient hyperparameter features and the reference features of the stage sub-block; wherein the reference features include all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction features of the i-1-th stage sub-block.
[0095] Exemplarily, the processing process at the encoding end is similar to the processing process at the decoding end, and the similarities are not repeated here.
[0096] For example, the above execution order is only for the convenience of describing the examples given. In actual applications, the execution order between the steps can also be changed, and this execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than the steps described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.
[0097] As can be seen from the above technical solutions, in the embodiments of the present application, a coding and decoding scheme with variable and adjustable bit rates is proposed for neural network-based coding and decoding technology, which improves parallelism, effectively saves feature storage cache, achieves higher bit rate control accuracy, and has less coding performance loss, with better coding performance and bit rate control accuracy. By adopting block coding and decoding, the peak memory usage is lower, the decoding time of a single block is shorter, and high-speed parallel decoding capabilities are provided, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve coding and decoding performance, and reduce complexity.
[0098] Example 3: For Example 1 and Example 2, the processing process of the encoding end can be seen in Figure 4. Of course, Figure 4 is only an example of the processing process of the encoding end and does not limit the processing process of this encoding end.
[0099] After obtaining the current image block x (which can be the original image block x, i.e., the input image block), the encoder can perform an analysis transformation on the current image block x using an analysis transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Performing feature transformation on the current image block x using the analysis transformation network means transforming the current image block x into image features y in the latent domain, thereby facilitating all subsequent processes to operate in the latent domain.
[0100] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.
[0101] After obtaining the image feature y, the encoding end performs a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of this hyperparameter encoding network, as long as it can perform a coefficient hyperparameter feature transformation on the image feature y. The image feature y in the latent domain obtains the super-prior latent information z (i.e., the coefficient hyperparameter feature z) after passing through the hyperparameter encoding network.
[0102] After obtaining the coefficient hyperparameter feature z, the encoding end can quantize the coefficient hyperparameter feature z to obtain the superparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, the Q operation in Figure 4 is the quantization process. After obtaining the superparameter quantization feature corresponding to the coefficient hyperparameter feature z, the superparameter quantization feature is encoded to obtain the Bitstream#1 (i.e., the first code stream) corresponding to the current image block, that is, the AE operation in Figure 4 represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or the coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the mean and parameters of the probability distribution model (i.e., probability distribution parameters).
[0103] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the Bitstream#1 corresponding to the current image block by the decoder, please refer to the subsequent embodiments.
[0104] After obtaining Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the super-parameter quantization feature. That is, AD in Figure 4 represents the decoding process. Then, the super-parameter quantization feature is dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. The IQ operation in Figure 4 is the dequantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the coefficient super-parameter feature z_hat without involving the dequantization process of the coefficient super-parameter feature z_hat.
[0105] For the encoding process of Bitstream#1, an encoding method of a fixed probability density model can be used. For the decoding process of Bitstream#1, a decoding method of a fixed probability density model can be used. There is no restriction on this encoding and decoding process.
[0106] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (see the subsequent embodiments for the determination process of the image feature y_hat) to obtain the predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input to the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat. The two are jointly input to obtain a more accurate predicted value mu. The predicted value mu is used to obtain a residual by subtracting the original feature, and is added to the decoded residual to obtain the reconstructed feature y_hat.
[0107] It should be noted that the mean prediction network is an optional neural network, that is, the mean prediction network may not be present, that is, the predicted value mu does not need to be determined through the mean prediction network. The dotted box in Figure 4 indicates that the mean prediction network is optional.
[0108] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the predicted value mu, such as taking the difference between the image feature y and the predicted value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the predicted value mu. Alternatively, after obtaining the image feature y, the encoding end can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.
[0109] After obtaining the image feature s, the encoder can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s. That is, the Q operation in Figure 4 represents the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoder can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block. That is, the AE operation in Figure 4 represents the encoding process, such as the entropy encoding process. Alternatively, the encoder can directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.
[0110] After obtaining Bitstream#2 corresponding to the current image block, the encoder can send Bitstream#2 corresponding to the current image block to the decoder. For details about the processing of Bitstream#2 corresponding to the current image block by the decoder, see the subsequent embodiments.
[0111] After obtaining Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain image quantization features. In other words, AD in Figure 4 represents the decoding process. The encoder can then dequantize the image quantization features to obtain image features s'. Image features s' can be the same as or different from image features s. The IQ operation in Figure 4 represents the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain image features s' without involving the dequantization process of the image quantization features.
[0112] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the predicted value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the predicted value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, which provides the predicted value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, there is no need to deploy a mean prediction network. The dotted box indicates that the residual process is an optional process.
[0113] After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.
[0114] In one possible implementation, when the encoder encodes the image quantization feature or image feature s to obtain Bitstream #2 corresponding to the current image block, the encoder needs to first determine a probability distribution model and then encode the image quantization feature or image feature s based on the probability distribution model. Furthermore, when decoding Bitstream #2, the encoder also needs to first determine a probability distribution model and then decode Bitstream #2 based on the probability distribution model.
[0115] In order to obtain the probability distribution model, as shown in Figure 4, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network. There is no restriction on the training process of this probability hyperparameter decoding network. It only needs to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.
[0116] In one possible implementation, the processing at the encoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without any restriction on the encoding process.
[0117] Example 4: For Example 1 and Example 2, the processing process of the decoding end can be seen in Figure 5. Of course, Figure 5 is only an example of the processing process of the decoding end and does not limit the processing process of the decoding end.
[0118] After obtaining Bitstream#1 corresponding to the current image block, the decoder can also decode Bitstream#1 to obtain the super-parameter quantization feature. That is, AD in Figure 5 represents the decoding process. Then, the super-parameter quantization feature is dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. The IQ operation in Figure 5 represents the dequantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoder can also decode Bitstream#1 to obtain the coefficient super-parameter feature z_hat without involving the dequantization process of the coefficient super-parameter feature z_hat.
[0119] For the decoding process of Bitstream#1, a decoding method using a fixed probability density model may be used, and there is no restriction on this.
[0120] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.
[0121] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (the determination process of the image feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input to the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded image feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.
[0122] It should be noted that the mean prediction network is an optional neural network, that is, the mean prediction network may not be present, that is, the predicted value mu does not need to be determined through the mean prediction network. The dotted box in Figure 5 indicates that the mean prediction network is optional.
[0123] After obtaining Bitstream#2 corresponding to the current image block, the decoder can also decode Bitstream#2 to obtain the image quantization feature. That is, AD in Figure 5 represents the decoding process. The decoder can then dequantize the image quantization feature to obtain image feature s'. Image feature s' can be the same as or different from image feature s. The IQ operation in Figure 5 represents the dequantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the decoder can also decode Bitstream#2 to obtain image feature s' without involving the dequantization process of the image quantization feature.
[0124] After obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoding end determines the image feature y_hat based on the residual feature r_hat and the predicted value mu. The image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the predicted value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, which provides the predicted value mu. Alternatively, after obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' to obtain the image feature y_hat. The image feature y_hat can be the same as or different from the image feature y. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.
[0125] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.
[0126] In one possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. In order to obtain the probability distribution model, referring to FIG5 , after obtaining the coefficient superparameter feature z_hat, the decoding end can perform a coefficient superparameter feature inverse transformation on the coefficient superparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient superparameter feature z_hat is input to the probability superparameter decoding network, and the probability superparameter decoding network performs a coefficient superparameter feature inverse transformation on the coefficient superparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. The probability superparameter decoding network can be a trained neural network, and there is no restriction on the training process of the probability superparameter decoding network. It is sufficient to perform a coefficient superparameter feature inverse transformation on the coefficient superparameter feature z_hat to obtain the probability distribution parameter p.
[0127] In one possible implementation, the processing at the decoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without limiting the decoding process.
[0128] Example 5: For Example 1, Example 2, Example 3 and Example 4, a mean prediction network may be deployed or not. In order to improve the feature encoding performance, the deployment of the mean prediction network is taken as an example. When deploying the mean prediction network, in order to obtain accurate prediction values of image features, it is necessary to use the reconstructed values of adjacent features (i.e., based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block, a context-based prediction is performed to obtain the prediction value mu corresponding to the current image block). Therefore, the feature value on the right (using the reconstructed value of the feature on the left or upper left side as a reference) can only be encoded after the feature value on the left or upper left side completes the encoding reconstruction. This dependency relationship makes the encoding between features only serially executed, and cannot be executed in parallel, resulting in higher complexity.
[0129] In response to the above findings, in this embodiment, by adopting block encoding and decoding, the degree of parallelism can be improved, the feature storage cache can be effectively saved, higher bit rate control accuracy can be achieved, the coding performance loss is smaller, and it has better coding performance and bit rate control accuracy, which makes the peak memory usage lower, the decoding time of a single block shorter, and has high-speed parallel decoding capabilities.
[0130] Regarding the processing process of the encoding end, please refer to FIG6 , and there is no limitation on the processing process of the encoding end.
[0131] 1. After obtaining the original image x, the block division unit determines whether to divide the original image x into blocks based on the image resolution. For example, if the image resolution is greater than a threshold, the original image x is divided into blocks; otherwise, the original image x is not divided into blocks. If yes, the original image x is divided into several image blocks x_p, with overlap between adjacent image blocks. If no, the image block x_p is the original image x, meaning that the original image x has only one image block. Each image block x_p can be denoted as the current image block. For ease of description, the processing of a current image block x_p will be used as an example.
[0132] 2. After obtaining the current image block x_p, the analysis transformation network is used to analyze and transform the current image block x_p to obtain the image feature y_p (feature block y_p) corresponding to the current image block x_p. For example, the analysis transformation network transforms the current image block x_p into the image feature y_p in the latent domain, making it easier for all subsequent processes to operate in the latent domain.
[0133] 3. The merging and classification unit merges and classifies the image feature y_p to obtain the to-be-encoded features y_i (i=1…N) of the N-stage sub-blocks, that is, splits the image feature y_p into the to-be-encoded features y_i of the N-stage sub-blocks.
[0134] 4. For each stage sub-block corresponding to the current image block, obtain the coefficient hyperparameter feature corresponding to the stage sub-block. For example, perform a coefficient hyperparameter feature transformation on the feature to be encoded y_i of stage sub-block i to obtain the coefficient hyperparameter feature z_i corresponding to stage sub-block i. For example, the feature to be encoded y_i can be input into a hyperparameter encoding network (i.e., a neural network), which performs a coefficient hyperparameter feature transformation on the feature to be encoded y_i to obtain the coefficient hyperparameter feature z_i.
[0135] 5. For each stage sub-block corresponding to the current image block, the coefficient hyperparameter feature of the stage sub-block is encoded into the first bitstream (Bitstream#1) of the current image block. For example, the coefficient hyperparameter feature z_i corresponding to the stage sub-block i is quantized to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_i, and the hyperparameter quantization feature is encoded to obtain the first bitstream corresponding to the current image block. Alternatively, the coefficient hyperparameter feature z_i corresponding to the stage sub-block i is directly encoded to obtain the first bitstream corresponding to the current image block. After obtaining the first bitstream corresponding to the current image block, the first bitstream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the first bitstream corresponding to the current image block, refer to the subsequent embodiments.
[0136] 6. After obtaining the first code stream corresponding to the current image block, the first code stream can also be decoded to obtain the coefficient hyperparameter features of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i (i=1…N).
[0137] For example, the first bitstream can be decoded to obtain the super-parameter quantization features of stage sub-block i. Then, the super-parameter quantization features can be dequantized to obtain the coefficient super-parameter features z_hat_i corresponding to stage sub-block i. Alternatively, the first bitstream can be decoded to obtain the coefficient super-parameter features z_hat_i corresponding to stage sub-block i without involving the dequantization process.
[0138] For the encoding process of the first code stream, an encoding method of a fixed probability density model can be adopted, and for the decoding process of the first code stream, a decoding method of a fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.
[0139] 7. For each stage sub-block corresponding to the current image block, the probability distribution parameters are determined based on the coefficient hyperparameter features of the stage sub-block. For example, the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i can be inversely transformed to obtain the probability distribution parameter p_i corresponding to the stage sub-block i. For example, the coefficient hyperparameter feature z_hat_i is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs an inverse transformation on the coefficient hyperparameter feature z_hat_i to obtain the probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, a probability distribution model can be generated based on the probability distribution parameter p_i.
[0140] 8. After obtaining the features to be encoded for the N stage sub-blocks, for the feature to be encoded y_1 of the first stage sub-block, input the coefficient hyperparameter feature z_hat_1 of the stage sub-block to the mean prediction network to obtain the mean feature of the stage sub-block (i.e., predicted value mu_1, mean mu_1). Alternatively, obtain the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 of the stage sub-block and the default reference feature to the mean prediction network to obtain the mean feature of the stage sub-block.
[0141] Based on the to-be-encoded feature y_1 and the mean feature mu_1 of the sub-block in the first stage, the residual feature r_1 is determined. For example, the difference between the to-be-encoded feature y_1 and the mean feature mu_1 is used as the residual feature r_1. Then, feature processing is performed on the residual feature r_1 to obtain the processed residual feature r_1. This feature processing process is not restricted and can be any feature processing method. Of course, feature processing is an optional step and it is not necessary to perform feature processing on the residual feature r_1.
[0142] The residual feature r_1 (or the residual feature r_1 after feature processing) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second code stream (Bitstream#2) corresponding to the current image block. Alternatively, the residual feature r_1 (or the residual feature r_1 after feature processing) can be directly encoded to obtain the second code stream corresponding to the current image block. After obtaining the second code stream corresponding to the current image block, the second code stream corresponding to the current image block is sent to the decoding end. Regarding the processing process of the decoding end for the second code stream corresponding to the current image block, refer to the subsequent embodiments. Among them, when encoding the image quantization feature or the residual feature r_1, the probability distribution model corresponding to the probability distribution parameter p_1 of the sub-block in the first stage can be used to encode the image quantization feature or the residual feature r_1 to obtain the second code stream.
[0143] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameter p_1 of the sub-block in the first stage can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_1. Alternatively, the image quantization features can be dequantized and then the residual features r_hat_1 can be directly obtained without involving a feature recovery process. Alternatively, after decoding the second bitstream, feature recovery can be performed on the decoded features to obtain residual features r_hat_1. Alternatively, the residual features r_hat_1 can be directly obtained after decoding the second bitstream.
[0144] After obtaining the residual feature r_hat_1, the reconstructed feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, such as taking the sum of the residual feature r_hat_1 and the mean feature mu_1 as the reconstructed feature y_hat_1.
[0145] 9. After obtaining the features to be encoded of N stage sub-blocks, for the feature to be encoded y_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i are input into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., predicted value mu_i, mean mu_i).
[0146] Among them, the reference features for stage sub-block i (i.e., stage sub-block i) can be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, they are obtained based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of stage sub-block i are obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include some of the reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on the reconstructed features of the i-1th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.
[0147] Based on the to-be-encoded feature y_i and the mean feature mu_i of the sub-block at the i-th stage, a residual feature r_i is determined. For example, the difference between the to-be-encoded feature y_i and the mean feature mu_i is used as the residual feature r_i. Feature processing is performed on the residual feature r_i to obtain the feature-processed residual feature r_i. Of course, feature processing can also be omitted for the residual feature r_i.
[0148] The residual feature r_i (or the residual feature r_i after feature processing) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second code stream corresponding to the current image block. Alternatively, the residual feature r_i (or the residual feature r_i after feature processing) is directly encoded to obtain the second code stream corresponding to the current image block. The second code stream corresponding to the current image block is sent to the decoding end. For the processing process of the second code stream corresponding to the current image block at the decoding end, refer to the subsequent embodiments. Among them, when encoding the image quantization feature or the residual feature r_i, the probability distribution model corresponding to the probability distribution parameter p_i of the sub-block in the i-th stage can be used to encode the image quantization feature or the residual feature r_i to obtain the second code stream.
[0149] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameter p_i of the sub-block in the i-th stage can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_i. Alternatively, the image quantization features can be dequantized and then the residual features r_hat_i can be directly obtained. Alternatively, after decoding the second bitstream, the decoded features can be restored to obtain residual features r_hat_i. Alternatively, the residual features r_hat_i can be directly obtained after decoding the second bitstream.
[0150] After obtaining the residual feature r_hat_i, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, such as taking the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.
[0151] 10. The clustering and blocking unit can obtain the reconstructed features y_hat_i (i=1...N) of the N stage sub-blocks, and perform feature aggregation on the reconstructed features y_hat_i of the N stage sub-blocks to obtain an aggregated feature. Then, the clustering and blocking unit blocks the aggregated feature to obtain at least one block feature y_hat_p, that is, a feature block y_hat_p.
[0152] 11. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.
[0153] 12. The merging unit can obtain multiple reconstructed image blocks x_hat_p corresponding to the block features y_hat_p, and merge these reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Obviously, if the original image x is divided into multiple current image blocks x_p, then the reconstructed image blocks x_hat corresponding to all current image blocks x_p can form a reconstructed image. If the original image x corresponds to a current image block x_p, then the reconstructed image block x_hat is the reconstructed image.
[0154] In the above process, the encoder can also assign different rate control parameters to sub-blocks at different stages of different image blocks based on the region of interest or rate information, so that sub-blocks at different stages of different image blocks consume different rate.
[0155] For example, if special processing is required for the region of interest (such as non-uniform processing to ensure higher quality of the region of interest) to avoid uniform quantization distortion of each image block, the rate control parameter can be a quantization parameter (the rate control parameter needs to be transmitted to the decoding end through the code stream). As shown in Figure 6, the rate control parameter affects the quantization process (Q) and inverse quantization process (IQ) of the first code stream, and affects the quantization process (Q) and inverse quantization process (IQ) of the second code stream.
[0156] For example, when quantizing the coefficient hyperparameter feature z_i corresponding to stage sub-block i, if stage sub-block i is located within the region of interest, the rate control parameter is used to quantize the coefficient hyperparameter feature z_i corresponding to stage sub-block i. The rate control parameter is used to ensure higher image quality corresponding to stage sub-block i. When dequantizing the hyperparameter quantized feature corresponding to stage sub-block i, the rate control parameter is also needed to dequantize the hyperparameter quantized feature corresponding to stage sub-block i.
[0157] When quantizing the residual feature r_i (or the residual feature r_i after feature processing) corresponding to the stage sub-block i, if the stage sub-block i is located within the region of interest, the rate control parameter is used to quantize the residual feature r_i corresponding to the stage sub-block i. The rate control parameter is used to ensure that the image quality corresponding to the stage sub-block i is higher. When dequantizing the image quantization feature corresponding to the stage sub-block i, the rate control parameter also needs to be used to dequantize the image quantization feature corresponding to the stage sub-block i.
[0158] For example, in order to perform precise bit rate control, the bit rate control parameter can be a control parameter of the synthesis transformation network (the bit rate control parameter needs to be transmitted to the decoding end through the code stream). As shown in Figure 6, the bit rate control parameter affects the analysis transformation network and the synthesis transformation network. For example, when the current image block x_p is analyzed and transformed by the analysis transformation network, the bit rate control parameter and the current image block x_p are input to the analysis transformation network, and the analysis transformation network outputs the image feature y_p corresponding to the current image block x_p. When the block feature y_hat_p is input to the synthesis transformation network, the block feature y_hat_p and the bit rate control parameter are input to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.
[0159] Based on the configured target bit rate (one target bit rate for each image block, one target bit rate for each image, or one target bit rate for multiple images), the encoder can try to use different rate control parameters. After using a certain rate control parameter, if the bit rate corresponding to the bit stream for the current image block (such as the first bit stream and the second bit stream) is the target bit rate or close to the target bit rate (that is, the difference between the two is less than a threshold), then this rate control parameter is the final rate control parameter used, and the encoder can send this rate control parameter to the decoder.
[0160] Example 6: The processing process of the decoding end can be shown in FIG7 , and there is no limitation on the processing process of the decoding end.
[0161] 1. After obtaining the first bitstream (Bitstream#1) corresponding to the current image block, decode the first bitstream to obtain the coefficient hyperparameter features of each stage sub-block of the current image block, such as the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i (i=1…N).
[0162] For example, the first bitstream can be decoded to obtain the hyperparametric quantization features of stage sub-block i. These hyperparametric quantization features are then dequantized to obtain the coefficient hyperparametric features z_hat_i corresponding to stage sub-block i. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparametric features z_hat_i corresponding to stage sub-block i without involving the dequantization process. For the decoding process of the first bitstream, a decoding method using a fixed probability density model can be used, and there are no restrictions on this decoding process.
[0163] 2. For each stage sub-block corresponding to the current image block, the probability distribution parameters are determined based on the coefficient hyperparameter features of the stage sub-block. For example, the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i can be inversely transformed to obtain the probability distribution parameter p_i corresponding to the stage sub-block i. For example, the coefficient hyperparameter feature z_hat_i is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs an inverse transformation on the coefficient hyperparameter feature z_hat_i to obtain the probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, a probability distribution model can be generated based on the probability distribution parameter p_i.
[0164] 3. For each stage sub-block corresponding to the current image block, decode the second bitstream (Bitstream#2) of the current image block based on the probability distribution parameters (probability distribution model) of the sub-block at that stage to obtain the residual features of the sub-block at that stage.
[0165] For example, the probability distribution model corresponding to the probability distribution parameter p_i of the sub-block at the i-th (i=1…N) stage can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_i, or dequantize the image quantization features and directly obtain residual features r_hat_i. Alternatively, after decoding the second bitstream, feature recovery is performed on the decoded features to obtain residual features r_hat_i, or after decoding the second bitstream, residual features r_hat_i are directly obtained.
[0166] 4. After obtaining the residual features of the N-stage sub-blocks, for the residual feature r_hat_1 of the first-stage sub-block, the coefficient hyperparameter feature z_hat_1 of the sub-block at that stage is input into the mean prediction network to obtain the mean feature of the sub-block at that stage (i.e., predicted value mu_1, mean mu_1). Alternatively, obtain the configured default reference feature, and input the coefficient hyperparameter feature z_hat_1 of the sub-block at that stage and the default reference feature into the mean prediction network to obtain the mean feature of the sub-block at that stage.
[0167] Then, the reconstructed feature y_hat_1 of the sub-block in the first stage is determined based on the residual feature r_hat_1 and the mean feature mu_1. For example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be used as the reconstructed feature y_hat_1.
[0168] 5. After obtaining the residual features of N stage sub-blocks, for the residual feature r_hat_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature z_hat_i of the stage sub-block i and the reference feature of the stage sub-block i are input into the mean prediction network to obtain the mean feature of the stage sub-block i (i.e., the predicted value mu_i, the mean mu_i).
[0169] Among them, the reference features for stage sub-block i (i.e., stage sub-block i) can be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, they can be obtained based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of stage sub-block i are obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include some of the reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on the reconstructed features of the i-1th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.
[0170] To control the cache size, the number of reconstructed features included in the reference features cannot exceed a preset number. Assuming the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference features include all reconstructed features from y_hat_1 to y_hat_i-1. If the number of reconstructed features is greater than M, the reference features include M reconstructed features from y_hat_1 to y_hat_i-1 (i.e., partially reconstructed features). To select M reconstructed features from y_hat_1 to y_hat_i-1, M reconstructed features can be selected based on a default strategy, such as selecting the M reconstructed features closest to the sub-block in the current stage in decoding order, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, and so on, until M reconstructed features are selected. Of course, other methods can also be used to select M reconstructed features, and this is not limited to this. Alternatively, the index values of the M reconstructed features can be obtained by decoding the bitstream, and the M reconstructed features can be selected based on the decoded index values.
[0171] Then, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstructed feature y_hat_i.
[0172] 6. The clustering and blocking unit can obtain the reconstructed features y_hat_i (i=1...N) of the N stage sub-blocks, and perform feature aggregation on the reconstructed features y_hat_i of the N stage sub-blocks to obtain an aggregated feature. Then, the clustering and blocking unit blocks the aggregated feature to obtain at least one block feature y_hat_p, that is, a feature block y_hat_p.
[0173] 7. For each block feature y_hat_p, the clustering block unit inputs the block feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.
[0174] 8. The merging unit can obtain multiple reconstructed image blocks x_hat_p corresponding to the block features y_hat_p, and merge these reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block. Obviously, if the original image x is divided into multiple current image blocks x_p, then the reconstructed image blocks x_hat corresponding to all current image blocks x_p can form a reconstructed image. If the original image x corresponds to a current image block x_p, then the reconstructed image block x_hat is the reconstructed image.
[0175] In the above process, different rate control parameters may be allocated to sub-blocks at different stages of different image blocks based on the region of interest or rate information, so that sub-blocks at different stages of different image blocks consume different rates.
[0176] For example, if special processing is required for the region of interest (such as non-uniform processing to ensure higher quality of the region of interest), the rate control parameter can be a quantization parameter (the rate control parameter needs to be transmitted to the decoding end through the code stream). As shown in Figure 7, the rate control parameter affects the inverse quantization process (IQ) of the first code stream and the inverse quantization process (IQ) of the second code stream.
[0177] For example, when dequantizing the hyperparametric quantization features corresponding to stage sub-block i, if the rate control parameters corresponding to stage sub-block i have been decoded from the bitstream, and the rate control parameters are used to ensure higher image quality corresponding to stage sub-block i, then the rate control parameters can be used to dequantize the hyperparametric quantization features corresponding to stage sub-block i.
[0178] For another example, when dequantizing the image quantization features corresponding to stage sub-block i, if the rate control parameter corresponding to stage sub-block i has been decoded from the bitstream, and the rate control parameter is used to ensure higher image quality corresponding to stage sub-block i, then the rate control parameter can be used to dequantize the image quantization features corresponding to stage sub-block i.
[0179] For example, to achieve precise bit rate control, the rate control parameters can be control parameters of the synthesis transformation network (the rate control parameters need to be transmitted to the decoder via the bitstream. The control parameters of the synthesis transformation network refer to the synthesis transformation network that can be determined based on the rate control parameters). As shown in FIG7 , the rate control parameters affect the synthesis transformation network. For example, when the block feature y_hat_p is input to the synthesis transformation network, the block feature y_hat_p and the rate control parameters are input to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p.
[0180] Example 7: In Example 5 and Example 6, for the i-th (i=1…N) stage sub-block, the reconstruction feature y_hat_i (i.e., the sum of the residual feature r_hat_i and the mean feature mu_i) of the i-th stage sub-block is obtained, and the reconstruction feature y_hat_i is used to determine the mean feature of the subsequent stage sub-block, and / or, the reconstruction feature y_hat_i is used to determine the reconstructed image block. Among them, the reconstruction feature y_hat_i is used to determine the mean feature of the subsequent stage sub-block means: the reconstruction feature y_hat_i is used as a reference feature of the subsequent stage sub-block, thereby determining the mean feature of the subsequent stage sub-block. The reconstruction feature y_hat_i is used to determine the reconstructed image block means: the reconstruction feature y_hat_i is input into the clustering block unit, thereby determining the reconstructed image block corresponding to the current image block.
[0181] For the i-th (i=1…N) stage sub-block, after obtaining the reconstruction feature y_hat_i of the i-th stage sub-block, the reconstruction feature y_hat_i can also be feature enhanced to obtain the enhanced reconstruction feature y_hat_i'. The enhanced reconstruction feature y_hat_i' is used to determine the mean feature of the subsequent stage sub-block, and / or the enhanced reconstruction feature y_hat_i' is used to determine the reconstructed image block. Among them, the enhanced reconstruction feature y_hat_i' is used to determine the mean feature of the subsequent stage sub-block, which means that the enhanced reconstruction feature y_hat_i' is used as a reference feature of the subsequent stage sub-block, thereby determining the mean feature of the subsequent stage sub-block. The enhanced reconstruction feature y_hat_i' is used to determine the reconstructed image block, which means that the enhanced reconstruction feature y_hat_i' is input into the clustering block unit, thereby determining the reconstructed image block corresponding to the current image block. For example, the bitstream can be decoded to obtain enhanced information, and the reconstructed feature y_hat_i can be enhanced based on the enhanced information to obtain a new reconstructed feature y_hat_i'. The new reconstructed feature y_hat_i' can be used for subsequent predictions of y_hat_i+1 to y_hat_N, and can also be used for subsequent clustering and blocking processes.
[0182] In a possible implementation, the enhancement process of reconstructing the feature y_hat may be performed in the following manner:
[0183] Method 1: Input is the pre-enhanced reconstructed feature y_hat, Threshold[idx], GreaterFlag[idx], Scale1[idx], and Scale2[idx], and output is the enhanced reconstructed feature y_hat_en. Threshold represents the threshold; different idx values may correspond to the same or different thresholds. idx represents the index, such as 0, 1, 2, 3, etc. GreaterFlag represents the flag; different idx values may correspond to the same or different flags. Scale1 represents the scaling factor; different idx values may correspond to the same or different Scale1 values. Scale2 represents the scaling factor; different idx values may correspond to the same or different Scale2 values.
[0184] Step 1: Determine mask[c,i,j] for each filter (indexed by idx). For example, you can use the following expression to determine mask[c,i,j]. Of course, the following expression is just an example and is not a limitation.
[0185] σ[c,i,j] represents the probability distribution parameter value of [c,i,j], c represents the channel, i represents the horizontal coordinate, and j represents the vertical coordinate. If the probability distribution parameter value is greater than Threshold[idx], GreaterFlag[idx] is true and mask[idx,c,i,j] is 1. If the probability distribution parameter value is less than Threshold[idx], GreaterFlag[idx] is false and mask[idx,c,i,j] is 1. Otherwise, mask[idx,c,i,j] is 0.
[0186] Step 2: Execute each filter set (indexed by idx) in sequence. If mask[idx,c,i,j] is 1, then y_hat_en[c,i,j] = y_hat[c,i,j] + mean_hat[c,i,j] * Scale2[idx] + residual_hat[c,i,j] * Scale1[idx]; y_hat[c,i,j] = y_hat_en[c,i,j]. If there are multiple sets of filters, y_hat is compensated multiple times. Where mean_hat represents the mean feature, i.e., mu_i mentioned above; residual_hat represents the residual feature, i.e., r_hat mentioned above.
[0187] Method 2: In method 1, the scaling factor for each channel c is the same. In method 2, different channels c allow different scaling factors, that is, the scaling factors are Scale1[c,idx] and Scale2[c,idx]. You can control whether the current feature is scaled by controlling the switch, or you can change the channel scaling factor to 0 to turn off scaling. Compared with method 1, the difference between method 2 and method 1 lies in the enhancement method: y_hat_en[c,i,j] = y_hat[c,i,j] + mean_hat[c,i,j] * Scale2[c,idx] + residual_hat[c,i,j] * Scale1[c,idx].
[0188] Method 3: Compared with Method 1, the implementation methods of Method 3 are similar. The difference is that the enhancement method is as follows: y_hat_en[c,i,j]=y_hat[c,i,j]*Scale1[idx]+mean_hat[c,i,j]*Scale2[idx].
[0189] Method 4: Method 4 is similar to Method 1 in implementation, differing in that the enhancement method is as follows: y_hat_en[c,i,j] = y_hat[c,i,j] * Scale1[idx] + residual_hat[c,i,j] * Scale2[idx]. In Method 4, Scale1[idx] can be set to 1 by default, and the adjustment method is affected only by changing the value of Scale2[idx].
[0190] Example 8: In Examples 5 and 6, the coefficient hyperparameter feature z_hat_1 of the first-stage sub-block can be input into the mean prediction network, and the mean prediction network outputs the mean feature of the first-stage sub-block. There is no restriction on the prediction process of this mean prediction network. For the i-th (i=1…N)-th stage sub-block, the coefficient hyperparameter feature z_hat_i of the i-th stage sub-block and the reference feature (which may be a default reference feature) of the i-th stage sub-block can be input into the mean prediction network, and the mean prediction network outputs the mean feature of the i-th stage sub-block. The prediction process of the mean prediction network is described below.
[0191] Referring to FIG8A , which is a schematic diagram of the structure of the mean prediction network, the mean prediction network may include a first prediction network, a second prediction network, and a prediction fusion network. Of course, this is just an example, and there is no limitation on the structure of the mean prediction network.
[0192] The coefficient hyperparameter feature z_hat_i of the i-th stage sub-block can be input into the first prediction network, and the first prediction feature corresponding to the coefficient hyperparameter feature z_hat_i can be obtained through the first prediction network. The reference feature of the i-th stage sub-block (such as the reconstructed features y_hat_1 to y_hat_i-1) can be input into the second prediction network, and the second prediction feature corresponding to the reference feature can be obtained through the second prediction network. Then, the first prediction feature and the second prediction feature are spliced to obtain the spliced feature, and the spliced feature is input into the prediction fusion network. The spliced feature is processed by the prediction fusion network to obtain the mean feature mu_i of the i-th stage sub-block.
[0193] For the first prediction network: perform feature enhancement and upsampling operations on the coefficient hyperparameter feature z_hat_i through the first prediction network to obtain a first prediction feature. The feature enhancement operation may include but is not limited to: a convolution operation, or a convolution operation and an activation operation. Of course, the above is only an example of a feature enhancement operation and is not limited to this feature enhancement operation. The upsampling operation may include but is not limited to: a deconvolution operation, a cropping operation, and an activation operation, or a deconvolution operation, a cropping operation, an activation operation, and a convolution operation. Of course, the above is only an example of an upsampling operation and is not limited to this upsampling operation. Exemplarily, the activation operation is a ReLU operation.
[0194] For example, referring to FIG8B , which is a schematic diagram of the structure of the first prediction network, the first prediction network consists of three enhancement networks and two 2x upsampling networks. Of course, the number of enhancement networks can be more or less, the number of 2x upsampling networks can be more or less, and the 2x upsampling network can also be replaced by an upsampling network of other multiples.
[0195] After the coefficient hyperparameter feature z_hat_i passes through enhancement network 1 and 2x upsampling network 1, enhanced large-size feature 1 is obtained. The spatial size (width and height) of enhanced large-size feature 1 is twice that of coefficient hyperparameter feature z_hat_i. After the enhanced large-size feature 1 passes through enhancement network 2 and 2x upsampling network 2, enhanced large-size feature 2 is obtained. The spatial size (width and height) of enhanced large-size feature 2 is twice that of enhanced large-size feature 1. After the enhanced large-size feature 2 passes through enhancement network 3, the first prediction feature is obtained.
[0196] Exemplarily, enhanced network 1 and enhanced network 2 may be the same or different, enhanced network 1 and enhanced network 3 may be the same or different, and enhanced network 2 and enhanced network 3 may be the same or different.
[0197] For example, an implementation of an enhanced network (such as enhanced network 1, and / or enhanced network 2, and / or enhanced network 3) can be shown in Figure 8C, that is, the enhanced network consists of a convolutional layer, which can be a 1x1 convolutional layer, or a 3x3 convolutional layer, or a 5x5 convolutional layer. There is no restriction on this, for example, a 3x3 convolutional layer can be selected.
[0198] For another example, an embodiment of an enhanced network (such as enhanced network 1, and / or enhanced network 2, and / or enhanced network 3) can be shown in FIG8D , that is, the enhanced network can be composed of convolution layer 1, activation layer 1, convolution layer 2, activation layer 2, and convolution layer 3. The convolution layer (such as convolution layer 1, and / or convolution layer 2, and / or convolution layer 3) can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, without limitation. For example, a 3x3 convolution layer can be selected.
[0199] The activation layer (such as activation layer 1 and / or activation layer 2) can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh (hyperbolic tangent) layer, or a Gelu (Gaussian Error Linear Units) layer. There is no restriction on the type of the activation layer, such as a relu layer can be selected.
[0200] Of course, the above are just two examples of enhancing the network, and there is no limitation thereto, as long as the enhanced function can be achieved.
[0201] Exemplarily, the 2x upsampling network 1 and the 2x upsampling network 2 may be the same or different.
[0202] For example, an embodiment of a 2x upsampling network (such as a 2x upsampling network 1 and / or a 2x upsampling network 2) can be shown in FIG8E , that is, the 2x upsampling network can be composed of a 2x deconvolution layer, a Crop layer, and an activation layer. The deconvolution layer can be a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer, and there is no restriction on this. For example, a 4x4 deconvolution layer can be selected. The activation layer can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. There is no restriction on the type of activation layer, and a relu layer can be selected. In addition, the crop layer is used to crop features so that the spatial resolution of the features becomes smaller and the number of channels remains unchanged.
[0203] For another example, an embodiment of a 2x upsampling network (such as a 2x upsampling network 1 and / or a 2x upsampling network 2) can be shown in FIG8F , that is, the 2x upsampling network can be composed of a 2x deconvolution layer, a Crop layer, an activation layer, and a convolution layer. The deconvolution layer can be a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer. The activation layer can be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. The Crop layer is used to crop features so that the spatial resolution of the features becomes smaller and the number of channels remains unchanged. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, without limitation, such as selecting a 3x3 convolution layer.
[0204] Of course, the above are just two examples of a 2x upsampling network, and there is no limitation to this, as long as the upsampling function can be achieved.
[0205] For example, the convolution layer in the first prediction network can be a normal 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation of the grouped convolution can be found in subsequent embodiments and will not be described in detail here.
[0206] For the second prediction network: all the reconstructed features in the reference features of the i-th stage sub-block are spliced according to the channel dimension through the second prediction network, and the spliced features are convolved to obtain the second prediction features. Alternatively, all the reconstructed features in the reference features of the i-th stage sub-block are added through the second prediction network, and the added features are convolved to obtain the second prediction features. The reference features include all the reconstructed features of the previous i-1 stage sub-blocks; or, part of the reconstructed features of the previous i-1 stage sub-blocks; or, the reconstructed features of the i-1-th stage sub-block.
[0207] For example, referring to FIG8G , which is a schematic diagram of the structure of the second prediction network, the second prediction network can be composed of a feature splicing layer and a convolution layer, and there is no restriction on the structure of the second prediction network. If the reference feature includes all the reconstructed features of the previous i-1 stage sub-blocks (such as y_hat_1 to y_hat_i-1), the feature splicing layer can perform feature splicing (concat) on these reconstructed features according to the channel dimension to obtain the spliced features. Alternatively, if the reference feature includes part of the reconstructed features of the previous i-1 stage sub-blocks (such as the part from y_hat_1 to y_hat_i-1), the feature splicing layer can perform feature splicing on these reconstructed features according to the channel dimension to obtain the spliced features. Alternatively, if the reference feature includes the reconstructed features of the i-1th stage sub-block, the feature splicing layer can directly use the reconstructed features of the i-1th stage sub-block as the spliced features.
[0208] The feature splicing layer can input the spliced features into the convolution layer, and the convolution layer performs a convolution operation on the spliced features to obtain the second prediction features. The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer can be selected. For the convolution layer in the second prediction network, it can be an ordinary 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation of the grouped convolution can be described in the subsequent embodiments.
[0209] For another example, the second prediction network can be composed of a feature addition layer and a convolution layer. If the reference feature includes all the reconstructed features of the previous i-1 stage sub-blocks, the feature addition layer adds (sums) these reconstructed features to obtain the added features. Alternatively, if the reference feature includes part of the reconstructed features of the previous i-1 stage sub-blocks, the feature addition layer adds these reconstructed features to obtain the added features. Alternatively, if the reference feature includes the reconstructed features of the i-1th stage sub-block, the feature addition layer can directly use the reconstructed features of the i-1th stage sub-block as the added features. The feature addition layer inputs the added features to the convolution layer, and the convolution layer performs a convolution operation on the added features to obtain the second prediction feature.
[0210] In order to control the cache size, the number of reconstructed features included in the reference features cannot exceed a preset number. Assuming the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference features include all reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, if the number of reconstructed features is greater than M, the reference features include M reconstructed features from y_hat_1 to y_hat_i-1. For example, M reconstructed features can be selected based on a default strategy, for example, the M reconstructed features that are closest to the sub-block in the current stage in decoding order can be selected. Alternatively, the index values of the M reconstructed features can be obtained by decoding the bitstream, and the M reconstructed features can be selected based on the decoded index values.
[0211] For the prediction fusion network: a convolution operation and at least one fusion operation are performed on the concatenated features (i.e., the concatenated features of the first prediction features and the second prediction features) through the prediction fusion network to obtain the mean feature mu_i of the i-th stage sub-block; wherein, for each fusion operation, the fusion operation may include an activation operation and a convolution operation. Exemplarily, the activation operation is a Relu operation.
[0212] For example, see Figure 8H, which is a schematic diagram of the structure of a prediction fusion network. The prediction fusion network can be composed of a convolutional layer and N fusion layers, where N is a positive integer, such as 1 or 2. Each fusion layer can be composed of an activation layer and a convolutional layer. There is no restriction on the structure of this prediction fusion network. Obviously, after the input features (i.e., the concatenated features) pass through the convolutional layer and N "activation layers + convolutional layers", the output features can be obtained, i.e., the mean feature mu_i of the sub-block at the i-th stage.
[0213] The convolution layer can be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, without limitation. For example, a 1x1 convolution layer can be selected. The convolution layer in the prediction fusion network can be a normal 2D convolution or a grouped convolution, such as a 2x grouped convolution. The implementation of grouped convolution is described in the subsequent embodiments. The activation layer can be a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. For example, a Relu layer can be selected.
[0214] Embodiment 9: For the encoding end and the decoding end, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage. For example, in Embodiment 5 and Embodiment 6, the clustering and blocking unit can perform feature aggregation on the reconstruction features (y_hat_1 to y_hat_N) of the N stage sub-blocks to obtain aggregated features (also referred to as full-size aggregated features y_hat_all), and block the aggregated features y_hat_all to obtain at least one block feature y_hat_p, such as block features y_hat_p_1 to block features y_hat_p_K. For each block feature y_hat_p, the clustering and blocking unit inputs the block feature y_hat_p to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p. The merging unit can obtain multiple block-reconstructed image blocks x_hat_p corresponding to the block features y_hat_p, and merge these block-reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.
[0215] First, the clustering block unit aggregates the reconstructed features of the N stage sub-blocks to obtain the aggregated features.
[0216] For example, as shown in Figure 9A , the reconstructed features (y_hat_1 to y_hat_N) of the N phase sub-blocks can be aggregated according to phase to obtain the aggregated feature y_hat_all. Figure 9A illustrates this using N=4 as an example. For example, an example of phase-based aggregation can be shown below: after obtaining y_hat_group1, y_hat_group1 can be used as the aggregated feature y_hat_all.
[0217] For example, as shown in FIG9B , the reconstructed features (y_hat_1 to y_hat_N) of the N-stage sub-blocks can be aggregated according to phase to obtain multiple phase-aggregated features, and the multiple phase-aggregated features can be spliced according to channels to obtain the aggregated feature y_hat_all. For example, an example of aggregation according to phase is shown in the above expression. According to the phase aggregation, y_hat_group1 and y_hat_group2 are obtained. If there are many reconstructed features, after obtaining y_hat_group1 and y_hat_group2, based on the multiple aggregated y_hat_groups, they can be spliced according to channels to obtain the aggregated feature y_hat_all.
[0218] Exemplarily, before the reconstructed features of the N stage sub-blocks are aggregated, useless feature values can also be removed. For example, since the height or width of the full-size feature is an odd number, when the encoding side is divided into four, in order to ensure that the size of the reconstructed features of the sub-blocks of each stage is the same, the reconstructed features of some stage sub-blocks need to be padded. The filling scheme can be to add a column to the right or a row below the full-size feature. Based on this, after the reconstructed features of the N stage sub-blocks are aggregated, useless features need to be removed. As shown in Figure 9C, an example with an odd width is given. If the height is an odd number, the bottom row is removed. Of course, the above is a case of dividing into four. If there is a division into 16, the features removed may be more than one row or one column.
[0219] Second, the clustering block unit blocks the aggregated feature y_hat_all to obtain K block features y_hat_p.
[0220] For example, the aggregated feature y_hat_all can also be partitioned to reduce subsequent memory overhead (the synthetic transformation operation requires multiple upsampling of the feature, which makes the feature data size exceed the memory capacity). Of course, this feature partitioning process is not necessary. If the size of the current feature is small, or it is determined based on the bitstream information that the current feature does not need to be partitioned, the subsequent partitioning process will be skipped directly, and y_hat_p will be equal to y_hat_all, that is, there is only one partitioned feature.
[0221] In order to block the aggregated feature y_hat_all, a feature blocking method without overlapping parts can be adopted. See FIG9D , which is an example of dividing the aggregated feature y_hat_all into four block features y_hat_p.
[0222] In the feature blocking method without overlapping parts, the target size of the block feature can be determined, and based on the target size, the aggregated feature y_hat_all can be evenly divided into multiple block features in order from top to bottom and from left to right, and the size of each block feature is the target size, such as the block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.
[0223] For example, the target size is the size of each block feature. The target size can be the size read from the bitstream or determined based on the feature size to be currently blocked. There is no restriction on this. If the number of channels of each block feature is tile_c, the width is tile_w, and the height is tile_h, then the aggregated feature y_hat_all is evenly divided into multiple tile_c*tile_w*tile_h block features in the order from top to bottom and from left to right, such as block features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.
[0224] In order to block the aggregated feature y_hat_all, a feature blocking method with overlapping parts can be used. See FIG9E , which is an example of dividing the aggregated feature y_hat_all into four block features y_hat_p.
[0225] In the feature blocking method with overlapping parts, the actual blocking size and overlapping size of the blocking feature can be determined. The actual blocking size can be the blocking size of the blocking feature after removing the overlapping parts, and the overlapping size can be the size of the overlapping parts of adjacent blocks. Based on the actual blocking size and overlapping size, the aggregated feature is divided into multiple blocking features, and the size of each blocking feature is the target size, such as the blocking features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.
[0226] For example, the actual block size and overlap size are determined. The actual block size is the block size after removing the overlapping part, and the overlap size is the size of the overlapping part of adjacent blocks. These size information can be read from the code stream or determined based on the feature size of the current block to be segmented. There is no restriction on this. Referring to Figure 9E, if the actual block size is tile_c*tile_w*tile_h and the overlap size is tile_c*padding_w*padding_h, then when overlapping up and down and left and right, the size of each block feature (i.e., the target size) is tile_c*(tile_w+2*padding_w)*(tile_h+2*padding_h). Based on this, for the aggregated feature y_hat_all, the features from the 1st to (tile_w+2*padding_w)th columns and the 1st to (tile_h+2*padding_h)th rows are classified as the 1st block feature. The features of the tile_h+padding_h+1 to (2*tile_h+3*padding_h) rows in the tile_w+padding_w+1 to 2*tile_w+3*padding_w columns are divided into the second block features, and so on.
[0227] As shown in FIG9E , the 4x4x4 aggregated feature y_hat_all can be divided into four 4x3x3 block features. In the block features, tile_c is 4, padding_w and padding_h are 1, and tile_w and tile_h are 1.
[0228] For example, if the number of tiles in the last row or column is insufficient, the overlapped area can be increased to ensure that the final tile size meets the requirements. As shown in Figure 9F , an example is shown where the size of y_hat_all is 11x13, padding_w and padding_h are 1, and tile_w and tile_h are 3. The first and second tiles in the first row are 5x5 in size, and the third tile is expanded to the left by two overlapping columns to ensure that the tile size is also 5x5.
[0229] Third, the clustering block unit inputs each block feature y_hat_p (such as K block features y_hat_p) to the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to each block feature y_hat_p.
[0230] For each block feature y_hat_p, after the block feature y_hat_p is input into the synthesis transformation network, the block feature y_hat_p is synthesized and transformed by the synthesis transformation network to obtain the block reconstructed image block x_hat_p, which is not restricted.
[0231] Fourth, the merging unit merges the multiple reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.
[0232] Exemplarily, the merging unit may obtain K block-reconstructed image blocks x_hat_p corresponding to the K block features y_hat_p, and merge the K block-reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.
[0233] For example, if the aggregated feature y_hat_all is partitioned using a feature partitioning method without overlapping portions, then the merging unit merges the K partitioned reconstructed image blocks x_hat_p using a feature merging method without overlapping portions. For example, the partitioned reconstructed image blocks x_hat_p corresponding to the K partitioned features are sorted from top to bottom and from left to right, and the partitioned reconstructed image blocks x_hat_p corresponding to the sorted K partitioned features are sequentially spliced together to obtain the reconstructed image block x_hat corresponding to the current image block. The splicing process can be the inverse of that of FIG. 9D and will not be described in detail here.
[0234] For example, if the aggregated feature y_hat_all is divided into blocks using a feature block division method with overlapping parts, then the merging unit merges the K block-reconstructed image blocks x_hat_p using a feature merging method with overlapping parts. For example, the merging unit can determine the actual block size and overlap size of each block-reconstructed image block x_hat_p. The merging unit can sort the block-reconstructed image blocks x_hat_p corresponding to the K block features y_hat_p in order from top to bottom and from left to right; based on the actual block size and overlap size, the block-reconstructed image blocks x_hat_p corresponding to the sorted K block features y_hat_p are sequentially spliced to obtain the reconstructed image block x_hat corresponding to the current image block. The reconstructed image block can include K block-reconstructed image blocks, the size of the non-overlapping portion of the block-reconstructed image block can be the actual block size of the block-reconstructed image block, and the size of the overlapping portion of the block-reconstructed image block with the adjacent block-reconstructed image block can be the overlap size of the block-reconstructed image block. The splicing process may be the reverse process of FIG. 9E , which will not be described in detail here.
[0235] For example, the actual tile size and overlap size are determined. The actual tile size is the tile size after removing the overlapped portion, and the overlap size is the size of the overlapped portion of adjacent tiles. These size information can be read from the bitstream or determined based on the characteristic size of the current tile to be tiled, without limitation. Assuming the actual tile size is x_c*xtile_w*xtile_h, the overlap width in the overlap size is xpadding_w, and the overlap height in the overlap size is xpadding_h, then the multiple reconstructed tiled image blocks x_hat_p are sequentially spliced in the order in which they were received.
[0236] For the overlapping portion of the left and right reconstructed image blocks, the value of the overlapping portion is the value of the left reconstructed image block, and the value of the right reconstructed image block is discarded. That is, the value of the earliest (left) reconstructed image block is used, and the values of the subsequent overlapping portions of the sub-blocks are directly discarded. Alternatively, the value of the overlapping portion is the average of the two reconstructed image blocks.
[0237] For the overlapping portion of the upper and lower reconstructed image blocks, the value of the overlapping portion is the value of the upper reconstructed image block, and the value of the lower reconstructed image block is discarded. That is, the value of the earliest (upper) reconstructed image block is used, and the values of the subsequent overlapping portions of the sub-blocks are directly discarded. Alternatively, the value of the overlapping portion is the average of the two reconstructed image blocks.
[0238] Example 10: For the encoding end and the decoding end, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks in each stage. For example, the clustering block unit can perform feature aggregation on the reconstruction features (y_hat_1 to y_hat_N) of the N stage sub-blocks to obtain the aggregated features (also called the full-size aggregated features y_hat_all). The clustering block unit inputs the aggregated features y_hat_all to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block corresponding to the aggregated features y_hat_all. The reconstructed image block is also the reconstructed image block x_hat corresponding to the current image block. Compared with Example 9, it does not involve blocking the aggregated features y_hat_all, nor does it involve merging multiple block reconstructed image blocks x_hat_p.
[0239] First, the clustering block unit aggregates the reconstructed features of the N stage sub-blocks to obtain the aggregated features.
[0240] For example, the reconstructed features (y_hat_1 to y_hat_N) of the N stage sub-blocks can be aggregated according to phase to obtain the aggregated feature y_hat_all. Alternatively, the reconstructed features (y_hat_1 to y_hat_N) of the N stage sub-blocks can be aggregated according to phase to obtain multiple phase-aggregated features, and the multiple phase-aggregated features are spliced according to channels to obtain the aggregated feature y_hat_all. Feature aggregation can be seen in Example 9 and will not be repeated here.
[0241] Second, the clustering block unit inputs the aggregated feature y_hat_all into the synthetic transformation network, and performs a synthetic transformation on the aggregated feature y_hat_all through the synthetic transformation network. There is no restriction on the synthetic transformation method. The synthetic transformation network outputs the reconstructed image block corresponding to the aggregated feature y_hat_all, that is, the reconstructed image block x_hat corresponding to the current image block.
[0242] Example 11: For the encoding end and the decoding end, it is necessary to determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage. For example, after obtaining the reconstruction features of the N stage sub-blocks (y_hat_1 to y_hat_N), the clustering block unit does not aggregate the reconstructed features, nor does it block the aggregated features. Instead, it directly inputs the reconstruction features of each stage sub-block into the synthesis transformation network, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to each stage sub-block. The merging unit can obtain the block reconstructed image blocks x_hat_p corresponding to multiple stage sub-blocks, and merge these block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat corresponding to the current image block.
[0243] First, after obtaining the reconstruction features of N stage sub-blocks, the clustering block unit inputs the reconstruction features of each stage sub-block into the synthesis transformation network, and performs a synthesis transformation on the reconstruction features through the synthesis transformation network. There is no restriction on this synthesis transformation method, and the synthesis transformation network outputs the block reconstructed image block x_hat_p corresponding to each stage sub-block.
[0244] Second, the merging unit merges the multiple reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.
[0245] Exemplarily, the merging unit may obtain N sub-block reconstructed image blocks x_hat_p corresponding to the N stage sub-blocks, and merge the N sub-block reconstructed image blocks x_hat_p to obtain the reconstructed image block x_hat.
[0246] Exemplarily, the merging unit merges the N block-reconstructed image blocks x_hat_p using a feature merging method in which there is no overlapping part. For example, the N block-reconstructed image blocks x_hat_p are sorted in order from top to bottom and from left to right, and the sorted N block-reconstructed image blocks x_hat_p are sequentially spliced to obtain the reconstructed image block x_hat corresponding to the current image block. Alternatively, the merging unit merges the N block-reconstructed image blocks x_hat_p using a feature merging method in which there is overlapping part. For example, the actual block size and overlap size of each block-reconstructed image block x_hat_p are determined, and the N block-reconstructed image blocks x_hat_p are sorted in order from top to bottom and from left to right; based on the actual block size and overlap size, the sorted N block-reconstructed image blocks x_hat_p are sequentially spliced to obtain the reconstructed image block x_hat corresponding to the current image block. The reconstructed image block includes N block-reconstructed image blocks. The size of the non-overlapping portion of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping portion of the block-reconstructed image block and the adjacent block-reconstructed image block is the overlapping size of the block-reconstructed image block. For the overlapping portion of the left and right block-reconstructed image blocks, the value of the overlapping portion is the value of the left block-reconstructed image block, and the value of the right block-reconstructed image block is discarded. Or the value of the overlapping portion is the average of the two block-reconstructed image blocks. For the overlapping portion of the upper and lower block-reconstructed image blocks, the value of the overlapping portion is the value of the upper block-reconstructed image block, and the value of the lower block-reconstructed image block is discarded. Or the value of the overlapping portion is the average of the two block-reconstructed image blocks.
[0247] Example 12: In Examples 1 to 11, a synthetic transformation network is involved. In order to perform precise bit rate control, the encoder can send rate control parameters for the synthetic transformation network to the decoder. For example, the encoder can encode the rate control parameters in the auxiliary bit stream corresponding to the current image block. The decoder can decode the auxiliary bit stream corresponding to the current image block to obtain the rate control parameters and determine the synthetic transformation network based on the rate control parameters, that is, determine the synthetic transformation network based on the bit stream information, especially the bit rate related information. The synthesis transformer network can be composed of at least one 2x deconvolution layer (e.g., a 2x2, 3x3, 4x4, or 5x5 2x deconvolution layer, e.g., a 4x4 2x deconvolution layer can be selected, or other multiples of deconvolution layer are also possible), at least one activation layer (e.g., a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, e.g., a Relu layer can be selected), and at least one convolution layer (e.g., a 1x1, 3x3, or 5x5 2x convolution layer, e.g., a 3x3 2x convolution layer can be selected. The convolution layer in the synthesis transformer network can be a normal 2D convolution or a grouped convolution, e.g., a 2x grouped convolution). The spatial resolution of the output features of the synthesis transformer network is a positive integer multiple of the spatial resolution of the input features, e.g., 16 times (16 times both the length and width), and the number of output feature channels is 1 or 3. Based on the synthesis transformer network, the synthesis transformer network can be controlled based on a rate control parameter, i.e., the rate control parameter is used as the input of the synthesis transformer network.
[0248] In one possible implementation, with respect to Example 9, the clustering and blocking unit inputs K block features y_hat_p into the synthesis transformation network. Therefore, the encoder encodes K rate control parameters corresponding to the K block features y_hat_p in the auxiliary bitstream corresponding to the current image block. There is no restriction on the source of the K rate control parameters. When the encoder uses these K rate control parameters to control the analysis transformation network and the synthesis transformation network (i.e., the K rate control parameters are input to the analysis transformation network and the synthesis transformation network, such as the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p are input together into the synthesis transformation network as input features), the bitrate corresponding to the bitstream of the current image block can be made to be the target bitrate or close to the target bitrate (i.e., the difference between the two is less than a threshold). The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the K rate control parameters corresponding to the K block features y_hat_p.
[0249] Based on this, the clustering block unit can obtain K block features y_hat_p and K rate control parameters corresponding to the K block features y_hat_p. For each block feature y_hat_p, the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p are input into the synthesis transformation network together, and the synthesis transformation network generates the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p.
[0250] Exemplarily, the synthetic transformation network generates a block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the block feature y_hat_p and the rate control parameter corresponding to the block feature y_hat_p, which may include but is not limited to: processing the block feature y_hat_p through the synthetic transformation network to obtain a first feature; processing the rate control parameter corresponding to the block feature y_hat_p through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p based on the third feature.
[0251] In one possible implementation, with respect to Example 10, the clustering and blocking unit inputs the aggregated feature y_hat_all into the synthesis transformation network. Consequently, the encoder encodes a rate control parameter corresponding to the aggregated feature y_hat_all in the auxiliary bitstream corresponding to the current image block, i.e., the rate control parameter corresponding to the current image block. The source of this rate control parameter is not restricted. When the encoder uses this rate control parameter to control the analysis transformation network and the synthesis transformation network (i.e., the rate control parameter serves as the input to the analysis transformation network and the synthesis transformation network, e.g., the aggregated feature y_hat_all is input together with the rate control parameter into the synthesis transformation network, and the current image block is input together with the rate control parameter into the analysis transformation network as input features), the rate corresponding to the bitstream of the current image block can be brought to a target rate or close to the target rate. The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the rate control parameter corresponding to the current image block.
[0252] Based on this, the clustering block unit can obtain the aggregated feature y_hat_all and the rate control parameter, and input the aggregated feature y_hat_all and the rate control parameter together into the synthesis transformation network, which generates the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the rate control parameter.
[0253] The synthetic transformation network generates a reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and the bit rate control parameter, which may include: processing the aggregated feature y_hat_all through the synthetic transformation network to obtain a first feature; processing the bit rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the third feature.
[0254] In one possible implementation, with respect to Example 11, the clustering and blocking unit inputs the reconstructed features of the N staged sub-blocks into the synthesis transform network. Consequently, the encoder encodes N rate control parameters corresponding to the N staged sub-blocks in the auxiliary bitstream corresponding to the current image block. The source of the N rate control parameters is not restricted. When the encoder uses these N rate control parameters to control the analysis transform network and the synthesis transform network (i.e., the N rate control parameters are input to the analysis transform network and the synthesis transform network, such as the reconstructed features of the staged sub-blocks and the rate control parameters corresponding to the staged sub-blocks are input to the synthesis transform network together), the bitrate corresponding to the bitstream of the current image block can be brought to a target bitrate or close to the target bitrate. The decoder decodes the auxiliary bitstream corresponding to the current image block to obtain the N rate control parameters corresponding to the N staged sub-blocks.
[0255] Based on this, the clustering block unit can obtain the reconstruction features of N stage sub-blocks and the N rate control parameters corresponding to the N stage sub-blocks. For each stage sub-block, the reconstruction features of the stage sub-block and the rate control parameters corresponding to the stage sub-block are input into the synthesis transformation network together. The synthesis transformation network generates the block reconstructed image block x_hat_p corresponding to the stage sub-block based on the reconstruction features of the stage sub-block and the rate control parameters corresponding to the stage sub-block.
[0256] Exemplarily, the synthetic transformation network generates a block-reconstructed image block x_hat_p corresponding to the sub-block at this stage based on the reconstruction features of the sub-block at this stage and the rate control parameters corresponding to the sub-block at this stage, which may include but is not limited to: processing the reconstruction features of the sub-block at this stage through the synthetic transformation network to obtain a first feature; processing the rate control parameters corresponding to the sub-block at this stage through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block-reconstructed image block x_hat_p corresponding to the sub-block at this stage based on the third feature.
[0257] Example 13: Based on Example 12, the synthesis transform network can be referred to as a synthesis transform network controlled by a rate control parameter. The rate control parameter can be denoted as the λ parameter, which is decoded from the auxiliary bitstream corresponding to the current image block. FIG10A shows a schematic diagram of the structure of a synthesis transform network controlled by the λ parameter. This synthesis transform network is a variable bitrate synthesis transform network. Based on this synthesis transform network, output features with 1 or 3 channels can be obtained. Of course, FIG10A is only an example of a synthesis transform network, and the structure of this synthesis transform network is not limited.
[0258] As shown in FIG10A , the synthetic transformation network controlled by the λ parameter may include at least one Conv (convolutional layer) and at least one λ-RSTB (Residual Swin Transformer Block). FIG10A takes 4 convolutional layers and 4 λ-RSTBs as an example. The number of convolutional layers may be more or less, and the number of λ-RSTBs may be more or less. The convolutional layer may be a 1x1, 3x3, or 5x5 convolutional layer, such as a 3x3 convolutional layer. The convolutional layer in the synthetic transformation network may be an ordinary 2D convolution or a grouped convolution, such as a 2x grouped convolution. There is no restriction on the structure of this convolutional layer. λ-RSTB represents an RSTB network layer controlled by the λ parameter. The structure of the RSTB network layer controlled by the λ parameter is shown in FIG10B .
[0259] As shown in Figure 10B, the λ-RSTB may include but is not limited to network layers such as FE (Feature Embedding), λ-STB (Swin Transformer Block), and FU (Feature Un-embedding). Of course, this is just an example and there is no restriction on the structure of this λ-RSTB. FE is a feature embedding layer used to implement feature embedding processing, and FU is a feature un-embedding layer used to implement feature un-embedding processing.
[0260] λ-STB represents the STB network layer by λ parameter control, the structure of the STB network layer by λ parameter control is referring to shown in Figure 10 B, λ-STB can include but not limited to LN (liner, linear), λ-WA (Weight Averaging, weighted average), λ-SWA (Stochastic Weight Averaging, stochastic weighted average), MLP (Multilayer Perceptron, multilayer perceptron) etc., certainly, here is just example, the structure of this λ-STB is not restricted.LN is linear layer, and λ-STB can include at least one LN, and the quantity of this LN is not restricted, and here is taken as example by 4 LN.MLP is multilayer perceptron, and λ-STB can include at least one MLP, and the quantity of this MLP is not restricted, and here is taken as example by 2 MLP.
[0261] The structure of λ-SWA is the same as that of λ-WA. Taking λ-WA as an example here, λ-WA represents the WA network layer controlled by the λ parameter. The structure of the WA network layer controlled by the λ parameter can be seen in Figure 10C. λ-WA may include but is not limited to FC (Full Connection) layer, MatMul (matrix multiplication), Scale, SoftMax (logistic regression, also known as normalization), MatMul, etc. Of course, this is just an example and there is no limitation on the structure of λ-WA.
[0262] The parameter λ may be mapped into a scaling factor through the FC layer, that is, the rate control parameter is processed through the FC layer of the synthetic transform network to obtain a second feature, which may be a scaling factor.
[0263] The input features (such as block features y_hat_p, or aggregated features y_hat_all, or reconstructed features of stage sub-blocks) can be processed through the transformer network to obtain Q features, K features, and V features. The Q features, K features, and V features all originate from the input features themselves and are feature vectors generated by multiplying the input features by corresponding weights. There is no restriction on this process. That is to say, the synthetic transformation network can include a transformer network, and the input features (such as block features, or aggregated features, or reconstructed features) are processed through the transformer network of the synthetic transformation network to obtain the first feature.
[0264] After obtaining the Q, K, and V features, matrix multiplication, scaling, and logistic regression operations can be performed on the Q and K features in sequence to obtain intermediate feature 1. The scaling coefficient (obtained by the FC layer mapping the parameter λ) can be multiplied by the V feature to obtain intermediate feature 2. Matrix multiplication of intermediate features 1 and 2 obtains the output feature of λ-WA. In summary, the scaling coefficient is the second feature, and the Q, K, and V features are the first features. Therefore, the third feature can be generated based on the first and second features. The third feature is the output feature of λ-WA.
[0265] After obtaining the output features of λ-WA, combined with Figure 10A and Figure 10B, the output features of the synthetic transformation network can be determined based on the output features of λ-WA (i.e., the third features), such as the block reconstructed image block x_hat_p corresponding to the block feature y_hat_p, the reconstructed image block x_hat corresponding to the aggregated feature y_hat_all, and the block reconstructed image block x_hat_p corresponding to the stage sub-block.
[0266] Referring to λ-WA shown in FIG10C , the calculation method of λ-WA can be as follows. In this way, the output of the image codec can be controlled by using the λ parameter, thereby obtaining bitstreams of different sizes and reconstructed images of different qualities.
[0267] Example 14: In Examples 1 to 13, a convolution layer may be involved. The convolution layer may be a normal 2D convolution or a grouped convolution, such as a 2x grouped convolution. For example, the steps of a normal 2D convolution can be shown in FIG11A , where the input feature is (H×W×C). Then, C' filters are applied (each filter has a size of (h×w×C), where the filter size C is the same as the input feature C), and the input layer is converted into an output feature of size (H'×W'×C').
[0268] The steps of grouped convolution can be seen in Figure 11B. For each filter group, it can contain C' / g (such as g is 2) filters, and the number of channels of each filter is half of that of the ordinary 2D convolution filter. Each filter group acts on 1 / g of the original number of channels corresponding to W×H×C, that is, W×H×C / g. In this way, the output of each filter group corresponds to the features of C' / g channels. Finally, the channels can be stacked to obtain the final C' channels, thereby achieving the same effect as the above-mentioned ordinary 2D convolution. Obviously, through grouped convolution, the number of parameters and the amount of calculation can be effectively reduced.
[0269] Example 15: In Example 1-Example 14, in order to perform precise bit rate control, the encoder needs to send bit rate control parameters to the decoder. For example, the bit rate control parameters are carried and sent via the auxiliary bit stream corresponding to the current image block. At the encoder, the bit rate control parameters affect the analysis transform network and the synthesis transform network. In order to make the bit rate corresponding to the bit stream of the current image block the target bit rate or close to the target bit rate (pre-configured bit rate), the encoder can try different bit rate control parameters to find the target bit rate control parameters. Under the target bit rate control parameters, the bit rate corresponding to the bit stream of the current image block is the target bit rate or close to the target bit rate, and the encoder encodes the target bit rate control parameters in the auxiliary bit stream corresponding to the current image block.
[0270] If the current image block corresponds to only one rate control parameter, such as the aggregated feature y_hat_all corresponds to one rate control parameter, then each rate control parameter can be tried in turn from multiple rate control parameters until the rate corresponding to the bitstream of the current image block is the target rate or close to the target rate, thereby obtaining the target rate control parameter.
[0271] If the current image block corresponds to multiple rate control parameters, such as K rate control parameters corresponding to K block features or N rate control parameters corresponding to N stage sub-blocks, initial values of the multiple rate control parameters are first set. If the bit rate corresponding to the code stream of the current image block does not meet the requirement, a candidate rate control parameter is selected from the multiple rate control parameters and the value of the candidate rate control parameter is reduced. If the bit rate corresponding to the code stream of the current image block does not meet the requirement, another candidate rate control parameter (the same as or different from the previous candidate rate control parameter) is selected from the multiple rate control parameters and the value of the candidate rate control parameter is reduced. This process is repeated until the bit rate corresponding to the code stream of the current image block meets the requirement (i.e., the bit rate corresponding to the code stream of the current image block is the target rate or close to the target rate), thereby obtaining target values for the multiple rate control parameters.
[0272] When selecting a candidate rate control parameter from multiple rate control parameters, a rate control parameter may be randomly selected as the candidate rate control parameter, the maximum rate control parameter may be selected as the candidate rate control parameter, or the candidate rate control parameter may be selected based on the relationship between rate and distortion. For example, if rate control parameter A decreases from its current level to the next level, the distortion is A1, and if rate control parameter B decreases from its current level to the next level, the distortion is B1. If distortion A1 is less than distortion B1, i.e., distortion A1 is the minimum distortion, rate control parameter A is selected as the candidate rate control parameter.
[0273] In one possible implementation, how does the encoder set the λ for each (feature or image) block? i One implementation of the parameters is as follows: For each block x′i (such as block features or stage sub-blocks), using different λ i Before encoding, several discrete λ values are taken for each block. These discrete λ values are defined as quality levels. The bitrate and distortion of each block and each quality level are then calculated before encoding. This process only needs to be performed once for each frame, before the first encoding.
[0274] During the encoding process, for each block, the quality level λ is selected for encoding. During the encoding process, first, all blocks are encoded using the same quality level, which is the minimum quality level that can make the total bit rate greater than the target bit rate, that is, satisfying the following formula, R t is the target bit rate, It represents the sum of the bit rates of all blocks at quality level q.
[0275] Then, the quality level of each block is gradually reduced until the total bitrate is less than the target bitrate. For each block, the gradient of the current quality level and the previous quality level on the rate-distortion curve can be calculated, that is, the gradient is calculated using the following formula:
[0276] in, is the reconstruction loss of block i at quality level q. When using a loss function that decreases with bit rate (such as mean square error) as a metric, the one with the minimum The block i with the smallest distortion (i.e., the candidate rate control parameter) is selected and its quality level is reduced by one. This process is repeated until the total bit rate is less than the target bit rate.
[0277] As shown in Figure 12, it is a schematic diagram of the decision of the λ parameter. Seven quality levels (i.e., seven values of the λ parameter) are designed on the rate-distortion curve. At quality level 3, it can be calculated That is, at quality level 3, reducing the quality level of block 2 will result in a greater coding performance loss, so the quality level of block 1 is reduced.
[0278] As can be seen from the technical solutions of the above embodiments, this embodiment proposes a variable-rate encoding and decoding scheme for neural network-based encoding and decoding technology. This scheme improves parallelism, effectively saves feature storage cache, achieves higher rate control accuracy, minimizes encoding performance loss, and provides better encoding performance and rate control accuracy. By adopting block encoding and decoding, peak memory usage is reduced, decoding time for a single block is shortened, and high-speed parallel decoding capabilities are enabled. This allows the neural network to maintain low complexity while effectively ensuring the quality of reconstructed image blocks, improving encoding and decoding performance and reducing complexity. For example, the rate control algorithm of this scheme can achieve a rate control accuracy of approximately 99%, while the encoding performance loss is only approximately 1% compared to no rate control. Taking the decoding of a 2k resolution image with a 256*256 block size as an example, the use of block encoding and decoding reduces the peak memory usage to 1 / 10 of that of full-image decoding, while decoding time for a single block is less than 0.1s, enabling high-speed parallel decoding capabilities. This achieves better encoding and decoding performance while adding fewer parameters.
[0279] Example 16: A decoding method is proposed in an embodiment of the present application, which can be applied to a decoding end (also called a video decoder). The method may include: decoding a first code stream of a current image block to obtain coefficient hyperparameter features of the current image block; determining probability distribution parameters based on the coefficient hyperparameter features, decoding a second code stream of the current image block based on the probability distribution parameters to obtain residual features of the current image block, and determining reconstruction features of the current image block based on the residual features; decoding an auxiliary code stream corresponding to the current image block to obtain a rate control parameter corresponding to the current image block, and inputting the reconstruction features and rate control parameters into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block.
[0280] Exemplarily, the reconstruction features and the rate control parameters are input into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block, which may include but is not limited to: processing the reconstruction features through the synthetic transformation network to obtain a first feature; processing the rate control parameters through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.
[0281] For example, based on Example 3 and Example 4, the synthetic transformation network may be adjusted. In Example 16, only the adjustment process of the synthetic transformation network is described. For other processes, please refer to Example 3 and Example 4.
[0282] Exemplarily, precise bit rate control can be achieved through rate control parameters, which affect the analysis transformation network and the synthesis transformation network. For example, when the encoder performs an analysis transformation on the current image block through the analysis transformation network, the rate control parameters and the current image block are input to the analysis transformation network, and the analysis transformation network outputs the image features corresponding to the current image block. When the encoder or decoder inputs the image feature y_hat to the synthesis transformation network, the image feature y_hat and the rate control parameters are input to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block corresponding to the current image block.
[0283] Based on the configured target bitrate, the encoder can try different rate control parameters. After adopting a certain rate control parameter, if the bitrate corresponding to the code stream for the current image block is at or close to the target bitrate, then this rate control parameter is the final rate control parameter adopted, and the encoder can send this rate control parameter to the decoder.
[0284] Exemplarily, the encoding end can send the rate control parameters for the synthesis transformation network to the decoding end. For example, the encoding end can encode the rate control parameters corresponding to the current image block in the auxiliary code stream corresponding to the current image block. The decoding end can decode the auxiliary code stream corresponding to the current image block to obtain the rate control parameters corresponding to the current image block, and determine the synthesis transformation network based on the rate control parameters. The synthesis transformer network can be composed of at least one 2x deconvolution layer (e.g., a 2x2, 3x3, 4x4, or 5x5 2x deconvolution layer, e.g., a 4x4 2x deconvolution layer can be selected, or other multiples of deconvolution layer are also possible), at least one activation layer (e.g., a Relu layer, a leaky Relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, e.g., a Relu layer can be selected), and at least one convolution layer (e.g., a 1x1, 3x3, or 5x5 2x convolution layer, e.g., a 3x3 2x convolution layer can be selected. The convolution layer in the synthesis transformer network can be a normal 2D convolution or a grouped convolution, e.g., a 2x grouped convolution). The spatial resolution of the output features of the synthesis transformer network is a positive integer multiple of the spatial resolution of the input features, e.g., 16 times (16 times both the length and width), and the number of output feature channels is 1 or 3. Based on the synthesis transformer network, the synthesis transformer network can be controlled based on a rate control parameter, i.e., the rate control parameter is used as the input of the synthesis transformer network.
[0285] Exemplarily, the encoder encodes the rate control parameters corresponding to the current image block in the auxiliary bitstream corresponding to the current image block. There is no restriction on the source of the rate control parameters. When the encoder uses the rate control parameters to control the analysis transform network and the synthesis transform network (i.e., the rate control parameters serve as input to the analysis transform network and the synthesis transform network), the rate corresponding to the bitstream of the current image block can be at or close to the target rate. The decoder can decode the auxiliary bitstream corresponding to the current image block to obtain the rate control parameters corresponding to the current image block. Based on this, the decoder can input the image feature y_hat and the rate control parameters together into the synthesis transform network. The synthesis transform network then generates a reconstructed image block corresponding to the current image block based on the image feature y_hat and the rate control parameters, and outputs the reconstructed image block.
[0286] Among them, the synthetic transformation network generates a reconstructed image block corresponding to the current image block based on the image feature y_hat and the bit rate control parameter, which may include: processing the image feature y_hat (i.e., the reconstructed feature y_hat) through the synthetic transformation network to obtain a first feature; processing the bit rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature.
[0287] For example, as shown in Figures 10A, 10B, and 10C, the synthetic transformation network may include at least one Conv (convolutional layer) and at least one λ-RSTB. The λ-RSTB may include but is not limited to network layers such as FE, λ-STB, and FU. The λ-STB may include but is not limited to LN, λ-WA, λ-SWA, and MLP. The structure of λ-SWA is the same as that of λ-WA. λ-WA may include but is not limited to FC layer, MatMul, Scale, SoftMax, and MatMul. On this basis, the parameter λ (i.e., the rate control parameter) can be mapped into a scaling factor through the FC layer, that is, the rate control parameter is processed through the FC layer of the synthetic transformation network to obtain the second feature. The image feature y_hat can be processed through the transformer network to obtain Q features, K features, and V features, that is, the image feature y_hat is processed through the transformer network of the synthetic transformation network to obtain the first feature. After obtaining the Q feature, K feature, and V feature, matrix multiplication, scaling, and logistic regression operations can be performed based on the Q feature and K feature in sequence to obtain intermediate feature 1; the scaling coefficient can be multiplied by the V feature to obtain intermediate feature 2; intermediate feature 1 and intermediate feature 2 are matrix multiplied to obtain the output feature of λ-WA, that is, a third feature is generated based on the first feature and the second feature, and the third feature is the output feature of λ-WA. After obtaining the output feature of λ-WA, in combination with Figures 10A and 10B, the output feature of the synthetic transformation network can be determined based on the output feature of λ-WA (i.e., the third feature). For example, the output feature can be the reconstructed image block corresponding to the current image block.
[0288] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 16 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 16 can be implemented in combination.
[0289] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.
[0290] For example, in the above-mentioned embodiments, the contents of different embodiments can be borrowed from each other, such as the contents of Example 3 can be applied to Example 16, the contents of Example 4 can be applied to Example 16, the contents of Examples 5-15 can be applied to Example 16, and the contents between Examples 5-15 can be borrowed from each other, and there is no restriction on this.
[0291] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The device is applied to the decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 16, that is, the processing flow of the decoding end.
[0292] For example, in one possible implementation, a decoder is configured to implement:
[0293] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;
[0294] For each stage sub-block, determine probability distribution parameters based on coefficient hyperparameter features of the sub-block at that stage, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the sub-block at that stage;
[0295] Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage;
[0296] A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.
[0297] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application. The device is applied to the encoding end, and the device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 16, that is, the processing flow of the encoding end.
[0298] For example, in one possible implementation, the encoder is configured to implement:
[0299] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;
[0300] Splitting the feature block into features to be encoded of multiple stage sub-blocks;
[0301] For each stage sub-block corresponding to the current image block, obtain coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into a first bitstream of the current image block;
[0302] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;
[0303] Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.
[0304] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present application, from a hardware perspective, can be specifically shown in FIG13A as a schematic diagram of its hardware architecture. The decoding end device includes: a processor 1301 and a machine-readable storage medium 1302, the machine-readable storage medium 1302 storing machine-executable instructions that can be executed by the processor 1301; the processor 1301 is used to execute the machine-executable instructions to implement the decoding method of the above embodiments 1-16 of the present application. For example, in one possible implementation, the processor 1301 performs the following steps when executing the machine-executable instructions:
[0305] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;
[0306] For each stage sub-block, determine probability distribution parameters based on coefficient hyperparameter features of the sub-block at that stage, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the sub-block at that stage;
[0307] Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage;
[0308] A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.
[0309] Based on the same application concept as the above method, the encoding end device (also referred to as a video encoder) provided in the embodiment of the present application, from a hardware perspective, its hardware architecture diagram can be specifically shown in Figure 13B. The encoding end device includes: a processor 1311 and a machine-readable storage medium 1312, the machine-readable storage medium 1312 stores machine-executable instructions that can be executed by the processor 1311; the processor 1311 is used to execute the machine-executable instructions to implement the encoding method of the above embodiments 1-16 of the present application. For example, in one possible implementation, the processor 1311 performs the following steps when executing the machine-executable instructions:
[0310] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;
[0311] Splitting the feature block into features to be encoded of multiple stage sub-blocks;
[0312] For each stage sub-block corresponding to the current image block, obtain coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into a first bitstream of the current image block;
[0313] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;
[0314] Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.
[0315] Based on the same application concept as the above-mentioned method, an embodiment of the present application provides an electronic device. The electronic device includes: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the decoding method or encoding method of the above-mentioned embodiments 1-16 of the present application.
[0316] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method or encoding method in the above embodiments.
[0317] Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.
[0318] Based on the same application concept as the above method, an embodiment of the present application also proposes a decoding device, which can be applied to the decoding end. The decoding device includes: a decoding module, used to decode the first code stream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, determine the probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block, decode the second code stream of the current image block based on the probability distribution parameters, and obtain the residual features of the stage sub-block; a determination module, used to determine the reconstruction features of the stage sub-block based on the residual features of the stage sub-block and the mean features of the stage sub-block; determine the reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block.
[0319] Exemplarily, the determination module is also used to obtain, for the first stage sub-block, the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block through a mean prediction network, or to obtain the configured default reference feature, and obtain the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature through a mean prediction network; or, for the i-th stage sub-block, i is greater than 1, obtain the reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; obtain the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through a mean prediction network; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction feature of the i-1th stage sub-block.
[0320] Exemplarily, the mean prediction network includes a first prediction network, a second prediction network and a prediction fusion network. When the determination module obtains the mean feature of the sub-block at this stage through the mean prediction network based on the coefficient hyperparameter feature of the sub-block at this stage and the reference feature, it is specifically used to: obtain the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network; obtain the second prediction feature corresponding to the reference feature through the second prediction network; splice the first prediction feature and the second prediction feature to obtain a spliced feature, input the spliced feature to the prediction fusion network, and process the spliced feature through the prediction fusion network to obtain the mean feature of the sub-block at this stage.
[0321] Exemplarily, when the determination module obtains the first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network, it is specifically used to: perform feature enhancement and upsampling operations on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature; the feature enhancement operation includes: convolution operation, or convolution operation and activation operation; the upsampling operation includes: deconvolution operation, cropping operation and activation operation, or deconvolution operation, cropping operation, activation operation and convolution operation. When the determination module obtains the second prediction feature corresponding to the reference feature through the second prediction network, it is specifically used to: perform feature splicing on all reconstructed features in the reference feature according to the channel dimension; perform convolution operation on the spliced features to obtain the second prediction feature; or perform feature addition on all reconstructed features in the reference feature, and perform convolution operation on the added features to obtain the second prediction feature. The determination module processes the spliced features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage, and is specifically used to: perform a convolution operation and at least one fusion operation on the spliced features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage; wherein the fusion operation includes an activation operation and a convolution operation.
[0322] Exemplarily, the determination module is also used to perform feature enhancement on the reconstruction features of the sub-block of the stage to obtain enhanced reconstruction features; wherein the enhanced reconstruction features are used to determine the mean features of the sub-block of the stage, and the enhanced reconstruction features are used to determine the reconstructed image block.
[0323] Exemplarily, when the determination module determines the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage, it is specifically used to: perform feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain aggregated features; input the aggregated features to the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; or, perform feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain aggregated features; block the aggregated features to obtain multiple block features; input each block feature to the synthesis transformation network to obtain a block reconstructed image block; merge the block reconstructed image blocks corresponding to the multiple block features to obtain the reconstructed image block corresponding to the current image block; or, input the reconstruction features of the sub-blocks at each stage to the synthesis transformation network to obtain a block reconstructed image block; merge the block reconstructed image blocks corresponding to all the sub-blocks at all stages to obtain the reconstructed image block corresponding to the current image block.
[0324] Exemplarily, the determination module performs feature aggregation on the reconstruction features of each stage sub-block to obtain the aggregated features, which is specifically used to: aggregate the reconstruction features of each stage sub-block according to the phase to obtain the aggregated features; or aggregate the reconstruction features of each stage sub-block according to the phase to obtain multiple phase-aggregated features, and splice the multiple phase-aggregated features according to the channel to obtain the aggregated features.
[0325] Exemplarily, when the determination module blocks the aggregated features to obtain multiple block features, it is specifically used to: determine the target size of the block features; based on the target size, divide the aggregated features into multiple block features in an even order from top to bottom and from left to right, and the size of each block feature is the target size; or, determine the actual block size and overlapping size of the block features, the actual block size is the block size of the block feature after removing the overlapping part, and the overlapping size is the size of the overlapping part of adjacent blocks; based on the actual block size and the overlapping size, divide the aggregated features into multiple block features, and the size of each block feature is the target size.
[0326] Exemplarily, the determination module merges the block-reconstructed image blocks corresponding to the multiple block features to obtain the reconstructed image block corresponding to the current image block, and is specifically used to: sort the multiple block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right, and sequentially splice the sorted multiple block-reconstructed image blocks to obtain the reconstructed image block corresponding to the current image block; or determine the actual block size and overlap size of each block-reconstructed image block; sort the multiple block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right; and splice the sorted multiple block-reconstructed image blocks based on the actual block size and the overlap size. The images are stitched together in sequence to obtain a reconstructed image block corresponding to the current image block; wherein the size of the non-overlapping portion of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping portion of the block-reconstructed image block and the adjacent block-reconstructed image block is the overlapping size of the block-reconstructed image block; for the overlapping portion of the left and right block-reconstructed image blocks, the value of the overlapping portion is the value of the left block-reconstructed image block, or the value of the overlapping portion is the average of the two block-reconstructed image blocks; for the overlapping portion of the upper and lower block-reconstructed image blocks, the value of the overlapping portion is the value of the upper block-reconstructed image block, or the value of the overlapping portion is the average of the two block-reconstructed image blocks.
[0327] Exemplarily, when the determination module inputs the aggregated features into a synthesis transformation network to obtain the reconstructed image block corresponding to the current image block, the determination module is specifically configured to: decode the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of the current image block, and input the rate control parameter and the aggregated features into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block; or, when the determination module inputs each block feature into the synthesis transformation network to obtain a block-reconstructed image block, the determination module is specifically configured to: decode the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of each block feature, and input the block feature and the rate control parameter of the block feature into the synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature; or, when the determination module inputs the reconstruction feature of each stage sub-block into the synthesis transformation network to obtain the block-reconstructed image block, the determination module is specifically configured to: decode the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of each stage sub-block, and input the reconstruction feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthesis transformation network to obtain the block-reconstructed image block.
[0328] Exemplarily, the determination module inputs the rate control parameters and the aggregated features into the synthesis transformation network, and when obtaining the reconstructed image block corresponding to the current image block, is specifically used for: processing the aggregated features through the synthesis transformation network to obtain a first feature; processing the rate control parameters through the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature; or, the determination module inputs the block feature and the rate control parameters of the block feature into the synthesis transformation network, and when obtaining the block reconstructed image block, is specifically used for: processing the block feature through the synthesis transformation network to obtain a first feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature. The synthetic transformation network processes the rate control parameter of the block feature to obtain a second feature; generates a third feature based on the first feature and the second feature, and determines the block reconstructed image block corresponding to the block feature based on the third feature; or, the determination module inputs the reconstruction feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthetic transformation network to obtain the block reconstructed image block, and is specifically used for: processing the reconstruction feature of the stage sub-block through the synthetic transformation network to obtain the first feature; processing the rate control parameter of the stage sub-block through the synthetic transformation network to obtain the second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block corresponding to the stage sub-block based on the third feature.
[0329] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application, which is applied to the encoding end, and the device includes: an acquisition module, used to input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks; an encoding module, used to obtain the coefficient hyperparameter features of the stage sub-block for each stage sub-block corresponding to the current image block, and encode the coefficient hyperparameter features of the stage sub-block into the first code stream of the current image block; a determination module, used to determine the residual features of the stage sub-block based on the features to be encoded of the stage sub-block and the mean features of the stage sub-block; the encoding module is also used to determine the probability distribution parameters based on the coefficient hyperparameter features of the stage sub-block, and encode the residual features of the stage sub-block into the second code stream of the current image block based on the probability distribution parameters.
[0330] Exemplarily, the determination module is also used to obtain, for the first stage sub-block, the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block through a mean prediction network, or to obtain the configured default reference feature, and obtain the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature through a mean prediction network; or, for the i-th stage sub-block, i is greater than 1, obtain the reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; obtain the mean feature of the stage sub-block based on the coefficient hyperparameter feature of the stage sub-block and the reference feature through a mean prediction network; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks; or, part of the reconstruction features of the previous i-1 stage sub-blocks; or, the reconstruction feature of the i-1th stage sub-block.
[0331] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The foregoing is merely an embodiment of the present application and is not intended to limit the present application.
[0332] For those skilled in the art, various modifications and variations are possible in this application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of the claims of this application.
Claims
1. A decoding method, characterized in that: The method comprises: Decoding a first bitstream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyper-parameter feature of the sub-block at the stage, and the second bitstream of the current image block is decoded based on the probability distribution parameter to obtain the residual feature of the sub-block at the stage; Determine the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage; A reconstructed image block corresponding to the current image block is determined based on the reconstruction features of the sub-blocks in each stage.
2. The method according to claim 1, characterized in that Before determining the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage, the method further includes: For the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through the mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the default reference feature.
3. The method according to claim 1 or 2, characterized in that: Before determining the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage, the method further includes: For the i-th stage sub-block, where i is greater than 1, a reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block; a mean feature of the stage sub-block is obtained through a mean prediction network based on the coefficient hyper-parameter features of the stage sub-block and the reference features; wherein the reference features include all reconstruction features of the previous i-1 stage sub-blocks; or, partial reconstruction features of the previous i-1 stage sub-blocks; or, reconstruction features of the i-1-th stage sub-block.
4. The method according to claim 2 or 3, characterized in that: The mean prediction network includes a first prediction network, a second prediction network and a prediction fusion network, and the coefficient hyper-parameter feature of the sub-block at this stage and the reference feature are used to obtain the mean feature of the sub-block at this stage through the mean prediction network, including: Acquire a first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network; Acquire, by the second prediction network, a second prediction feature corresponding to the reference feature; The first prediction feature and the second prediction feature are concatenated to obtain a concatenated feature, the concatenated feature is input to the prediction fusion network, and the concatenated feature is processed by the prediction fusion network to obtain a mean feature of the sub-block at this stage.
5. The method according to claim 4, characterized in that The obtaining, through the first prediction network, a first prediction feature corresponding to the coefficient hyperparameter feature comprises: performing a feature enhancement operation and an upsampling operation on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature; The feature enhancement operation includes: a convolution operation, or a convolution operation and an activation operation; the upsampling operation includes: a deconvolution operation, a cropping operation and an activation operation, or a deconvolution operation, a cropping operation, an activation operation and a convolution operation.
6. The method according to claim 5, characterized in that The activation operation is a Relu operation.
7. The method according to claim 4, characterized in that The acquiring, by the second prediction network, a second prediction feature corresponding to the reference feature includes: All reconstructed features in the reference features are spliced according to the channel dimension; the spliced features are convolved to obtain the second prediction feature; or all reconstructed features in the reference features are added, and the added features are convolved to obtain the second prediction feature.
8. The method according to claim 4, characterized in that The step of processing the spliced features through the prediction fusion network to obtain the mean features of the sub-blocks at this stage includes: The prediction fusion network is used to perform a convolution operation and at least one fusion operation on the spliced features to obtain the mean features of the sub-blocks at this stage; wherein the fusion operation includes an activation operation and a convolution operation.
9. The method according to claim 8, characterized in that The activation operation is a Relu operation.
10. The method according to any one of claims 1 to 9, characterized in that After determining the reconstruction feature of the sub-block at this stage based on the residual feature of the sub-block at this stage and the mean feature of the sub-block at this stage, the method includes: The reconstruction features of the sub-blocks at this stage are enhanced to obtain enhanced reconstruction features; wherein the enhanced reconstruction features are used to determine the mean features of the sub-blocks at the stage, and the enhanced reconstruction features are used to determine the reconstructed image blocks.
11. The method according to any one of claims 1 to 9, characterized in that The determining the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage includes: Performing feature aggregation on the reconstructed features of the sub-blocks at each stage to obtain aggregated features; inputting the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; Alternatively, feature aggregation is performed on the reconstructed features of the sub-blocks at each stage to obtain aggregated features; the aggregated features are divided into blocks to obtain multiple block features; each block feature is input into the synthetic transformation network to obtain a block-reconstructed image block; the block-reconstructed image blocks corresponding to the multiple block features are merged to obtain a reconstructed image block corresponding to the current image block; Alternatively, the reconstruction features of each stage sub-block are input into the synthetic transformation network to obtain a block-by-block reconstructed image block; the block-by-block reconstructed image blocks corresponding to all stage sub-blocks are merged to obtain a reconstructed image block corresponding to the current image block.
12. The method according to claim 11, characterized in that The reconstructed features of the sub-blocks at each stage are aggregated to obtain aggregated features, including: Aggregate the reconstructed features of the sub-blocks at each stage according to the phase to obtain the aggregated features; or, The reconstructed features of the sub-blocks in each stage are aggregated according to the phase to obtain a plurality of phase-aggregated features, and the plurality of phase-aggregated features are spliced according to the channel to obtain the aggregated features.
13. The method according to claim 11, characterized in that The step of dividing the aggregated features into blocks to obtain a plurality of block features includes: Determine a target size of the block feature; based on the target size, divide the aggregated feature into a plurality of block features in an order from top to bottom and from left to right, and the size of each block feature is the target size; or, Determine the actual block size and overlap size of the block feature, wherein the actual block size is the block size of the block feature excluding the overlapped part, and the overlap size is the size of the overlapped part of adjacent blocks; divide the aggregated feature into multiple block features based on the actual block size and the overlap size, and the size of each block feature is the target size.
14. The method according to claim 13, characterized in that The step of merging the block-reconstructed image blocks corresponding to the plurality of block features to obtain the reconstructed image block corresponding to the current image block includes: Sorting the multiple block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right, and sequentially splicing the sorted multiple block-reconstructed image blocks to obtain a reconstructed image block corresponding to the current image block; or, Determine the actual block size and overlap size of each block-reconstructed image block; sort the multiple block-reconstructed image blocks corresponding to the multiple block features in order from top to bottom and from left to right; based on the actual block size and the overlap size, sequentially splice the sorted multiple block-reconstructed image blocks to obtain a reconstructed image block corresponding to the current image block; wherein the size of the non-overlapping portion of the block-reconstructed image block is the actual block size of the block-reconstructed image block, and the size of the overlapping portion of the block-reconstructed image block and the adjacent block-reconstructed image block is the overlap size of the block-reconstructed image block; For the overlapping part of the left and right sub-block reconstructed image blocks, the value of the overlapping part is the value of the left sub-block reconstructed image block, or the value of the overlapping part is the average value of the two sub-block reconstructed image blocks; For the overlapping portion of the upper and lower sub-block reconstructed image blocks, the value of the overlapping portion is the value of the upper sub-block reconstructed image block, or the value of the overlapping portion is the average value of the two sub-block reconstructed image blocks.
15. The method according to any one of claims 11 to 14, characterized in that The aggregated features are input into a synthetic transformation network to obtain a reconstructed image corresponding to the current image block. The block comprises: decoding the auxiliary code stream corresponding to the current image block to obtain a rate control parameter of the current image block, inputting the rate control parameter and the aggregated features into the synthesis transformation network to obtain a reconstructed image block corresponding to the current image block; or The step of inputting each block feature to the synthesis transformation network to obtain a block-reconstructed image block comprises: decoding the auxiliary bitstream corresponding to the current image block to obtain a rate control parameter of each block feature, inputting the block feature and the rate control parameter of the block feature to the synthesis transformation network to obtain a block-reconstructed image block corresponding to the block feature; or The step of inputting the reconstruction features of each stage sub-block into the synthesis transformation network to obtain the block-by-block reconstructed image block comprises: decoding the auxiliary code stream corresponding to the current image block to obtain the rate control parameter of each stage sub-block, inputting the reconstruction features of the stage sub-block and the rate control parameter of the stage sub-block into the synthesis transformation network to obtain the block-by-block reconstructed image block.
16. The method according to claim 15, characterized in that Inputting the rate control parameter and the aggregated feature into the synthesis transformation network to obtain the reconstructed image block corresponding to the current image block includes: processing the aggregated feature through the synthesis transformation network to obtain a first feature; processing the rate control parameter through the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the reconstructed image block corresponding to the current image block based on the third feature; or, The step of inputting the block feature and the rate control parameter of the block feature to the synthesis transformation network to obtain the block reconstructed image block corresponding to the block feature comprises: processing the block feature through the synthesis transformation network to obtain a first feature; processing the rate control parameter of the block feature through the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block reconstructed image block corresponding to the block feature based on the third feature; or The step of inputting the reconstruction feature of the stage sub-block and the rate control parameter of the stage sub-block into the synthesis transformation network to obtain the block-reconstructed image block includes: processing the reconstruction feature of the stage sub-block through the synthesis transformation network to obtain a first feature; processing the rate control parameter of the stage sub-block through the synthesis transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining the block-reconstructed image block corresponding to the stage sub-block based on the third feature.
17. A coding method, characterized in that: The method comprises: Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block; Splitting the feature block into features to be encoded of multiple stage sub-blocks; For each stage sub-block corresponding to the current image block, obtain the coefficient hyper-parameter feature of the stage sub-block, and encode the coefficient hyper-parameter feature of the stage sub-block into the first bitstream of the current image block; Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage; Probability distribution parameters are determined based on the coefficient hyperparameter features of the sub-block at this stage, and the residual features of the sub-block at this stage are encoded into the second code stream of the current image block based on the probability distribution parameters.
18. The method according to claim 17, characterized in that Before determining the residual feature of the sub-block at this stage based on the feature to be encoded of the sub-block at this stage and the mean feature of the sub-block at this stage, the method further includes: For the first stage sub-block, the mean feature of the sub-block at this stage is obtained through a mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage, or the configured default reference feature is obtained, and the mean feature of the sub-block at this stage is obtained through the mean prediction network based on the coefficient hyper-parameter feature of the sub-block at this stage and the default reference feature.
19. The method according to claim 17 or 18, characterized in that Before determining the residual feature of the sub-block at this stage based on the feature to be encoded of the sub-block at this stage and the mean feature of the sub-block at this stage, the method further includes: For the i-th stage sub-block, i is greater than 1, the reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the i-1 stage sub-blocks before the i-th stage sub-block; the mean feature of the sub-block of this stage is obtained through the mean prediction network based on the coefficient hyper-parameter feature of the sub-block of this stage and the reference feature; wherein the reference feature includes the previous i-1 All the reconstructed features of the sub-blocks in the previous stage; or, part of the reconstructed features of the sub-blocks in the previous i-1 stages; or, the reconstructed features of the sub-block in the i-1th stage.
20. A decoding method, characterized in that: The method comprises: Decoding a first bitstream of a current image block to obtain coefficient hyperparameter features of the current image block; Determine probability distribution parameters based on the coefficient hyperparameter features, decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the current image block, and determine reconstruction features of the current image block based on the residual features; The auxiliary code stream corresponding to the current image block is decoded to obtain a rate control parameter corresponding to the current image block, and the reconstruction feature and the rate control parameter are input into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block.
21. The method according to claim 20, characterized in that The step of inputting the reconstruction feature and the rate control parameter into a synthesis transformation network to obtain a reconstructed image block corresponding to the current image block includes: Processing the reconstructed feature through the synthetic transformation network to obtain a first feature; Processing the rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature; A reconstructed image block corresponding to the current image block is determined based on the third feature.
22. A decoding device, characterized in that: The device comprises: A decoding module is used to decode the first bitstream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, determine the probability distribution parameter based on the coefficient hyperparameter features of the sub-block at the stage, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain the residual features of the sub-block at the stage; The determination module is used to determine the reconstruction features of the sub-blocks at this stage based on the residual features of the sub-blocks at this stage and the mean features of the sub-blocks at this stage; and determine the reconstructed image block corresponding to the current image block based on the reconstruction features of the sub-blocks at each stage.
23. A coding device, characterized in that: The device comprises: The acquisition module is used to input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks; An encoding module, configured to obtain, for each stage sub-block corresponding to the current image block, coefficient hyper-parameter features of the stage sub-block, and encode the coefficient hyper-parameter features of the stage sub-block into a first bitstream of the current image block; A determination module, used to determine the residual features of the sub-block at this stage based on the to-be-encoded features of the sub-block at this stage and the mean features of the sub-block at this stage; The encoding module is further used to determine probability distribution parameters based on coefficient hyperparameter features of the sub-block at this stage, and encode the residual features of the sub-block at this stage into the second code stream of the current image block based on the probability distribution parameters.
24. A decoding end device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method according to any one of claims 1 to 16.
25. A coding end device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method according to any one of claims 17 to 19.
26. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by the processor, the method described in any one of claims 1 to 16 is implemented; or, when the computer instructions are executed by the processor, the method described in any one of claims 17 to 19 is implemented.
27. A computer application, characterized in that When the computer application is executed by a processor, the method described in any one of claims 1 to 16 is implemented; or, when the computer application is executed by a processor, the method described in any one of claims 17 to 19 is implemented.
