Decoding and coding method, device and equipment

Abstract

The invention provides a decoding and coding method, device and equipment, and the decoding method comprises the steps: decoding a first code stream of a current image block, and obtaining a coefficient hyper-parameter feature of the current image block; for each stage sub-block of the current image block, determining a probability distribution parameter of the stage sub-block based on the coefficient hyper-parameter feature, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; based on a mean value prediction network, determining mean value features of the sub-blocks in the stage; determining a reconstruction feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block; performing feature aggregation on the reconstruction features of the sub-blocks in each stage to obtain the reconstruction features of the current image block; a reconstructed image block of the current image block is determined based on the reconstruction feature of the current image block. Through the scheme of the invention, the coding performance and the decoding performance are improved.

Description

Technical Field

[0001] The present application relates to the field of coding and decoding technology, and in particular to a decoding and encoding method, device and equipment thereof. Background Art

[0002] To save space, video images are encoded before transmission. Complete video encoding involves prediction, transformation, quantization, entropy coding, and filtering. Prediction can include intra-frame prediction and inter-frame prediction. Inter-frame prediction leverages temporal correlations in the video to predict the current pixel using pixels from adjacent coded images, effectively removing temporal redundancy. Intra-frame prediction leverages spatial correlations in the video to predict the current pixel using pixels from coded blocks in the current frame, effectively removing spatial redundancy.

[0003] With the rapid development of deep learning, it has achieved success in many high-level computer vision problems, such as image classification and object detection. Deep learning is also gradually being applied to codecs, enabling the use of neural networks to encode and decode images. Although neural network-based codecs demonstrate significant performance potential, they still suffer from issues such as poor encoding and decoding performance and high complexity. Summary of the Invention

[0004] In view of this, the present application provides a decoding and encoding method, apparatus and device thereof to improve encoding and decoding performance.

[0005] The present application provides a decoding method, applied to a decoding end, the method comprising:

[0006] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block;

[0007] For each stage sub-block of the current image block, determine a probability distribution parameter of the sub-block at that stage based on the coefficient hyperparameter feature, and decode the second bitstream of the current image block based on the probability distribution parameter to obtain a residual feature of the sub-block at that stage;

[0008] Determining a mean feature of the sub-block at the current stage based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the sub-block at the current stage by the first prediction network and the output features of the sub-block at the current stage by the second prediction network are used to determine the mean feature; or, the output features of the sub-block at the current stage by the first prediction network and the output features of the sub-block at the previous stage by the second prediction network are used to determine the mean feature;

[0009] Determine the reconstruction feature of the sub-block at this stage based on the mean feature of the sub-block at this stage and the residual feature of the sub-block at this stage;

[0010] The reconstruction features of the sub-blocks at each stage are aggregated to obtain the reconstruction features of the current image block;

[0011] A reconstructed image block of the current image block is determined based on the reconstruction feature of the current image block.

[0012] The present application provides an encoding method, applied to an encoding end, the method comprising:

[0013] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0014] Splitting the feature block into features to be encoded of multiple stage sub-blocks;

[0015] Obtaining coefficient hyperparameter features of the current image block, encoding the coefficient hyperparameter features to obtain a first bitstream of the current image block, and determining probability distribution parameters of sub-blocks in each stage based on the coefficient hyperparameter features;

[0016] For each stage sub-block, a mean feature of the stage sub-block is determined based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the stage sub-block are used to determine the mean feature; or, the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the previous stage sub-block are used to determine the mean feature;

[0017] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0018] For each stage sub-block, the residual features of the stage sub-block are encoded based on the probability distribution parameters of the stage sub-block to obtain a second code stream of the current image block.

[0019] The present application provides a decoding device, applied to a decoding end, comprising:

[0020] a decoding module configured to decode the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block; determine, for each stage sub-block of the current image block, probability distribution parameters of the sub-block at that stage based on the coefficient hyperparameter features, and decode the second bitstream of the current image block based on the probability distribution parameters to obtain residual features of the sub-block at that stage;

[0021] A determination module is configured to determine a mean feature of the sub-block at the current stage based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the first prediction network for the sub-block at the current stage and the output features of the second prediction network for the sub-block at the current stage are used to determine the mean feature; or, the output features of the first prediction network for the sub-block at the current stage and the output features of the second prediction network for the sub-block at the previous stage are used to determine the mean feature; and a reconstruction feature of the sub-block at the current stage is determined based on the mean feature of the sub-block at the current stage and the residual feature of the sub-block at the current stage;

[0022] The processing module is used to perform feature aggregation on the reconstruction features of the sub-blocks in each stage to obtain the reconstruction features of the current image block; and determine the reconstructed image block of the current image block based on the reconstruction features of the current image block.

[0023] The present application provides an encoding device, applied to an encoding end, comprising:

[0024] A processing module is used to input the current image block into the analysis and transformation network to obtain a feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks;

[0025] an encoding module, configured to obtain coefficient hyperparameter features of the current image block, encode the coefficient hyperparameter features to obtain a first bitstream of the current image block, and determine probability distribution parameters of sub-blocks at each stage based on the coefficient hyperparameter features;

[0026] A determination module is configured to determine, for each stage sub-block, a mean feature of the sub-block at that stage based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the first prediction network for the sub-block at that stage and the output features of the second prediction network for the sub-block at that stage are used to determine the mean feature; or, the output features of the first prediction network for the sub-block at that stage and the output features of the second prediction network for the sub-block at a previous stage are used to determine the mean feature; and a residual feature of the sub-block at that stage is determined based on the to-be-encoded feature of the sub-block at that stage and the mean feature of the sub-block at that stage;

[0027] The encoding module is further configured to encode the residual features of each stage sub-block based on the probability distribution parameters of the stage sub-block to obtain a second code stream of the current image block.

[0028] The present application provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0029] The processor is configured to execute machine-executable instructions to implement the decoding method of the above example.

[0030] The present application provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0031] The processor is configured to execute machine-executable instructions to implement the encoding method of the above example.

[0032] The present application provides an electronic device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned decoding method; or, the processor is used to execute the machine-executable instructions to implement the above-mentioned encoding method.

[0033] The present application provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the above-mentioned decoding method is implemented; or, the above-mentioned encoding method is implemented.

[0034] It can be seen from the above technical solution that in the embodiment of the present application, by adopting a staged prediction model, the peak memory usage is lower, the decoding time of a single block is shorter, and it has high-speed parallel decoding capabilities, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve encoding performance and decoding performance, and reduce complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application;

[0036] Figure 2 is a flowchart of a decoding method in one embodiment of the present application;

[0037] Figure 3 is a flowchart of an encoding method in one embodiment of the present application;

[0038] Figure 4 is a schematic diagram of the processing process of the encoding end in one embodiment of the present application;

[0039] Figure 5 is a schematic diagram of a processing process of a decoding end in one embodiment of the present application;

[0040] Figure 6 is a schematic diagram of the processing process of the encoding end in one embodiment of the present application;

[0041] Figures 7A-7E is a schematic diagram of a mean prediction network in one embodiment of the present application;

[0042] Figures 8A-8D is a schematic diagram of a mean prediction network in one embodiment of the present application;

[0043] Figures 9A-9L is a schematic diagram of a first prediction network in one embodiment of the present application;

[0044] Figures 10A-10D is a schematic diagram of a second prediction network in one embodiment of the present application;

[0045] Figures 11A-11B is a schematic diagram of a prediction fusion network in one embodiment of the present application;

[0046] Figures 12A-12B is a schematic diagram of a feature fusion network in one embodiment of the present application;

[0047] Figures 13A-13B is a schematic diagram of feature separation in one embodiment of the present application;

[0048] Figure 14A This is a hardware structure diagram of a decoding end device in one embodiment of the present application;

[0049] Figure 14B This is a hardware structure diagram of an encoding terminal device in one embodiment of the present application. DETAILED DESCRIPTION

[0050] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than for limiting the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at the time of...", or "when...", or "in response to determination".

[0051] In the embodiments of the present application, a decoding method and an encoding method are proposed, which may involve the following concepts:

[0052] Entropy encoding: Entropy encoding is a method of encoding that does not lose any information during the encoding process according to the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0053] Neural Network (NN): A neural network refers to an artificial neural network. A NN is a computational model composed of a large number of interconnected nodes (or neurons). In a NN, neuronal processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. Processing units in a NN can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; and hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information is reflected in the connections between processing units. A NN is a non-programmed, brain-like information processing method. Its essence is to achieve parallel and distributed information processing capabilities through the transformations and dynamics of the neural network, mimicking the information processing capabilities of the human brain to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.

[0054] Convolutional Neural Network (CNN): A convolutional neural network (CNN) is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to surrounding cells within a certain coverage area, making it an excellent choice for processing large images. The basic structure of a CNN consists of two layers. The first is the feature extraction layer (also known as the convolution layer). Each neuron's input is connected to the local receptive field of the previous layer, extracting local features. Once a local feature is extracted, its positional relationship with other features is determined. The second is the feature mapping layer (also known as the activation layer). Each computational layer of the neural network consists of multiple feature maps. Each feature map is a plane where all neurons have equal weights. The feature map structure can use Sigmoid, ReLU, Leaky-ReLU, PReLU, GDN, and other activation functions. Furthermore, because neurons within a mapping plane share weights, the number of free parameters in the network is reduced.

[0055] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid complex pre-processing of images (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected. This will result in a huge number of parameters, making network training time-consuming or even difficult to train. Convolutional neural networks avoid this difficulty through methods such as local connections and weight sharing.

[0056] Deconvolution: Deconvolution is also called transposed convolution. It works similarly to convolution, but uses padding to make the output larger than the input (though they can also remain the same). If the stride is 1, the output size is equal to the input size. If the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0057] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns implicit in the data pairs. For data outside the learning set with the same patterns, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0058] Feature: A feature is a three-dimensional feature matrix or tensor of C*W*H, see Figure 1 The figure below is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input or output of a neural network.

[0059] Rate-Distortion Optimized (RDO): Coding efficiency is evaluated using two key metrics: bit rate and Peak Signal to Noise Ratio (PSNR). A smaller bitrate results in a higher compression ratio, while a higher PSNR indicates better reconstructed image quality. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of these two metrics. For example, the cost of a mode is: J(mode) = D + λ*R, where D represents distortion. This is typically measured using the SSE metric (mean square sum of squared differences between the reconstructed image block and the source image). For cost considerations, the SAD metric (the sum of absolute differences between the reconstructed image block and the source image) can also be used. λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in that mode, including the bits required for coding mode information, motion information, and residual information. Using the rate-distortion principle to compare coding modes during mode selection generally ensures optimal coding performance.

[0060] Numerous coding tools have been proposed for various modules on the encoding side, and each tool often has multiple modes. Different coding tools often achieve optimal coding performance for different video sequences. Therefore, during the encoding process, RDO (Rate-Distortion Optimize) is often used to compare the coding performance of different tools or modes to select the optimal mode. After determining the optimal tool or mode, the decision-making information for the tool or mode is transmitted by encoding marker information in the bitstream. Although this approach introduces higher coding complexity, it can adaptively select the optimal mode combination for different content, achieving optimal coding performance. The decoding side can obtain relevant mode information by directly parsing the marker information, with minimal complexity impact.

[0061] The following describes in detail the decoding method and encoding method in the embodiments of the present application in combination with several embodiments.

[0062] Example 1: A decoding method is proposed in the embodiment of this application, see Figure 2 FIG. 1 is a flow chart of the decoding method, which can be applied to a decoding end (also called a video decoder). The method may include:

[0063] Step 201: Decode the first code stream of the current image block to obtain coefficient hyperparameter features of the current image block.

[0064] Step 202: For each stage sub-block of the current image block, determine the probability distribution parameters of the sub-block at that stage based on the coefficient hyperparameter features (i.e., the coefficient hyperparameter features of the current image block), decode the second code stream of the current image block based on the probability distribution parameters (i.e., the coefficient hyperparameter features of the current image block), and obtain the residual features of the sub-block at that stage.

[0065] Step 203: For each stage sub-block, determine the mean feature of the stage sub-block based on the mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the first prediction network for the stage sub-block and the output features of the second prediction network for the stage sub-block are used to determine the mean feature; or, the output features of the first prediction network for the stage sub-block and the output features of the second prediction network for the previous stage sub-block are used to determine the mean feature.

[0066] Exemplarily, for the 1st stage sub-block, the coefficient hyperparameter feature can be input into the mean prediction network to obtain the mean feature of the stage sub-block. For the i-th stage sub-block, where i is greater than 1, the coefficient hyperparameter feature and the reference feature of the i-th stage sub-block can be input into the mean prediction network to obtain the mean feature of the stage sub-block. The reference feature of the i-th stage sub-block can be obtained based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; for example, the reference feature can include all the reconstruction features of the previous i-1 stage sub-blocks, or part of the reconstruction features of the previous i-1 stage sub-blocks, or the reconstruction features of the i-1 stage sub-block, and there is no limitation on this reference feature.

[0067] Step 204: Determine the reconstruction features of the sub-blocks at this stage based on the mean features and the residual features of the sub-blocks at this stage, and perform feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain the reconstruction features of the current image block.

[0068] Step 205: Determine a reconstructed image block of the current image block based on the reconstruction features of the current image block.

[0069] Exemplarily, for each stage sub-block of the current image block, determining the probability distribution parameters of the stage sub-block based on the coefficient hyperparameter feature can include: inputting the coefficient hyperparameter feature (i.e., the coefficient hyperparameter feature of the current image block) into a probability hyperparameter decoding network to obtain the probability distribution parameters of the current image block, and the probability distribution parameters are the probability distribution parameters of each stage sub-block of the current image block. Alternatively, for the first stage sub-block, determining the probability distribution parameters of the stage sub-block based on the coefficient hyperparameter feature of the current image block; for the i-th stage sub-block, where i is greater than 1, inputting the coefficient hyperparameter feature of the current image block and the reference feature of the i-th stage sub-block into a mean prediction network to obtain the probability distribution parameters of the stage sub-block.

[0070] Exemplarily, the mean prediction network may include a first prediction network and a second prediction network, and the coefficient hyperparameter feature and the reference feature are input into the mean prediction network to obtain the mean feature of the sub-block at that stage, which may include but is not limited to: inputting the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature; inputting the reference feature into the second prediction network to obtain the second prediction feature; performing feature separation on the first prediction feature to obtain the third prediction feature; and generating the mean feature of the sub-block at that stage based on the third prediction feature and the second prediction feature. For example, generating the mean feature of the sub-block at that stage based on the third prediction feature and the second prediction feature may include but is not limited to: if the mean prediction network also includes a prediction fusion network, then the third prediction feature and the second prediction feature may be concatenated, and the concatenated feature may be input into the prediction fusion network, and the concatenated feature may be fused by the prediction fusion network to obtain the mean feature of the sub-block at that stage; or, if the mean prediction network also includes an addition layer, then the third prediction feature and the second prediction feature may be input into the addition layer, and the third prediction feature and the second prediction feature may be added by the addition layer to obtain the mean feature of the sub-block at that stage.

[0071] Exemplarily, the mean prediction network includes a first prediction network, a second prediction network, and a prediction fusion network. The coefficient hyperparameter feature and the reference feature are input into the mean prediction network to obtain the mean feature of the sub-block at that stage. This may include but is not limited to: when obtaining the reconstruction feature of each stage sub-block, the reconstruction feature of the stage sub-block is input into the second prediction network to obtain the second prediction feature of the stage sub-block, and the second prediction feature of the stage sub-block is stored. The coefficient hyperparameter feature is input into the first prediction network to obtain the first prediction feature; the first prediction feature is feature separated to obtain the third prediction feature. On this basis, the second prediction feature corresponding to the reference feature can be selected from all the stored second prediction features, and the third prediction feature and the selected second prediction feature are feature spliced; then, the spliced ​​feature can be input into the prediction fusion network, and the spliced ​​feature is fused through the prediction fusion network to obtain the mean feature of the sub-block at that stage.

[0072] Exemplarily, the mean prediction network includes a first prediction network, a second prediction network, a feature fusion network and an addition layer, and the coefficient hyperparameter feature and the reference feature are input into the mean prediction network to obtain the mean feature of the sub-block at that stage, which may include but is not limited to: when obtaining the reconstruction feature of each stage sub-block, the reconstruction feature of the stage sub-block is input into the second prediction network to obtain the second prediction feature of the stage sub-block, and the second prediction feature of the stage sub-block is stored; the coefficient hyperparameter feature is input into the first prediction network to obtain the first prediction feature; the first prediction feature is feature separated to obtain the third prediction feature; the second prediction feature corresponding to the reference feature is selected from all stored second prediction features, and the selected second prediction feature is input into the feature fusion network to obtain the fused feature; the third prediction feature and the fused feature are input into the addition layer, and the third prediction feature and the fused feature are added by the addition layer to obtain the mean feature of the sub-block at that stage.

[0073] Exemplarily, inputting the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature may include, but is not limited to: if the first prediction network includes at least one enhancement network and at least one upsampling network, inputting the coefficient hyperparameter feature into the first prediction network, performing enhancement operations and upsampling operations on the coefficient hyperparameter feature through the enhancement network and the upsampling network to obtain the first prediction feature. Alternatively, if the first prediction network includes at least one linear layer, at least one enhancement network, and at least one upsampling network, inputting the coefficient hyperparameter feature into the first prediction network, performing linear operations, enhancement operations, and upsampling operations on the coefficient hyperparameter feature through the linear layer, the enhancement network, and the upsampling network to obtain the first prediction feature.

[0074] Exemplarily, inputting the reference feature into the second prediction network to obtain the second prediction feature may include, but is not limited to: if the second prediction network includes at least one linear layer and at least one enhancement network, inputting the reference feature into the second prediction network, performing linear operations and enhancement operations on the reference feature through the linear layer and the enhancement network to obtain the second prediction feature. Alternatively, if the second prediction network includes at least one enhancement network and at least one activation layer, inputting the reference feature into the second prediction network, performing enhancement operations and activation operations on the reference feature through the enhancement network and the activation layer to obtain the second prediction feature.

[0075] Exemplarily, inputting the reconstructed features of the sub-block at the stage into the second prediction network to obtain the second prediction features of the sub-block at the stage may include, but is not limited to: if the second prediction network includes at least one linear layer and at least one enhancement network, inputting the reconstructed features of the sub-block at the stage into the second prediction network, performing linear operations and enhancement operations on the reconstructed features of the sub-block at the stage through the linear layer and the enhancement network, and obtaining the second prediction features of the sub-block at the stage. Alternatively, if the second prediction network includes at least one enhancement network and at least one activation layer, inputting the reconstructed features of the sub-block at the stage into the second prediction network, performing enhancement operations and activation operations on the reconstructed features of the sub-block at the stage through the enhancement network and the activation layer, and obtaining the second prediction features.

[0076] Exemplarily, the spliced ​​features are input into the prediction fusion network, and the spliced ​​features are fused through the prediction fusion network to obtain the mean features of the sub-blocks at this stage. This may include but is not limited to: if the prediction fusion network includes at least one linear layer and at least one enhancement network, the spliced ​​features are input into the prediction fusion network, and linear operations and enhancement operations are performed on the spliced ​​features through the linear layer and the enhancement network to obtain the mean features of the sub-blocks at this stage.

[0077] Exemplarily, the selected second prediction feature is input into the feature fusion network to obtain the fused feature, which may include but is not limited to: if the feature fusion network includes at least one linear layer and at least one enhancement network, then the selected second prediction feature is input into at least one linear layer and at least one enhancement network of the feature fusion network, and linear operations and enhancement operations are performed on the selected second prediction feature through the linear layer and the enhancement network to obtain the fused feature.

[0078] Exemplarily, the enhanced network includes a convolutional layer; or the enhanced network includes a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer in sequence; or the enhanced network includes a convolutional layer, an activation layer, and a convolutional layer in sequence; or the enhanced network includes a convolutional layer, an activation layer, a convolutional layer, and an addition layer in sequence; or the enhanced network includes a convolutional layer, an activation layer, a convolutional layer, and a multiplication layer in sequence. The upsampling network includes an upsampling layer, a crop layer, and an activation layer in sequence; or the upsampling network includes an upsampling layer and a crop layer in sequence; or the upsampling network includes an upsampling layer, a crop layer, an activation layer, and a convolutional layer in sequence. The linear layer includes N convolutional layers, where N is a non-negative integer. The upsampling layer is a deconvolutional layer, a Pixshuffle layer, an Unpooling layer, a Nearest layer, a Bilinear layer, or a Bicubic layer; and the activation layer is a Relu activation layer, a leaky Relu activation layer, a sigmoid activation layer, a tanh activation layer, or a gelu activation layer.

[0079] Exemplarily, feature separation of the first prediction feature to obtain the third prediction feature may include but is not limited to: uniformly dividing the first prediction feature into K parts according to the channel dimension to obtain K prediction group features; wherein each prediction group feature may contain N spatial feature maps, and each spatial feature map may contain multiple spatial feature points; wherein K may be a positive integer greater than 1, and N may be a positive integer greater than or equal to 1. Then, the feature separation method corresponding to each prediction group feature is determined; wherein the feature separation methods corresponding to different prediction group features may be different. For each prediction group feature, the spatial feature points of the prediction group feature may be uniformly sampled in the spatial domain based on the feature separation method corresponding to the prediction group feature to obtain K sampling result groups, and the third prediction feature is generated based on the K sampling result groups. wherein, uniform sampling with different sampling starting points or sampling step sizes may be regarded as different feature separation methods.

[0080] Exemplarily, based on the feature separation method corresponding to the prediction group feature, the spatial feature points of the prediction group feature are evenly sampled in the spatial domain to obtain K sampling result groups, which may include but is not limited to: for the N spatial feature maps within the prediction group feature, the spatial feature map is divided into multiple sub-blocks according to the spatial domain, and each sub-block includes N*2*2 spatial feature points; wherein K can be 2, or K can be 4; for each sub-block of the prediction group feature, the sampling result group to which each spatial feature point in the sub-block belongs can be determined based on the feature separation method corresponding to the prediction group feature; wherein the feature separation method is used to indicate the correspondence between the spatial feature points and the sampling result group, and the 2*2 spatial feature points in the sub-block correspond to K sampling result groups; wherein different feature separation methods corresponding to different prediction group features are different, which means that the same spatial feature point of the spatial feature maps in different prediction group features corresponds to different sampling result groups.

[0081] Exemplarily, feature aggregation is performed on the reconstruction features of each stage sub-block to obtain the reconstruction features of the current image block, which may include but is not limited to: for the reconstruction features of each stage sub-block, the reconstruction features of the stage sub-block are divided into K group reconstruction features; based on the K group reconstruction features corresponding to each stage sub-block, the group reconstruction features with the same index are spliced ​​in the channel dimension to obtain the spliced ​​reconstruction features corresponding to the K channel groups; for each channel group, a feature aggregation method corresponding to the feature separation method of the channel group is obtained, and the feature aggregation method is used to indicate the correspondence between the spatial feature points in the spliced ​​reconstruction features and the spatial feature points in the aggregated reconstruction features; based on the feature aggregation method corresponding to the channel group, the spatial feature points in the spliced ​​reconstruction features corresponding to the channel group are feature rearranged to obtain the aggregated reconstruction features corresponding to the channel group; the aggregated reconstruction features corresponding to the K channel groups are spliced ​​in the channel dimension to obtain the reconstruction features of the current image block.

[0082] For example, the above execution order is only for the convenience of describing the examples given. In actual applications, the execution order between the steps can also be changed, and this execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0083] It can be seen from the above technical solution that in the embodiment of the present application, by adopting a staged prediction model, the peak memory usage is lower, the decoding time of a single block is shorter, and it has high-speed parallel decoding capabilities, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve encoding performance and decoding performance, and reduce complexity.

[0084] Example 2: This application proposes an encoding method, see Figure 3 FIG. 1 is a flow chart of the encoding method, which can be applied to an encoding end (also called a video encoder). The method may include:

[0085] Step 301: Input the current image block into the analysis transformation network to obtain the feature block corresponding to the current image block.

[0086] Step 302: Split the feature block corresponding to the current image block into features to be encoded of multiple stage sub-blocks.

[0087] Step 303: Obtain coefficient hyperparameter features of the current image block, encode the coefficient hyperparameter features to obtain a first code stream of the current image block, and determine the probability distribution parameters of the sub-blocks in each stage based on the coefficient hyperparameter features.

[0088] Step 304: For each stage sub-block, determine the mean feature of the stage sub-block based on the mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the first prediction network for the stage sub-block and the output features of the second prediction network for the stage sub-block are used to determine the mean feature; or, the output features of the first prediction network for the stage sub-block and the output features of the second prediction network for the previous stage sub-block are used to determine the mean feature.

[0089] Exemplarily, for the first stage sub-block, the coefficient hyperparameter feature can be input into the mean prediction network to obtain the mean feature of the sub-block at this stage; for the i-th stage sub-block, where i is greater than 1, the coefficient hyperparameter feature and the reference feature of the i-th stage sub-block can be input into the mean prediction network to obtain the mean feature of the sub-block at this stage. The reference feature of the i-th stage sub-block can be obtained based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block, and the reference feature can include all the reconstruction features of the previous i-1 stage sub-blocks, or part of the reconstruction features of the previous i-1 stage sub-blocks, or the reconstruction features of the i-1 stage sub-block, and there is no limitation on the reference feature of this stage sub-block.

[0090] Step 305: For each stage sub-block corresponding to the current image block, determine the residual features of the stage sub-block based on the to-be-encoded features of the stage sub-block and the mean features of the stage sub-block, and encode the residual features of the stage sub-block based on the probability distribution parameters of the stage sub-block to obtain a second code stream of the current image block.

[0091] Exemplarily, the processing process at the encoding end is similar to the processing process at the decoding end, and the similarities are not repeated here.

[0092] For example, the above execution order is only for the convenience of describing the examples given. In actual applications, the execution order between the steps can also be changed, and this execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification, and the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0093] It can be seen from the above technical solution that in the embodiment of the present application, by adopting a staged prediction model, the peak memory usage is lower, the encoding time of a single block is shorter, and it has high-speed parallel encoding capabilities, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve encoding performance and decoding performance, and reduce complexity.

[0094] Example 3: For Example 1 and Example 2, the processing process of the encoding end can be found in Figure 4 As shown, of course, Figure 4 This is just an example of the processing process at the encoding end, and does not limit the processing process at the encoding end.

[0095] After obtaining the current image block x (which may be the original image block x, i.e., the input image block), the encoder can perform an analysis transformation on the current image block x using an analysis transformation network (e.g., a neural network) to obtain the image features y corresponding to the current image block x. Performing feature transformation on the current image block x using the analysis transformation network means transforming the current image block x into image features y in the latent domain, thereby facilitating all subsequent processes to operate in the latent domain.

[0096] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0097] After obtaining the image feature y, the encoding end performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (such as a neural network), and the hyperparameter encoding network performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network. There is no restriction on the training process of this hyperparameter encoding network. It only needs to be able to perform a coefficient hyperparameter feature transformation on the image feature y. The image feature y in the latent domain obtains the super-prior latent information z after passing through the hyperparameter encoding network.

[0098] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 4 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain the Bitstream#1 (i.e., the first code stream) corresponding to the current image block, i.e. Figure 4 The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoder can directly encode the coefficient hyperparameter feature z to obtain the first bitstream corresponding to the current image block. The hyperparameter quantization feature or coefficient hyperparameter feature z carried in the first bitstream is mainly used to obtain the mean and probability distribution model parameters.

[0099] After obtaining the first code stream corresponding to the current image block, the encoder may send the first code stream corresponding to the current image block to the decoder. For details on the processing of the first code stream corresponding to the current image block by the decoder, please refer to the subsequent embodiments.

[0100] After obtaining the first code stream corresponding to the current image block, the encoder can also decode the first code stream to obtain the super-parameter quantization feature, that is, Figure 4AD in it represents the decoding process. Then, the super-parameter quantization feature is dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the first bitstream corresponding to the current image block, the encoder can also decode the first bitstream to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0101] For the encoding process of the first code stream, an encoding method of a fixed probability density model can be adopted, and for the decoding process of the first code stream, a decoding method of a fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0102] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block (see the subsequent embodiments for the determination process of the reconstruction feature y_hat) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstruction feature y_hat. The two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain a residual by subtracting from the original feature and adding it to the decoded residual to obtain the reconstructed y.

[0103] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the predicted value mu. For example, the difference between the image feature y and the predicted value mu can be used as the residual feature r. The residual feature r is then subjected to feature processing to obtain the image feature s. This feature processing is not restricted and can be performed in any manner. It should be noted that feature processing is optional; that is, the residual feature r can be used as the image feature s without feature processing.

[0104] After obtaining the image feature s, the encoding end can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 4 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoder can encode the image quantization feature to obtain Bitstream#2 (i.e., the second code stream) corresponding to the current image block, i.e. Figure 4The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the image feature s to obtain a second bitstream corresponding to the current image block without involving the quantization process of the image feature s.

[0105] After obtaining the second code stream corresponding to the current image block, the encoder can send the second code stream corresponding to the current image block to the decoder. The decoding end processes the second code stream corresponding to the current image block in detail in the subsequent embodiments.

[0106] After obtaining the second code stream corresponding to the current image block, the encoder can also decode the second code stream to obtain the image quantization feature, that is, Figure 4 AD in the image represents the decoding process. Then, the encoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 Alternatively, after obtaining the second bitstream corresponding to the current image block, the encoder can also decode the second bitstream to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0107] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and any feature recovery method can be used to obtain the residual feature r_hat. The residual feature r_hat can be the same as or different from the residual feature r. It should be noted that the feature recovery process is optional, that is, if the feature processing process is not performed, the feature recovery process will not be performed. In other words, the image feature s' can be the residual feature r_hat.

[0108] After the encoder obtains the residual feature r_hat, the reconstructed feature y_hat is determined based on the residual feature r_hat and the predicted value mu. For example, the sum of the residual feature r_hat and the predicted value mu can be used as the reconstructed feature y_hat.

[0109] After obtaining the reconstructed feature y_hat, the encoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0110] In one possible implementation, when the encoder encodes the image quantization feature or image feature s to obtain a second bitstream corresponding to the current image block, the encoder needs to first determine a probability distribution model and then encode the image quantization feature or image feature s based on the probability distribution model. Furthermore, when decoding the second bitstream, the encoder also needs to first determine a probability distribution model and then decode the second bitstream based on the probability distribution model.

[0111] To obtain the probability distribution model, continue to refer to Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probabilistic hyperparameter decoding network can be a trained neural network. There is no restriction on the training process of this probabilistic hyperparameter decoding network. It only needs to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0112] In one possible implementation, the processing at the encoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without any restriction on the encoding process.

[0113] Example 4: For Example 1 and Example 2, the processing process of the decoding end can be found in Figure 5 As shown, of course, Figure 5 This is just an example of the processing process at the decoding end, and does not limit the processing process at the decoding end.

[0114] After obtaining the first code stream corresponding to the current image block, the decoding end can also decode the first code stream to obtain the super-parameter quantization feature, that is, Figure 5 AD in it represents the decoding process. Then, the super-parameter quantization feature is dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. Figure 5 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the first bitstream corresponding to the current image block, the decoder can also decode the first bitstream to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0115] For the decoding process of the first code stream, a decoding method of a fixed probability density model may be used, and there is no limitation on this.

[0116] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0117] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstruction feature y_hat of the previous image block (the determination process of the reconstruction feature y_hat is described in subsequent embodiments) to obtain the predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat are input to the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstruction feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0118] After obtaining the second code stream corresponding to the current image block, the decoding end can also decode the second code stream to obtain the image quantization feature, that is, Figure 5 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 5 Alternatively, after obtaining the second bitstream corresponding to the current image block, the decoder can also decode the second bitstream to obtain the image feature s' without involving the dequantization process of the image quantization feature.

[0119] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and any feature recovery method can be used to obtain the residual feature r_hat. The residual feature r_hat is the same as or different from the residual feature r. It should be noted that the feature recovery process is an optional process, that is, if the feature processing process is not performed, the feature recovery process will not be performed. In other words, the image feature s' can be the residual feature r_hat.

[0120] After the decoding end obtains the residual feature r_hat, the reconstructed feature y_hat is determined based on the residual feature r_hat and the predicted value mu. For example, the sum of the residual feature r_hat and the predicted value mu can be used as the reconstructed feature y_hat.

[0121] After obtaining the reconstructed feature y_hat, the decoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0122] In a possible implementation, when decoding the second bitstream, the decoding end needs to first determine the probability distribution model, and then decode the second bitstream based on the probability distribution model. Figure 5 As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probabilistic hyperparameter decoding network can be a trained neural network. There is no restriction on the training process of this probabilistic hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0123] In one possible implementation, the processing at the decoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without limiting the decoding process.

[0124] Example 5: For Example 1 and Example 2, the processing process of the encoding end can be found in Figure 6 As shown, of course, Figure 6 This is just an example of the processing process at the encoding end, and does not limit the processing process at the encoding end.

[0125] After obtaining the current image block x, the encoder can perform an analysis and transformation on the current image block x through the analysis and transformation network to obtain the image feature y corresponding to the current image block x. The image can be divided into a single image block or multiple image blocks. If the image is divided into a single image block, the current image block x can also be an image, that is, the encoding and decoding process is directly applied to the image.

[0126] The encoding end performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. For example, the image feature y is input into the hyperparameter encoding network, and the hyperparameter encoding network performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z.

[0127] The encoder quantizes the coefficient hyperparameter feature z to obtain a hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, encodes the hyperparameter quantization feature, and obtains a first bitstream corresponding to the current image block. Alternatively, the encoder can directly encode the coefficient hyperparameter feature z to obtain the first bitstream corresponding to the current image block.

[0128] After obtaining the first code stream corresponding to the current image block, the encoder may send the first code stream corresponding to the current image block to the decoder. For details on the processing of the first code stream corresponding to the current image block by the decoder, please refer to the subsequent embodiments.

[0129] After obtaining the first code stream corresponding to the current image block, the encoding end can also decode the first code stream to obtain super-parameter quantization features, dequantize the super-parameter quantization features, and obtain the coefficient super-parameter features z_hat. Alternatively, the encoding end can also decode the first code stream to obtain the coefficient super-parameter features z_hat without involving the dequantization process of the coefficient super-parameter features z_hat. For the encoding process of the first code stream, an encoding method of a fixed probability density model can be adopted, and for the decoding process of the first code stream, a decoding method of a fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0130] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can input the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat of the previous image block into the mean prediction network. The mean prediction network determines the predicted value mu and the probability distribution parameter p based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat, and generates a probability distribution model based on the probability distribution parameter p. Obviously, the difference from Example 3 is that the probability distribution parameter p is determined by the mean prediction network based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat, rather than by the probability hyperparameter decoding network based on the coefficient hyperparameter feature z_hat.

[0131] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the predicted value mu. For example, the difference between the image feature y and the predicted value mu is used as the residual feature r. The residual feature r is then processed to obtain the image feature s. Alternatively, the residual feature r can be used as the image feature s without undergoing feature processing.

[0132] After obtaining the image feature s, the encoder can quantize the image feature s to obtain a quantized image feature corresponding to the image feature s, encode the quantized image feature, and obtain a second bitstream corresponding to the current image block. Alternatively, the encoder can directly encode the image feature s to obtain the second bitstream corresponding to the current image block.

[0133] After obtaining the second code stream corresponding to the current image block, the encoder can send the second code stream corresponding to the current image block to the decoder. The decoding end processes the second code stream corresponding to the current image block in detail in the subsequent embodiments.

[0134] After obtaining the second bitstream corresponding to the current image block, the encoder can use the probability distribution model to decode the second bitstream to obtain image quantization features, and then dequantize the image quantization features to obtain image features s'. Alternatively, the encoder can use the probability distribution model to decode the second bitstream to obtain image features s' without involving the dequantization process.

[0135] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' to obtain the residual feature r_hat. After obtaining the residual feature r_hat, the encoder determines the reconstructed feature y_hat based on the residual feature r_hat and the predicted value mu. For example, the sum of the residual feature r_hat and the predicted value mu can be used as the reconstructed feature y_hat.

[0136] After obtaining the reconstructed feature y_hat, the encoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0137] Example 6: With respect to Example 1 and Example 2, regarding the processing process at the decoding end, the processing process may include:

[0138] After obtaining the first bitstream corresponding to the current image block, the decoder can also decode the first bitstream to obtain superparameter quantization features, and dequantize the superparameter quantization features to obtain the coefficient superparameter features z_hat. Alternatively, the decoder can also decode the first bitstream to obtain the coefficient superparameter features z_hat without involving the dequantization process of the coefficient superparameter features z_hat. For the decoding process of the first bitstream, a decoding method using a fixed probability density model can be used.

[0139] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0140] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can input the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat of the previous image block into the mean prediction network. The mean prediction network determines the predicted value mu and the probability distribution parameter p based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat, and generates a probability distribution model based on the probability distribution parameter p. Obviously, the difference from Example 4 is that the probability distribution parameter p is determined by the mean prediction network based on the coefficient hyperparameter feature z_hat and the reconstruction feature y_hat, rather than by the probability hyperparameter decoding network based on the coefficient hyperparameter feature z_hat.

[0141] After obtaining the second bitstream corresponding to the current image block, the decoder can use the probability distribution model to decode the second bitstream to obtain image quantization features, and then dequantize the image quantization features to obtain image features s'. Alternatively, the decoder can use the probability distribution model to decode the second bitstream to obtain image features s' without involving the dequantization process.

[0142] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the residual feature r_hat. After obtaining the residual feature r_hat, the decoder determines the reconstructed feature y_hat based on the residual feature r_hat and the predicted value mu. For example, the sum of the residual feature r_hat and the predicted value mu can be used as the reconstructed feature y_hat.

[0143] After obtaining the reconstructed feature y_hat, the decoding end can perform a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0144] Example 7: For Example 1 to Example 6, a mean prediction network can be deployed. In order to obtain accurate prediction values ​​of image features, the mean prediction network needs to use the reconstructed values ​​of adjacent features (i.e., based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block, a context-based prediction is performed to obtain the prediction value mu of the current image block, or the prediction value mu of the current image block and the probability distribution parameter p). Therefore, the feature value on the right (using the reconstructed value of the feature on the left or upper left side as a reference) can only be encoded after the feature value on the left or upper left side completes the encoding and reconstruction before the current feature can be encoded. This dependency relationship makes the encoding between features only serially executed and cannot be executed in parallel, resulting in high complexity. The feature value on the right can only be decoded after the feature value on the left or upper left side completes the decoding and reconstruction before the current feature can be decoded. This dependency relationship makes the decoding between features only serially executed and cannot be executed in parallel, resulting in high complexity.

[0145] In response to the above findings, in this embodiment, by adopting a staged prediction model, the degree of parallelism can be improved, the feature storage cache can be effectively saved, higher bit rate control accuracy can be achieved, the coding performance loss is smaller, and it has better coding performance and bit rate control accuracy, which makes the peak memory usage lower, the decoding time of a single block shorter, and has high-speed parallel decoding capabilities.

[0146] For the processing of the encoding end, please refer to Figure 4 or Figure 6 As shown, there is no restriction on the processing process of the encoding end.

[0147] 1. After obtaining the original image, determine whether to divide the original image into blocks based on the image resolution. For example, if the image resolution is greater than a threshold, divide the original image into blocks; otherwise, do not divide the original image into blocks.

[0148] If so, the original image is divided into several image blocks, with adjacent image blocks overlapping, and each image block serves as the current image block in the above embodiment. If not, the original image is not divided into blocks, and the original image serves as the current image block in the above embodiment. For ease of description, the processing of a current image block will be used as an example.

[0149] 2. After obtaining the current image block, the analysis transform network performs an analysis transform on the current image block to obtain the image features corresponding to the current image block. This image feature is recorded as a feature block. For example, the analysis transform network transforms the current image block into image features in the latent domain, making it easier for all subsequent processes to operate in the latent domain.

[0150] 3. Perform feature separation on the feature block corresponding to the current image block to obtain the features y_i (i=1…K) to be encoded of K stage sub-blocks. For example, split the feature block corresponding to the current image block into features y_i to be encoded of multiple stage sub-blocks.

[0151] 4. Obtain the coefficient hyperparameter features corresponding to the current image block. For example, perform a coefficient hyperparameter feature transformation on the feature block corresponding to the current image block to obtain the coefficient hyperparameter features corresponding to the current image block. For example, input the feature block corresponding to the current image block into a hyperparameter encoding network, which performs a coefficient hyperparameter feature transformation on the feature block to obtain the coefficient hyperparameter features.

[0152] 5. Encode the coefficient hyperparameter features of the current image block into the first bitstream of the current image block. For example, quantize the coefficient hyperparameter features to obtain the hyperparameter quantization features corresponding to the coefficient hyperparameter features, and encode the hyperparameter quantization features to obtain the first bitstream corresponding to the current image block. Alternatively, directly encode the coefficient hyperparameter features to obtain the first bitstream corresponding to the current image block. After obtaining the first bitstream corresponding to the current image block, send the first bitstream corresponding to the current image block to the decoding end. For the processing process of the first bitstream corresponding to the current image block by the decoding end, refer to the subsequent embodiments.

[0153] 6. After obtaining the first bitstream corresponding to the current image block, the first bitstream can be decoded to obtain coefficient hyperparameter features corresponding to the current image block. For example, the first bitstream can be decoded to obtain hyperparameter quantization features, and then the hyperparameter quantization features can be dequantized to obtain coefficient hyperparameter features corresponding to the current image block. Alternatively, the first bitstream can be decoded to obtain coefficient hyperparameter features corresponding to the current image block without involving a dequantization process.

[0154] For the encoding process of the first code stream, an encoding method of a fixed probability density model can be adopted, and for the decoding process of the first code stream, a decoding method of a fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0155] 7. For each stage sub-block corresponding to the current image block, determine the probability distribution parameters corresponding to the stage sub-block based on the coefficient hyperparameter characteristics of the current image block, that is, obtain the probability distribution parameters corresponding to each stage sub-block corresponding to the current image block.

[0156] For example, with respect to Example 3, the coefficient hyperparameter features corresponding to the current image block can be subjected to an inverse transformation of the coefficient hyperparameter features to obtain probability distribution parameters. For example, the coefficient hyperparameter features are input into a probabilistic hyperparameter decoding network, which performs an inverse transformation of the coefficient hyperparameter features on the coefficient hyperparameter features to obtain probability distribution parameters, which are probability distribution parameters for each stage sub-block of the current image block. After obtaining the probability distribution parameters, a probability distribution model can be generated based on the probability distribution parameters, which is a probability distribution model for each stage sub-block of the current image block.

[0157] For Example 5, the probability distribution parameters corresponding to each stage sub-block can be determined based on the coefficient hyperparameter features of the current image block, that is, the probability distribution parameters corresponding to each stage sub-block can be determined separately. For example, for the first stage sub-block, the coefficient hyperparameter features of the current image block can be input into the mean prediction network to obtain the probability distribution parameters of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameters of the stage sub-block. For the i-th stage sub-block, i is greater than 1, the coefficient hyperparameter features of the current image block and the reference features of the i-th stage sub-block can be input into the mean prediction network to obtain the probability distribution parameters of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameters of the stage sub-block.

[0158] 8. After obtaining the features to be encoded for the K stage sub-blocks, for the features to be encoded y_1 for the first stage sub-block, the coefficient hyperparameter features of the current image block are input into the mean prediction network to obtain the mean features of the sub-blocks at that stage (i.e., predicted value mu_1, mean mu_1). With respect to Example 5, after the coefficient hyperparameter features of the current image block are input into the mean prediction network, the mean prediction network simultaneously outputs the mean features of the sub-blocks at that stage and the probability distribution parameters of the sub-blocks at that stage.

[0159] Based on the to-be-encoded feature y_1 and the mean feature mu_1 of the sub-block in the first stage, the residual feature r_1 is determined. For example, the difference between the to-be-encoded feature y_1 and the mean feature mu_1 is used as the residual feature r_1. Then, feature processing is performed on the residual feature r_1 to obtain the processed residual feature r_1. This feature processing process is not restricted and can be any feature processing method. Of course, feature processing is an optional step and it is not necessary to perform feature processing on the residual feature r_1.

[0160] The residual feature r_1 (or the residual feature r_1 after feature processing) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second code stream corresponding to the current image block. Alternatively, the residual feature r_1 (or the residual feature r_1 after feature processing) can be directly encoded to obtain the second code stream corresponding to the current image block. After obtaining the second code stream corresponding to the current image block, the second code stream corresponding to the current image block is sent to the decoding end. Regarding the processing process of the decoding end for the second code stream corresponding to the current image block, refer to the subsequent embodiments. Among them, when encoding the image quantization feature or the residual feature r_1, the probability distribution model corresponding to the probability distribution parameters of the sub-block in the first stage can be used to encode the image quantization feature or the residual feature r_1 to obtain the second code stream.

[0161] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameters of the sub-blocks in the first stage can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_1. Alternatively, the image quantization features can be dequantized to directly obtain residual features r_hat_1 without involving a feature recovery process. Alternatively, after decoding the second bitstream, feature recovery can be performed on the decoded features to obtain residual features r_hat_1. Alternatively, the residual features r_hat_1 can be directly obtained after decoding the second bitstream.

[0162] After obtaining the residual feature r_hat_1, the reconstructed feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, such as taking the sum of the residual feature r_hat_1 and the mean feature mu_1 as the reconstructed feature y_hat_1.

[0163] 9. After obtaining the features to be encoded of the K stage sub-blocks, for the feature to be encoded y_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature of the current image block and the reference feature of stage sub-block i are input into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., predicted value mu_i, mean mu_i). For Example 5, after inputting the coefficient hyperparameter feature of the current image block and the reference feature of stage sub-block i into the mean prediction network, the mean prediction network can simultaneously output the mean feature of stage sub-block i and the probability distribution parameter of stage sub-block i.

[0164] Among them, the reference features for stage sub-block i (i.e., stage sub-block i) can be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, they can be obtained based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of stage sub-block i are obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include some of the reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on the reconstructed features of the i-1th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.

[0165] Based on the to-be-encoded feature y_i and the mean feature mu_i of the sub-block at the i-th stage, a residual feature r_i is determined. For example, the difference between the to-be-encoded feature y_i and the mean feature mu_i is used as the residual feature r_i. Feature processing is performed on the residual feature r_i to obtain the feature-processed residual feature r_i. Of course, feature processing can also be omitted for the residual feature r_i.

[0166] The residual feature r_i (or the residual feature r_i after feature processing) is quantized to obtain the image quantization feature, and the image quantization feature is encoded to obtain the second code stream corresponding to the current image block. Alternatively, the residual feature r_i (or the residual feature r_i after feature processing) is directly encoded to obtain the second code stream corresponding to the current image block. The second code stream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the second code stream corresponding to the current image block, refer to the subsequent embodiments. Among them, when encoding the image quantization feature or the residual feature r_i, the probability distribution model corresponding to the probability distribution parameter of the sub-block in the i-th stage can be used to encode the image quantization feature or the residual feature r_i to obtain the second code stream.

[0167] After obtaining the second bitstream, the probability distribution model corresponding to the probability distribution parameter p_i of the sub-block in the i-th stage can be used to decode the second bitstream to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_i. Alternatively, the image quantization features can be dequantized and then the residual features r_hat_i can be directly obtained. Alternatively, after decoding the second bitstream, the decoded features can be restored to obtain residual features r_hat_i. Alternatively, the residual features r_hat_i can be directly obtained after decoding the second bitstream.

[0168] After obtaining the residual feature r_hat_i, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, such as taking the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0169] 10. After obtaining the reconstruction features y_hat_i (i=1...K) of the K stage sub-blocks, feature aggregation can be performed on the reconstruction features y_hat_i of the K stage sub-blocks to obtain the reconstruction features of the current image block.

[0170] 11. Input the reconstructed features of the current image block to the synthesis transformation network, which outputs a reconstructed image block of the current image block. For example, the synthesis transformation network performs a synthesis transformation based on the reconstructed features of the current image block to obtain a reconstructed image block.

[0171] Example 8: The processing process of the decoding end is shown in Figure 5 or Figure 6 As shown, there is no restriction on the processing process of the decoding end.

[0172] 1. After obtaining the first bitstream corresponding to the current image block, the first bitstream can be decoded to obtain the coefficient hyperparameter features corresponding to the current image block. For example, the first bitstream can be decoded to obtain the hyperparameter quantization features, and then the hyperparameter quantization features can be dequantized to obtain the coefficient hyperparameter features corresponding to the current image block. Alternatively, the first bitstream can be decoded to obtain the coefficient hyperparameter features corresponding to the current image block without involving the dequantization process. For the decoding process of the first bitstream, a decoding method using a fixed probability density model can be used, and there is no restriction on this decoding process.

[0173] 2. For each stage sub-block corresponding to the current image block, determine the probability distribution parameters corresponding to the stage sub-block based on the coefficient hyperparameter characteristics of the current image block, that is, obtain the probability distribution parameters corresponding to each stage sub-block corresponding to the current image block.

[0174] For example, with respect to Example 4, the coefficient hyperparameter features corresponding to the current image block can be subjected to an inverse transformation of the coefficient hyperparameter features to obtain probability distribution parameters. For example, the coefficient hyperparameter features are input into a probabilistic hyperparameter decoding network, which performs an inverse transformation of the coefficient hyperparameter features on the coefficient hyperparameter features to obtain probability distribution parameters, which are probability distribution parameters for each stage sub-block of the current image block. After obtaining the probability distribution parameters, a probability distribution model can be generated based on the probability distribution parameters, which is a probability distribution model for each stage sub-block of the current image block.

[0175] For Example 6, the probability distribution parameters corresponding to each stage sub-block can be determined based on the coefficient hyperparameter features of the current image block, that is, the probability distribution parameters corresponding to each stage sub-block can be determined separately. For example, for the first stage sub-block, the coefficient hyperparameter features of the current image block can be input into the mean prediction network to obtain the probability distribution parameters of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameters of the stage sub-block. For the i-th stage sub-block, i is greater than 1, the coefficient hyperparameter features of the current image block and the reference features of the i-th stage sub-block can be input into the mean prediction network to obtain the probability distribution parameters of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameters of the stage sub-block.

[0176] 3. For each stage sub-block corresponding to the current image block, decode the second code stream of the current image block based on the probability distribution parameters (probability distribution model) of the stage sub-block to obtain the residual features of the stage sub-block.

[0177] For example, the second bitstream can be decoded using a probability distribution model corresponding to the probability distribution parameters of the sub-block at the i-th (i=1…K) stage to obtain image quantization features, dequantize the image quantization features, and perform feature recovery on the dequantized features to obtain residual features r_hat_i. Alternatively, the image quantization features can be dequantized and then the residual features r_hat_i can be directly obtained. Alternatively, after decoding the second bitstream, the decoded features can be restored to obtain residual features r_hat_i. Alternatively, after decoding the second bitstream, the residual features r_hat_i can be directly obtained.

[0178] 4. After obtaining the residual features of the K-stage sub-blocks, for the residual feature r_hat_1 of the first-stage sub-block, the coefficient hyperparameter feature of the current image block is input into the mean prediction network to obtain the mean feature of the sub-block at that stage (i.e., the predicted value mu_1 and the mean mu_1). For Example 6, after the coefficient hyperparameter feature of the current image block is input into the mean prediction network, the mean prediction network simultaneously outputs the mean feature of the sub-block at that stage and the probability distribution parameters of the sub-block at that stage.

[0179] Then, the reconstructed feature y_hat_1 of the sub-block in the first stage is determined based on the residual feature r_hat_1 and the mean feature mu_1. For example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be used as the reconstructed feature y_hat_1.

[0180] 5. After obtaining the residual features of K stage sub-blocks, for the residual feature r_hat_i of the i-th (i>1, and i is less than or equal to K) stage sub-block, the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i are input into the mean prediction network to obtain the mean feature of the stage sub-block i (i.e., the predicted value mu_i, the mean mu_i). For Example 5, after inputting the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i into the mean prediction network, the mean prediction network can simultaneously output the mean feature of the stage sub-block i and the probability distribution parameter of the stage sub-block i.

[0181] Among them, the reference features for stage sub-block i (i.e., stage sub-block i) can be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, they can be obtained based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of stage sub-block i are obtained based on all the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on some of the reconstructed features from y_hat_1 to y_hat_i-1, that is, the reference features include some of the reconstructed features from y_hat_1 to y_hat_i-1. Alternatively, the reference features of stage sub-block i are obtained based on the reconstructed features of the i-1th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.

[0182] To control the cache size, the number of reconstructed features included in the reference features cannot exceed a preset number. Assuming the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference features include all reconstructed features from y_hat_1 to y_hat_i-1. If the number of reconstructed features is greater than M, the reference features include M reconstructed features from y_hat_1 to y_hat_i-1 (i.e., partially reconstructed features). To select M reconstructed features from y_hat_1 to y_hat_i-1, M reconstructed features can be selected based on a default strategy, such as selecting the M reconstructed features closest to the sub-block in the current stage in decoding order, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, and so on, until M reconstructed features are selected. Of course, other methods can also be used to select M reconstructed features, and this is not limited to this. Alternatively, the index values ​​of the M reconstructed features can be obtained by decoding the bitstream, and the M reconstructed features can be selected based on the decoded index values.

[0183] Then, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstructed feature y_hat_i.

[0184] 6. After obtaining the reconstruction features y_hat_i (i=1...K) of the K stage sub-blocks, feature aggregation can be performed on the reconstruction features y_hat_i of the K stage sub-blocks to obtain the reconstruction features of the current image block.

[0185] 7. Input the reconstructed features of the current image block to the synthesis transformation network, which outputs a reconstructed image block of the current image block. For example, the synthesis transformation network performs a synthesis transformation based on the reconstructed features of the current image block to obtain a reconstructed image block.

[0186] Example 9: For Example 1 to Example 8, the input of the mean prediction network can be the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i, and the output of the mean prediction network can be the mean feature mu_i of the stage sub-block i. The mean prediction network can refer to Figure 7A Alternatively, the input of the mean prediction network can be the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i, and the output of the mean prediction network can be the mean feature mu_i of the stage sub-block i and the probability distribution parameter sigma_i of the stage sub-block i. The mean prediction network can be found in Figure 7B shown.

[0187] See also Figure 7A and Figure 7B As shown, the mean prediction network may include a first prediction network, a second prediction network and a prediction fusion network. The coefficient hyperparameter feature z_hat may be input to the first prediction network to obtain a first prediction feature. The reference feature of stage sub-block i (such as y_hat_1 to y_hat_i-1) may be input to the second prediction network to obtain a second prediction feature. If stage sub-block i is stage sub-block 1, the reference feature is empty, that is, the second prediction feature is empty. The first prediction feature may be feature separated to obtain a third prediction feature Phi_i. The third prediction feature Phi_i and the second prediction feature may be feature spliced. The spliced ​​feature is input to the prediction fusion network, and the spliced ​​feature is fused by the prediction fusion network to obtain the mean feature mu_i of stage sub-block i (see Figure 7A As shown), or, the spliced ​​features are fused through the prediction fusion network to obtain the mean feature mu_i and probability distribution parameter sigma_i of the stage sub-block i (see Figure 7B shown).

[0188] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstructed feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0189] In a possible implementation, in order to make the current prediction point more accurate, the second prediction network uses the decoded reconstructed feature y_hat as prior information to assist in predicting the current feature point and obtain the corresponding mean feature mu_i.

[0190] See also Figure 7C As shown, the channel Divide equally and use the simFC1 network to decode half Then use simFC2 to decode the other half See Figure 7D As shown in the figure, it is a structural diagram of the decoding network Spatial4Net-2, which can decode according to the spatial domain position. The network structure on the left is the network structure of Param_Net.

[0191] According to the channel, y is divided into two parts, y1 and y2. The internal parts of y1 and y2 are also designed with multi-stage decoding according to the phase. 2 For example, Preprocessing, the According to the special downsampling division method - feature separation. Transformation y2∈R {H / 2,W / 2,2c} , the spatial dimensions H and W are converted to 1 / 2 of the original, the number of channels is increased to 4 times the original, and then the data is divided into 4 parts according to the channel, which is equivalent to the data before conversion, and y is divided according to the phase in the spatial domain. 2 It is divided into four parts, denoted as y2_1, y2_2, y2_3, and y2_4. The phase description is as follows: y2_1 represents the spatial position in y2, which is in the even rows and even columns of the y2 spatial index, and contains data information of all channels. y2_2 represents the spatial position in y2, which is in the odd rows and odd columns of the y2 spatial index, and contains data information of all channels. y2_2 represents the spatial position in y2, which is in the even rows and odd columns of the y2 spatial index, and contains data information of all channels. y2_2 represents the spatial position in y2, which is in the odd rows and even columns of the y2 spatial index, and contains data information of all channels.

[0192] The four parts are then decoded in sequence, divided into four stages. Taking y1 decoding as an example: In the first stage, z_hat is passed through the first prediction network to obtain the first predicted feature, and Psi information is obtained through feature separation. The cc_ctx information comes from the reconstructed y1_hat and is padded with 0 to keep the number of input channels consistent with the number of input channels required by Param_Net. The mean information mu of the first part is output through Param_Net. The residual is combined with the decoder to obtain the reconstructed value y2_hat_1 of the first part. In the second stage, y2_hat_1 is used as prior information and passed through the convolution kernel to obtain features. The Psi information and cc_ctx information channels are spliced ​​and input into Param_Net to obtain the mean information mu of the second part. The residual is combined with the decoder to obtain the reconstructed value y2_hat_2 of the second part. In the third stage, y2_hat_1 and y2_hat_2 are used as prior information. After channel concatenation, they are concatenated and fed into a convolution kernel to obtain features. This information is then combined with the Psi and cc_ctx information channels and fed into Param_Net to obtain the mean information mu of the third part. The residual is then decoded and the reconstructed value y2_hat_3 of the third part is obtained. In the fourth stage, y2_hat_1, y2_hat_2, and y2_hat_3 are used as prior information. After channel concatenation, they are concatenated and fed into a convolution kernel to obtain features. This information is then combined with the Psi and cc_ctx information channels and fed into Param_Net to obtain the mean information mu of the fourth part. The residual is then decoded and the reconstructed value y2_hat_4 of the fourth part is obtained.

[0193] The four parts are reconstructed together: y2_hat_1, y2_hat_2, y2_hat_3, and y2_hat_4. After the inverse of the defined downsampling partitioning method, y2_hat is reassembled to complete the decoding of y2. A similar mechanism is used for decoding y1, but there is no cc_ctx in the entire process of y1. Therefore, zero padding is used to ensure the number of input channels received by the param_Net network is consistent. Finally, y1_hat and y2_hat are combined to form y_hat, resulting in the final reconstruction.

[0194] If the above eight-stage scheme does not perform channel division and only divides the spatial domain, it can be degenerated into a four-stage scheme to reduce complexity. In this case, the input of each stage of the four-stage scheme does not come from context information. In addition, the four-stage scheme can be further degenerated into a two-stage scheme, see Figure 7E As shown, this is a schematic diagram of the 2-stage prediction network structure.

[0195] For example, the 8-stage approach includes two 4-stage processes, but the 8-stage approach performs sequential reconstruction in the channel domain, while the 4-stage approach does not. Analyzing the 4-stage approach, the four reconstruction stages require different amounts of input data. In subsequent stages, the amount of input data increases due to the increased number of reconstructed features, leading to a linear increase in complexity with the number of stages. The prediction fusion network consumes a significant amount of computation; simplifying or eliminating this module can reduce the complexity of the prediction network.

[0196] Example 10: In this embodiment, a coding method is proposed. The original image x is generated by the corresponding analysis transformation network to generate y. Y is subjected to feature separation to obtain K-stage features to be encoded y_i (i=1...K). After y passes through the hyperparameter coding network, z is obtained. z is quantized into z_hat and encoded into the first bitstream. Based on the decoding of the first bitstream, z_hat is obtained, and based on the decoding of z_hat, the first predicted feature is obtained. For the first stage feature y_1, Phi is obtained, and the mean mu_1 is obtained through the mean prediction network. Based on mu_1, the residual r_1 of the feature y_1 is obtained, and r_1 is quantized to obtain r_q_1 and encoded into the second bitstream. Based on the decoding of the second bitstream, r_q_1 is obtained, and r_hat_1 is inversely quantized. Feature recovery is performed in combination with the mean mu_1 to obtain the reconstructed feature y_hat_1.

[0197] For the i-th stage feature y_i (i>1), the first predicted feature Phi_i is obtained based on the first bitstream, and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stage are combined to obtain mu_i. The residual r_i of feature y_i is obtained, and r_q_i is encoded into the second bitstream. Based on the decoded second bitstream, r_hat_i is obtained, and the feature is recovered by combining it with the mean mu_i to obtain the reconstructed feature y_hat_i.

[0198] The feature y_hat is obtained by clustering y_hat_i (i=1...K). The feature y_hat is passed through a synthesis transformation network to obtain a Y component reconstructed image. Thus, the encoding method is completed. The encoding method can be referred to in Example 7.

[0199] In this embodiment, a method for Figure 3 and Figure 4The decoding method is based on the following process to obtain y_hat_i (i=1…K): Decode the first bitstream to obtain z_hat, and obtain the probability parameter sigma of all residuals based on z_hat. Then decode the second bitstream based on the probability parameter sigma and perform inverse quantization to obtain r_hat. Decode the first-stage feature y_hat_1. Based on z_hat (the first stage cannot use features from other stages), use a mean prediction network to obtain the mean mu_1. Then obtain the corresponding first-stage residual in r_hat, denoted as r_hat_1, and add the mean mu_1 to perform feature recovery to obtain the reconstructed feature y_hat_1. Decode the i-th stage feature y_hat_i (i>1). Based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stage, use a mean prediction network to obtain the mean mu_i. Then obtain the corresponding stage residual in r_hat, denoted as r_hat_i, and add the mean mu_i to perform feature recovery to obtain the reconstructed feature y_hat_i.

[0200] The reconstructed feature y_hat is obtained by subjecting y_hat_i (i=1...K) to feature aggregation. The reconstructed feature y_hat is then merged to obtain a reconstructed image. Thus, the decoding method is completed. The decoding method can be referred to in Example 8.

[0201] In this embodiment, a method for Figure 5 The decoding method is based on the following process to obtain y_hat_i (i=1…K): Decode the first bitstream to obtain z_hat. Decode the first-stage feature y_hat_1, and use the mean prediction network based on z_hat (the first stage cannot use features from other stages) to obtain the mean mu_1 and probability parameter sigma_1. Using this probability parameter sigma_1, decode the residual from the second bitstream and perform dequantization to obtain r_hat_1. Add the mean mu_1 to perform feature recovery to obtain the reconstructed feature y_hat_1. Decode the i-th stage feature y_hat_i (i>1), based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 of the previous stage, obtain the mean mu_i and probability parameter sigma_i through the mean prediction network, and use this probability parameter sigma_i to decode the i-th stage residual from the second bitstream, and perform inverse quantization to obtain r_hat_i, and add the mean mu_1 to perform feature recovery to obtain the reconstructed feature y_hat_i.

[0202] The reconstructed feature y_hat is obtained by subjecting y_hat_i (i=1...K) to feature aggregation. The reconstructed feature y_hat is then merged to obtain a reconstructed image. Thus, the decoding method is completed. The decoding method can be referred to in Example 8.

[0203] Example 11: For Example 1 to Example 8, the input of the mean prediction network can be the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i, and the output of the mean prediction network can be the mean feature mu_i of the stage sub-block i. The output of the mean prediction network can be the mean feature mu_i of the stage sub-block i and the probability distribution parameter sigma_i of the stage sub-block i.

[0204] See also Figure 8A Figure 2 shows a schematic diagram of the structure of a mean prediction network, which may include a first prediction network, a second prediction network, and a prediction fusion network. The coefficient hyperparameter feature z_hat can be input into the first prediction network to obtain a first prediction feature (i.e., the output feature of the first prediction network for the sub-block at that stage). The reference feature of sub-block i (e.g., y_hat_1 to y_hat_i-1) can be input into the second prediction network to obtain a second prediction feature (i.e., the output feature of the second prediction network for the sub-block at that stage). If sub-block i is sub-block 1, the reference feature is 0, i.e., the second prediction feature is 0. Feature separation can be performed on the first prediction feature to obtain a third prediction feature Phi_i. The third prediction feature Phi_i and the second prediction feature can be concatenated, and the concatenated feature can be input into the prediction fusion network. The concatenated features can be fused by the prediction fusion network to obtain the mean feature mu_i for sub-block i at that stage. Alternatively, the concatenated features can be fused by the prediction fusion network to obtain the mean feature mu_i and probability distribution parameter sigma_i for sub-block i at that stage.

[0205] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstructed feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0206] See also Figure 8B As shown in the figure, it is a structural diagram of the mean prediction network. Figure 8AIn comparison, the prediction fusion network is cancelled and the feature concatenation layer is modified into an element-by-element addition layer, that is, the mean prediction network can include a first prediction network, a second prediction network and an addition layer (ADD). The coefficient hyperparameter feature z_hat can be input to the first prediction network to obtain the first prediction feature (that is, the output feature of the first prediction network for the sub-block of this stage). The reference feature of the stage sub-block i (such as y_hat_1 to y_hat_i-1) can be input to the second prediction network to obtain the second prediction feature (that is, the output feature of the second prediction network for the sub-block of this stage). If the stage sub-block i is the stage sub-block 1, the reference feature is 0, that is, the second prediction feature is 0. The first prediction feature can be feature separated to obtain the third prediction feature Phi_i. The third prediction feature and the second prediction feature can be input into the addition layer, and the third prediction feature and the second prediction feature can be added together through the addition layer (such as element-by-element addition) to obtain the mean feature mu_i of the stage sub-block i, or the third prediction feature and the second prediction feature can be added together through the addition layer (such as element-by-element addition) to obtain the mean feature mu_i and probability distribution parameter sigma_i of the stage sub-block i.

[0207] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstructed feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0208] See also Figure 8C As shown in the figure, it is a structural diagram of the mean prediction network. Figure 8A In contrast, the second prediction network is placed after the feature reconstruction. In this way, only one reconstructed feature (i.e., y_hat_i) needs to be input to the second prediction network each time, instead of inputting all reconstructed features (e.g., y_hat_1 to y_hat_i-1) to the second prediction network each time. This simplifies the complexity of the second prediction network, that is, the second prediction network can extract features in a single pass. Figure 8C , which is a structural diagram of a mean prediction network. The mean prediction network may include a first prediction network, a second prediction network, and a prediction fusion network.

[0209] Exemplarily, when the reconstructed feature y_hat_i of each stage sub-block is obtained, the reconstructed feature y_hat_i of the stage sub-block can be input into the second prediction network to obtain the second prediction feature y_f_i of the stage sub-block, and the second prediction feature y_f_i of the stage sub-block is stored. For example, when the reconstructed feature y_hat_1 of stage sub-block 1 is obtained, the reconstructed feature y_hat_1 of the stage sub-block can be input into the second prediction network to obtain the second prediction feature y_f_1 of the stage sub-block, and the second prediction feature y_f_1 of the stage sub-block is stored. When the reconstructed feature y_hat_2 of stage sub-block 2 is obtained, the reconstructed feature y_hat_2 of the stage sub-block can be input into the second prediction network to obtain the second prediction feature y_f_2 of the stage sub-block, and the second prediction feature y_f_2 of the stage sub-block is stored. By analogy, when determining the reconstructed feature y_hat_i of stage sub-block i, the second prediction features corresponding to the stage sub-blocks preceding stage sub-block i have been stored, such as the second prediction features y_f_1 to y_f_i-1.

[0210] When determining the reconstructed feature y_hat_i of the sub-block i at stage i, the coefficient hyperparameter feature z_hat can be input to the first prediction network to obtain the first prediction feature (i.e., the output feature of the first prediction network for the sub-block at this stage). The first prediction feature can be feature separated to obtain the third prediction feature Phi_i.

[0211] The second prediction feature corresponding to the reference feature (i.e., the output feature of the second prediction network for the previous stage sub-block of stage sub-block i) can be selected from all stored second prediction features. For example, if the reference feature includes all the reconstructed features of the previous i-1 stage sub-blocks, then all the stored second prediction features, i.e., y_f_1 to y_f_i-1, are selected. If the reference feature includes part of the reconstructed features of the previous i-1 stage sub-blocks, then part of the stored second prediction features are selected. For example, if the reference feature includes y_hat_1 and y_hat_2, then the second prediction feature y_f_1 corresponding to y_hat_1 and the second prediction feature y_f_2 corresponding to y_hat_2 are selected. If the reference feature includes the reconstructed features of the i-1th stage sub-block, then the stored second prediction feature y_f_i-1, i.e., the second prediction feature corresponding to y_hat_i, is selected. If stage sub-block i is stage sub-block 1, then the stored second prediction feature is 0, i.e., y_f_1 to y_f_i-1 are 0.

[0212] The third prediction feature Phi_i and the selected second prediction features y_f_1 to y_f_i-1 can be concatenated and the concatenated features can be input into the prediction fusion network. On this basis, the concatenated features can be fused through the prediction fusion network to obtain the mean feature mu_i of the stage sub-block i. Alternatively, the concatenated features can be fused through the prediction fusion network to obtain the mean feature mu_i and probability distribution parameter sigma_i of the stage sub-block i.

[0213] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstructed feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0214] After obtaining the reconstructed feature y_hat_i corresponding to stage sub-block i, the reconstructed feature y_hat_i of stage sub-block i can also be input into the second prediction network to obtain the second predicted feature y_f_i of stage sub-block i, and the second predicted feature y_f_i of stage sub-block i is stored. Obviously, this second predicted feature y_f_i can be used in the subsequent processing of stage sub-block i.

[0215] See also Figure 8D As shown in the figure, it is a structural diagram of the mean prediction network. Figure 8A In contrast, the second prediction network is placed after the feature reconstruction. In this way, only one reconstructed feature (i.e., y_hat_i) needs to be input to the second prediction network each time, instead of inputting all reconstructed features (e.g., y_hat_1 to y_hat_i-1) to the second prediction network each time. This simplifies the complexity of the second prediction network, that is, the second prediction network can extract features in a single pass. Figure 8D As shown in FIG, it is a structural diagram of the mean prediction network, which includes a first prediction network, a second prediction network, a feature fusion network and an addition layer ADD.

[0216] Exemplarily, when the reconstructed feature y_hat_i of each stage sub-block is obtained, the reconstructed feature y_hat_i of the stage sub-block can be input into the second prediction network to obtain the second prediction feature y_f_i of the stage sub-block, and the second prediction feature y_f_i of the stage sub-block is stored. Obviously, when the reconstructed feature y_hat_i of stage sub-block i is determined, the second prediction features corresponding to the stage sub-blocks preceding stage sub-block i have already been stored, such as the second prediction features y_f_1 to y_f_i-1.

[0217] When determining the reconstructed feature y_hat_i of stage sub-block i, the coefficient hyperparameter feature z_hat can be input to the first prediction network to obtain the first prediction feature (i.e., the output feature of the second prediction network for the previous stage sub-block of stage sub-block i). The first prediction feature can be feature separated to obtain the third prediction feature Phi_i. The second prediction feature corresponding to the reference feature (i.e., the output feature of the second prediction network for the previous stage sub-block of stage sub-block i) can be selected from all stored second prediction features, such as y_f_1 to y_f_i-1.

[0218] Then, the selected second prediction feature is input into the feature fusion network to obtain the fused feature. For example, the feature fusion network can perform feature fusion processing based on the selected second prediction feature to obtain the fused feature.

[0219] After obtaining the third prediction feature and the fused feature, the third prediction feature and the fused feature can be input into the addition layer, and the third prediction feature and the fused feature can be added together through the addition layer (such as element-by-element addition) to obtain the mean feature mu_i of the stage sub-block i, or the third prediction feature and the fused feature can be added together through the addition layer (such as element-by-element addition) to obtain the mean feature mu_i and probability distribution parameter sigma_i of the stage sub-block i.

[0220] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstructed feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0221] After obtaining the reconstructed feature y_hat_i corresponding to stage sub-block i, the reconstructed feature y_hat_i of stage sub-block i can also be input into the second prediction network to obtain the second predicted feature y_f_i of stage sub-block i, and the second predicted feature y_f_i of stage sub-block i is stored. Obviously, this second predicted feature y_f_i can be used in the subsequent processing of stage sub-block i.

[0222] certainly, Figures 8A-8D These are just a few examples of mean prediction networks, and their structure is not restricted. In each of the aforementioned mean prediction networks, model complexity can be reduced by optimizing the network structure of each subnetwork, or by optimizing individual network structures, such as eliminating the prediction fusion network. Furthermore, by optimizing feature separation techniques and adding specific partitioning rules, performance can be improved even with a small number of stages.

[0223] Example 12: With respect to Example 11, the mean prediction network may include a first prediction network, and the coefficient hyperparameter feature may be input into the first prediction network to obtain the first prediction feature. For example, the first prediction network may include at least one enhancement network and at least one upsampling network, and the coefficient hyperparameter feature may be input into the first prediction network, and the coefficient hyperparameter feature may be enhanced and upsampled by the enhancement network and the upsampling network to obtain the first prediction feature. For another example, the first prediction network may include at least one linear layer, at least one enhancement network, and at least one upsampling network, and the coefficient hyperparameter feature may be input into the first prediction network, and the coefficient hyperparameter feature may be linearly, enhanced, and upsampled by the linear layer, the enhancement network, and the upsampling network to obtain the first prediction feature. The structure of the first prediction network is described below.

[0224] See also Figure 9A Figure 2 shows the structure of the first prediction network, which includes enhancement network 1, 2x upsampling network 1, enhancement network 2, 2x upsampling network 2, and enhancement network 3. The coefficient hyperparameter feature z_hat_i (the coefficient hyperparameter feature z_hat of the current image block) is processed through enhancement network 1 and 2x upsampling network 1 to obtain enhanced large-size feature 1. The spatial size (width and height) of enhanced large-size feature 1 is twice that of coefficient hyperparameter feature z_hat_i. The enhanced large-size feature 1 is processed through enhancement network 2 and 2x upsampling network 2 to obtain enhanced large-size feature 2. The spatial size (width and height) of enhanced large-size feature 2 is twice that of enhanced large-size feature 1. The enhanced large-size feature 2 is processed through enhancement network 3 to obtain the first prediction feature.

[0225] against Figure 9A In the first prediction network shown, the number of enhancement networks can be more or less, the number of 2x upsampling networks can be more or less, and the 2x upsampling network can also be replaced with an upsampling network of other multiples.

[0226] against Figure 9A In the first prediction network shown, enhancement network 1 and enhancement network 2 can be the same or different, enhancement network 1 and enhancement network 3 can be the same or different, and enhancement network 2 and enhancement network 3 can be the same or different. In addition, 2x upsampling network 1 and 2x upsampling network 2 can be the same or different.

[0227] See also Figure 9BFigure 2 shows the structure of the first prediction network, which includes a linear layer 1, an enhancement network, a linear layer 2, and a 4x upsampling network. After the coefficient hyperparameter feature z_hat passes through the linear layer 1, the enhancement network, the linear layer 2, and the 4x upsampling network, an enhanced large-scale feature is obtained. The spatial size (width and height) of the enhanced large-scale feature is four times that of the coefficient hyperparameter feature z_hat. This enhanced large-scale feature is the first prediction feature output by the first prediction network.

[0228] against Figure 9B In the first prediction network shown, the number of enhancement networks can be greater or less, the number of 4x upsampling networks can be greater or less, the 4x upsampling network can be replaced with an upsampling network of other multiples, and the number of linear layers can be greater or less. Furthermore, linear layer 1 and linear layer 2 can be the same or different.

[0229] See also Figure 9C Figure 2 shows the structure of the first prediction network, which includes an enhancement network and a 4x upsampling network. After the coefficient hyperparameter feature z_hat passes through the enhancement network and the 4x upsampling network, it is transformed into an enhanced large-scale feature. The spatial size (width and height) of the enhanced large-scale feature is four times that of the coefficient hyperparameter feature z_hat, which is the first prediction feature.

[0230] against Figure 9C In the first prediction network shown, the number of enhancement networks can be more or less, the number of 4x upsampling networks can be more or less, and the 4x upsampling network can also be replaced by an upsampling network of other multiples.

[0231] For the enhanced network in the first prediction network described above, the enhanced network may include a convolutional layer. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, and a convolutional layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, and an addition layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, and a multiplication layer in sequence. Of course, the above are just a few examples of enhanced networks, and there is no limitation on the structure of the enhanced network. It is sufficient that the enhanced network can achieve the enhanced function.

[0232] See also Figure 9D Figure 1 is a schematic diagram of the structure of an enhanced network. The enhanced network may include at least one convolutional layer. Of course, the number of convolutional layers can be greater and there is no limit to this. The convolutional layer may be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, and there is no limit to this. For example, a 3x3 convolutional layer may be selected.

[0233] See also Figure 9EAs shown in FIG, a schematic diagram of the structure of an enhanced network is shown, which may include convolution layer 1, activation layer 1, convolution layer 2, activation layer 2, and convolution layer 3 in sequence. Of course, the number of convolution layers may be more or less, and there is no restriction on this. The number of activation layers may be more or less, and there is no restriction on this. For each convolution layer (such as convolution layer 1, and / or convolution layer 2, and / or convolution layer 3), the convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer may be selected. For each activation layer (such as activation layer 1 and / or activation layer 2), the activation layer may be a relu activation layer, a leaky relu activation layer, a sigmoid activation layer, a tanh activation layer, or a gelu activation layer. There is no restriction on the type of activation layer, and there may be a relu activation layer.

[0234] See also Figure 9F As shown in FIG, it is a structural diagram of an enhanced network, which may include convolution layer 1, activation layer 1, and convolution layer 2 in sequence. Of course, the number of convolution layers may be more or less, and there is no restriction on this. The number of activation layers may be more, and there is no restriction on this. For each convolution layer (such as convolution layer 1 and / or convolution layer 2), the convolution layer may be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, and there is no restriction on this. For example, a 3x3 convolution layer may be selected. For each activation layer, the activation layer may be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer. There is no restriction on the type of activation layer, and there may be a relu activation layer.

[0235] See also Figure 9G As shown, it is a structural diagram of the enhanced network, which can include convolution layer 1, activation layer 1, convolution layer 2, and addition layer (the addition layer can be replaced by a multiplication layer) in sequence. Of course, the number of convolution layers can be more or less, and there is no limit to this. The number of activation layers can be more, and there is no limit to this. For each convolution layer (such as convolution layer 1 and / or convolution layer 2), the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 3x3 convolution layer. Activation layer 1 can be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer. There is no limit to the type of activation layer, such as a relu activation layer.

[0236] See also Figure 9HAs shown in FIG, it is a structural diagram of the enhanced network, which can include a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 3x3 convolution layer (i.e., conv3x3), a relu activation layer, and a 1x1 convolution layer (i.e., conv1x1) in sequence.

[0237] For the upsampling network in the first prediction network, the upsampling network may include an upsampling layer, a crop layer, and an activation layer in sequence. Alternatively, the upsampling network may include an upsampling layer and a crop layer in sequence. Alternatively, the upsampling network may include an upsampling layer, a crop layer, an activation layer, and a convolutional layer in sequence. Of course, the above are just a few examples of upsampling networks, and there is no limitation on the structure of the upsampling network. The upsampling network can implement the upsampling operation function. The sampling multiple of the upsampling layer can be 2, and the sampling multiple of the upsampling layer can also be 4, without limitation.

[0238] See also Figure 9I The figure shows a schematic diagram of the structure of an upsampling network, which may include an upsampling layer, a crop layer, and an activation layer in sequence. Of course, there may be more upsampling layers, more crop layers, and more activation layers. The upsampling layer may be a deconvolution layer (such as a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer, without limitation, such as a 4x4 deconvolution layer), a Pixshuffle layer, an Unpooling layer, a Nearest layer, a Bilinear layer, or a Bicubic layer. The activation layer may be a Relu activation layer, a Leaky Relu activation layer, a Sigmoid activation layer, a Tanh activation layer, or a Gelu activation layer, without limitation, such as a Relu activation layer. The crop layer is used to crop features so that the spatial resolution of the features is reduced while the number of channels remains unchanged.

[0239] See also Figure 9J Figure 1 is a schematic diagram of the structure of an upsampling network, which may include an upsampling layer and a crop layer in sequence. There may be more upsampling layers and more crop layers. The upsampling layer may be a deconvolution layer (such as a 4x4 deconvolution layer), a Pixshuffle layer, an Unpooling layer, a Nearest layer, a Bilinear layer, or a Bicubic layer.

[0240] See also Figure 9KAs shown in FIG, it is a schematic diagram of the structure of the upsampling network, which may include an upsampling layer, a crop layer, an activation layer, and a convolution layer in sequence. Among them, the upsampling layer may be a deconvolution layer (such as a 4x4 deconvolution layer), or a Pixshuffle layer, or an Unpooling layer, or a Nearest layer, or a Bilinear layer, or a Bicubic layer. The activation layer may be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer, such as a relu activation layer. The convolution layer may be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 3x3 convolution layer.

[0241] For the linear layer in the first prediction network above, see Figure 9L Figure 1 shows the structure of a linear layer. This linear layer can include N convolutional layers, where N is a non-negative integer. Each convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected. N represents the number of convolutional layers. The value of N is not limited. N can be 0, indicating that the linear layer does not perform any processing on the input features.

[0242] Example 13: With respect to Example 11, the mean prediction network may include a second prediction network, and the reference feature (or the reconstructed feature of the stage sub-block, which will be described later using the processing process of the reference feature as an example) may be input into the second prediction network to obtain the second prediction feature. For example, the second prediction network may include at least one linear layer and at least one enhancement network, and the reference feature may be input into the second prediction network, and the linear operation and enhancement operation are performed on the reference feature through the linear layer and the enhancement network to obtain the second prediction feature. For another example, the second prediction network may include at least one enhancement network and at least one activation layer, and the reference feature may be input into the second prediction network, and the enhancement operation and activation operation are performed on the reference feature through the enhancement network and the activation layer to obtain the second prediction feature. The structure of the second prediction network is described below.

[0243] See also Figure 10A As shown in FIG, the second prediction network structure diagram, the second prediction network may include a linear layer 1, an enhanced network and a linear layer 2. After the reference features (such as y_hat_1 to y_hat_i-1) pass through the linear layer 1, the enhanced network and the linear layer 2, the second prediction features can be obtained. Figure 10A In the second prediction network shown, the number of enhancement networks can be more or less, the number of linear layers can be more or less, and linear layer 1 and linear layer 2 can be the same or different.

[0244] For the enhanced network in the second prediction network, the enhanced network may include a convolutional layer. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, and a convolutional layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, and an addition layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, and a multiplication layer in sequence. Of course, the above are just a few examples of enhanced networks, and there is no limitation on the structure of the enhanced network. The enhanced network can achieve the enhanced function. For the enhanced network in the second prediction network, please refer to the enhanced network in the first prediction network, and no further details will be given here.

[0245] For the linear layer in the second prediction network, the linear layer can include N convolutional layers, where N is a non-negative integer. Each convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected. N represents the number of convolutional layers. The value of N is not limited and can be 0.

[0246] If the number of convolutional layers N of the linear layer is 0, then when the second prediction network includes linear layer 1, enhanced network and linear layer 2, the network structure can also be equivalent to that the second prediction network includes the enhanced network, see Figure 10B As shown in FIG, it is a schematic diagram of the structure of the second prediction network, which can include a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 3x3 convolution layer (i.e., conv3x3), a relu activation layer, and a 1x1 convolution layer (i.e., conv1x1) in sequence.

[0247] See also Figure 10C Figure 2 shows a schematic diagram of the structure of the second prediction network. The second prediction network may include an enhancement network 1, an activation layer, and an enhancement network 2. After the reference features (e.g., y_hat_1 to y_hat_i-1) pass through the enhancement network 1, the activation layer, and the enhancement network 2, the second prediction features can be obtained. For this second prediction network, the number of enhancement networks can be greater or less, the number of activation layers can be greater or less, and the enhancement network 1 and the enhancement network 2 can be the same or different.

[0248] For each enhanced network in the second prediction network, the enhanced network may include a convolutional layer. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, and a convolutional layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, and an addition layer in sequence. Alternatively, the enhanced network may include a convolutional layer, an activation layer, a convolutional layer, and a multiplication layer in sequence. Of course, the above are just a few examples of enhanced networks, and there is no limitation on the structure of the enhanced network. The enhanced network can achieve the enhanced function. For the enhanced network in the second prediction network, please refer to the enhanced network in the first prediction network, and no further details will be given here.

[0249] For the activation layer in the second prediction network, the activation layer can be a relu activation layer, a leaky relu activation layer, a sigmoid activation layer, a tanh activation layer, or a gelu activation layer, and there is no limitation on this. For example, a relu activation layer can be selected.

[0250] For example, for Figure 10C The second prediction network shown, see Figure 10D As shown, the second prediction network may include, in sequence, a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 3x3 convolution layer (i.e., conv3x3), a relu activation layer, a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, and a 1x1 convolution layer (i.e., conv1x1). Of course, the above is only an example of the second prediction network and is not limited thereto.

[0251] Example 14: With respect to Example 11, the mean prediction network may include a prediction fusion network, and the spliced ​​features may be input to the prediction fusion network, and the spliced ​​features may be fused by the prediction fusion network to obtain the mean features of the sub-block at that stage, or the spliced ​​features may be fused by the prediction fusion network to obtain the mean features and probability distribution parameters of the sub-block at that stage. For example, the prediction fusion network may include at least one linear layer and at least one enhancement network, and the spliced ​​features may be input to the prediction fusion network, and the spliced ​​features may be subjected to linear and enhancement operations by the linear layer and the enhancement network to obtain the mean features of the sub-block at that stage, or the mean features and probability distribution parameters of the sub-block at that stage.

[0252] See also Figure 11AAs shown in FIG, the structure diagram of the prediction fusion network is shown, and the prediction fusion network may include a linear layer 1, an enhanced network, and a linear layer 2. After the concatenated features pass through the linear layer 1, the enhanced network, and the linear layer 2, the mean features can be obtained, or after the concatenated features pass through the linear layer 1, the enhanced network, and the linear layer 2, the mean features and probability distribution parameters can be obtained. Figure 11A In the prediction fusion network shown, the number of enhancement networks can be more or less, the number of linear layers can be more or less, and linear layer 1 and linear layer 2 can be the same or different.

[0253] For the enhanced network in the prediction fusion network, the enhanced network may include a convolution layer. Alternatively, the enhanced network may include a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer in sequence. Alternatively, the enhanced network may include a convolution layer, an activation layer, and a convolution layer in sequence. Alternatively, the enhanced network may include a convolution layer, an activation layer, a convolution layer, and an addition layer in sequence. Alternatively, the enhanced network may include a convolution layer, an activation layer, a convolution layer, and a multiplication layer in sequence. For the enhanced network in the prediction fusion network, please refer to the enhanced network in the first prediction network, which will not be repeated here.

[0254] For the linear layer in the prediction fusion network, it can include N convolutional layers, where N is a non-negative integer. Each convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected. N represents the number of convolutional layers. The value of N is not limited and can be 0.

[0255] If the number of convolutional layers N of the linear layer is 0, then when the prediction fusion network includes linear layer 1, enhanced network and linear layer 2, the network structure can also be equivalent to that, the prediction fusion network includes the enhanced network, see Figure 11B As shown in FIG, it is a structural diagram of the prediction fusion network, which can include a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, and a 1x1 convolution layer (i.e., conv1x1) in sequence.

[0256] Example 15: With respect to Example 11, the mean prediction network may include a feature fusion network, and the selected second prediction features (y_f_1 to y_f_i-1) may be input into the feature fusion network to obtain fused features. For example, the feature fusion network includes at least one linear layer and at least one enhancement network. The selected second prediction features are input into the feature fusion network, and linear and enhancement operations are performed on the selected second prediction features through the linear layer and the enhancement network to obtain fused features.

[0257] See also Figure 12AAs shown in Figure 1, it is a structural diagram of the feature fusion network. The feature fusion network can include linear layer 1, enhancement network and linear layer 2. After the second prediction feature (y_f_1 to y_f_i-1) passes through linear layer 1, enhancement network and linear layer 2, the fused feature can be obtained. Figure 12A In the feature fusion network shown, the number of enhancement networks can be more or less, the number of linear layers can be more or less, and linear layer 1 and linear layer 2 can be the same or different.

[0258] For the enhanced network in the feature fusion network, the enhanced network may include a convolution layer. Alternatively, the enhanced network may include a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer in sequence. Alternatively, the enhanced network may include a convolution layer, an activation layer, and a convolution layer in sequence. Alternatively, the enhanced network may include a convolution layer, an activation layer, a convolution layer, and an addition layer in sequence. Alternatively, the enhanced network may include a convolution layer, an activation layer, a convolution layer, and a multiplication layer in sequence. For the enhanced network in the feature fusion network, please refer to the enhanced network in the first prediction network, which will not be repeated here.

[0259] The linear layer in the feature fusion network can include N convolutional layers, where N is a non-negative integer. Each convolutional layer can be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer. For example, a 1x1 convolutional layer can be selected. N represents the number of convolutional layers. The value of N is not limited and can be 0.

[0260] If the number of convolutional layers N of the linear layer is 0, then when the feature fusion network includes linear layer 1, enhanced network and linear layer 2, the network structure can also be equivalent to that the feature fusion network includes the enhanced network, see Figure 12B As shown in FIG, it is a structural diagram of the feature fusion network, which can include a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, and a 1x1 convolution layer (i.e., conv1x1) in sequence.

[0261] Example 16: In Examples 1 to 15, for the encoding end and the decoding end, the first prediction feature will be feature separated to obtain the third prediction feature. For the encoding end, the feature block corresponding to the current image block will be feature separated to obtain the to-be-encoded features y_i (i=1…K) of K stage sub-blocks. The feature separation methods of the two are the same. Taking the feature separation of the first prediction feature to obtain the third prediction feature as an example, the feature separation process in the embodiment of the present application is introduced.

[0262] For example, the first prediction feature can be evenly divided into K parts according to the channel dimension to obtain K prediction group features; each prediction group feature can include N spatial feature maps, and each spatial feature map can include multiple spatial feature points. K can be a positive integer greater than 1, and N can be a positive integer greater than or equal to 1. For example, K can be 2 or 4.

[0263] For example, see Figure 13A As shown in the figure, taking K as 2 as an example, the left side represents the first prediction feature, which includes a spatial feature map of 2 channels (the number of channels of the spatial feature map can be greater than 2). The first prediction feature can be evenly divided into 2 parts according to the channel dimension to obtain 2 prediction group features. Prediction group feature 1 includes 1 spatial feature map (i.e., a spatial feature map with a value of 1), and prediction group feature 2 includes 1 spatial feature map (i.e., a spatial feature map with a value of 2). Of course, when the number of channels of the spatial feature map is greater than 2, the number of spatial feature maps included in the prediction group feature will be greater than 1.

[0264] See also Figure 13B As shown in the figure, taking K as 4 as an example, the left side represents the first prediction feature, which includes the spatial feature maps of 4 channels. The first prediction feature is evenly divided into 4 parts according to the channel dimension to obtain 4 prediction group features. Prediction group feature 1 contains 1 spatial feature map (i.e., the spatial feature map with a value of 1), prediction group feature 2 contains 1 spatial feature map (i.e., the spatial feature map with a value of 2), prediction group feature 3 contains the first spatial feature map (i.e., the spatial feature map with a value of 3), and prediction group feature 4 contains 1 spatial feature map (i.e., the spatial feature map with a value of 4).

[0265] For example, a feature separation method corresponding to each prediction group feature can be determined, and different prediction group features can correspond to different feature separation methods. The feature separation method is used to indicate the correspondence between spatial feature points and sampling result groups. Different feature separation methods corresponding to different prediction group features mean that the same spatial feature point in the spatial feature maps within different prediction group features (i.e., the spatial feature point at the same position in different spatial feature maps) corresponds to different sampling result groups.

[0266] Exemplarily, for each prediction group feature, the spatial feature points of the prediction group feature can be uniformly sampled in the spatial domain based on the feature separation method corresponding to the prediction group feature to obtain K sampling result groups, and a third prediction feature is generated based on the K sampling result groups. For example, for the N spatial feature maps within the prediction group feature, the spatial feature map is divided into multiple sub-blocks based on the spatial domain, and each sub-block includes N*2*2 spatial feature points; wherein, for each sub-block of the prediction group feature, the sampling result group to which each spatial feature point in the sub-block belongs can be determined based on the feature separation method corresponding to the prediction group feature; wherein, the feature separation method is used to indicate the correspondence between the spatial feature point and the sampling result group, and the 1*2*2 spatial feature points in the sub-block correspond to K sampling result groups; wherein, different feature separation methods corresponding to different prediction group features are different, which means that the same spatial feature point in the spatial feature maps within different prediction group features corresponds to different sampling result groups. wherein, uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation methods.

[0267] For example, see Figure 13A As shown, the feature separation method 1 corresponding to the prediction group feature 1 can be determined, and based on the feature separation method 1, the spatial feature points of the prediction group feature 1 are evenly sampled in the spatial domain to obtain two sampling result groups.

[0268] In the uniformly spaced sampling process, the N spatial feature maps (such as the spatial feature map with a value of 1) in the prediction group feature 1 are divided into multiple sub-blocks, and each sub-block includes N*2*2 spatial feature points. For each sub-block, the sampling result group to which each spatial feature point in the sub-block belongs is determined based on the feature separation method 1. For example, the feature separation method 1 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to the sampling result group 1, the spatial feature point in the upper right corner of the sub-block corresponds to the sampling result group 2, the spatial feature point in the lower left corner of the sub-block corresponds to the sampling result group 2, and the spatial feature point in the lower right corner of the sub-block corresponds to the sampling result group 1. Therefore, based on the feature separation method 1, it is determined that the spatial feature point in the upper left corner and the spatial feature point in the lower right corner of each sub-block belong to the sampling result group 1, and it is determined that the spatial feature point in the upper right corner and the spatial feature point in the lower left corner of each sub-block belong to the sampling result group 2.

[0269] In the uniformly spaced sampling process, the N spatial feature maps within the prediction group feature 2 (such as the spatial feature map with a value of 2) are divided into multiple sub-blocks, each sub-block includes N*2*2 spatial feature points, N=1 in the figure, each sub-block includes 1*2*2 spatial feature points, and N*2*2 is used as an example for subsequent explanation. For each sub-block, the sampling result group to which each spatial feature point in the sub-block belongs is determined based on feature separation method 2. Moreover, since different prediction group features correspond to different feature separation methods, that is, the same spatial feature point of the spatial feature maps within different prediction group features needs to correspond to different sampling result groups, and uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation methods.

[0270] Based on this, since feature separation method 1 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1, when different prediction group features correspond to different feature separation methods (the same spatial feature point needs to correspond to different sampling result groups), feature separation method 2 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 2. Since feature separation method 1 is used to indicate that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 2, feature separation method 2 is used to indicate that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 1. Since feature separation method 1 is used to indicate that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 2, feature separation method 2 is used to indicate that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 1. Since feature separation method 1 is used to indicate that the spatial feature point in the lower right corner of the sub-block corresponds to sampling result group 1, feature separation method 2 is used to indicate that the spatial feature point in the lower right corner of the sub-block corresponds to sampling result group 2. To sum up, based on feature separation method 2, it is determined that the spatial feature points in the upper left corner and the spatial feature points in the lower right corner of each sub-block belong to sampling result group 2, and it is determined that the spatial feature points in the upper right corner and the spatial feature points in the lower left corner of each sub-block belong to sampling result group 1.

[0271] See also Figure 13A As shown, all the spatial feature points of the sampling result group 1 can be spliced ​​together to obtain a combined feature, namely Part1, and all the spatial feature points of the sampling result group 2 can be spliced ​​together to obtain a combined feature, namely Part2. In this way, Part1 and Part2 can be combined to obtain the third prediction feature corresponding to the first prediction feature.

[0272] against Figure 13A The feature separation method shown can also be called a 2-stage feature separation method. Based on the 2-stage feature separation, the channel domain information can be extracted when the feature points are reconstructed.

[0273] For example, see Figure 13BAs shown, feature separation method 1 corresponding to prediction group feature 1 can be determined. Based on feature separation method 1, the spatial feature points of prediction group feature 1 are evenly sampled in the spatial domain, obtaining four sampling result groups. Feature separation method 2 corresponding to prediction group feature 2 can be determined. Based on feature separation method 2, the spatial feature points of prediction group feature 2 are evenly sampled in the spatial domain, obtaining four sampling result groups. Feature separation method 3 corresponding to prediction group feature 3 can be determined. Based on feature separation method 3, the spatial feature points of prediction group feature 3 are evenly sampled in the spatial domain, obtaining four sampling result groups. Feature separation method 4 corresponding to prediction group feature 4 can be determined. Based on feature separation method 4, the spatial feature points of prediction group feature 4 are evenly sampled in the spatial domain, obtaining four sampling result groups.

[0274] In the uniformly spaced sampling process, the N spatial feature maps (such as the spatial feature map with a value of 1) in the prediction group feature 1 are divided into multiple sub-blocks, each of which includes N*2*2 spatial feature points. For each sub-block, the sampling result group to which each spatial feature point in the sub-block belongs is determined based on the feature separation method 1. For example, the feature separation method 1 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to the sampling result group 1, the spatial feature point in the upper right corner of the sub-block corresponds to the sampling result group 3, the spatial feature point in the lower left corner of the sub-block corresponds to the sampling result group 4, and the spatial feature point in the lower right corner of the sub-block corresponds to the sampling result group 2. Therefore, based on the feature separation method 1, it is determined that the spatial feature point in the upper left corner of each sub-block belongs to the sampling result group 1, the spatial feature point in the upper right corner of each sub-block belongs to the sampling result group 3, the spatial feature point in the lower left corner of each sub-block belongs to the sampling result group 4, and the spatial feature point in the lower right corner of each sub-block belongs to the sampling result group 2.

[0275] In the uniformly spaced sampling process, the N spatial feature maps within the prediction group feature 2 (such as the spatial feature map with a value of 2) are divided into multiple sub-blocks, each of which includes N*2*2 spatial feature points. For each sub-block, the sampling result group to which each spatial feature point within the sub-block belongs is determined based on feature separation method 2. Moreover, different prediction group features correspond to different feature separation methods, that is, the same spatial feature point of the spatial feature maps within different prediction group features needs to correspond to different sampling result groups, and uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation methods.

[0276] Based on this, since feature separation method 1 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1, feature separation method 2 cannot indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1. Take the example of feature separation method 2 indicating that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 2. Since feature separation method 1 is used to indicate that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 3, feature separation method 2 cannot indicate that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 3 and sampling result group 2 (occupied by the upper left corner of the sub-block). Take the example of feature separation method 2 indicating that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 4. Since feature separation method 1 is used to indicate that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 4, feature separation method 2 cannot indicate that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 4 and sampling result group 2 (occupied by the upper left corner of the sub-block). Take the example of feature separation method 2 indicating that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 3. Since feature separation method 1 is used to indicate that the spatial feature point in the lower right corner of the sub-block corresponds to sampling result group 2, feature separation method 2 cannot indicate that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 2, sampling result group 3 (occupied), or sampling result group 4 (occupied), take the case where feature separation method 2 indicates that the spatial feature point in the lower right corner of the sub-block corresponds to sampling result group 1 as an example. In summary, based on feature separation method 2, it is determined that the spatial feature point in the upper left corner of each sub-block belongs to sampling result group 2, the spatial feature point in the upper right corner of each sub-block belongs to sampling result group 4, the spatial feature point in the lower left corner of each sub-block belongs to sampling result group 3, and the spatial feature point in the lower right corner of each sub-block belongs to sampling result group 1.

[0277] In the uniformly spaced sampling process, the N spatial feature maps within the prediction group feature 3 (such as the spatial feature map with a value of 3) are divided into multiple sub-blocks, each of which includes N*2*2 spatial feature points. For each sub-block, the sampling result group to which each spatial feature point within the sub-block belongs is determined based on the feature separation method 3. Moreover, different prediction group features correspond to different feature separation methods, that is, the same spatial feature point in the spatial feature maps within different prediction group features needs to correspond to different sampling result groups, and uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation methods.

[0278] Based on this, since feature separation method 1 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1, and feature separation method 2 indicates that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 2, feature separation method 3 cannot indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1 and sampling result group 2. Take, for example, the case where feature separation method 3 indicates that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 3. Similarly, assume that feature separation method 3 indicates that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 1, feature separation method 3 indicates that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 2, and feature separation method 3 indicates that the spatial feature point in the lower right corner of the sub-block corresponds to sampling result group 4.

[0279] To sum up, based on feature separation method 3, it is determined that the spatial feature point in the upper left corner of each sub-block belongs to sampling result group 3, the spatial feature point in the upper right corner of each sub-block belongs to sampling result group 1, the spatial feature point in the lower left corner of each sub-block belongs to sampling result group 2, and the spatial feature point in the lower right corner of each sub-block belongs to sampling result group 4.

[0280] In the uniformly spaced sampling process, the N spatial feature maps within the prediction group feature 4 (such as the spatial feature map with a value of 4) are divided into multiple sub-blocks, each of which includes N*2*2 spatial feature points. For each sub-block, the sampling result group to which each spatial feature point within the sub-block belongs is determined based on the feature separation method 4. Moreover, different prediction group features correspond to different feature separation methods, that is, the same spatial feature point of the spatial feature maps within different prediction group features needs to correspond to different sampling result groups, and uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation methods.

[0281] Based on this, since feature separation method 1 is used to indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1, feature separation method 2 indicates that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 2, and feature separation method 3 indicates that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 3, feature separation method 4 cannot indicate that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 1, sampling result group 2, and sampling result group 3. Take the example of feature separation method 4 indicating that the spatial feature point in the upper left corner of the sub-block corresponds to sampling result group 4. Similarly, assume that feature separation method 4 indicates that the spatial feature point in the upper right corner of the sub-block corresponds to sampling result group 2, feature separation method 4 indicates that the spatial feature point in the lower left corner of the sub-block corresponds to sampling result group 1, and feature separation method 4 indicates that the spatial feature point in the lower right corner of the sub-block corresponds to sampling result group 3.

[0282] To sum up, based on feature separation method 4, it is determined that the spatial feature point in the upper left corner of each sub-block belongs to sampling result group 4, the spatial feature point in the upper right corner of each sub-block belongs to sampling result group 2, the spatial feature point in the lower left corner of each sub-block belongs to sampling result group 1, and the spatial feature point in the lower right corner of each sub-block belongs to sampling result group 3.

[0283] See also Figure 13B As shown, all the spatial feature points of the sampling result group 1 can be spliced ​​together to obtain a combined feature, namely Part1, all the spatial feature points of the sampling result group 2 can be spliced ​​together to obtain a combined feature, namely Part2, all the spatial feature points of the sampling result group 3 can be spliced ​​together to obtain a combined feature, namely Part3, and all the spatial feature points of the sampling result group 4 can be spliced ​​together to obtain a combined feature, namely Part4. In this way, Part1, Part2, Part3 and Part4 can be combined to obtain the third prediction feature corresponding to the first prediction feature.

[0284] against Figure 13B The feature separation method shown can also be called a 4-stage feature separation method. Based on the 4-stage feature separation, the channel domain information can be extracted when the feature points are reconstructed.

[0285] Example 17: Based on Example 1 to Example 16, feature aggregation can be performed on the reconstruction features of the sub-blocks at each stage to obtain the reconstruction features of the current image block. Feature aggregation is the inverse process of feature separation. That is, after the feature separation method is known, the feature aggregation method corresponding to the feature separation method can be obtained. The feature aggregation method is the inverse process of the feature separation method. Based on the feature aggregation method, feature aggregation is performed on the reconstruction features of the sub-blocks at each stage to obtain the reconstruction features of the current image block. When performing feature aggregation on the reconstruction features of the sub-blocks at each stage, the following feature aggregation methods can be used:

[0286] For the reconstruction features of each stage sub-block, the reconstruction features of the sub-block of this stage are divided into K group reconstruction features. For example, see Figure 13A As shown in the figure, Part1 corresponds to the reconstruction features of the sub-blocks in the first stage. The reconstruction features of the sub-blocks in this stage can be divided into two grouped reconstruction features. Grouped reconstruction feature 1 corresponds to the spatial feature points with a value of 1 (spatial feature map), and grouped reconstruction feature 2 corresponds to the spatial feature points with a value of 2 (spatial feature map). Part2 corresponds to the reconstruction features of the sub-blocks in the second stage. The reconstruction features of the sub-blocks in this stage can be divided into two grouped reconstruction features. Grouped reconstruction feature 1 corresponds to the spatial feature points with a value of 1, and grouped reconstruction feature 2 corresponds to the spatial feature points with a value of 2.

[0287] For example, see Figure 13B As shown, Part1 corresponds to the reconstruction features of the sub-blocks in the first stage, and the reconstruction features of the sub-blocks in this stage can be divided into 4 grouped reconstruction features, wherein the grouped reconstruction feature 1 corresponds to the spatial feature points with a value of 1, the grouped reconstruction feature 2 corresponds to the spatial feature points with a value of 2, the grouped reconstruction feature 3 corresponds to the spatial feature points with a value of 3, and the grouped reconstruction feature 4 corresponds to the spatial feature points with a value of 4. Part2 corresponds to the reconstruction features of the sub-blocks in the second stage, and the reconstruction features of the sub-blocks in this stage can be divided into 4 grouped reconstruction features, wherein the grouped reconstruction feature 1 corresponds to the spatial feature points with a value of 1, the grouped reconstruction feature 2 corresponds to the spatial feature points with a value of 2, the grouped reconstruction feature 3 corresponds to the spatial feature points with a value of 3, and the grouped reconstruction feature 4 corresponds to the spatial feature points with a value of 4. Part3 corresponds to the reconstruction features of the sub-blocks in the third stage. The reconstruction features of the sub-blocks in this stage can be divided into four grouped reconstruction features, among which grouped reconstruction feature 1 corresponds to the spatial feature points with a value of 1, grouped reconstruction feature 2 corresponds to the spatial feature points with a value of 2, grouped reconstruction feature 3 corresponds to the spatial feature points with a value of 3, and grouped reconstruction feature 4 corresponds to the spatial feature points with a value of 4. Part4 corresponds to the reconstruction features of the sub-blocks in the fourth stage. The reconstruction features of the sub-blocks in this stage can be divided into four grouped reconstruction features, among which grouped reconstruction feature 1 corresponds to the spatial feature points with a value of 1, grouped reconstruction feature 2 corresponds to the spatial feature points with a value of 2, grouped reconstruction feature 3 corresponds to the spatial feature points with a value of 3, and grouped reconstruction feature 4 corresponds to the spatial feature points with a value of 4.

[0288] Based on the K grouped reconstruction features corresponding to the sub-blocks in each stage, the grouped reconstruction features with the same index are spliced ​​in the channel dimension to obtain the spliced ​​reconstruction features corresponding to the K channel groups. Figure 13A As shown in the figure, the grouped reconstructed feature 1 of Part 1 and the grouped reconstructed feature 1 of Part 2 are spliced ​​in the channel dimension to obtain the spliced ​​reconstructed feature 1, which corresponds to all spatial feature points with a value of 1. The grouped reconstructed feature 2 of Part 1 and the grouped reconstructed feature 2 of Part 2 are spliced ​​in the channel dimension to obtain the spliced ​​reconstructed feature 2, which corresponds to all spatial feature points with a value of 2.

[0289] See also Figure 13BAs shown, the grouped reconstructed features 1 of Part 1, the grouped reconstructed features 1 of Part 2, the grouped reconstructed features 1 of Part 3, and the grouped reconstructed features 1 of Part 4 are spliced ​​in the channel dimension to obtain the spliced ​​reconstructed feature 1. The spliced ​​reconstructed feature 1 corresponds to all spatial feature points with a value of 1. Similarly, the grouped reconstructed features 2 of all parts are spliced ​​in the channel dimension to obtain the spliced ​​reconstructed feature 2, the grouped reconstructed features 3 of all parts are spliced ​​in the channel dimension to obtain the spliced ​​reconstructed feature 3, and the grouped reconstructed features 4 of all parts are spliced ​​in the channel dimension to obtain the spliced ​​reconstructed feature 4.

[0290] For each channel group, obtain the feature aggregation method corresponding to the feature separation method of the channel group. The feature aggregation method is used to indicate the correspondence between the spatial feature points in the reconstructed features after splicing and the spatial feature points in the reconstructed features after aggregation. The feature aggregation method is the inverse process of the feature separation method. For example, see Figure 13A As shown in , the reconstructed feature 1 after splicing corresponds to all spatial feature maps with a value of 1, and the feature separation method 1 is used to indicate the separation method of the spatial feature map with a value of 1. Therefore, the feature aggregation method 1 corresponding to the reconstructed feature 1 after splicing corresponds to the feature separation method 1. Similarly, the feature aggregation method 2 corresponding to the reconstructed feature 2 after splicing corresponds to the feature separation method 2. Figure 13B As shown, the feature aggregation method 1 corresponding to the reconstructed feature 1 after splicing corresponds to the feature separation method 1, the feature aggregation method 2 corresponding to the reconstructed feature 2 after splicing corresponds to the feature separation method 2, the feature aggregation method 3 corresponding to the reconstructed feature 3 after splicing corresponds to the feature separation method 3, and the feature aggregation method 4 corresponding to the reconstructed feature 4 after splicing corresponds to the feature separation method 4.

[0291] .Based on the feature aggregation method corresponding to the channel group, the spatial feature points in the spliced ​​reconstructed features corresponding to the channel group are rearranged to obtain the aggregated reconstructed features corresponding to the channel group. That is, the feature aggregation method (that is, the inverse process of the feature separation method) is used to rearrange all the spatial feature points to obtain the aggregated reconstructed features, that is, the corresponding spatial feature map.

[0292] The aggregated reconstructed features corresponding to the K channel groups are spliced ​​in the channel dimension to obtain the reconstructed features of the current image block.

[0293] It can be seen from the above technical solutions that in the embodiment of the present application, by adopting a staged prediction model, the peak memory usage is lower, the decoding time of a single block is shorter, and the high-speed parallel decoding capability is provided, so that the neural network can effectively ensure the quality of the reconstructed image block while maintaining low complexity, improve the encoding performance and decoding performance, and reduce complexity. For the neural network-based coding technology, a low-complexity context model solution is proposed, which is applied to the encoding and decoding process of the relevant neural network. By optimizing the overall framework of the mean prediction network, such as adjusting the position of the internal subnet and canceling some subnets, the model complexity is effectively reduced. For each subnet, a lower complexity optimization scheme is proposed to reduce the complexity while keeping the performance degradation small. Two optimization schemes for feature separation and feature aggregation are proposed to make full use of the information between channels and improve performance.

[0294] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 17 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 17 can be implemented in combination.

[0295] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0296] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application. The device is applied to the decoding end, and the device includes: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 17, that is, the processing flow of the decoding end.

[0297] For example, in one possible implementation, a decoder is configured to implement:

[0298] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block;

[0299] For each stage sub-block of the current image block, determine a probability distribution parameter of the sub-block at that stage based on the coefficient hyperparameter feature, and decode the second bitstream of the current image block based on the probability distribution parameter to obtain a residual feature of the sub-block at that stage;

[0300] Determining a mean feature of the sub-block at the current stage based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the sub-block at the current stage by the first prediction network and the output features of the sub-block at the current stage by the second prediction network are used to determine the mean feature; or, the output features of the sub-block at the current stage by the first prediction network and the output features of the sub-block at the previous stage by the second prediction network are used to determine the mean feature;

[0301] Determine the reconstruction feature of the sub-block at this stage based on the mean feature of the sub-block at this stage and the residual feature of the sub-block at this stage;

[0302] The reconstruction features of the sub-blocks at each stage are aggregated to obtain the reconstruction features of the current image block;

[0303] A reconstructed image block of the current image block is determined based on the reconstruction feature of the current image block.

[0304] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application. The device is applied to the encoding end, and the device includes: a memory, which is configured to store video data; an encoder, which is configured to implement the encoding method in the above embodiments 1 to 17, that is, the processing flow of the encoding end.

[0305] For example, in one possible implementation, the encoder is configured to implement:

[0306] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0307] Splitting the feature block into features to be encoded of multiple stage sub-blocks;

[0308] Obtaining coefficient hyperparameter features of the current image block, encoding the coefficient hyperparameter features to obtain a first bitstream of the current image block, and determining probability distribution parameters of sub-blocks in each stage based on the coefficient hyperparameter features;

[0309] For each stage sub-block, a mean feature of the stage sub-block is determined based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the stage sub-block are used to determine the mean feature; or, the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the previous stage sub-block are used to determine the mean feature;

[0310] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0311] For each stage sub-block, the residual features of the stage sub-block are encoded based on the probability distribution parameters of the stage sub-block to obtain a second code stream of the current image block.

[0312] Based on the same application concept as the above method, the decoding end device (also called video decoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Figure 14A As shown. It includes: a processor 1401 and a machine-readable storage medium 1402, the machine-readable storage medium 1402 storing machine-executable instructions that can be executed by the processor 1401; the processor 1401 is used to execute the machine-executable instructions to implement the decoding method of the above-mentioned embodiments 1-17 of this application. For example, in one possible implementation, when the processor 1401 executes the machine-executable instructions, it performs the following steps:

[0313] Decode the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block;

[0314] For each stage sub-block of the current image block, determine a probability distribution parameter of the sub-block at that stage based on the coefficient hyperparameter feature, and decode the second bitstream of the current image block based on the probability distribution parameter to obtain a residual feature of the sub-block at that stage;

[0315] Determining a mean feature of the sub-block at the current stage based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the sub-block at the current stage by the first prediction network and the output features of the sub-block at the current stage by the second prediction network are used to determine the mean feature; or, the output features of the sub-block at the current stage by the first prediction network and the output features of the sub-block at the previous stage by the second prediction network are used to determine the mean feature;

[0316] Determine the reconstruction feature of the sub-block at this stage based on the mean feature of the sub-block at this stage and the residual feature of the sub-block at this stage;

[0317] The reconstruction features of the sub-blocks at each stage are aggregated to obtain the reconstruction features of the current image block;

[0318] A reconstructed image block of the current image block is determined based on the reconstruction feature of the current image block.

[0319] Based on the same application concept as the above method, the encoding end device (also called video encoder) provided in the embodiment of the present application, from the hardware level, its hardware architecture diagram can be specifically referred to as Figure 14BAs shown. It includes: a processor 1411 and a machine-readable storage medium 1412, the machine-readable storage medium 1412 storing machine-executable instructions that can be executed by the processor 1411; the processor 1411 is used to execute the machine-executable instructions to implement the encoding method of the above-mentioned embodiments 1-17 of this application. For example, in one possible implementation, when the processor 1411 executes the machine-executable instructions, it performs the following steps:

[0320] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0321] Splitting the feature block into features to be encoded of multiple stage sub-blocks;

[0322] Obtaining coefficient hyperparameter features of the current image block, encoding the coefficient hyperparameter features to obtain a first bitstream of the current image block, and determining probability distribution parameters of sub-blocks in each stage based on the coefficient hyperparameter features;

[0323] For each stage sub-block, a mean feature of the stage sub-block is determined based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the stage sub-block are used to determine the mean feature; or, the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the previous stage sub-block are used to determine the mean feature;

[0324] Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage;

[0325] For each stage sub-block, the residual features of the stage sub-block are encoded based on the probability distribution parameters of the stage sub-block to obtain a second code stream of the current image block.

[0326] Based on the same application concept as the above-mentioned method, an embodiment of the present application provides an electronic device. The electronic device includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor. The processor is configured to execute the machine-executable instructions to implement the decoding method or encoding method of the above-mentioned embodiments 1-17 of the present application.

[0327] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented, such as the decoding method or encoding method in the above embodiments.

[0328] Based on the same application concept as the above method, an embodiment of the present application also provides a computer application, which, when executed by a processor, can implement the decoding method or encoding method disclosed in the above example of the present application.

[0329] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present application, which can be applied to the decoding end, and the decoding device includes: a decoding module, which is used to decode the first code stream of the current image block to obtain the coefficient hyperparameter feature of the current image block; for each stage sub-block of the current image block, the probability distribution parameter of the stage sub-block is determined based on the coefficient hyperparameter feature, and the second code stream of the current image block is decoded based on the probability distribution parameter to obtain the residual feature of the stage sub-block; a determination module, which is used to determine the mean feature of the stage sub-block based on the mean prediction network; wherein the mean prediction network includes a first A prediction network and a second prediction network, wherein the output features of the sub-block at this stage by the first prediction network and the output features of the sub-block at this stage by the second prediction network are used to determine the mean feature; or, the output features of the sub-block at this stage by the first prediction network and the output features of the sub-block at the previous stage by the second prediction network are used to determine the mean feature; based on the mean feature of the sub-block at this stage and the residual feature of the sub-block at this stage, the reconstruction feature of the sub-block at this stage is determined; a processing module is used to perform feature aggregation on the reconstruction features of the sub-block at each stage to obtain the reconstruction features of the current image block; and determine the reconstructed image block of the current image block based on the reconstruction features of the current image block.

[0330] Exemplarily, for each stage sub-block of the current image block, the decoding module is specifically used to determine the probability distribution parameters of the stage sub-block based on the coefficient hyperparameter feature: input the coefficient hyperparameter feature to the probability hyperparameter decoding network to obtain the probability distribution parameters of the current image block, and the probability distribution parameters are the probability distribution parameters of each stage sub-block of the current image block; or, for the first stage sub-block, determine the probability distribution parameters of the stage sub-block based on the coefficient hyperparameter feature; for the i-th stage sub-block, where i is greater than 1, input the coefficient hyperparameter feature and the reference feature of the i-th stage sub-block to the mean prediction network to obtain the probability distribution parameters of the stage sub-block.

[0331] Exemplarily, when the determination module determines the mean feature of the sub-block of this stage based on the mean prediction network, it is specifically used to: for the first stage sub-block, input the coefficient hyperparameter feature to the mean prediction network to obtain the mean feature of the sub-block of this stage; for the i-th stage sub-block, i is greater than 1, input the coefficient hyperparameter feature and the reference feature of the i-th stage sub-block to the mean prediction network to obtain the mean feature of the sub-block of this stage; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks, or, part of the reconstruction features of the previous i-1 stage sub-blocks, or, the reconstruction features of the i-1th stage sub-block.

[0332] Exemplarily, when the determination module inputs the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the sub-block at that stage, it is specifically configured to: input the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; input the reference feature into the second prediction network to obtain a second prediction feature; perform feature separation on the first prediction feature to obtain a third prediction feature; and generate the mean feature of the sub-block at that stage based on the third prediction feature and the second prediction feature. When the determination module generates the mean feature of the sub-block at that stage based on the third prediction feature and the second prediction feature, it is specifically configured to: if the mean prediction network also includes a prediction fusion network, perform feature concatenation on the third prediction feature and the second prediction feature, input the concatenated feature into the prediction fusion network, and fuse the concatenated feature through the prediction fusion network to obtain the mean feature of the sub-block at that stage; or, if the mean prediction network also includes an addition layer, input the third prediction feature and the second prediction feature into the addition layer, and perform addition processing on the third prediction feature and the second prediction feature through the addition layer to obtain the mean feature of the sub-block at that stage.

[0333] Exemplarily, the mean prediction network also includes a prediction fusion network, and the determination module inputs the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the sub-block at this stage, and is specifically used to: when obtaining the reconstruction feature of each stage sub-block, input the reconstruction feature of the stage sub-block into the second prediction network to obtain the second prediction feature of the stage sub-block, and store the second prediction feature of the stage sub-block; input the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature; perform feature separation on the first prediction feature to obtain the third prediction feature; select the second prediction feature corresponding to the reference feature from all stored second prediction features, and perform feature splicing on the third prediction feature and the selected second prediction feature; input the spliced ​​feature into the prediction fusion network, and perform fusion processing on the spliced ​​feature through the prediction fusion network to obtain the mean feature of the sub-block at this stage.

[0334] Exemplarily, the mean prediction network also includes a feature fusion network and an addition layer, and the determination module inputs the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the sub-block at this stage, and is specifically used to: when obtaining the reconstruction feature of each stage sub-block, input the reconstruction feature of the stage sub-block into the second prediction network to obtain the second prediction feature of the stage sub-block, and store the second prediction feature of the stage sub-block; input the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature; perform feature separation on the first prediction feature to obtain the third prediction feature; select the second prediction feature corresponding to the reference feature from all stored second prediction features, and input the selected second prediction feature into the feature fusion network to obtain the fused feature; input the third prediction feature and the fused feature into the addition layer, and perform addition processing on the third prediction feature and the fused feature through the addition layer to obtain the mean feature of the sub-block at this stage.

[0335] Exemplarily, when the determination module inputs the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature, it is specifically used to: if the first prediction network includes at least one enhancement network and at least one upsampling network, then the coefficient hyperparameter feature is input into the first prediction network, and the enhancement operation and upsampling operation are performed on the coefficient hyperparameter feature through the enhancement network and the upsampling network to obtain the first prediction feature; or, if the first prediction network includes at least one linear layer, at least one enhancement network and at least one upsampling network, then the coefficient hyperparameter feature is input into the first prediction network, and the linear operation, enhancement operation and upsampling operation are performed on the coefficient hyperparameter feature through the linear layer, the enhancement network and the upsampling network to obtain the first prediction feature.

[0336] Exemplarily, when the determination module inputs the reference feature into the second prediction network to obtain the second prediction feature, it is specifically configured to: if the second prediction network includes at least one linear layer and at least one enhancement network, input the reference feature into the second prediction network, and perform a linear operation and an enhancement operation on the reference feature through the linear layer and the enhancement network to obtain the second prediction feature; or, if the second prediction network includes at least one enhancement network and at least one activation layer, input the reference feature into the second prediction network, and perform an enhancement operation and an activation operation on the reference feature through the enhancement network and the activation layer to obtain the second prediction feature;

[0337] The determination module is specifically used to input the reconstruction features of the stage sub-block into the second prediction network to obtain the second prediction features of the stage sub-block: if the second prediction network includes at least one linear layer and at least one enhancement network, then the reconstruction features of the stage sub-block are input into the second prediction network, and linear operations and enhancement operations are performed on the reconstruction features of the stage sub-block through the linear layer and the enhancement network to obtain the second prediction features of the stage sub-block; or, if the second prediction network includes at least one enhancement network and at least one activation layer, then the reconstruction features of the stage sub-block are input into the second prediction network, and enhancement operations and activation operations are performed on the reconstruction features of the stage sub-block through the enhancement network and the activation layer to obtain the second prediction features.

[0338] Exemplarily, the determination module inputs the spliced ​​features into the prediction fusion network, performs fusion processing on the spliced ​​features through the prediction fusion network, and obtains the mean features of the sub-blocks at this stage. Specifically used for: if the prediction fusion network includes at least one linear layer and at least one enhancement network, inputs the spliced ​​features into the prediction fusion network, performs linear operations and enhancement operations on the spliced ​​features through the linear layer and the enhancement network, and obtains the mean features of the sub-blocks at this stage.

[0339] Exemplarily, when the determination module inputs the selected second prediction feature into the feature fusion network to obtain the fused feature, it is specifically used as follows: if the feature fusion network includes at least one linear layer and at least one enhancement network, the selected second prediction feature is input into the feature fusion network, and linear operations and enhancement operations are performed on the selected second prediction feature through the linear layer and the enhancement network to obtain the fused feature.

[0340] Exemplarily, when the determination module performs feature separation on the first prediction feature to obtain the third prediction feature, it is specifically used to: evenly divide the first prediction feature into K parts according to the channel dimension to obtain K prediction group features; wherein each prediction group feature contains N spatial feature maps, and each spatial feature map contains multiple spatial feature points; the K is a positive integer greater than 1, and the N is a positive integer greater than or equal to 1; determine the feature separation method corresponding to each prediction group feature, and different prediction group features correspond to different feature separation methods; for each prediction group feature, based on the feature separation method corresponding to the prediction group feature, the spatial feature points of the prediction group feature are evenly sampled in the spatial domain to obtain K sampling result groups; generate the third prediction feature based on the K sampling result groups; wherein, uniform sampling with different sampling starting points or sampling step sizes is regarded as different feature separation methods.

[0341] Exemplarily, the determination module performs uniformly spaced sampling of the spatial feature points of the prediction group feature in the spatial domain based on the feature separation method corresponding to the prediction group feature, and obtains K sampling result groups, which is specifically used for: for the N spatial feature maps within the prediction group feature, dividing the spatial feature map into multiple sub-blocks in the spatial domain, each sub-block includes N*2*2 spatial feature points; wherein, K is 2, or, K is 4; for each sub-block of the prediction group feature, determining the sampling result group to which each spatial feature point in the sub-block belongs based on the feature separation method corresponding to the prediction group feature; wherein, the feature separation method is used to indicate the correspondence between the spatial feature points and the sampling result group, and the spatial feature points in the sub-block correspond to K sampling result groups; different feature separation methods corresponding to different prediction group features mean that the same spatial feature point of the spatial feature maps in different prediction group features corresponds to different sampling result groups.

[0342] Exemplarily, when the processing module performs feature aggregation on the reconstruction features of the sub-blocks of each stage to obtain the reconstruction features of the current image block, it is specifically used to: for the reconstruction features of the sub-blocks of each stage, divide the reconstruction features of the sub-blocks of the stage into K grouped reconstruction features; based on the K grouped reconstruction features corresponding to the sub-blocks of each stage, perform channel-dimensional splicing on the grouped reconstruction features with the same index to obtain the spliced ​​reconstruction features corresponding to the K channel groups; for each channel group, obtain the feature aggregation method corresponding to the feature separation method of the channel group, and the feature aggregation method is used to indicate the correspondence between the spatial feature points in the spliced ​​reconstruction features and the spatial feature points in the aggregated reconstruction features; based on the feature aggregation method corresponding to the channel group, perform feature rearrangement on the spatial feature points in the spliced ​​reconstruction features corresponding to the channel group to obtain the aggregated reconstruction features corresponding to the channel group; perform channel-dimensional splicing on the aggregated reconstruction features corresponding to the K channel groups to obtain the reconstruction features of the current image block.

[0343] Based on the same application concept as the above method, an encoding device is also proposed in an embodiment of the present application, which is applied to the encoding end, and the device includes: a processing module, which is used to input the current image block into an analysis transformation network to obtain a feature block corresponding to the current image block; split the feature block into features to be encoded for multiple stage sub-blocks; an encoding module, which is used to obtain the coefficient hyperparameter features of the current image block, encode the coefficient hyperparameter features to obtain the first code stream of the current image block, and determine the probability distribution parameters of each stage sub-block based on the coefficient hyperparameter features; a determination module, which is used to determine the mean features of the stage sub-block based on the mean prediction network for each stage sub-block ; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output features of the first prediction network for the sub-block at this stage and the output features of the second prediction network for the sub-block at this stage are used to determine the mean feature; or, the output features of the first prediction network for the sub-block at this stage and the output features of the second prediction network for the sub-block at the previous stage are used to determine the mean feature; the residual features of the sub-block at this stage are determined based on the to-be-encoded features of the sub-block at this stage and the mean features of the sub-block at this stage; the encoding module is further used to encode the residual features of the sub-block at this stage based on the probability distribution parameters of the sub-block at this stage for each sub-block at this stage to obtain a second code stream of the current image block.

[0344] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. The present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. The embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The foregoing is merely an embodiment of the present application and is not intended to limit the present application.

[0345] For those skilled in the art, various modifications and variations are possible in this application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of the claims of this application.

Claims

1. A decoding method, characterized in that: Applied to a decoding end, the method comprises: Decoding the first bitstream of the current image block to obtain coefficient hyperparameter features of the current image block; For each stage sub-block of the current image block, determine the probability distribution parameters of the sub-block at the stage based on the coefficient hyper-parameter feature, decode the second bitstream of the current image block based on the probability distribution parameter, and obtain the residual feature of the sub-block at the stage; Determine the mean feature of the sub-block at this stage based on the mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the sub-block at this stage by the first prediction network and the output feature of the sub-block at this stage by the second prediction network are used to determine the mean feature; or, the output feature of the sub-block at this stage by the first prediction network and the output feature of the sub-block at the previous stage by the second prediction network are used to determine the mean feature; Determine the reconstruction feature of the sub-block at this stage based on the mean feature of the sub-block at this stage and the residual feature of the sub-block at this stage; The reconstruction features of the sub-blocks at each stage are aggregated to obtain the reconstruction features of the current image block; A reconstructed image block of the current image block is determined based on the reconstruction feature of the current image block.

2. The method according to claim 1, characterized in that: The step of determining the probability distribution parameters of the sub-block at each stage of the current image block based on the coefficient hyper-parameter feature comprises: Input the coefficient hyperparameter feature to a probability hyperparameter decoding network to obtain probability distribution parameters of the current image block, where the probability distribution parameters are probability distribution parameters of each stage sub-block of the current image block; or, For the first stage sub-block, determine the probability distribution parameters of the sub-block in this stage based on the coefficient hyper-parameter feature; For the i-th stage sub-block, where i is greater than 1, the coefficient hyper-parameter features and the reference features of the i-th stage sub-block are input into the mean prediction network to obtain the probability distribution parameters of the stage sub-block.

3. The method according to claim 1, characterized in that The determining the mean feature of the sub-block at this stage based on the mean prediction network includes: for the first stage sub-block, inputting the coefficient hyper-parameter feature into the mean prediction network to obtain the mean feature of the sub-block at this stage; For the i-th stage sub-block, i is greater than 1, the coefficient hyper-parameter feature and the reference feature of the i-th stage sub-block are input into the mean prediction network to obtain the mean feature of the stage sub-block; wherein the reference feature includes all the reconstruction features of the previous i-1 stage sub-blocks, or, part of the reconstruction features of the previous i-1 stage sub-blocks, or, the reconstruction features of the i-1-th stage sub-block.

4. The method according to claim 3, characterized in that The step of inputting the coefficient hyperparameter feature and the reference feature into a mean prediction network to obtain the mean feature of the sub-block at this stage includes: Inputting the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; Inputting the reference feature into the second prediction network to obtain a second prediction feature; Performing feature separation on the first prediction feature to obtain a third prediction feature; A mean feature of the sub-block at this stage is generated based on the third prediction feature and the second prediction feature.

5. The method according to claim 4, characterized in that The generating the mean feature of the sub-block at this stage based on the third prediction feature and the second prediction feature comprises: If the mean prediction network also includes a prediction fusion network, the third prediction feature and the second prediction feature are feature spliced, and the spliced ​​features are input into the prediction fusion network, and the spliced ​​features are fused through the prediction fusion network to obtain the mean feature of the sub-block at this stage; or, if the mean prediction network also includes an addition layer, the third prediction feature and the second prediction feature are input into the addition layer, and the third prediction feature and the second prediction feature are added through the addition layer to obtain the mean feature of the sub-block at this stage.

6. The method according to claim 3, characterized in that The mean prediction network also includes a prediction fusion network, and the step of inputting the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the sub-block at this stage includes: When the reconstruction feature of each stage sub-block is obtained, the reconstruction feature of the stage sub-block is input to the second prediction network to obtain the second prediction feature of the stage sub-block, and the second prediction feature of the stage sub-block is stored; Inputting the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; Performing feature separation on the first prediction feature to obtain a third prediction feature; The second prediction feature corresponding to the reference feature is selected from all the stored second prediction features, and the third prediction feature and the selected second prediction feature are feature spliced; the spliced ​​feature is input to the prediction fusion network, and the spliced ​​feature is fused through the prediction fusion network to obtain the mean feature of the sub-block at this stage.

7. The method according to claim 3, characterized in that The mean prediction network also includes a feature fusion network and an addition layer, and the coefficient hyperparameter feature and the reference feature are input into the mean prediction network to obtain the mean feature of the sub-block at this stage, including: When the reconstruction feature of each stage sub-block is obtained, the reconstruction feature of the stage sub-block is input to the second prediction network to obtain the second prediction feature of the stage sub-block, and the second prediction feature of the stage sub-block is stored; Inputting the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; Performing feature separation on the first prediction feature to obtain a third prediction feature; Selecting a second prediction feature corresponding to the reference feature from all stored second prediction features, and inputting the selected second prediction feature into the feature fusion network to obtain a fused feature; The third prediction feature and the fused feature are input to the addition layer, and the third prediction feature and the fused feature are added by the addition layer to obtain the mean feature of the sub-block at this stage.

8. The method according to any one of claims 4 to 7, characterized in that: The step of inputting the coefficient hyperparameter feature to the first prediction network to obtain a first prediction feature includes: If the first prediction network includes at least one enhancement network and at least one upsampling network, the coefficient hyperparameter feature is input to the first prediction network, and the coefficient hyperparameter feature is enhanced and upsampled by the enhancement network and the upsampling network to obtain the first prediction feature; or If the first prediction network includes at least one linear layer, at least one enhancement network and at least one upsampling network, the coefficient hyperparameter feature is input into the first prediction network, and the coefficient hyperparameter feature is subjected to linear operation, enhancement operation and upsampling operation through the linear layer, the enhancement network and the upsampling network to obtain a first prediction feature.

9. The method according to any one of claims 4 to 7, characterized in that: The step of inputting the reference feature to the second prediction network to obtain the second prediction feature comprises: if the second prediction network includes at least one linear layer and at least one enhancement network, inputting the reference feature to the second prediction network, performing linear operation and enhancement operation on the reference feature through the linear layer and the enhancement network, and obtaining the second prediction feature; or, if the second prediction network includes at least one enhancement network and at least one activation layer, inputting the reference feature to the second prediction network, performing enhancement operation and activation operation on the reference feature through the enhancement network and the activation layer, and obtaining the second prediction feature; The step of inputting the reconstructed features of the sub-block at the stage into the second prediction network to obtain the second prediction features of the sub-block at the stage comprises: if the second prediction network comprises at least one linear layer and at least one enhancement network, inputting the reconstructed features of the sub-block at the stage into the second prediction network, performing linear operations and enhancement operations on the reconstructed features of the sub-block at the stage through the linear layer and the enhancement network, and obtaining the second prediction features of the sub-block at the stage; or If the second prediction network includes at least one enhancement network and at least one activation layer, the reconstructed features of the sub-block at this stage are input into the second prediction network, and the enhancement network and the activation layer are used to perform enhancement operations and activation operations on the reconstructed features of the sub-block at this stage to obtain the second prediction features.

10. The method according to any one of claims 4 to 7, characterized in that: The step of inputting the spliced ​​features to the prediction fusion network, fusing the spliced ​​features through the prediction fusion network, and obtaining the mean features of the sub-blocks at this stage includes: if the prediction fusion network includes at least one linear layer and at least one enhancement network, inputting the spliced ​​features to the prediction fusion network, performing linear operations and enhancement operations on the spliced ​​features through the linear layer and the enhancement network, and obtaining the mean features of the sub-blocks at this stage.

11. The method according to any one of claims 4 to 7, characterized in that: The step of inputting the selected second prediction feature into the feature fusion network to obtain the fused feature includes: If the feature fusion network includes at least one linear layer and at least one enhancement network, the selected second prediction feature is input into the feature fusion network, and linear operations and enhancement operations are performed on the selected second prediction feature through the linear layer and the enhancement network to obtain the fused feature.

12. The method according to any one of claims 8 to 11, characterized in that: The enhanced network includes a convolutional layer; or, the enhanced network includes a convolutional layer, an activation layer, a convolutional layer, an activation layer, and a convolutional layer in sequence; or, the enhanced network includes a convolutional layer, an activation layer, and a convolutional layer in sequence; or, the enhanced network includes a convolutional layer, an activation layer, a convolutional layer, and an addition layer in sequence; or, the enhanced network includes a convolutional layer, an activation layer, a convolutional layer, and a multiplication layer in sequence; The upsampling network includes an upsampling layer, a crop layer, and an activation layer in sequence; or, the upsampling network includes an upsampling layer and a crop layer in sequence; or, the upsampling network includes an upsampling layer, a crop layer, an activation layer, and a convolutional layer in sequence; The linear layer includes N convolutional layers, and N is a non-negative integer; The upsampling layer is a deconvolution layer, a Pixshuffle layer, an Unpooling layer, a Nearest layer, a Bilinear layer, or a Bicubic layer; The activation layer is a relu activation layer, a leaky relu activation layer, a sigmoid activation layer, a tanh activation layer, or a gelu activation layer.

13. The method according to any one of claims 4 to 7, characterized in that: The step of performing feature separation on the first prediction feature to obtain a third prediction feature includes: The first prediction feature is evenly divided into K parts according to the channel dimension to obtain K prediction group features; wherein each prediction group feature includes N spatial feature maps, each spatial feature map includes multiple spatial feature points; K is a positive integer greater than 1, and N is a positive integer greater than or equal to 1; Determine the feature separation method corresponding to each prediction group feature, and the feature separation methods corresponding to different prediction group features are different; For each prediction group feature, the spatial feature points of the prediction group feature are sampled evenly in the spatial domain based on the feature separation method corresponding to the prediction group feature, and K sampling result groups are obtained; Generating the third prediction feature based on the K sampling result groups; Among them, uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation methods.

14. The method according to claim 13, characterized in that The feature separation method corresponding to the prediction group feature is used to uniformly sample the spatial feature points of the prediction group feature in the spatial domain, and the K sampling result groups obtained include: For the N spatial domain feature maps in the prediction group feature, the spatial domain feature maps are divided into a plurality of sub-blocks in the spatial domain, each sub-block includes N*2*2 spatial domain feature points; wherein the K is 2, or the K is 4; For each sub-block of the prediction group feature, the sampling result group to which each spatial feature point in the sub-block belongs is determined based on the feature separation method corresponding to the prediction group feature; wherein the feature separation method is used to indicate the correspondence between the spatial feature points and the sampling result groups, and the spatial feature points in the sub-block correspond to K sampling result groups; different feature separation methods corresponding to different prediction group features mean that the same spatial feature point in the spatial feature maps in different prediction group features corresponds to different sampling result groups.

15. The method according to claim 13 or 14, characterized in that The step of performing feature aggregation on the reconstruction features of the sub-blocks at each stage to obtain the reconstruction features of the current image block includes: For the reconstruction features of the sub-blocks at each stage, the reconstruction features of the sub-blocks at this stage are divided into K grouped reconstruction features; Based on the K grouped reconstruction features corresponding to the sub-blocks in each stage, the grouped reconstruction features with the same index are spliced ​​in the channel dimension to obtain the spliced ​​reconstruction features corresponding to the K channel groups; For each channel group, a feature aggregation method corresponding to the feature separation method of the channel group is obtained, where the feature aggregation method is used to indicate the correspondence between the spatial feature points in the reconstructed features after splicing and the spatial feature points in the reconstructed features after aggregation; Based on the feature aggregation method corresponding to the channel group, feature rearrangement is performed on the spatial feature points in the spliced ​​reconstructed features corresponding to the channel group to obtain the aggregated reconstructed features corresponding to the channel group; The aggregated reconstructed features corresponding to the K channel groups are concatenated in the channel dimension to obtain the reconstructed features of the current image block.

16. A coding method, characterized in that: Applied to the encoding end, the method includes: Inputting the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block; Splitting the feature block into features to be encoded of multiple stage sub-blocks; Acquire coefficient hyper-parameter features of the current image block, encode the coefficient hyper-parameter features to obtain a first bitstream of the current image block, and determine probability distribution parameters of sub-blocks at each stage based on the coefficient hyper-parameter features; For each stage sub-block, the mean feature of the stage sub-block is determined based on the mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the stage sub-block are used to determine the mean feature; or, the output feature of the first prediction network for the stage sub-block and the output feature of the second prediction network for the previous stage sub-block are used to determine the mean feature; Determine the residual feature of the sub-block at this stage based on the to-be-encoded feature of the sub-block at this stage and the mean feature of the sub-block at this stage; For each stage sub-block, the residual features of the stage sub-block are encoded based on the probability distribution parameters of the stage sub-block to obtain the second code stream of the current image block.

17. A decoding device, characterized in that: Applied to a decoding end, the device comprises: A decoding module is used to decode the first bitstream of the current image block to obtain the coefficient hyperparameter feature of the current image block; for each stage sub-block of the current image block, determine the probability distribution parameter of the sub-block at this stage based on the coefficient hyperparameter feature, and decode the second bitstream of the current image block based on the probability distribution parameter to obtain the residual feature of the sub-block at this stage; A determination module, used to determine the mean feature of the sub-block at this stage based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the sub-block at this stage by the first prediction network and the output feature of the sub-block at this stage by the second prediction network are used to determine the mean feature; or, the output feature of the sub-block at this stage by the first prediction network and the output feature of the sub-block at a previous stage by the second prediction network are used to determine the mean feature; the reconstruction feature of the sub-block at this stage is determined based on the mean feature of the sub-block at this stage and the residual feature of the sub-block at this stage; The processing module is used to perform feature aggregation on the reconstruction features of the sub-blocks in each stage to obtain the reconstruction features of the current image block; and determine the reconstructed image block of the current image block based on the reconstruction features of the current image block.

18. An encoding device, characterized in that: Applied to the encoding end, the device includes: A processing module is used to input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block; split the feature block into features to be encoded of multiple stage sub-blocks; An encoding module, used for obtaining coefficient hyper-parameter features of the current image block, encoding the coefficient hyper-parameter features to obtain a first bitstream of the current image block, and determining probability distribution parameters of sub-blocks at each stage based on the coefficient hyper-parameter features; A determination module, for determining, for each stage sub-block, the mean feature of the sub-block at the stage based on the mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the sub-block at the stage by the first prediction network and the output feature of the sub-block at the stage by the second prediction network are used to determine the mean feature; or, the output feature of the sub-block at the stage by the first prediction network and the output feature of the sub-block at the previous stage by the second prediction network are used to determine the mean feature; the residual feature of the sub-block at the stage is determined based on the feature to be encoded of the sub-block at the stage and the mean feature of the sub-block at the stage; The encoding module is further used to encode the residual features of each stage sub-block based on the probability distribution parameters of the stage sub-block to obtain a second code stream of the current image block.

19. A decoding end device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method described in any one of claims 1-15.

20. A coding end device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 16.

21. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by the processor, the method described in any one of claims 1 to 15 is implemented; or, when the computer instructions are executed by the processor, the method described in claim 16 is implemented.

Images