A decoding, encoding method, apparatus and device thereof

Abstract

The application provides a decoding method, an encoding method, a decoding device, an encoding device, and a decoding apparatus and an encoding apparatus. The decoding method comprises: decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block; determining, for each stage sub-block of the current image block, a probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature, decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; determining a mean feature of the stage sub-block based on a mean prediction network; determining a reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block; performing feature aggregation on the reconstructed feature of each stage sub-block to obtain a reconstructed feature of the current image block; and determining a reconstructed image block of the current image block based on the reconstructed feature of the current image block. Through the application, the encoding performance and the decoding performance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of coding and decoding, in particular to a decoding and encoding method, device and equipment. BACKGROUND

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy encoding, filtering and other processes. For the prediction process, the prediction process can include intra prediction and inter prediction. Inter prediction refers to using the correlation in the time domain of the video to predict the current pixel using the pixels of the adjacent encoded image, so as to effectively remove the temporal redundancy of the video. Intra prediction refers to using the correlation in the spatial domain of the video to predict the current pixel using the pixels of the encoded block of the current frame image, so as to remove the spatial redundancy of the video.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems such as image classification and object detection. Deep learning has also gradually begun to be applied in the field of coding and decoding, that is, a neural network can be used to encode and decode images. Although the neural network-based coding and decoding method has shown great performance potential, the neural network-based coding and decoding method still has problems such as poor coding performance, poor decoding performance and high complexity. SUMMARY

[0004] Therefore, the present application provides a decoding and encoding method, device and equipment to improve the coding and decoding performance.

[0005] The present application provides a decoding method applied to a decoding end, the method comprising:

[0006] decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block;

[0007] For each stage sub-block of the current image block, determining a probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block;

[0008] determining a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, and the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature;

[0009] determine a reconstruction feature of the current image block based on the reconstruction features of the stage sub-blocks.

[0010] perform feature aggregation on the reconstruction features of each stage sub-block to obtain a reconstruction feature of the current image block;

[0011] determine a reconstruction image block of the current image block based on the reconstruction feature of the current image block.

[0012] The present application provides an encoding method applied to an encoding end, which comprises:

[0013] input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block;

[0014] split the feature block into to-be-encoded features of multiple stage sub-blocks;

[0015] obtain a coefficient hyperparameter feature of the current image block, encode the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determine a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature;

[0016] for each stage sub-block, determine a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, and the output feature of the stage sub-block by the first prediction network and the output feature of the stage sub-block by the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block by the first prediction network and the output feature of the previous stage sub-block by the second prediction network are used to determine the mean feature;

[0017] determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block;

[0018] for each stage sub-block, encode the residual feature of the stage sub-block based on the probability distribution parameter of the stage sub-block to obtain a second code stream of the current image block.

[0019] The present application provides a decoding device applied to a decoding end, which comprises:

[0020] a decoding module, configured to decode a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block, determine a probability distribution parameter of each stage sub-block of the current image block based on the coefficient hyperparameter feature, and decode a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block;

[0021] determining, for each stage sub-block, a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature; determining a residual feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block;

[0022] processing the reconstruction features of each stage sub-block to obtain a reconstruction feature of the current image block; and determining a reconstructed image block of the current image block based on the reconstruction feature of the current image block.

[0023] The application provides an encoding device applied to an encoding end, the device comprising:

[0024] processing the current image block to obtain a feature block corresponding to the current image block; and splitting the feature block into to-be-encoded features of a plurality of stage sub-blocks;

[0025] encoding the coefficient hyperparameter feature of the current image block to obtain a first code stream of the current image block, and determining a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature;

[0026] determining, for each stage sub-block, a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature; determining a residual feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block;

[0027] The encoding module is further configured to, for each stage sub-block, encode the residual feature of the stage sub-block based on the probability distribution parameter of the stage sub-block to obtain a second code stream of the current image block.

[0028] The application provides a decoding end device, comprising a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor;

[0029] The processor is configured to execute the machine executable instructions to implement the decoding method of the above examples.

[0030] The application provides an encoding end device, comprising a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by the processor;

[0031] The processor is configured to execute the machine executable instructions to implement the encoding method of the above examples.

[0032] The application provides an electronic device, comprising a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by the processor; the processor is configured to execute the machine executable instructions to implement the decoding method described above; or the processor is configured to execute the machine executable instructions to implement the encoding method described above.

[0033] The application provides a machine readable storage medium, the machine readable storage medium storing a plurality of computer instructions, the computer instructions being executed by a processor to implement the decoding method described above; or to implement the encoding method described above.

[0034] From the above technical solutions, in the embodiments of the application, by adopting the stage-by-stage prediction model, the peak memory occupation is lower, the decoding time of a single block is smaller, and the neural network has high-speed parallel decoding capability, so that the quality of the reconstructed image block is effectively ensured while maintaining low complexity, the encoding performance and decoding performance are improved, and the complexity is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a schematic diagram of a three-dimensional feature matrix in an embodiment of the application;

[0036] Figure 2 is a flowchart of a decoding method in an embodiment of the application;

[0037] Figure 3 is a flowchart of an encoding method in an embodiment of the application;

[0038] Figure 4 is a schematic diagram of the processing process of an encoding end in an embodiment of the application;

[0039] Figure 5 is a schematic diagram of the processing process of a decoding end in an embodiment of the application;

[0040] Figure 6 is a schematic diagram of the processing process of an encoding end in an embodiment of the application;

[0041] Figures 7A-7E is a schematic diagram of a mean prediction network in an embodiment of the application;

[0042] Figures 8A-8D is a schematic diagram of a mean prediction network in an embodiment of the present application;

[0043] Figures 9A-9L is a schematic diagram of a first prediction network in an embodiment of the present application;

[0044] Figures 10A-10D is a schematic diagram of a second prediction network in an embodiment of the present application;

[0045] Figures 11A-11B is a schematic diagram of a prediction fusion network in an embodiment of the present application;

[0046] Figures 12A-12B is a schematic diagram of a feature fusion network in an embodiment of the present application;

[0047] Figures 13A-13B is a schematic diagram of feature separation in an embodiment of the present application;

[0048] Figure 14A is a hardware structure diagram of a decoding end device in an embodiment of the present application;

[0049] Figure 14B is a hardware structure diagram of an encoding end device in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The terminology used in the embodiments of the present application merely describes specific embodiments and is not intended to limit the present application. The singular forms "a", "an" and "the" used in the embodiments of the present application and the claims are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, third, etc. can be used in the embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "when", or "upon", or "in response to determining".

[0051] The embodiments of the present application propose a decoding method and an encoding method, which can involve the following concepts:

[0052] Entropy Encoding: Entropy encoding refers to an encoding process that does not lose any information according to the entropy principle. The entropy of information is the average amount of information of a source (a measure of uncertainty). The encoding method of entropy encoding can include but is not limited to: Shannon encoding, Huffman encoding and arithmetic encoding.

[0053] Neural Network (NN): Neural network refers to an artificial neural network. The neural network is an operation model composed of a large number of nodes (or neurons) connected to each other. In the neural network, the neuron processing unit can represent different objects such as features, letters, concepts or some meaningful abstract patterns. The types of processing units in the neural network can be divided into three categories: input units, output units and hidden units. The input unit accepts signals and data from the external world; the output unit realizes the output of the processing result; the hidden unit is between the input and output units and cannot be observed from the outside of the system. The connection weight between neurons reflects the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamics of neural network, and to simulate the information processing function of the nervous system of the human brain to different degrees and levels. In the field of video processing, commonly used neural networks can include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0054] Convolutional Neural Network (CNN): Convolutional neural network is a kind of feedforward neural network, which is one of the most representative network structures in deep learning technology. The artificial neuron of convolutional neural network can respond to a part of the surrounding units in the coverage range, and has excellent performance for large image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer), the input of each neuron is connected to the local receptive field of the previous layer, and the local feature is extracted. Once the local feature is extracted, the positional relationship between the local feature and other features is also determined. The second is the feature mapping layer (also called activation layer). Each calculation layer of the neural network is composed of multiple feature mappings. Each feature mapping is a plane, and all the neurons on the plane have equal weights. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of convolutional network. In addition, since the neurons on a mapping plane share weights, the number of free parameters of the network is reduced.

[0055] For example, one of the advantages of the convolutional neural network over the image processing algorithm is that the complex pre-processing process of the image (extracting artificial features, etc.) is avoided, and the original image can be directly inputted for end-to-end learning. One of the advantages of the convolutional neural network over the ordinary neural network is that the ordinary neural network is in a full connection mode, i.e., all the neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, and the network training will be time-consuming or even difficult to train, while the convolutional neural network avoids this difficulty through local connection and weight sharing.

[0056] Deconvolution: The deconvolution layer is also called the transpose convolution layer. The working process of the deconvolution layer is very similar to that of the convolution layer, and the main difference is that the deconvolution layer will be padded to make the output larger than the input (of course, it can also be kept the same). If the stride is 1, the output size is equal to the input size; if the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0057] Generalization Ability: The generalization ability can refer to the adaptability of the machine learning algorithm to new samples. The purpose of learning is to learn the rule hidden in the data pair. The network trained for the data outside the learning set with the same rule can also give a suitable output, and this ability can be called the generalization ability.

[0058] Feature: The feature is a three-dimensional feature matrix or tensor of C*W*H, which is shown in FIG. 1. Figure 1 As shown in FIG. 1, it is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of the neural network or the output of the neural network.

[0059] Rate-Distortion Optimized: There are two indicators to evaluate the coding efficiency: code rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream is, the greater the compression rate is, and the greater the PSNR is, the better the reconstructed image quality is. In mode selection, the judgment formula is essentially a comprehensive evaluation of the two. For example, the cost of a mode: J (mode) = D + λ * R, where D represents Distortion, which can usually be measured using the SSE indicator, which is the sum of the squares of the differences between the reconstructed image block and the source image. In order to achieve cost consideration, the SAD indicator can also be used, which is the sum of the absolute values of the differences between the reconstructed image block and the source image. λ is the Lagrange multiplier, and R is the actual number of bits required for image block coding under the mode, including the total number of bits required for coding mode information, motion information, and residual error. In mode selection, if the rate-distortion principle is used to compare and decide the coding mode, the best coding performance can usually be guaranteed.

[0060] A large number of encoding tools are proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the encoding tool that can obtain the optimal coding performance is often different. Therefore, in the encoding process, RDO (Rate-Distortion Opitimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bit stream. Although this method brings higher encoding complexity, it can adaptively select the optimal mode combination for different contents to obtain the optimal coding performance. The decoding end can obtain the related mode information by directly parsing the marker information, and the complexity is less affected.

[0061] The decoding method and the encoding method in the embodiments of the present application will be described in detail in combination with several embodiments.

[0062] Embodiment 1: A decoding method is proposed in the embodiments of the present application, as shown in Figure 2 The method can be applied to a decoding end (also referred to as a video decoder), and the method can include the following steps.

[0063] Step 201: Decoding the first code stream of the current image block to obtain the coefficient hyperparameter feature of the current image block.

[0064] Step 202, for each stage sub-block of the current image block, determining a probability distribution parameter of the stage sub-block based on the coefficient hyper-parameter feature (i.e., the coefficient hyper-parameter feature of the current image block), and decoding the second code stream of the current image block based on the probability distribution parameter (i.e., the coefficient hyper-parameter feature of the current image block) to obtain a residual feature of the stage sub-block.

[0065] Step 203, for each stage sub-block, determining a mean feature of the stage sub-block based on the mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature.

[0066] For example, for the first stage sub-block, the coefficient hyper-parameter feature can be input into the mean prediction network to obtain the mean feature of the stage sub-block. For the i-th stage sub-block, i is greater than 1, the coefficient hyper-parameter feature and the reference feature of the i-th stage sub-block can be input into the mean prediction network to obtain the mean feature of the stage sub-block. Wherein, the reference feature of the i-th stage sub-block can be obtained based on the reconstructed features of the previous i-1 stage sub-blocks; for example, the reference feature can include all reconstructed features of the previous i-1 stage sub-blocks, or part of the reconstructed features of the previous i-1 stage sub-blocks, or the reconstructed feature of the i-1 stage sub-block, and the reference feature is not limited in this regard.

[0067] Step 204, determining a reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block, and performing feature aggregation on the reconstructed features of each stage sub-block to obtain a reconstructed feature of the current image block.

[0068] Step 205, determining a reconstructed image block of the current image block based on the reconstructed feature of the current image block.

[0069] For example, for each stage sub-block of the current image block, determining the probability distribution parameter of the stage sub-block based on the coefficient hyper-parameter feature can include: inputting the coefficient hyper-parameter feature (i.e., the coefficient hyper-parameter feature of the current image block) into the probability hyper-parameter decoding network to obtain the probability distribution parameter of the current image block, which is the probability distribution parameter of each stage sub-block of the current image block. Or, for the first stage sub-block, the probability distribution parameter of the stage sub-block is determined based on the coefficient hyper-parameter feature of the current image block; for the i-th stage sub-block, i is greater than 1, the coefficient hyper-parameter feature of the current image block and the reference feature of the i-th stage sub-block are input into the mean prediction network to obtain the probability distribution parameter of the stage sub-block.

[0070] For example, the mean prediction network can include a first prediction network and a second prediction network, and inputting the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the subblock in the stage can include but is not limited to: inputting the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; inputting the reference feature into the second prediction network to obtain a second prediction feature; performing feature separation on the first prediction feature to obtain a third prediction feature; and generating the mean feature of the subblock in the stage based on the third prediction feature and the second prediction feature. For example, generating the mean feature of the subblock in the stage based on the third prediction feature and the second prediction feature can include but is not limited to: if the mean prediction network further includes a prediction fusion network, then the third prediction feature and the second prediction feature can be spliced to obtain spliced features, and the spliced features can be input into the prediction fusion network to obtain the mean feature of the subblock in the stage by performing fusion processing on the spliced features by the prediction fusion network; or if the mean prediction network further includes an addition layer, then the third prediction feature and the second prediction feature can be input into the addition layer, and the third prediction feature and the second prediction feature can be added by the addition layer to obtain the mean feature of the subblock in the stage.

[0071] For example, the mean prediction network includes a first prediction network, a second prediction network, and a prediction fusion network, and inputting the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the subblock in the stage can include but is not limited to: when obtaining the reconstructed feature of each subblock in the stage, inputting the reconstructed feature of the subblock in the stage into the second prediction network to obtain a second prediction feature of the subblock in the stage, and storing the second prediction feature of the subblock in the stage. Inputting the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; and performing feature separation on the first prediction feature to obtain a third prediction feature. On this basis, the second prediction feature corresponding to the reference feature can be selected from all the stored second prediction features, and the third prediction feature and the selected second prediction feature can be spliced. Then, the spliced features can be input into the prediction fusion network to obtain the mean feature of the subblock in the stage by performing fusion processing on the spliced features by the prediction fusion network.

[0072] Exemplarily, the mean prediction network comprises a first prediction network, a second prediction network, a feature fusion network and an addition layer, the coefficient hyperparameter feature and the reference feature are input into the mean prediction network to obtain the mean feature of the sub-block in the stage, which can include but is not limited to: when the reconstructed feature of each sub-block in the stage is obtained, the reconstructed feature of the sub-block in the stage is input into the second prediction network to obtain the second prediction feature of the sub-block in the stage, and the second prediction feature of the sub-block in the stage is stored; the coefficient hyperparameter feature is input into the first prediction network to obtain the first prediction feature; the first prediction feature is separated to obtain the third prediction feature; the second prediction feature corresponding to the reference feature is selected from all the stored second prediction features, and the selected second prediction feature is input into the feature fusion network to obtain the fused feature; the third prediction feature and the fused feature are input into the addition layer, and the third prediction feature and the fused feature are added by the addition layer to obtain the mean feature of the sub-block in the stage.

[0073] Exemplarily, the coefficient hyperparameter feature is input into the first prediction network to obtain the first prediction feature, which can include but is not limited to: if the first prediction network comprises at least one enhancement network and at least one up-sampling network, the coefficient hyperparameter feature is input into the first prediction network, and the coefficient hyperparameter feature is enhanced and up-sampled by the enhancement network and the up-sampling network to obtain the first prediction feature. Alternatively, if the first prediction network comprises at least one linear layer, at least one enhancement network and at least one up-sampling network, the coefficient hyperparameter feature is input into the first prediction network, and the coefficient hyperparameter feature is linearly operated, enhanced and up-sampled by the linear layer, the enhancement network and the up-sampling network to obtain the first prediction feature.

[0074] Exemplarily, the reference feature is input into the second prediction network to obtain the second prediction feature, which can include but is not limited to: if the second prediction network comprises at least one linear layer and at least one enhancement network, the reference feature is input into the second prediction network, and the reference feature is linearly operated and enhanced by the linear layer and the enhancement network to obtain the second prediction feature. Alternatively, if the second prediction network comprises at least one enhancement network and at least one activation layer, the reference feature is input into the second prediction network, and the reference feature is enhanced and activated by the enhancement network and the activation layer to obtain the second prediction feature.

[0075] Exemplarily, inputting the reconstructed feature of the stage sub-block to the second prediction network to obtain the second prediction feature of the stage sub-block can include but is not limited to: if the second prediction network includes at least one linear layer and at least one enhancement network, inputting the reconstructed feature of the stage sub-block to the second prediction network, performing linear operation and enhancement operation on the reconstructed feature of the stage sub-block through the linear layer and the enhancement network to obtain the second prediction feature of the stage sub-block. Or, if the second prediction network includes at least one enhancement network and at least one activation layer, inputting the reconstructed feature of the stage sub-block to the second prediction network, performing enhancement operation and activation operation on the reconstructed feature of the stage sub-block through the enhancement network and the activation layer to obtain the second prediction feature.

[0076] Exemplarily, inputting the spliced feature to the prediction fusion network to obtain the mean feature of the stage sub-block by performing fusion processing on the spliced feature through the prediction fusion network can include but is not limited to: if the prediction fusion network includes at least one linear layer and at least one enhancement network, inputting the spliced feature to the prediction fusion network, performing linear operation and enhancement operation on the spliced feature through the linear layer and the enhancement network to obtain the mean feature of the stage sub-block.

[0077] Exemplarily, inputting the selected second prediction feature to the feature fusion network to obtain the fused feature can include but is not limited to: if the feature fusion network includes at least one linear layer and at least one enhancement network, inputting the selected second prediction feature to the at least one linear layer and the at least one enhancement network of the feature fusion network, performing linear operation and enhancement operation on the selected second prediction feature through the linear layer and the enhancement network to obtain the fused feature.

[0078] Exemplarily, the enhancement network includes a convolution layer; or, the enhancement network sequentially includes a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer; or, the enhancement network sequentially includes a convolution layer, an activation layer, and a convolution layer; or, the enhancement network sequentially includes a convolution layer, an activation layer, a convolution layer, and an addition layer; or, the enhancement network sequentially includes a convolution layer, an activation layer, a convolution layer, and a multiplication layer. The up-sampling network sequentially includes an up-sampling layer, a crop layer, and an activation layer; or, the up-sampling network sequentially includes an up-sampling layer and a crop layer; or, the up-sampling network sequentially includes an up-sampling layer, a crop layer, an activation layer, and a convolution layer. The linear layer includes N convolution layers, and N is a non-negative integer. Wherein, the up-sampling layer is a deconvolution layer, or a Pixshuffle layer, or an Unpooling layer, or a Nearest layer, or a Bilinear layer, or a Bicubic layer; wherein, the activation layer is a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer.

[0079] Exemplarily, the feature separation on the first prediction feature to obtain the third prediction feature can include but is not limited to: dividing the first prediction feature into K parts uniformly according to a channel dimension to obtain K prediction group features; wherein each prediction group feature can include N spatial feature maps, and each spatial feature map can include a plurality of spatial feature points; wherein K can be a positive integer greater than 1, and N can be a positive integer greater than or equal to 1. Then, determine the feature separation mode corresponding to each prediction group feature; wherein the feature separation modes corresponding to different prediction group features can be different. For each prediction group feature, the spatial feature points of the prediction group feature can be uniformly sampled in the spatial domain based on the feature separation mode corresponding to the prediction group feature to obtain K sampling result groups, and the third prediction feature is generated based on the K sampling result groups. Wherein, the uniform sampling with different sampling starting points or sampling step lengths can be regarded as different feature separation modes.

[0080] Exemplarily, the feature separation on the first prediction feature to obtain the third prediction feature can include but is not limited to: dividing the first prediction feature into K parts uniformly according to a channel dimension to obtain K prediction group features; wherein each prediction group feature can include N spatial feature maps, and each spatial feature map can include a plurality of spatial feature points; wherein K can be a positive integer greater than 1, and N can be a positive integer greater than or equal to 1. Then, determine the feature separation mode corresponding to each prediction group feature; wherein the feature separation modes corresponding to different prediction group features can be different. For each prediction group feature, the spatial feature points of the prediction group feature can be uniformly sampled in the spatial domain based on the feature separation mode corresponding to the prediction group feature to obtain K sampling result groups, and the third prediction feature is generated based on the K sampling result groups. Wherein, the uniform sampling with different sampling starting points or sampling step lengths can be regarded as different feature separation modes.

[0081] Exemplarily, the feature separation on the first prediction feature to obtain the third prediction feature can include but is not limited to: dividing the first prediction feature into K parts uniformly according to a channel dimension to obtain K prediction group features; wherein each prediction group feature can include N spatial feature maps, and each spatial feature map can include a plurality of spatial feature points; wherein K can be a positive integer greater than 1, and N can be a positive integer greater than or equal to 1. Then, determine the feature separation mode corresponding to each prediction group feature; wherein the feature separation modes corresponding to different prediction group features can be different. For each prediction group feature, the spatial feature points of the prediction group feature can be uniformly sampled in the spatial domain based on the feature separation mode corresponding to the prediction group feature to obtain K sampling result groups, and the third prediction feature is generated based on the K sampling result groups. Wherein, the uniform sampling with different sampling starting points or sampling step lengths can be regarded as different feature separation modes.

[0082] For example, the execution sequence described above is only an example given for the convenience of description, and in actual application, the execution sequence between steps can also be changed, and the execution sequence is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in the specification, and the steps included in the method can be more or less than those described in the specification. In addition, a single step described in the specification can be divided into multiple steps for description in other embodiments; multiple steps described in the specification can also be combined into a single step for description in other embodiments.

[0083] From the above technical solutions, in the embodiments of the present application, by adopting the stage-by-stage prediction model, the peak memory occupancy is lower, the decoding time of a single block is smaller, and the neural network has high-speed parallel decoding capability, so that the quality of the reconstructed image block is effectively guaranteed while maintaining low complexity, the encoding performance and decoding performance are improved, and the complexity is reduced.

[0084] Embodiment 2: An encoding method is proposed in the embodiments of the present application, as shown in Figure 3 The method can be applied to an encoding end (also referred to as a video encoder), and the method can include the following steps:

[0085] Step 301: inputting a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block.

[0086] Step 302: splitting the feature block corresponding to the current image block into to-be-encoded features of a plurality of stage sub-blocks.

[0087] Step 303: obtaining a coefficient hyperparameter feature of the current image block, encoding the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determining a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature.

[0088] Step 304: for each stage sub-block, determining a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature.

[0089] For the first stage sub-block, the coefficient hyper-parameter feature can be input into the mean prediction network to obtain the mean feature of the stage sub-block. For the i-th stage sub-block, i is greater than 1, the coefficient hyper-parameter feature and the reference feature of the i-th stage sub-block can be input into the mean prediction network to obtain the mean feature of the stage sub-block. The reference feature of the i-th stage sub-block can be obtained based on the reconstructed features of the previous i-1 stage sub-blocks, and the reference feature can include all reconstructed features of the previous i-1 stage sub-blocks, or part of the reconstructed features of the previous i-1 stage sub-blocks, or the reconstructed feature of the i-1 stage sub-block. The reference feature of the stage sub-block is not limited.

[0090] In step 305, for each stage sub-block corresponding to the current image block, the residual feature of the stage sub-block is determined based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block, and the residual feature of the stage sub-block is encoded based on the probability distribution parameter of the stage sub-block to obtain the second code stream of the current image block.

[0091] For example, the processing process at the encoding end is similar to the processing process at the decoding end, and the same parts will not be repeated.

[0092] For example, the above execution order is only an example given for convenience of description, and in actual application, the execution order between steps can also be changed, and the execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in the present specification, and the steps included in the method can be more or less than those described in the present specification. In addition, a single step described in the present specification can be divided into multiple steps for description in other embodiments, and multiple steps described in the present specification can be combined into a single step for description in other embodiments.

[0093] From the above technical solutions, in the embodiments of the present application, by using the stage-by-stage prediction model, the peak memory occupation is lower, the encoding time of a single sub-block is smaller, and the neural network has high-speed parallel coding capability. The neural network effectively guarantees the quality of the reconstructed image block while maintaining low complexity, improves the encoding performance and decoding performance, and reduces the complexity.

[0094] Embodiment 3: For embodiments 1 and 2, the processing process at the encoding end can be seen from Figure 4 of course, Figure 4 is only an example of the processing process at the encoding end, and the processing process at the encoding end is not limited.

[0095] After obtaining the current image block x (the current image block x can be the original image block x, i.e., the input image block), the encoding end can analyze and transform the current image block x through an analysis transformation network (such as a neural network) to obtain the image feature y corresponding to the current image block x. Wherein, the feature transformation of the current image block x through the analysis transformation network refers to: transforming the current image block x into the image feature y in the latent domain, so as to facilitate the operation of all subsequent processes in the latent domain.

[0096] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the image, that is, the coding and decoding process for the image block can also be directly used for the image.

[0097] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. For example, the image feature y can be input to a hyperparameter encoding network (such as a neural network) to perform coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. Wherein, the hyperparameter encoding network can be a trained neural network, and the training process of the hyperparameter encoding network is not limited, as long as it can perform coefficient hyperparameter feature transformation on the image feature y. Wherein, the image feature y in the latent domain obtains the hyper-prior latent information z after passing through the hyperparameter encoding network.

[0098] After obtaining the coefficient hyperparameter feature z, the encoding end can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 4 The Q operation in the above formula is a quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain the Bitstream #1 (i.e., the first code stream) corresponding to the current image block, that is, Figure 4 The AE operation in the above formula represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the first code stream corresponding to the current image block. Wherein, the hyperparameter quantization feature or the coefficient hyperparameter feature z carried in the first code stream is mainly used to obtain the parameters of the mean and the probability distribution model.

[0099] After obtaining the first code stream corresponding to the current image block, the encoding end can send the first code stream corresponding to the current image block to the decoding end. The processing process of the decoding end for the first code stream corresponding to the current image block is described in subsequent embodiments.

[0100] After obtaining the first code stream corresponding to the current image block, the encoding end can also decode the first code stream to obtain the hyperparameter quantization feature, that is, Figure 4AD in the formula (1) represents a decoding process, and then the hyper-quantized feature is dequantized to obtain the coefficient hyper-feature z_hat, which can be the same as or different from the coefficient hyper-feature z, Figure 4 The IQ operation in the formula (1) is a dequantization process. Alternatively, the encoding end can also decode the first code stream corresponding to the current image block after obtaining the first code stream, to obtain the coefficient hyper-feature z_hat, without involving the dequantization process of the coefficient hyper-feature z_hat.

[0101] For the encoding process of the first code stream, a fixed probability density model encoding method can be used, and for the decoding process of the first code stream, a fixed probability density model decoding method can be used, and the encoding and decoding processes are not limited.

[0102] After the encoding end obtains the coefficient hyper-feature z_hat, context-based prediction can be performed based on the coefficient hyper-feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block (the determination process of the reconstructed feature y_hat is described in subsequent embodiments), to obtain the prediction value mu (i.e., the mean value mu) of the current image block, for example, the coefficient hyper-feature z_hat and the reconstructed feature y_hat are input into a mean value prediction network, and the mean value prediction network determines the prediction value mu based on the coefficient hyper-feature z_hat and the reconstructed feature y_hat, and the prediction process is not limited. For the context-based prediction process, the input includes the coefficient hyper-feature z_hat and the decoded reconstructed feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu, which is used to obtain a residual by subtracting the original feature, and the reconstructed y is obtained by adding the decoded residual.

[0103] After the encoding end obtains the image feature y, the residual feature r can be determined based on the image feature y and the prediction value mu, for example, the difference between the image feature y and the prediction value mu is taken as the residual feature r. Then, the residual feature r is processed to obtain the image feature s, and the feature processing process is not limited and can be any feature processing mode. It should be noted that the feature processing process is an optional process, that is, the residual feature r can also not be processed, that is, the residual feature r is taken as the image feature s.

[0104] After the encoding end obtains the image feature s, the image feature s can be quantized to obtain the image quantization feature corresponding to the image feature s, that is, Figure 4 The Q operation in the formula (1) is a quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain the Bitstream#2 (i.e., the second code stream) corresponding to the current image block, that is, Figure 4The AE operation in the s represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the image feature s to obtain the second code stream corresponding to the current image block without involving the quantization process of the image feature s.

[0105] After obtaining the second code stream corresponding to the current image block, the encoding end can send the second code stream corresponding to the current image block to the decoding end. The processing process of the decoding end for the second code stream corresponding to the current image block is described in subsequent embodiments.

[0106] After obtaining the second code stream corresponding to the current image block, the encoding end can also decode the second code stream to obtain the image quantized feature, that is, Figure 4 The AD in the s represents a decoding process. Then, the encoding end can dequantize the image quantized feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 4 The IQ operation in the s is a dequantization process. Alternatively, after obtaining the second code stream corresponding to the current image block, the encoding end can also decode the second code stream to obtain the image feature s' without involving the dequantization process of the image quantized feature.

[0107] After obtaining the image feature s', the encoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s'. The feature restoration process is not limited and can be any feature restoration manner to obtain the residual feature r_hat, which can be the same as or different from the residual feature r. It should be noted that the feature restoration process is an optional process. That is, when the feature processing process is not performed, the feature restoration process is not performed. In other words, the image feature s' can be the residual feature r_hat.

[0108] After obtaining the residual feature r_hat, the encoding end determines the reconstructed feature y_hat based on the residual feature r_hat and the prediction value mu. For example, the sum of the residual feature r_hat and the prediction value mu can be taken as the reconstructed feature y_hat.

[0109] After obtaining the reconstructed feature y_hat, the encoding end can perform synthesis transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into a synthesis transformation network, and the synthesis transformation network performs synthesis transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0110] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the second code stream corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encodes the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes the second code stream, the encoding end also needs to first determine the probability distribution model, and then decodes the second code stream based on the probability distribution model.

[0111] To obtain the probability distribution model, continue to refer to Figure 4 As shown in the figure, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, the probability distribution model can be generated based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network, and the training process of the probability hyperparameter decoding network is not limited. The probability hyperparameter decoding network can only perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0112] In a possible implementation, the processing process of the encoding end described above can be performed by a deep learning model or a neural network model, so as to realize an end-to-end image compression and encoding process, and the encoding process is not limited.

[0113] For the processing process of the decoding end of embodiments 1 and 2, refer to Figure 5 of course, Figure 5 which is only an example of the processing process of the decoding end, and the processing process of the decoding end is not limited.

[0114] After obtaining the first code stream corresponding to the current image block, the decoding end can also decode the first code stream to obtain the hyperparameter quantization feature, that is, Figure 5 AD in the figure represents the decoding process, and then the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat, which can be the same as or different from the coefficient hyperparameter feature z, Figure 5 the IQ operation in the figure is the dequantization process. Alternatively, after obtaining the first code stream corresponding to the current image block, the decoding end can also decode the first code stream to obtain the coefficient hyperparameter feature z_hat, without involving the dequantization process of the coefficient hyperparameter feature z_hat.

[0115] For the decoding process of the first code stream, a decoding method of a fixed probability density model can be used, and the decoding method is not limited.

[0116] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the image, that is, the decoding process for the image block can also be directly used for the image.

[0117] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block (the determination process of the reconstructed feature y_hat is described in subsequent embodiments), to obtain the prediction value mu (that is, the mean value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat are input into the mean value prediction network, and the mean value prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat. The prediction process is not limited. For the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded reconstructed feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0118] After obtaining the second code stream corresponding to the current image block, the decoding end can also decode the second code stream to obtain the image quantization feature, that is, Figure 5 AD in the above formula represents the decoding process. Then, the decoding end can perform inverse quantization on the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 5 the IQ operation in the above formula is the inverse quantization process. Alternatively, after obtaining the second code stream corresponding to the current image block, the decoding end can decode the second code stream to obtain the image feature s', without involving the inverse quantization process of the image quantization feature.

[0119] After obtaining the image feature s', the decoding end can perform feature recovery (that is, the inverse process of feature processing) on the image feature s', and the feature recovery process is not limited and can be any feature recovery manner, to obtain the residual feature r_hat, which can be the same as or different from the residual feature r. It should be noted that the feature recovery process is an optional process, that is, when the feature processing process is not performed, the feature recovery process is not performed, that is, the image feature s' can be the residual feature r_hat.

[0120] After obtaining the residual feature r_hat, the decoding end determines the reconstructed feature y_hat based on the residual feature r_hat and the prediction value mu, for example, the sum of the residual feature r_hat and the prediction value mu can be taken as the reconstructed feature y_hat.

[0121] After obtaining the reconstructed feature y_hat, the decoding end can perform a synthesis transformation on the reconstructed feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x, for example, inputting the reconstructed feature y_hat into a synthesis transformation network, performing a synthesis transformation on the reconstructed feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat, thereby completing the image reconstruction process.

[0122] In a possible implementation, when decoding the second code stream, the decoding end needs to first determine the probability distribution model, and then decodes the second code stream based on the probability distribution model. In order to obtain the probability distribution model, continue to refer to Figure 5 As shown in the figure, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p, for example, inputting the coefficient hyperparameter feature z_hat into a probability hyperparameter decoding network, performing a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat by the probability hyperparameter decoding network to obtain the probability distribution parameter p, and after obtaining the probability distribution parameter p, generating the probability distribution model based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network, and the training process of the probability hyperparameter decoding network is not limited. The probability hyperparameter decoding network can only perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0123] In a possible implementation, the processing process of the decoding end described above can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, and the decoding process is not limited.

[0124] For the processing process of the encoding end in Embodiments 1 and 2, refer to Figure 6 As shown in the figure, of course, Figure 6 This is only an example of the processing process of the encoding end, and the processing process of the encoding end is not limited.

[0125] After obtaining the current image block x, the encoding end can perform an analysis transformation on the current image block x by an analysis transformation network to obtain an image feature y corresponding to the current image block x. The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the image, that is, the coding and decoding process is directly used for the image.

[0126] The encoding end performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z, for example, inputting the image feature y into a hyperparameter encoding network, performing a coefficient hyperparameter feature transformation on the image feature y by the hyperparameter encoding network to obtain the coefficient hyperparameter feature z.

[0127] The encoding end quantizes the coefficient hyperparameter feature z to obtain a hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, encodes the hyperparameter quantization feature, and obtains a first code stream corresponding to the current image block. Alternatively, the encoding end can directly encode the coefficient hyperparameter feature z to obtain the first code stream corresponding to the current image block.

[0128] After obtaining the first code stream corresponding to the current image block, the encoding end can send the first code stream corresponding to the current image block to the decoding end. The processing process of the decoding end with respect to the first code stream corresponding to the current image block is described in subsequent embodiments.

[0129] After obtaining the first code stream corresponding to the current image block, the encoding end can also decode the first code stream to obtain the hyperparameter quantization feature, and dequantize the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat. Alternatively, the encoding end can also decode the first code stream to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the coefficient hyperparameter feature z_hat. The encoding process of the first code stream can use a fixed probability density model, and the decoding process of the first code stream can use a fixed probability density model. The encoding and decoding processes are not limited.

[0130] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can input the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat of the previous image block into the mean prediction network. The mean prediction network determines the prediction value mu and the probability distribution parameter p based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat, and generates a probability distribution model based on the probability distribution parameter p. Obviously, unlike the embodiment 3, the probability distribution parameter p is determined by the mean prediction network based on the coefficient hyperparameter feature z_hat and the reconstructed feature y_hat, rather than by the probability hyperparameter decoding network based on the coefficient hyperparameter feature z_hat.

[0131] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is processed to obtain the image feature s, or the residual feature r can not be processed, i.e., the residual feature r is taken as the image feature s.

[0132] After obtaining the image feature s, the encoding end can quantize the image feature s to obtain an image quantization feature corresponding to the image feature s, encode the image quantization feature, and obtain a second code stream corresponding to the current image block. Alternatively, the encoding end can directly encode the image feature s to obtain the second code stream corresponding to the current image block.

[0133] After obtaining the second code stream corresponding to the current image block, the encoding end can send the second code stream corresponding to the current image block to the decoding end. The processing process of the decoding end for the second code stream corresponding to the current image block is described in subsequent embodiments.

[0134] After obtaining the second code stream corresponding to the current image block, the encoding end can decode the second code stream by using the probability distribution model to obtain the image quantization feature, and dequantize the image quantization feature to obtain the image feature s'.

[0135] After obtaining the image feature s', the encoding end can perform feature restoration on the image feature s' to obtain the residual feature r_hat. After obtaining the residual feature r_hat, the encoding end determines the reconstructed feature y_hat based on the residual feature r_hat and the prediction value mu. For example, the sum of the residual feature r_hat and the prediction value mu can be taken as the reconstructed feature y_hat.

[0136] After obtaining the reconstructed feature y_hat, the encoding end can perform synthesis transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the reconstructed feature y_hat is input into the synthesis transformation network, and the synthesis transformation network performs synthesis transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0137] Embodiment 6: For embodiments 1 and 2, regarding the processing process of the decoding end, the processing process can include:

[0138] After obtaining the first code stream corresponding to the current image block, the decoding end can further decode the first code stream to obtain the hyperparameter quantization feature, and dequantize the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat. Alternatively, the decoding end can further decode the first code stream to obtain the coefficient hyperparameter feature z_hat without involving the dequantization process of the coefficient hyperparameter feature z_hat. The decoding process for the first code stream can use a decoding method of a fixed probability density model.

[0139] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the image. That is, the decoding process for the image block can also be directly used for the image.

[0140] After obtaining the coefficient hyper-parameter feature z_hat, the decoding end can input the coefficient hyper-parameter feature z_hat and the reconstructed feature y_hat of the previous image block to the mean prediction network, determine the prediction value mu and the probability distribution parameter p based on the coefficient hyper-parameter feature z_hat and the reconstructed feature y_hat by the mean prediction network, and generate the probability distribution model based on the probability distribution parameter p. Obviously, unlike Embodiment 4, the probability distribution parameter p is determined based on the coefficient hyper-parameter feature z_hat and the reconstructed feature y_hat by the mean prediction network, instead of being determined based on the coefficient hyper-parameter feature z_hat by the probability hyper-parameter decoding network.

[0141] After obtaining the second code stream corresponding to the current image block, the decoding end can decode the second code stream by using the probability distribution model to obtain the image quantization feature, and dequantize the image quantization feature to obtain the image feature s'.

[0142] After obtaining the image feature s', the decoding end can perform feature restoration on the image feature s' to obtain the residual feature r_hat. After obtaining the residual feature r_hat, the decoding end determines the reconstructed feature y_hat based on the residual feature r_hat and the prediction value mu, for example, the sum of the residual feature r_hat and the prediction value mu can be taken as the reconstructed feature y_hat.

[0143] After obtaining the reconstructed feature y_hat, the decoding end can perform synthesis transformation on the reconstructed feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x, for example, input the reconstructed feature y_hat to the synthesis transformation network, perform synthesis transformation on the reconstructed feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat, and thus complete the image reconstruction process.

[0144] Embodiment 7: For Embodiments 1-6, a mean prediction network can be deployed, and in order to obtain an accurate prediction value of an image feature, the mean prediction network needs to use the reconstructed value of the adjacent feature (i.e., based on the coefficient hyperparameter feature z_hat of the current image block and the reconstructed feature y_hat of the previous image block, a context-based prediction is performed to obtain the prediction value mu of the current image block, or the prediction value mu of the current image block and the probability distribution parameter p), therefore, the feature value on the right side (using the reconstructed value of the feature on the left side or the upper left side as a reference) can only be subjected to the encoding process of the current feature after the feature value on the left side or the upper left side is completed, and this dependency relationship makes the encoding between features only serially executable, not in parallel, resulting in a high complexity. The feature value on the right side can only be subjected to the decoding process of the current feature after the feature value on the left side or the upper left side is completed, and this dependency relationship makes the decoding between features only serially executable, not in parallel, resulting in a high complexity.

[0145] In view of the above finding, in this embodiment, by adopting a staged prediction model, the parallelism can be improved, the feature storage cache can be effectively saved, higher rate control accuracy can be achieved, the encoding performance loss is smaller, and better encoding performance and rate control accuracy are achieved, so that the peak memory occupation is lower, the decoding time of a single block is smaller, and high-speed parallel decoding capability is possessed.

[0146] As to the processing process at the encoding end, reference can be made to Figure 4 or Figure 6 The processing process at the encoding end is not limited.

[0147] 1. After obtaining the original image, it is determined whether to block the original image based on the image resolution. For example, if the image resolution is greater than a threshold, the original image is blocked, otherwise, the original image is not blocked.

[0148] If yes, the original image is divided into a plurality of image blocks, and there is an overlapping part between adjacent image blocks. Each image block is the current image block in the above embodiments. If no, the original image is not blocked, and the original image is the current image block in the above embodiments. For the convenience of description, the processing process of one current image block is taken as an example for subsequent description.

[0149] 2. After obtaining the current image block, the current image block is analyzed and transformed by an analysis transformation network to obtain an image feature corresponding to the current image block, which is denoted as a feature block. For example, the current image block is transformed into an image feature in the latent domain by the analysis transformation network, so that all subsequent processes can be operated in the latent domain.

[0150] 3. Perform feature separation on the feature block corresponding to the current image block to obtain K stage sub-blocks of to-be-encoded features y_i (i = 1…K), for example, the feature block corresponding to the current image block is split into to-be-encoded features y_i of multiple stage sub-blocks.

[0151] 4. Obtain the coefficient hyperparameter feature corresponding to the current image block. For example, perform coefficient hyperparameter feature transformation on the feature block corresponding to the current image block to obtain the coefficient hyperparameter feature corresponding to the current image block. For example, input the feature block corresponding to the current image block into the hyperparameter encoding network, and perform coefficient hyperparameter feature transformation on the feature block by the hyperparameter encoding network to obtain the coefficient hyperparameter feature.

[0152] 5. Encode the coefficient hyperparameter feature of the current image block into the first code stream of the current image block. For example, quantize the coefficient hyperparameter feature to obtain the hyperparameter quantized feature corresponding to the coefficient hyperparameter feature, and encode the hyperparameter quantized feature to obtain the first code stream corresponding to the current image block. Alternatively, directly encode the coefficient hyperparameter feature to obtain the first code stream corresponding to the current image block. After obtaining the first code stream corresponding to the current image block, the first code stream corresponding to the current image block is sent to the decoding end. The processing process of the decoding end for the first code stream corresponding to the current image block is described in subsequent embodiments.

[0153] 6. After obtaining the first code stream corresponding to the current image block, the first code stream can also be decoded to obtain the coefficient hyperparameter feature corresponding to the current image block. For example, the first code stream can be decoded to obtain the hyperparameter quantized feature, and then the hyperparameter quantized feature is dequantized to obtain the coefficient hyperparameter feature corresponding to the current image block. Alternatively, the first code stream can be decoded to obtain the coefficient hyperparameter feature corresponding to the current image block without involving the dequantization process.

[0154] For the encoding process of the first code stream, a fixed probability density model encoding method can be used, and for the decoding process of the first code stream, a fixed probability density model decoding method can be used, and the encoding and decoding processes are not limited.

[0155] 7. For each stage sub-block corresponding to the current image block, determine the probability distribution parameter corresponding to the stage sub-block based on the coefficient hyperparameter feature of the current image block, that is, obtain the probability distribution parameter corresponding to each stage sub-block of the current image block.

[0156] For example, for example 3, the coefficient hyperparameter feature corresponding to the current image block can be inverse transformed to obtain the probability distribution parameter. For example, the coefficient hyperparameter feature is input into the probability hyperparameter decoding network, and the coefficient hyperparameter feature is inverse transformed by the probability hyperparameter decoding network to obtain the probability distribution parameter. The probability distribution parameter is the probability distribution parameter of each stage sub-block of the current image block. After obtaining the probability distribution parameter, the probability distribution model can be generated based on the probability distribution parameter. The probability distribution model is the probability distribution model of each stage sub-block of the current image block.

[0157] For example 5, the probability distribution parameter corresponding to each stage sub-block can be determined based on the coefficient hyperparameter feature of the current image block, that is, the probability distribution parameter corresponding to each stage sub-block is determined respectively. For example, for the first stage sub-block, the coefficient hyperparameter feature of the current image block can be input into the mean prediction network to obtain the probability distribution parameter of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameter of the stage sub-block. For the i-th stage sub-block, i is greater than 1, the coefficient hyperparameter feature of the current image block and the reference feature of the i-th stage sub-block can be input into the mean prediction network to obtain the probability distribution parameter of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameter of the stage sub-block.

[0158] 8、After obtaining the to-be-encoded features of the K stage sub-blocks, for the to-be-encoded feature y_1 of the first stage sub-block, the coefficient hyperparameter feature of the current image block is input into the mean prediction network to obtain the mean feature (i.e. the predicted value mu_1, the mean mu_1) of the stage sub-block. For example 5, after the coefficient hyperparameter feature of the current image block is input into the mean prediction network, the mean prediction network simultaneously outputs the mean feature of the stage sub-block and the probability distribution parameter of the stage sub-block.

[0159] The residual feature r_1 is determined based on the to-be-encoded feature y_1 and the mean feature mu_1 of the first stage sub-block, such as the difference between the to-be-encoded feature y_1 and the mean feature mu_1 as the residual feature r_1. Then, the residual feature r_1 is processed to obtain the feature-processed residual feature r_1. The feature processing process is not limited and can be any feature processing mode. Of course, the feature processing is an optional step, and the residual feature r_1 can not be processed.

[0160] The residual feature r_1 (or the residual feature r_1 processed by the feature) is quantized to obtain an image quantized feature, and the image quantized feature is encoded to obtain a second code stream corresponding to the current image block. Alternatively, the residual feature r_1 (or the residual feature r_1 processed by the feature) can be directly encoded to obtain a second code stream corresponding to the current image block. After obtaining the second code stream corresponding to the current image block, the second code stream corresponding to the current image block is sent to the decoding end. For the processing process of the decoding end for the second code stream corresponding to the current image block, see the subsequent embodiments. When encoding the image quantized feature or the residual feature r_1, the probability distribution model corresponding to the probability distribution parameter of the first stage sub-block can be used to encode the image quantized feature or the residual feature r_1 to obtain the second code stream.

[0161] After obtaining the second code stream, the image quantized feature can be obtained by decoding the second code stream using the probability distribution model corresponding to the probability distribution parameter of the first stage sub-block, and the image quantized feature is dequantized, and the dequantized feature is recovered to obtain the residual feature r_hat_1, or the residual feature r_hat_1 is directly obtained after the image quantized feature is dequantized, without involving the feature recovery process. Alternatively, after decoding the second code stream, the decoded feature is recovered to obtain the residual feature r_hat_1, or the residual feature r_hat_1 is directly obtained after decoding the second code stream.

[0162] After obtaining the residual feature r_hat_1, the reconstructed feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, such as the sum of the residual feature r_hat_1 and the mean feature mu_1 as the reconstructed feature y_hat_1.

[0163] 9、After obtaining the to-be-encoded features of the K stage sub-blocks, for the to-be-encoded feature y_i of the i-th (i>1, and i is less than or equal to N) stage sub-block, the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i are input into the mean prediction network to obtain the mean feature (i.e., the predicted value mu_i, the mean mu_i) of the stage sub-block i. For example 5, after the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i are input into the mean prediction network, the mean prediction network can simultaneously output the mean feature of the stage sub-block i and the probability distribution parameter of the stage sub-block i.

[0164] The reference feature for the stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the i-1 stage sub-blocks before the stage sub-block i, i.e., based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature for the stage sub-block i is obtained based on all the reconstructed features y_hat_1 to y_hat_i-1, i.e., the reference feature includes y_hat_1 to y_hat_i-1. Alternatively, the reference feature for the stage sub-block i is obtained based on part of the reconstructed features y_hat_1 to y_hat_i-1, i.e., the reference feature includes part of the reconstructed features y_hat_1 to y_hat_i-1. Alternatively, the reference feature for the stage sub-block i is obtained based on the reconstructed feature of the i-1-th stage sub-block, i.e., the reference feature for the stage sub-block i is y_hat_i-1.

[0165] The residual feature r_i is determined based on the to-be-encoded feature y_i of the i-th stage sub-block and the mean feature mu_i, e.g., the difference between the to-be-encoded feature y_i and the mean feature mu_i is taken as the residual feature r_i. The residual feature r_i is processed to obtain a processed residual feature r_i. Alternatively, the residual feature r_i is not processed.

[0166] The residual feature r_i (or the processed residual feature r_i) is quantized to obtain an image quantized feature, and the image quantized feature is encoded to obtain a second code stream corresponding to the current image block. Alternatively, the residual feature r_i (or the processed residual feature r_i) is directly encoded to obtain a second code stream corresponding to the current image block. The second code stream corresponding to the current image block is sent to a decoding end. For the processing process of the decoding end with respect to the second code stream corresponding to the current image block, see subsequent embodiments. When the image quantized feature or the residual feature r_i is encoded, a probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block can be used to encode the image quantized feature or the residual feature r_i to obtain the second code stream.

[0167] After obtaining the second code stream, the image quantized feature can be obtained by decoding the second code stream using a probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block, and the image quantized feature is dequantized, and the dequantized feature is recovered to obtain the residual feature r_hat_i. Alternatively, the residual feature r_hat_i is directly obtained after the image quantized feature is dequantized. Alternatively, after the second code stream is decoded, the decoded feature is recovered to obtain the residual feature r_hat_i, or the residual feature r_hat_i is directly obtained after the second code stream is decoded.

[0168] After obtaining the residual feature r_hat_i, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i, i.e., the sum of the residual feature r_hat_i and the mean feature mu_i is taken as the reconstructed feature y_hat_i.

[0169] 10、After obtaining the reconstructed features y_hat_i (i = 1…K) of the K stage sub-blocks, the reconstructed features y_hat_i of the K stage sub-blocks can be aggregated to obtain the reconstructed feature of the current image block.

[0170] 11、The reconstructed feature of the current image block is input to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block of the current image block. For example, the synthesis transformation network performs synthesis transformation based on the reconstructed feature of the current image block to obtain the reconstructed image block.

[0171] Embodiment 8: The processing process of the decoding end is not limited. Figure 5 or Figure 6 The processing process of the decoding end is not limited.

[0172] 1、After obtaining the first code stream corresponding to the current image block, the first code stream can be decoded to obtain the coefficient hyperparameter feature corresponding to the current image block. For example, the first code stream can be decoded to obtain the hyperparameter quantization feature, and then the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature corresponding to the current image block. Alternatively, the first code stream can be decoded to obtain the coefficient hyperparameter feature corresponding to the current image block without involving the dequantization process. The decoding process of the first code stream can adopt a decoding method of a fixed probability density model, and the decoding process is not limited.

[0173] 2、For each stage sub-block corresponding to the current image block, the probability distribution parameters corresponding to the stage sub-block are determined based on the coefficient hyperparameter feature of the current image block, i.e., the probability distribution parameters corresponding to each stage sub-block of the current image block are obtained.

[0174] For example, for embodiment 4, the coefficient hyperparameter feature of the current image block can be subjected to coefficient hyperparameter feature inverse transformation to obtain the probability distribution parameters. For example, the coefficient hyperparameter feature is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature to obtain the probability distribution parameters, which are the probability distribution parameters of each stage sub-block of the current image block. After obtaining the probability distribution parameters, the probability distribution model can be generated based on the probability distribution parameters, which is the probability distribution model of each stage sub-block of the current image block.

[0175] For example, for the first stage sub-block, the coefficient hyper-parameter feature of the current image block can be input into the mean prediction network to obtain the probability distribution parameter of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameter of the stage sub-block. For the i-th stage sub-block, i is greater than 1, the coefficient hyper-parameter feature of the current image block and the reference feature of the i-th stage sub-block can be input into the mean prediction network to obtain the probability distribution parameter of the stage sub-block, and the probability distribution model of the stage sub-block is generated based on the probability distribution parameter of the stage sub-block.

[0176] 3. For each stage sub-block corresponding to the current image block, the second code stream of the current image block is decoded based on the probability distribution parameter (probability distribution model) of the stage sub-block to obtain the residual feature of the stage sub-block.

[0177] For example, the probability distribution model corresponding to the probability distribution parameter of the i-th (i = 1…K) stage sub-block can be used to decode the second code stream to obtain the image quantization feature, and the image quantization feature is dequantized, and the dequantized feature is recovered to obtain the residual feature r_hat_i, or the image quantization feature is dequantized to directly obtain the residual feature r_hat_i. Alternatively, after decoding the second code stream, the decoded feature is recovered to obtain the residual feature r_hat_i, or the residual feature r_hat_i can be directly obtained after decoding the second code stream.

[0178] 4. After obtaining the residual features of the K stage sub-blocks, for the residual feature r_hat_1 of the first stage sub-block, the coefficient hyper-parameter feature of the current image block is input into the mean prediction network to obtain the mean feature (i.e. the predicted value mu_1, the mean mu_1) of the stage sub-block. For example 6, after inputting the coefficient hyper-parameter feature of the current image block into the mean prediction network, the mean prediction network simultaneously outputs the mean feature of the stage sub-block and the probability distribution parameter of the stage sub-block.

[0179] Then, the reconstruction feature y_hat_1 of the first stage sub-block is determined based on the residual feature r_hat_1 and the mean feature mu_1, for example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be taken as the reconstruction feature y_hat_1.

[0180] 5、After obtaining the residual features of K stage sub-blocks, for the residual features r_hat_i of the i-th (i>1, and i is less than or equal to K) stage sub-block, the coefficient hyperparameter features of the current image block and the reference features of the stage sub-block i are input into the mean prediction network to obtain the mean features (i.e., the predicted value mu_i, the mean mu_i) of the stage sub-block i. For embodiment 5, after inputting the coefficient hyperparameter features of the current image block and the reference features of the stage sub-block i into the mean prediction network, the mean prediction network can simultaneously output the mean features of the stage sub-block i and the probability distribution parameters of the stage sub-block i.

[0181] Wherein, the reference features of the stage sub-block i (i.e., the i-th stage sub-block) can be obtained based on the reconstructed features of the i-1 stage sub-blocks before the stage sub-block i, i.e., based on the reconstructed features y_hat_1 to y_hat_i-1. For example, the reference features of the stage sub-block i are obtained based on all the reconstructed features in y_hat_1 to y_hat_i-1, i.e., the reference features include y_hat_1 to y_hat_i-1. Or, the reference features of the stage sub-block i are obtained based on part of the reconstructed features in y_hat_1 to y_hat_i-1, i.e., the reference features include part of the reconstructed features in y_hat_1 to y_hat_i-1. Or, the reference features of the stage sub-block i are obtained based on the reconstructed features of the i-1 stage sub-block, i.e., the reference features of the stage sub-block i are y_hat_i-1.

[0182] Wherein, in order to control the cache size, the number of reconstructed features included in the reference features cannot exceed the preset number, assuming that the preset number is M, then if the number of reconstructed features is less than or equal to M, the reference features include all the reconstructed features in y_hat_1 to y_hat_i-1, if the number of reconstructed features is greater than M, the reference features include M reconstructed features (i.e., part of the reconstructed features) in y_hat_1 to y_hat_i-1. In order to select M reconstructed features from y_hat_1 to y_hat_i-1, M reconstructed features can be selected based on a default strategy, such as selecting M reconstructed features closest to the current stage sub-block in the decoding order, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, and so on until M reconstructed features are selected, of course, other ways of selecting M reconstructed features can also be used, which are not limited. Or, the index values of M reconstructed features can also be obtained by decoding the code stream, and M reconstructed features are selected based on the decoded index values.

[0183] Then, the reconstructed feature y_hat_i of the i-th stage sub-block is determined based on the residual feature r_hat_i and the mean feature mu_i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be taken as the reconstructed feature y_hat_i.

[0184] 6、After obtaining the reconstructed features y_hat_i (i = 1…K) of the K stage sub-blocks, the reconstructed features y_hat_i of the K stage sub-blocks can be aggregated to obtain the reconstructed feature of the current image block.

[0185] 7、The reconstructed feature of the current image block is input to the synthesis transformation network, and the synthesis transformation network outputs the reconstructed image block of the current image block. For example, the synthesis transformation network performs synthesis transformation based on the reconstructed feature of the current image block to obtain the reconstructed image block.

[0186] Embodiment 9: For embodiments 1-8, the input of the mean prediction network can be the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i, and the output of the mean prediction network can be the mean feature mu_i of the stage sub-block i. The mean prediction network can be seen from Figure 7A . Alternatively, the input of the mean prediction network can be the coefficient hyperparameter feature of the current image block and the reference feature of the stage sub-block i, and the output of the mean prediction network can be the mean feature mu_i of the stage sub-block i and the probability distribution parameter sigma_i of the stage sub-block i. The mean prediction network can be seen from Figure 7B .

[0187] See Figure 7A and Figure 7B , the mean prediction network can include a first prediction network, a second prediction network and a prediction fusion network. The coefficient hyperparameter feature z_hat can be input to the first prediction network to obtain a first prediction feature. The reference feature (such as y_hat_1 to y_hat_i-1) of the stage sub-block i can be input to the second prediction network to obtain a second prediction feature, and if the stage sub-block i is the first stage sub-block, the reference feature is empty, i.e. the second prediction feature is empty. The first prediction feature can be separated to obtain a third prediction feature Phi_i. The third prediction feature Phi_i and the second prediction feature can be spliced. The spliced feature is input to the prediction fusion network, and the spliced feature is fused by the prediction fusion network to obtain the mean feature mu_i of the stage sub-block i (see Figure 7A ), or the spliced feature is fused by the prediction fusion network to obtain the mean feature mu_i and the probability distribution parameter sigma_i of the stage sub-block i (see Figure 7B ).

[0188] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstruction feature y_hat_i corresponding to stage sub-block i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstruction feature y_hat_i.

[0189] In one possible implementation, in order to make the current prediction point more accurate, the second prediction network uses the already decoded reconstructed feature y_hat as prior information to assist the current feature point in prediction and obtain the corresponding mean feature mu_i.

[0190] See Figure 7C As shown, this is based on the channel. Divide the code evenly and decode half of it using a simFC1 network. Then use simFC2 to decode the other half. A schematic diagram. See also... Figure 7D The diagram shows the structure of the Spatial4Net-2 decoding network. This decoding network can perform decoding based on spatial location. The network structure on the left is the Param_Net network structure.

[0191] Based on the channel, y is divided into two parts, y1 and y2. Within y1 and y2, a multi-stage decoding process is designed based on phase. This is used to decode y. 2 For example, pre-planning Preprocessing is performed to... Based on a special downsampling partitioning method – feature separation. Transform y2∈R {H / 2,W / 2,2c} The spatial dimensions H and W are halved, the number of channels is increased to four times, and the data is divided into four parts based on the channels. This is equivalent to the data before the transformation. The y-axis is then divided according to the phase in the spatial domain. 2 The data is divided into four parts, denoted as y2_1, y2_2, y2_3, and y2_4. The phase descriptions are as follows: y2_1 represents the data in y2 where the spatial domain position is in the even-numbered row and even-numbered column of the y2 spatial domain index, and contains data information for all channels. y2_2 represents the data in y2 where the spatial domain position is in the odd-numbered row and odd-numbered column of the y2 spatial domain index, and contains data information for all channels. y2_3 represents the data in y2 where the spatial domain position is in the even-numbered row and odd-numbered column of the y2 spatial domain index, and contains data information for all channels. y2_4 represents the data in y2 where the spatial domain position is in the odd-numbered row and even-numbered column of the y2 spatial domain index, and contains data information for all channels.

[0192] The four parts are decoded in turn, and the decoding is divided into four stages. Taking y1 decoding as an example: in the first stage, a first prediction feature is obtained by z_hat through a first prediction network, and Psi information is obtained through feature separation, cc_ctx information is obtained from the reconstructed y1_hat through a network, and is filled with 0 to keep consistent with the input channel number required by the Param_Net, and the Param_Net outputs the mean value information mu of the first part. The first part of the reconstructed value y2_hat_1 is obtained by combining the residual error and the Decoder. In the second stage, y2_hat_1 is used as prior information, a feature is obtained through a convolution kernel, and the Psi information and the cc_ctx information channel are spliced and input into the Param_Net to obtain the mean value information mu of the second part. The second part of the reconstructed value y2_hat_2 is obtained by combining the residual error and the Decoder. In the third stage, y2_hat_1 and y2_hat_2 are used as prior information, and a feature is obtained through a convolution kernel after channel splicing, and the Psi information and the cc_ctx information channel are spliced and input into the Param_Net to obtain the mean value information mu of the third part. The third part of the reconstructed value y2_hat_3 is obtained by combining the residual error and the Decoder. In the fourth stage, y2_hat_1, y2_hat_2 and y2_hat_3 are used as prior information, and a feature is obtained through a convolution kernel after channel splicing, and the Psi information and the cc_ctx information channel are spliced and input into the Param_Net to obtain the mean value information mu of the fourth part. The fourth part of the reconstructed value y2_hat_4 is obtained by combining the residual error and the Decoder.

[0193] The reconstructed values of the four parts y2_hat_1, y2_hat_2, y2_hat_3 and y2_hat_4 are obtained jointly, and y2_hat is reorganized through the inverse process of the special downsampling division mode defined, to complete the decoding of y2. Similarly, the decoding process of y1 also has a similar mechanism, but the entire process of y1 does not have cc_ctx, so 0 is filled to ensure that the input channel number received by the param_Net network is consistent, and finally y1_hat and y2_hat are combined to form y_hat, to obtain the final reconstruction.

[0194] The above eight-stage scheme can be degraded to a four-stage scheme if channel division is not performed, only spatial domain division is performed, the complexity is reduced, and at this time, the input of each stage of the four-stage scheme does not come from the context information. In addition, the four-stage scheme can be further degraded to a two-stage scheme, as shown in Figure 7E Fig. 2 shows a schematic diagram of a 2-stage prediction network structure.

[0195] For example, the 8-stage scheme includes two 4-stage processes, but the 8-stage scheme has the process of sequential reconstruction in the channel domain, while the 4-stage scheme does not have the process. From the analysis of the 4-stage scheme: the data input amounts of the four different reconstruction stages are different, and in the subsequent stages, due to the increase of the features of the reconstruction, the input data amount increases, resulting in that the complexity increases linearly with the number of stages. The prediction fusion network occupies a large amount of calculation, and simplifying or canceling the module can reduce the complexity of the prediction network.

[0196] In this embodiment, an encoding method is proposed. An original image x is generated by a corresponding analysis transformation network to obtain y. The y is separated by a feature separation network to obtain K stages of to-be-encoded features y_i (i = 1…K). The y is input into a hyperparameter encoding network to obtain z. The z is quantized to obtain z_hat, and is encoded into a first code stream. The z_hat is decoded based on the first code stream, and the first prediction feature is decoded based on the z_hat. For the first stage feature y_1, Phi is obtained, and the mean value mu_1 is obtained by a mean value prediction network. Based on the mu_1, the residual r_1 of the feature y_1 is obtained, and the r_q_1 is quantized to obtain the second code stream. The r_q_1 is decoded based on the second code stream, and the r_hat_1 is obtained by inverse quantization. The feature is recovered by combining the mean value mu_1 to obtain the reconstructed feature y_hat_1.

[0197] For the i-th stage feature y_i (i > 1), the first prediction feature Phi_i is obtained based on the first code stream, and the mu_i is obtained based on the reconstructed features y_hat_1 to y_hat_i-1 of the previous stages. The residual r_i of the feature y_i is obtained, and the r_q_i is encoded into the second code stream. The r_hat_i is decoded based on the second code stream, and the feature is recovered by combining the mean value mu_i to obtain the reconstructed feature y_hat_i.

[0198] The y_hat_i (i = 1…K) is processed by aggregation to obtain the feature y_hat. The feature y_hat is input into a synthesis transformation network to obtain the Y component reconstructed image. Thus, the encoding method is completed, and the encoding method can be referred to in Embodiment 7.

[0199] In this embodiment, an encoding method is proposed. Figure 3 and Figure 4The decoding method is based on the following process to decode y_hat_i (i = 1…K): Based on the first bitstream, decode z_hat, and obtain the probability parameter sigma of all residuals based on z_hat. Then, decode the second bitstream based on the probability parameter sigma and perform inverse quantization to obtain r_hat. Decode the first-stage feature y_hat_1, and based on z_hat (features from other stages cannot be used in the first stage), obtain the mean mu_1 through a mean prediction network. Obtain the corresponding first-stage residual in r_hat, denoted as r_hat_1, and add the mean mu_1. Perform feature recovery to obtain the reconstructed feature y_hat_1. Decode the i-th stage feature y_hat_i (i>1), based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stages, obtain the mean mu_i through a mean prediction network. Obtain the corresponding stage residual in r_hat, denoted as r_hat_i, and add the mean mu_i. Perform feature recovery to obtain the reconstructed feature y_hat_i.

[0200] The reconstructed features y_hat are obtained by feature aggregation of y_hat_i (i = 1…K). The reconstructed features y_hat are then merged to obtain the reconstructed image. This completes the decoding method, which can be found in Example 8.

[0201] This embodiment proposes a method for... Figure 5 The decoding method is based on the following process to decode y_hat_i (i=1…K): Based on the first bitstream, z_hat is decoded. The first-stage feature y_hat_1 is decoded, and based on z_hat (features from other stages cannot be used in the first stage), the mean mu_1 and probability parameter sigma_1 are obtained through the mean prediction network. The residual is then decoded from the second bitstream using this probability parameter sigma_1, and dequantized to obtain r_hat_1. The mean mu_1 is added to r_hat_1, and feature recovery is performed to obtain the reconstructed feature y_hat_1. Decode the i-th stage feature y_hat_i (i>1). Based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stages, obtain the mean mu_i and probability parameter sigma_i through the mean prediction network. Use this probability parameter sigma_i to decode the i-th stage residual from the second bitstream, and perform inverse quantization to obtain r_hat_i. Add the mean mu_1 to perform feature recovery to obtain the reconstructed feature y_hat_i.

[0202] The reconstructed features y_hat are obtained by feature aggregation of y_hat_i (i = 1…K). The reconstructed features y_hat are then merged to obtain the reconstructed image. This completes the decoding method, which can be found in Example 8.

[0203] Example 11: For Examples 1-8, the input of the mean prediction network can be the coefficient hyperparameter features of the current image block and the reference features of stage sub-block i. The output of the mean prediction network can be the mean feature mu_i of stage sub-block i. The output of the mean prediction network can be the mean feature mu_i of stage sub-block i and the probability distribution parameter sigma_i of stage sub-block i.

[0204] See Figure 8A The diagram illustrates the structure of a mean prediction network, which can include a first prediction network, a second prediction network, and a prediction fusion network. The hyperparameter feature z_hat can be input into the first prediction network to obtain the first prediction feature (i.e., the output feature of the first prediction network for this stage sub-block). Reference features of stage sub-block i (such as y_hat_1 to y_hat_i-1) can be input into the second prediction network to obtain the second prediction feature (i.e., the output feature of the second prediction network for this stage sub-block). If stage sub-block i is stage sub-block 1, then the reference feature is 0, i.e., the second prediction feature is 0. The first prediction feature can be separated to obtain a third prediction feature Phi_i. The third prediction feature Phi_i and the second prediction feature can be concatenated, and the concatenated feature is input into the prediction fusion network. The prediction fusion network can then fuse the concatenated features to obtain the mean feature mu_i of stage sub-block i, or it can fuse the concatenated features to obtain the mean feature mu_i and the probability distribution parameter sigma_i of stage sub-block i.

[0205] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstruction feature y_hat_i corresponding to stage sub-block i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstruction feature y_hat_i.

[0206] See Figure 8B The diagram shown is a schematic of the mean prediction network. Figure 8AIn contrast, the prediction fusion network is removed, and the feature concatenation layer is modified to an element-wise additive layer. That is, the mean prediction network can include a first prediction network, a second prediction network, and an additive layer (ADD). The hyperparameter feature z_hat can be input into the first prediction network to obtain the first prediction feature (i.e., the output feature of the first prediction network for this stage sub-block). The reference features of stage sub-block i (such as y_hat_1 to y_hat_i-1) can be input into the second prediction network to obtain the second prediction feature (i.e., the output feature of the second prediction network for this stage sub-block). If stage sub-block i is stage sub-block 1, then the reference feature is 0, i.e., the second prediction feature is 0. The first prediction feature can be separated to obtain the third prediction feature Phi_i. The third and second prediction features can be input into the additive layer, and the additive layer can perform addition processing on the third and second prediction features (such as element-wise addition processing) to obtain the mean feature mu_i of stage sub-block i. Alternatively, the additive layer can perform addition processing on the third and second prediction features (such as element-wise addition processing) to obtain the mean feature mu_i and probability distribution parameter sigma_i of stage sub-block i.

[0207] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstruction feature y_hat_i corresponding to stage sub-block i. For example, the sum of the residual feature r_hat_i and the mean feature mu_i can be used as the reconstruction feature y_hat_i.

[0208] See Figure 8C The diagram shown is a schematic of the mean prediction network. Figure 8A In contrast, placing the second prediction network after the feature reconstruction process allows it to extract features in a single step. This means that only one reconstructed feature (y_hat_i) needs to be input into the second prediction network at a time, instead of inputting all reconstructed features (y_hat_1 to y_hat_i-1) each time. This simplifies the complexity of the second prediction network, enabling it to extract features in a single pass. See also... Figure 8C The diagram shown is a schematic of the mean prediction network, which may include a first prediction network, a second prediction network, and a prediction fusion network.

[0209] For example, when the reconstructed feature y_hat_i of the stage sub-block i is determined, the coefficient hyper-parameter feature z_hat can be input into the first prediction network to obtain the first prediction feature (i.e., the output feature of the first prediction network for the stage sub-block i). The first prediction feature can be separated to obtain the third prediction feature Phi_i.

[0210] For example, when the reconstructed feature y_hat_i of the stage sub-block i is determined, the coefficient hyper-parameter feature z_hat can be input into the first prediction network to obtain the first prediction feature (i.e., the output feature of the first prediction network for the stage sub-block i). The first prediction feature can be separated to obtain the third prediction feature Phi_i.

[0211] For example, when the reconstructed feature y_hat_i of the stage sub-block i is determined, the coefficient hyper-parameter feature z_hat can be input into the first prediction network to obtain the first prediction feature (i.e., the output feature of the first prediction network for the stage sub-block i). The first prediction feature can be separated to obtain the third prediction feature Phi_i.

[0212] The third prediction feature Phi_i and the selected second prediction features y_f_1 to y_f_i-1 can be concatenated, and the concatenated features can be input into the prediction fusion network. On this basis, the prediction fusion network can be used to fuse the concatenated features to obtain the mean feature mu_i of the stage sub-block i, or the prediction fusion network can be used to fuse the concatenated features to obtain the mean feature mu_i and the probability distribution parameter sigma_i of the stage sub-block i.

[0213] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain the reconstructed feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstructed feature y_hat_i.

[0214] After obtaining the reconstructed feature y_hat_i corresponding to the stage sub-block i, the reconstructed feature y_hat_i of the stage sub-block i can also be input into the second prediction network to obtain the second prediction feature y_f_i of the stage sub-block i, and the second prediction feature y_f_i of the stage sub-block i can be stored. Obviously, this second prediction feature y_f_i can participate in the processing process of the subsequent stage sub-block i.

[0215] Referring to Figure 8D , which is a structural schematic diagram of the mean prediction network, and Figure 8A compared with the mean prediction network, the second prediction network is placed behind the feature reconstruction, so that only one reconstructed feature (i.e. y_hat_i) needs to be input into the second prediction network each time, instead of inputting all reconstructed features (such as y_hat_1 to y_hat_i-1) into the second prediction network each time, which simplifies the complexity of the second prediction network, i.e. the second prediction network can extract features at a time. Referring to Figure 8D , which is a structural schematic diagram of the mean prediction network, the mean prediction network includes a first prediction network, a second prediction network, a feature fusion network and an addition layer ADD.

[0216] For example, when obtaining the reconstructed feature y_hat_i of each stage sub-block, the reconstructed feature y_hat_i of the stage sub-block can be input into the second prediction network to obtain the second prediction feature y_f_i of the stage sub-block, and the second prediction feature y_f_i of the stage sub-block can be stored. Obviously, when determining the reconstructed feature y_hat_i of the stage sub-block i, the second prediction features corresponding to the stage sub-blocks before the stage sub-block i have been stored, such as the second prediction features y_f_1 to y_f_i-1 have been stored.

[0217] In determining the reconstruction feature y_hat_i of the stage sub-block i, the coefficient hyper-parameter feature z_hat can be input into the first prediction network to obtain a first prediction feature (i.e. the output feature of the second prediction network for the previous stage sub-block of the stage sub-block i). The first prediction feature can be separated to obtain a third prediction feature Phi_i. The second prediction features (i.e. the output features of the second prediction network for the previous stage sub-block of the stage sub-block i) corresponding to the reference features can be selected from all the stored second prediction features, such as y_f_1 to y_f_i-1.

[0218] Then, the selected second prediction features are input into the feature fusion network to obtain a fused feature. For example, the feature fusion network can perform feature fusion processing based on the selected second prediction features to obtain the fused feature.

[0219] After obtaining the third prediction feature and the fused feature, the third prediction feature and the fused feature can be input into an addition layer, and the third prediction feature and the fused feature are added (e.g. element-wise addition) by the addition layer to obtain a mean feature mu_i of the stage sub-block i, or the third prediction feature and the fused feature are added (e.g. element-wise addition) by the addition layer to obtain a mean feature mu_i and a probability distribution parameter sigma_i of the stage sub-block i.

[0220] After obtaining the mean feature mu_i, feature reconstruction can be performed based on the mean feature mu_i to obtain a reconstruction feature y_hat_i corresponding to the stage sub-block i, such as the sum of the residual feature r_hat_i and the mean feature mu_i as the reconstruction feature y_hat_i.

[0221] After obtaining the reconstruction feature y_hat_i corresponding to the stage sub-block i, the reconstruction feature y_hat_i of the stage sub-block i can also be input into the second prediction network to obtain a second prediction feature y_f_i of the stage sub-block i, and the second prediction feature y_f_i of the stage sub-block i is stored. Obviously, this second prediction feature y_f_i can participate in the subsequent processing process of the stage sub-block i.

[0222] Of course, Figures 8A-8D The above are only several examples of the mean prediction network, and the structure of the mean prediction network is not limited thereto. In the above mean prediction networks, the model complexity of the mean prediction network can be reduced by optimizing the network structure of each sub-network or optimizing each network structure, such as canceling the prediction fusion network. In addition, by optimizing the feature separation technology and adding certain division rules, the performance can be improved in the case of a small number of stages.

[0223] Example 12: Regarding Example 11, the mean prediction network may include a first prediction network. Hyperparameter features of the coefficients can be input into the first prediction network to obtain first predicted features. For example, the first prediction network may include at least one enhancement network and at least one upsampling network. Hyperparameter features of the coefficients can be input into the first prediction network, and enhancement and upsampling operations can be performed on the hyperparameter features of the coefficients through the enhancement and upsampling networks to obtain the first predicted features. As another example, the first prediction network may include at least one linear layer, at least one enhancement network, and at least one upsampling network. Hyperparameter features of the coefficients can be input into the first prediction network, and linear, enhancement, and upsampling operations can be performed on the hyperparameter features of the coefficients through the linear layer, enhancement network, and upsampling network to obtain the first predicted features. The structure of the first prediction network is described below.

[0224] See Figure 9A The diagram shows the structure of the first prediction network, which includes enhancement network 1, 2x upsampling network 1, enhancement network 2, 2x upsampling network 2, and enhancement network 3. The hyperparameter feature z_hat_i (the hyperparameter feature z_hat of the current image patch) is processed by enhancement network 1 and 2x upsampling network 1 to obtain enhanced large-size feature 1, whose spatial dimensions (width and height) are twice that of the hyperparameter feature z_hat_i. Enhanced large-size feature 1 is processed by enhancement network 2 and 2x upsampling network 2 to obtain enhanced large-size feature 2, whose spatial dimensions (width and height) are twice that of enhanced large-size feature 1. Enhanced large-size feature 2 is then processed by enhancement network 3 to obtain the first prediction feature.

[0225] against Figure 9A The first prediction network shown can have more or fewer augmentation networks, more or fewer 2x upsampling networks, and the 2x upsampling networks can also be replaced with upsampling networks of other factors.

[0226] against Figure 9A The first prediction network shown can have augmentation networks 1 and 2 that are the same or different, augmentation networks 1 and 3 that are the same or different, and augmentation networks 2 and 3 that are the same or different. Furthermore, 2x upsampling network 1 and 2x upsampling network 2 can be the same or different.

[0227] See Figure 9BAs shown in FIG. 1, it is a structural schematic diagram of the first prediction network, the first prediction network includes linear layer 1, enhancement network, linear layer 2, 4 times up-sampling network. After the coefficient hyperparameter feature z_hat passes through the linear layer 1, the enhancement network, the linear layer 2 and the 4 times up-sampling network, the enhanced large size feature is obtained, and the spatial size (width and height) of the enhanced large size feature is four times that of the coefficient hyperparameter feature z_hat. The enhanced large size feature is the first prediction feature output by the first prediction network.

[0228] For the first prediction network shown in FIG. 1, the number of enhancement networks can be more or less, the number of 4 times up-sampling networks can be more or less, and the 4 times up-sampling network can be replaced by other multiple up-sampling networks. Figure 9B For the first prediction network shown in FIG. 1, the number of enhancement networks can be more or less, the number of 4 times up-sampling networks can be more or less, and the 4 times up-sampling network can be replaced by other multiple up-sampling networks.

[0229] Figure 9C As shown in FIG. 2, it is a structural schematic diagram of the first prediction network, the first prediction network includes enhancement network, 4 times up-sampling network. After the coefficient hyperparameter feature z_hat passes through the enhancement network and the 4 times up-sampling network, the enhanced large size feature is obtained, and the spatial size (width and height) of the enhanced large size feature is four times that of the coefficient hyperparameter feature z_hat, that is, the first prediction feature.

[0230] For the first prediction network shown in FIG. 2, the number of enhancement networks can be more or less, and the number of 4 times up-sampling networks can be more or less. The 4 times up-sampling network can be replaced by other multiple up-sampling networks. Figure 9C For the enhancement network in the above first prediction network, the enhancement network can include a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, and a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, and an addition layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, and a multiplication layer. Of course, the above is only a few examples of the enhancement network, and the structure of the enhancement network is not limited. The enhancement network can realize the enhancement function.

[0231] As shown in FIG. 3, it is a structural schematic diagram of the enhancement network, the enhancement network can include at least one convolution layer, and the number of convolution layers can be more, which is not limited. The convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, which is not limited. For example, a 3x3 convolution layer can be selected.

[0232] Figure 9D As shown in FIG. 3, it is a structural schematic diagram of the enhancement network, the enhancement network can include at least one convolution layer, and the number of convolution layers can be more, which is not limited. The convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, which is not limited. For example, a 3x3 convolution layer can be selected.

[0233] As shown in FIG. 3, it is a structural schematic diagram of the enhancement network, the enhancement network can include at least one convolution layer, and the number of convolution layers can be more, which is not limited. The convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, which is not limited. For example, a 3x3 convolution layer can be selected. Figure 9E ​​As shown, it is a structural schematic diagram of the enhancement network, the enhancement network can sequentially include convolution layer 1, activation layer 1, convolution layer 2, activation layer 2, convolution layer 3. Of course, the number of convolution layers can be more or less, which is not limited. The number of activation layers can be more or less, which is not limited. For each convolution layer (such as convolution layer 1, and / or convolution layer 2, and / or convolution layer 3), the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, which is not limited, such as a 3x3 convolution layer can be selected. For each activation layer (such as activation layer 1 and / or activation layer 2), the activation layer can be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer, and the type of the activation layer is not limited, such as a relu activation layer can be selected.

[0234] Referring to Figure 9F As shown, it is a structural schematic diagram of the enhancement network, the enhancement network can sequentially include convolution layer 1, activation layer 1, convolution layer 2. Of course, the number of convolution layers can be more or less, which is not limited. The number of activation layers can be more, which is not limited. For each convolution layer (such as convolution layer 1 and / or convolution layer 2), the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, which is not limited, such as a 3x3 convolution layer can be selected. For each activation layer, the activation layer can be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer, and the type of the activation layer is not limited, such as a relu activation layer can be selected.

[0235] Referring to Figure 9G As shown, it is a structural schematic diagram of the enhancement network, the enhancement network can sequentially include convolution layer 1, activation layer 1, convolution layer 2, addition layer (the addition layer can be replaced by multiplication layer). Of course, the number of convolution layers can be more or less, which is not limited. The number of activation layers can be more, which is not limited. For each convolution layer (such as convolution layer 1 and / or convolution layer 2), the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 3x3 convolution layer can be selected. Activation layer 1 can be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer, and the type of the activation layer is not limited, such as a relu activation layer can be selected.

[0236] Referring to Figure 9HAs shown, the structure diagram of the enhancement network, which can sequentially include a 1x1 convolutional layer (i.e., conv1x1), a relu activation layer, a 3x3 convolutional layer (i.e., conv3x3), a relu activation layer, a 1x1 convolutional layer (i.e., conv1x1).

[0237] For the upsampling network in the first prediction network, the upsampling network can sequentially include an upsampling layer, a crop layer, and an activation layer. Alternatively, the upsampling network can sequentially include an upsampling layer and a crop layer. Alternatively, the upsampling network can sequentially include an upsampling layer, a crop layer, an activation layer, and a convolutional layer. Of course, the above is only a few examples of the upsampling network, and the structure of the upsampling network is not limited. The upsampling network can only implement the function of upsampling operation. The sampling multiple of the upsampling layer can be 2, and the sampling multiple of the upsampling layer can also be 4, which is not limited.

[0238] Referring to Figure 9I As shown, the structure diagram of the upsampling network, which can sequentially include an upsampling layer, a crop layer, and an activation layer. Of course, there can be more upsampling layers, more crop layers, and more activation layers. The upsampling layer can be a deconvolutional layer (such as a 2x2 deconvolutional layer, or a 3x3 deconvolutional layer, or a 4x4 deconvolutional layer, or a 5x5 deconvolutional layer, which is not limited, such as a 4x4 deconvolutional layer), or a Pixshuffle layer, or an Unpooling layer, or a Nearest layer, or a Bilinear layer, or a Bicubic layer. The activation layer can be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer, and the type of the activation layer is not limited, such as a relu activation layer. The crop layer is used to crop the features, so that the spatial resolution of the features becomes smaller, and the number of channels remains unchanged.

[0239] Referring to Figure 9J As shown, the structure diagram of the upsampling network, which can sequentially include an upsampling layer and a crop layer. There can be more upsampling layers and more crop layers. The upsampling layer can be a deconvolutional layer (such as a 4x4 deconvolutional layer), or a Pixshuffle layer, or an Unpooling layer, or a Nearest layer, or a Bilinear layer, or a Bicubic layer.

[0240] Referring to Figure 9KAs shown in FIG. 13, it is a structural schematic diagram of the upsampling network, which can sequentially include an upsampling layer, a crop layer, an activation layer, and a convolution layer. The upsampling layer can be a deconvolution layer (such as a 4x4 deconvolution layer), or a Pixshuffle layer, or an Unpooling layer, or a Nearest layer, or a Bilinear layer, or a Bicubic layer. The activation layer can be a relu activation layer, or a leaky relu activation layer, or a sigmoid activation layer, or a tanh activation layer, or a gelu activation layer, such as a relu activation layer. The convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 3x3 convolution layer.

[0241] For the linear layer in the first prediction network, refer to FIG. 14. Figure 9L As shown in FIG. 14, it is a structural schematic diagram of the linear layer, which can include N convolution layers, and N is a non-negative integer. For each convolution layer, the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 1x1 convolution layer. N represents the number of convolution layers, and the value of N is not limited, N can be 0, which means that the linear layer does not process the input feature.

[0242] For example 11, the mean prediction network can include a second prediction network, and the reference feature (or the reconstructed feature of the stage sub-block, and the processing process of the reference feature is described later) can be input into the second prediction network to obtain a second prediction feature. For example, the second prediction network can include at least one linear layer and at least one enhancement network, and the reference feature can be input into the second prediction network to obtain the second prediction feature through the linear layer and the enhancement network for linear operation and enhancement operation. For another example, the second prediction network can include at least one enhancement network and at least one activation layer, and the reference feature can be input into the second prediction network to obtain the second prediction feature through the enhancement network and the activation layer for enhancement operation and activation operation. The structure of the second prediction network is described below.

[0243] For example 11, the mean prediction network can include a second prediction network, and the reference feature (or the reconstructed feature of the stage sub-block, and the processing process of the reference feature is described later) can be input into the second prediction network to obtain a second prediction feature. For example, the second prediction network can include at least one linear layer and at least one enhancement network, and the reference feature can be input into the second prediction network to obtain the second prediction feature through the linear layer and the enhancement network for linear operation and enhancement operation. For another example, the second prediction network can include at least one enhancement network and at least one activation layer, and the reference feature can be input into the second prediction network to obtain the second prediction feature through the enhancement network and the activation layer for enhancement operation and activation operation. The structure of the second prediction network is described below. Figure 10A As shown in FIG. 15, it is a structural schematic diagram of the second prediction network, which can include a linear layer 1, an enhancement network, and a linear layer 2. After the reference feature (such as y_hat_1 to y_hat_i-1) passes through the linear layer 1, the enhancement network, and the linear layer 2, the second prediction feature can be obtained. For the second prediction network shown in FIG. 15, the number of enhancement networks can be more or less, the number of linear layers can be more or less, and the linear layer 1 and the linear layer 2 can be the same or different. Figure 10A As shown in FIG. 15, it is a structural schematic diagram of the second prediction network, which can include a linear layer 1, an enhancement network, and a linear layer 2. After the reference feature (such as y_hat_1 to y_hat_i-1) passes through the linear layer 1, the enhancement network, and the linear layer 2, the second prediction feature can be obtained. For the second prediction network shown in FIG. 15, the number of enhancement networks can be more or less, the number of linear layers can be more or less, and the linear layer 1 and the linear layer 2 can be the same or different.

[0244] For the enhancement network in the second prediction network, the enhancement network can include one convolution layer. Alternatively, the enhancement network can include, in sequence, a convolution layer, an activation layer, a convolution layer, an activation layer, a convolution layer. Alternatively, the enhancement network can include, in sequence, a convolution layer, an activation layer, a convolution layer. Alternatively, the enhancement network can include, in sequence, a convolution layer, an activation layer, a convolution layer, an addition layer. Alternatively, the enhancement network can include, in sequence, a convolution layer, an activation layer, a convolution layer, a multiplication layer. Of course, the above are only several examples of the enhancement network, and the structure of the enhancement network is not limited thereto, as long as the enhancement network can achieve the enhancement function. For the enhancement network in the second prediction network, reference can be made to the enhancement network in the first prediction network, which will not be repeated here.

[0245] For the linear layer in the second prediction network, the linear layer can include N convolution layers, and N is a non-negative integer. For each convolution layer, the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 1x1 convolution layer. N represents the number of convolution layers, and N is not limited in value, and N can be 0.

[0246] If the number N of convolution layers of the linear layer is 0, when the second prediction network includes the linear layer 1, the enhancement network, and the linear layer 2, the network structure can also be equivalent to that the second prediction network includes the enhancement network, as shown in FIG. 8B, which is a structural schematic diagram of the second prediction network. The second prediction network can include, in sequence, a 1x1 convolution layer (i.e., conv1x1), a relu activation layer, a 3x3 convolution layer (i.e., conv3x3), a relu activation layer, and a 1x1 convolution layer (i.e., conv1x1). Figure 10B

[0247] As shown in FIG. 8C, which is a structural schematic diagram of the second prediction network, the second prediction network can include the enhancement network 1, the activation layer, and the enhancement network 2. After the reference features (such as y_hat_1 to y_hat_i-1) pass through the enhancement network 1, the activation layer, and the enhancement network 2, the second prediction features can be obtained. For the second prediction network, the number of enhancement networks can be more or less, the number of activation layers can be more or less, the enhancement network 1 and the enhancement network 2 can be the same or different. Figure 10C

[0248] ​​For each enhancement network in the second prediction network, the enhancement network may include a convolutional layer. Alternatively, the enhancement network may sequentially include a convolutional layer, an activation layer, a convolutional layer, an activation layer, and another convolutional layer. Alternatively, the enhancement network may sequentially include a convolutional layer, an activation layer, and another convolutional layer. Alternatively, the enhancement network may sequentially include a convolutional layer, an activation layer, a convolutional layer, and an addition layer. Alternatively, the enhancement network may sequentially include a convolutional layer, an activation layer, a convolutional layer, and a multiplication layer. Of course, the above are just a few examples of enhancement networks, and the structure of the enhancement network is not limited, as long as the enhancement network can achieve the enhancement function. For the enhancement networks in the second prediction network, please refer to the enhancement networks in the first prediction network, which will not be repeated here.

[0249] For the activation layer in the second prediction network, the activation layer can be a ReLU activation layer, a leaky ReLU activation layer, a sigmoid activation layer, a tanh activation layer, or a GELU activation layer. There are no restrictions on this, such as choosing a ReLU activation layer.

[0250] For example, targeting Figure 10C The second prediction network shown is referred to in [reference]. Figure 10D As shown, the second prediction network can sequentially include a 1x1 convolutional layer (i.e., conv1x1), a ReLU activation layer, another 1x1 convolutional layer (i.e., conv1x1), a ReLU activation layer, a 3x3 convolutional layer (i.e., conv3x3), a ReLU activation layer, another 1x1 convolutional layer (i.e., conv1x1), a ReLU activation layer, and another 1x1 convolutional layer (i.e., conv1x1). Of course, the above is merely an example of a second prediction network and is not intended to be limiting.

[0251] Example 14: Regarding Example 11, the mean prediction network may include a prediction fusion network. The concatenated features can be input into the prediction fusion network, which then performs fusion processing on the concatenated features to obtain the mean features of the sub-block at that stage. Alternatively, the prediction fusion network can perform fusion processing on the concatenated features to obtain the mean features and probability distribution parameters of the sub-block at that stage. For example, the prediction fusion network may include at least one linear layer and at least one enhancement network. The concatenated features can be input into the prediction fusion network, and the linear layer and enhancement network can perform linear and enhancement operations on the concatenated features to obtain the mean features of the sub-block at that stage, or the mean features and probability distribution parameters of the sub-block at that stage.

[0252] See Figure 11AAs shown in FIG. 1, the prediction fusion network can include a linear layer 1, an enhancement network, and a linear layer 2. After the post-splicing features pass through the linear layer 1, the enhancement network, and the linear layer 2, the mean feature can be obtained, or after the post-splicing features pass through the linear layer 1, the enhancement network, and the linear layer 2, the mean feature and the probability distribution parameter can be obtained. For the prediction fusion network Figure 11A As shown in the prediction fusion network, the number of enhancement networks can be more or less, the number of linear layers can be more or less, the linear layer 1 and the linear layer 2 can be the same or different.

[0253] For the enhancement network in the prediction fusion network, the enhancement network can include a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, and a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, and an addition layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, and a multiplication layer. For the enhancement network in the prediction fusion network, please refer to the enhancement network in the first prediction network, which will not be repeated here.

[0254] For the linear layer in the prediction fusion network, the linear layer can include N convolution layers, and N is a non-negative integer. For each convolution layer, the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, such as a 1x1 convolution layer. N represents the number of convolution layers, and the value of N is not limited. N can be 0.

[0255] If the number of convolution layers N of the linear layer is 0, when the prediction fusion network includes a linear layer 1, an enhancement network, and a linear layer 2, the network structure can also be equivalent to that the prediction fusion network includes an enhancement network, as shown in FIG. 1. Figure 11B As shown in FIG. 1, the prediction fusion network can include a linear layer 1, an enhancement network, and a linear layer 2. After the post-splicing features pass through the linear layer 1, the enhancement network, and the linear layer 2, the mean feature can be obtained, or after the post-splicing features pass through the linear layer 1, the enhancement network, and the linear layer 2, the mean feature and the probability distribution parameter can be obtained. For the prediction fusion network

[0256] In embodiment 15, for embodiment 11, the mean prediction network can include a feature fusion network. The selected second prediction features (y_f_1 to y_f_i-1) can be input into the feature fusion network to obtain the fused features. For example, the feature fusion network includes at least one linear layer and at least one enhancement network. The selected second prediction features are input into the feature fusion network, and the linear layer and the enhancement network are used to perform linear operation and enhancement operation on the selected second prediction features to obtain the fused features.

[0257] As shown in FIG. 1, the prediction fusion network can include a linear layer 1, an enhancement network, and a linear layer 2. After the post-splicing features pass through the linear layer 1, the enhancement network, and the linear layer 2, the mean feature can be obtained, or after the post-splicing features pass through the linear layer 1, the enhancement network, and the linear layer 2, the mean feature and the probability distribution parameter can be obtained. For the prediction fusion network Figure 12AAs shown in FIG. 1, the structure diagram of the feature fusion network can include linear layer 1, an enhancement network, and linear layer 2. After the second predicted feature (y_f_1 to y_f_i-1) passes through linear layer 1, the enhancement network, and linear layer 2, the fused feature can be obtained. For the feature fusion network, the number of enhancement networks can be more or less, the number of linear layers can be more or less, linear layer 1 and linear layer 2 can be the same or different. Figure 12A As shown in FIG. 1, the structure diagram of the feature fusion network can include linear layer 1, an enhancement network, and linear layer 2. After the second predicted feature (y_f_1 to y_f_i-1) passes through linear layer 1, the enhancement network, and linear layer 2, the fused feature can be obtained. For the feature fusion network, the number of enhancement networks can be more or less, the number of linear layers can be more or less, linear layer 1 and linear layer 2 can be the same or different.

[0258] For the enhancement network in the feature fusion network, the enhancement network can include a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, and a convolution layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, and an addition layer. Alternatively, the enhancement network can sequentially include a convolution layer, an activation layer, a convolution layer, and a multiplication layer. For the enhancement network in the feature fusion network, the enhancement network in the first prediction network can be referred to, and thus the description is not repeated here.

[0259] For the linear layer in the feature fusion network, the linear layer can include N convolution layers, and N is a non-negative integer. For each convolution layer, the convolution layer can be a 1x1 convolution layer, or a 3x3 convolution layer, or a 5x5 convolution layer, for example, a 1x1 convolution layer can be selected. N represents the number of convolution layers, and the value of N is not limited, and N can be 0.

[0260] If the number of convolution layers N of the linear layer is 0, when the feature fusion network includes linear layer 1, an enhancement network, and linear layer 2, the network structure can also be equivalent to that the feature fusion network includes an enhancement network, as shown in FIG. 1. Figure 12B As shown in FIG. 1, the structure diagram of the feature fusion network can include linear layer 1, an enhancement network, and linear layer 2. After the second predicted feature (y_f_1 to y_f_i-1) passes through linear layer 1, the enhancement network, and linear layer 2, the fused feature can be obtained. For the feature fusion network, the number of enhancement networks can be more or less, the number of linear layers can be more or less, linear layer 1 and linear layer 2 can be the same or different.

[0261] In embodiments 1-15, for the encoding end and the decoding end, the first predicted feature can be separated into a third predicted feature, for the encoding end, the feature blocks corresponding to the current image block can be separated into K stage sub-blocks to be encoded y_i (i=1…K), and the feature separation manners of the two are the same. Taking the feature separation of the first predicted feature into the third predicted feature as an example, the feature separation process in the embodiments of the present application is introduced.

[0262] Exemplarily, the first prediction feature can be evenly divided into K parts in the channel dimension to obtain K prediction group features; each prediction group feature can include N spatial feature maps, and each spatial feature map can include a plurality of spatial feature points. K can be a positive integer greater than 1, and N can be a positive integer greater than or equal to 1. For example, K can be 2 or 4.

[0263] For example, referring to FIG. 6A, Figure 13A For example, referring to FIG. 6A,

[0264] For example, referring to FIG. 6B, Figure 13B For example, referring to FIG. 6B,

[0265] Exemplarily, a feature separation mode corresponding to each prediction group feature can be determined, and the feature separation modes corresponding to different prediction group features can be different. The feature separation mode is used to indicate the correspondence between the spatial feature points and the sampling result groups, and the feature separation modes corresponding to different prediction group features are different, that is, the same spatial feature point in the spatial feature maps in the different prediction group features (that is, the spatial feature points at the same position in the different spatial feature maps) correspond to different sampling result groups.

[0266] For example, for each prediction group feature, the spatial domain feature points of the prediction group feature can be uniformly sampled based on the corresponding feature separation manner of the prediction group feature, to obtain K sampling result groups, and the third prediction feature is generated based on the K sampling result groups. For example, for N spatial domain feature maps in the prediction group feature, the spatial domain feature maps are divided into multiple sub-blocks according to the spatial domain, and each sub-block includes N*2*2 spatial domain feature points. For each sub-block of the prediction group feature, the sampling result group to which each spatial domain feature point in the sub-block belongs can be determined based on the corresponding feature separation manner of the prediction group feature. The feature separation manner is used to indicate the correspondence between the spatial domain feature point and the sampling result group, and 1*2*2 spatial domain feature points in the sub-block correspond to K sampling result groups. Different prediction group features correspond to different feature separation manners, that is, the same spatial domain feature point in the spatial domain feature map in different prediction group features corresponds to different sampling result groups. Uniform sampling with different sampling starting points or sampling steps is regarded as different feature separation manners.

[0267] For example, as shown in FIG. 1, the feature separation manner 1 corresponding to the prediction group feature 1 can be determined, and the spatial domain feature points of the prediction group feature 1 are uniformly sampled based on the feature separation manner 1 in the spatial domain to obtain 2 sampling result groups. Figure 13A

[0268] In the uniform sampling process, N spatial domain feature maps (such as the spatial domain feature map with a value of 1) in the prediction group feature 1 are divided into multiple sub-blocks, and each sub-block includes N*2*2 spatial domain feature points. For each sub-block, the sampling result group to which each spatial domain feature point in the sub-block belongs is determined based on the feature separation manner 1. For example, the feature separation manner 1 is used to indicate that the spatial domain feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, the spatial domain feature point at the upper right corner of the sub-block corresponds to the sampling result group 2, the spatial domain feature point at the lower left corner of the sub-block corresponds to the sampling result group 2, and the spatial domain feature point at the lower right corner of the sub-block corresponds to the sampling result group 1. Therefore, based on the feature separation manner 1, it is determined that the spatial domain feature point at the upper left corner of each sub-block and the spatial domain feature point at the lower right corner of each sub-block belong to the sampling result group 1, and the spatial domain feature point at the upper right corner of each sub-block and the spatial domain feature point at the lower left corner of each sub-block belong to the sampling result group 2.

[0269] ​In the uniform sampling process, the N spatial feature maps in the prediction group feature 2 (for example, the spatial feature map with a value of 2) are divided into a plurality of sub-blocks, each of which includes N*2*2 spatial feature points, N=1 in the figure, each sub-block includes 1*2*2 spatial feature points, and subsequent descriptions are all based on N*2*2. For each sub-block, determine the sampling result group to which each spatial feature point in the sub-block belongs based on the feature separation mode 2. Moreover, since the feature separation modes corresponding to different prediction group features are different, that is, the same spatial feature point in different prediction group features needs to correspond to different sampling result groups, and different sampling starting points or sampling step lengths are regarded as different feature separation modes.

[0270] Based on this, since the feature separation mode 1 is used to indicate that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, when the feature separation modes corresponding to different prediction group features are different (the same spatial feature point needs to correspond to different sampling result groups), the feature separation mode 2 is used to indicate that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 2. Since the feature separation mode 1 is used to indicate that the spatial feature point at the upper right corner of the sub-block corresponds to the sampling result group 2, the feature separation mode 2 is used to indicate that the spatial feature point at the upper right corner of the sub-block corresponds to the sampling result group 1. Since the feature separation mode 1 is used to indicate that the spatial feature point at the lower left corner of the sub-block corresponds to the sampling result group 2, the feature separation mode 2 is used to indicate that the spatial feature point at the lower left corner of the sub-block corresponds to the sampling result group 1. Since the feature separation mode 1 is used to indicate that the spatial feature point at the lower right corner of the sub-block corresponds to the sampling result group 1, the feature separation mode 2 is used to indicate that the spatial feature point at the lower right corner of the sub-block corresponds to the sampling result group 2. In summary, based on the feature separation mode 2, it is determined that the spatial feature point at the upper left corner of each sub-block and the spatial feature point at the lower right corner of each sub-block belong to the sampling result group 2, and the spatial feature point at the upper right corner of each sub-block and the spatial feature point at the lower left corner of each sub-block belong to the sampling result group 1.

[0271] Referring to Figure 13A As shown, all spatial feature points of the sampling result group 1 can be spliced to obtain a combined feature, that is, Part 1, and all spatial feature points of the sampling result group 2 can be spliced to obtain a combined feature, that is, Part 2. In this way, Part 1 and Part 2 can be combined to obtain a third prediction feature corresponding to the first prediction feature.

[0272] For Figure 13A The feature separation mode shown in FIG. 10 can also be referred to as a 2-stage feature separation mode. Based on the 2-stage feature separation, the information in the channel domain can be extracted when the feature points are reconstructed.

[0273] For example, referring to Figure 13BAs shown, the feature separation mode 1 corresponding to the prediction group feature 1 can be determined, and based on the feature separation mode 1, the spatial domain feature points of the prediction group feature 1 are uniformly sampled in the spatial domain to obtain 4 sampling result groups. The feature separation mode 2 corresponding to the prediction group feature 2 can be determined, and based on the feature separation mode 2, the spatial domain feature points of the prediction group feature 2 are uniformly sampled in the spatial domain to obtain 4 sampling result groups. The feature separation mode 3 corresponding to the prediction group feature 3 can be determined, and based on the feature separation mode 3, the spatial domain feature points of the prediction group feature 3 are uniformly sampled in the spatial domain to obtain 4 sampling result groups. The feature separation mode 4 corresponding to the prediction group feature 4 can be determined, and based on the feature separation mode 4, the spatial domain feature points of the prediction group feature 4 are uniformly sampled in the spatial domain to obtain 4 sampling result groups.

[0274] In the uniform sampling process, the N spatial domain feature maps (such as the spatial domain feature map with the value of 1) in the prediction group feature 1 are divided into multiple sub-blocks, and each sub-block includes N*2*2 spatial domain feature points. For each sub-block, based on the feature separation mode 1, the sampling result group to which each spatial domain feature point in the sub-block belongs is determined. For example, the feature separation mode 1 is used to indicate that the spatial domain feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, the spatial domain feature point at the upper right corner of the sub-block corresponds to the sampling result group 3, the spatial domain feature point at the lower left corner of the sub-block corresponds to the sampling result group 4, and the spatial domain feature point at the lower right corner of the sub-block corresponds to the sampling result group 2. Therefore, based on the feature separation mode 1, it is determined that the spatial domain feature point at the upper left corner of each sub-block belongs to the sampling result group 1, the spatial domain feature point at the upper right corner of each sub-block belongs to the sampling result group 3, the spatial domain feature point at the lower left corner of each sub-block belongs to the sampling result group 4, and the spatial domain feature point at the lower right corner of each sub-block belongs to the sampling result group 2.

[0275] In the uniform sampling process, the N spatial domain feature maps (such as the spatial domain feature map with the value of 2) in the prediction group feature 2 are divided into multiple sub-blocks, and each sub-block includes N*2*2 spatial domain feature points. For each sub-block, based on the feature separation mode 2, the sampling result group to which each spatial domain feature point in the sub-block belongs is determined. Moreover, the feature separation modes corresponding to different prediction group features are different, that is, the same spatial domain feature point in different prediction group features needs to correspond to different sampling result groups, and uniform sampling with different sampling starting points or sampling step lengths is regarded as different feature separation modes.

[0276] Based on this, since feature separation mode 1 is used to indicate that the spatial feature point at the upper left corner of the sub-block corresponds to sampling result group 1, feature separation mode 2 cannot indicate that the spatial feature point at the upper left corner of the sub-block corresponds to sampling result group 1. Taking the case where feature separation mode 2 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to sampling result group 2 as an example. Since feature separation mode 1 is used to indicate that the spatial feature point at the upper right corner of the sub-block corresponds to sampling result group 3, feature separation mode 2 cannot indicate that the spatial feature point at the upper right corner of the sub-block corresponds to sampling result group 3 and sampling result group 2 (occupied by the upper left corner of the sub-block). Taking the case where feature separation mode 2 indicates that the spatial feature point at the upper right corner of the sub-block corresponds to sampling result group 4 as an example. Since feature separation mode 1 is used to indicate that the spatial feature point at the lower left corner of the sub-block corresponds to sampling result group 4, feature separation mode 2 cannot indicate that the spatial feature point at the lower left corner of the sub-block corresponds to sampling result group 4 and sampling result group 2 (occupied by the upper left corner of the sub-block). Taking the case where feature separation mode 2 indicates that the spatial feature point at the lower left corner of the sub-block corresponds to sampling result group 3 as an example. Since feature separation mode 1 is used to indicate that the spatial feature point at the lower right corner of the sub-block corresponds to sampling result group 2, feature separation mode 2 cannot indicate that the spatial feature point at the lower right corner of the sub-block corresponds to sampling result group 2, sampling result group 3 (occupied), and sampling result group 4 (occupied). Taking the case where feature separation mode 2 indicates that the spatial feature point at the lower right corner of the sub-block corresponds to sampling result group 1 as an example. In summary, based on feature separation mode 2, it is determined that the spatial feature point at the upper left corner of each sub-block belongs to sampling result group 2, that the spatial feature point at the upper right corner of each sub-block belongs to sampling result group 4, that the spatial feature point at the lower left corner of each sub-block belongs to sampling result group 3, and that the spatial feature point at the lower right corner of each sub-block belongs to sampling result group 1.

[0277] In the uniform point sampling process, the N spatial feature maps (such as the spatial feature map with a value of 3) in the prediction group feature 3 are divided into multiple sub-blocks, and each sub-block includes N*2*2 spatial feature points. For each sub-block, based on feature separation mode 3, it is determined which sampling result group each spatial feature point in the sub-block belongs to. Moreover, different prediction group features correspond to different feature separation modes, that is, the same spatial feature point of the spatial feature map in different prediction group features needs to correspond to different sampling result groups, and different sampling starting points or sampling step lengths for uniform sampling are regarded as different feature separation modes.

[0278] Based on this, since the feature separation mode 1 is used to indicate that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, the feature separation mode 2 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 2, and the feature separation mode 3 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 3, the feature separation mode 4 cannot indicate that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, the sampling result group 2 and the sampling result group 3. For example, the feature separation mode 4 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 4. Similarly, assuming that the feature separation mode 4 indicates that the spatial feature point at the upper right corner of the sub-block corresponds to the sampling result group 2, the feature separation mode 4 indicates that the spatial feature point at the lower left corner of the sub-block corresponds to the sampling result group 1, and the feature separation mode 4 indicates that the spatial feature point at the lower right corner of the sub-block corresponds to the sampling result group 3.

[0279] Based on the above, based on the feature separation mode 3, it is determined that the spatial feature point at the upper left corner of each sub-block belongs to the sampling result group 3, it is determined that the spatial feature point at the upper right corner of each sub-block belongs to the sampling result group 1, it is determined that the spatial feature point at the lower left corner of each sub-block belongs to the sampling result group 2, and it is determined that the spatial feature point at the lower right corner of each sub-block belongs to the sampling result group 4.

[0280] In the uniform point sampling process, the N spatial feature maps (such as the spatial feature map with a value of 4) in the prediction group feature 4 are divided into multiple sub-blocks, and each sub-block includes N*2*2 spatial feature points. For each sub-block, based on the feature separation mode 4, it is determined that each spatial feature point in the sub-block belongs to a sampling result group. Moreover, different prediction group features correspond to different feature separation modes, that is, the same spatial feature point of the spatial feature map in different prediction group features needs to correspond to different sampling result groups, and different sampling starting points or sampling step lengths for uniform sampling are regarded as different feature separation modes.

[0281] Based on this, since the feature separation mode 1 is used to indicate that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, the feature separation mode 2 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 2, and the feature separation mode 3 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 3, the feature separation mode 4 cannot indicate that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 1, the sampling result group 2 and the sampling result group 3. For example, the feature separation mode 4 indicates that the spatial feature point at the upper left corner of the sub-block corresponds to the sampling result group 4. Similarly, assuming that the feature separation mode 4 indicates that the spatial feature point at the upper right corner of the sub-block corresponds to the sampling result group 2, the feature separation mode 4 indicates that the spatial feature point at the lower left corner of the sub-block corresponds to the sampling result group 1, and the feature separation mode 4 indicates that the spatial feature point at the lower right corner of the sub-block corresponds to the sampling result group 3.

[0282] In summary, based on the feature separation mode 4, the spatial feature points at the top left corner of each sub-block belong to the sampling result group 4, the spatial feature points at the top right corner of each sub-block belong to the sampling result group 2, the spatial feature points at the bottom left corner of each sub-block belong to the sampling result group 1, and the spatial feature points at the bottom right corner of each sub-block belong to the sampling result group 3.

[0283] Referring to Figure 13B As shown in FIG. 6, all the spatial feature points in the sampling result group 1 can be spliced to obtain a combined feature, i.e., Part 1, all the spatial feature points in the sampling result group 2 can be spliced to obtain a combined feature, i.e., Part 2, all the spatial feature points in the sampling result group 3 can be spliced to obtain a combined feature, i.e., Part 3, and all the spatial feature points in the sampling result group 4 can be spliced to obtain a combined feature, i.e., Part 4. In this way, Part 1, Part 2, Part 3, and Part 4 can be combined to obtain the third prediction feature corresponding to the first prediction feature.

[0284] For Figure 13B As shown in FIG. 6, the feature separation mode can also be referred to as a 4-stage feature separation mode. Based on the 4-stage feature separation, the information of the feature points in the channel domain can be extracted during reconstruction.

[0285] In embodiments 1-16, the reconstructed features of each stage sub-block can be aggregated to obtain the reconstructed feature of the current image block. Feature aggregation is the inverse process of feature separation. That is, after the feature separation mode is known, the feature aggregation mode corresponding to the feature separation mode can be obtained. The feature aggregation mode is the inverse process of the feature separation mode. The reconstructed features of each stage sub-block are aggregated based on the feature aggregation mode to obtain the reconstructed feature of the current image block. When the reconstructed features of each stage sub-block are aggregated, the following feature aggregation mode can be used:

[0286] For the reconstructed features of each stage sub-block, the reconstructed features of the stage sub-block are divided into K grouped reconstructed features. For example, referring to Figure 13A As shown in FIG. 6, Part 1 corresponds to the reconstructed features of the first stage sub-block. The reconstructed features of the stage sub-block can be divided into two grouped reconstructed features. Grouped reconstructed feature 1 corresponds to the spatial feature points with a value of 1 (spatial feature map), and grouped reconstructed feature 2 corresponds to the spatial feature points with a value of 2. Part 2 corresponds to the reconstructed features of the second stage sub-block. The reconstructed features of the stage sub-block can be divided into two grouped reconstructed features. Grouped reconstructed feature 1 corresponds to the spatial feature points with a value of 1, and grouped reconstructed feature 2 corresponds to the spatial feature points with a value of 2.

[0287] For example, referring toFigure 13B As shown in FIG. 7, Part 1 corresponds to the reconstructed features of the first stage sub-block, and the reconstructed features of the first stage sub-block can be divided into four grouped reconstructed features. Grouped reconstructed feature 1 corresponds to the spatial feature points with a value of 1, grouped reconstructed feature 2 corresponds to the spatial feature points with a value of 2, grouped reconstructed feature 3 corresponds to the spatial feature points with a value of 3, and grouped reconstructed feature 4 corresponds to the spatial feature points with a value of 4. Part 2 corresponds to the reconstructed features of the second stage sub-block, and the reconstructed features of the second stage sub-block can be divided into four grouped reconstructed features. Grouped reconstructed feature 1 corresponds to the spatial feature points with a value of 1, grouped reconstructed feature 2 corresponds to the spatial feature points with a value of 2, grouped reconstructed feature 3 corresponds to the spatial feature points with a value of 3, and grouped reconstructed feature 4 corresponds to the spatial feature points with a value of 4. Part 3 corresponds to the reconstructed features of the third stage sub-block, and the reconstructed features of the third stage sub-block can be divided into four grouped reconstructed features. Grouped reconstructed feature 1 corresponds to the spatial feature points with a value of 1, grouped reconstructed feature 2 corresponds to the spatial feature points with a value of 2, grouped reconstructed feature 3 corresponds to the spatial feature points with a value of 3, and grouped reconstructed feature 4 corresponds to the spatial feature points with a value of 4. Part 4 corresponds to the reconstructed features of the fourth stage sub-block, and the reconstructed features of the fourth stage sub-block can be divided into four grouped reconstructed features. Grouped reconstructed feature 1 corresponds to the spatial feature points with a value of 1, grouped reconstructed feature 2 corresponds to the spatial feature points with a value of 2, grouped reconstructed feature 3 corresponds to the spatial feature points with a value of 3, and grouped reconstructed feature 4 corresponds to the spatial feature points with a value of 4.

[0288] Based on the K grouped reconstructed features corresponding to each stage sub-block, the grouped reconstructed features with the same index are spliced in the channel dimension to obtain the spliced reconstructed features corresponding to K channel groups. Referring to FIG. 7, the grouped reconstructed feature 1 of Part 1 and the grouped reconstructed feature 1 of Part 2 are spliced in the channel dimension to obtain the spliced reconstructed feature 1, and the spliced reconstructed feature 1 corresponds to all spatial feature points with a value of 1. The grouped reconstructed feature 2 of Part 1 and the grouped reconstructed feature 2 of Part 2 are spliced in the channel dimension to obtain the spliced reconstructed feature 2, and the spliced reconstructed feature 2 corresponds to all spatial feature points with a value of 2. Figure 13A

[0289] Referring to FIG. 7, the grouped reconstructed feature 1 of Part 1 and the grouped reconstructed feature 1 of Part 2 are spliced in the channel dimension to obtain the spliced reconstructed feature 1, and the spliced reconstructed feature 1 corresponds to all spatial feature points with a value of 1. The grouped reconstructed feature 2 of Part 1 and the grouped reconstructed feature 2 of Part 2 are spliced in the channel dimension to obtain the spliced reconstructed feature 2, and the spliced reconstructed feature 2 corresponds to all spatial feature points with a value of 2. Figure 13B ​As shown, the channel dimension splicing is performed on the group reconstruction feature 1 of Part 1, the group reconstruction feature 1 of Part 2, the group reconstruction feature 1 of Part 3, and the group reconstruction feature 1 of Part 4 to obtain the spliced reconstruction feature 1, which corresponds to all spatial feature points with a value of 1. Similarly, the channel dimension splicing is performed on the group reconstruction feature 2 of all Parts to obtain the spliced reconstruction feature 2, the channel dimension splicing is performed on the group reconstruction feature 3 of all Parts to obtain the spliced reconstruction feature 3, and the channel dimension splicing is performed on the group reconstruction feature 4 of all Parts to obtain the spliced reconstruction feature 4.

[0290] For each channel group, a feature aggregation manner corresponding to the feature separation manner of the channel group is obtained, the feature aggregation manner being used to indicate the correspondence between the spatial feature points in the spliced reconstruction feature and the spatial feature points in the aggregated reconstruction feature, and the feature aggregation manner being the inverse process of the feature separation manner. For example, referring to FIG. 6, the feature aggregation manner 1 corresponds to the feature separation manner 1, the feature aggregation manner 2 corresponds to the feature separation manner 2, the feature aggregation manner 3 corresponds to the feature separation manner 3, and the feature aggregation manner 4 corresponds to the feature separation manner 4. Figure 13A As shown, the spliced reconstruction feature 1 corresponds to all spatial feature maps with a value of 1, and the feature separation manner 1 is used to indicate the separation manner of the spatial feature map with a value of 1. Therefore, the feature aggregation manner 1 corresponding to the spliced reconstruction feature 1 corresponds to the feature separation manner 1. Similarly, the feature aggregation manner 2 corresponding to the spliced reconstruction feature 2 corresponds to the feature separation manner 2. Referring to FIG. 6, the feature aggregation manner 1 corresponds to the feature separation manner 1, the feature aggregation manner 2 corresponds to the feature separation manner 2, the feature aggregation manner 3 corresponds to the feature separation manner 3, and the feature aggregation manner 4 corresponds to the feature separation manner 4. Figure 13B As shown, the feature aggregation manner 1 corresponding to the spliced reconstruction feature 1 corresponds to the feature separation manner 1, the feature aggregation manner 2 corresponding to the spliced reconstruction feature 2 corresponds to the feature separation manner 2, the feature aggregation manner 3 corresponding to the spliced reconstruction feature 3 corresponds to the feature separation manner 3, and the feature aggregation manner 4 corresponding to the spliced reconstruction feature 4 corresponds to the feature separation manner 4.

[0291] Based on the feature aggregation manner corresponding to the channel group, the spatial feature points in the spliced reconstruction feature corresponding to the channel group are subjected to feature rearrangement to obtain the aggregated reconstruction feature corresponding to the channel group, that is, all spatial feature points are subjected to feature rearrangement by using the feature aggregation manner (that is, the inverse process of the feature separation manner) to obtain the aggregated reconstruction feature, that is, the spatial feature map.

[0292] The aggregated reconstruction features corresponding to the K channel groups are subjected to channel dimension splicing to obtain the reconstruction feature of the current image block.

[0293] From the above technical solutions, in the embodiments of the present application, by adopting the stage-by-stage prediction model, the peak memory occupancy is lower, the decoding time of a single block is smaller, and the high-speed parallel decoding capability is possessed, so that the neural network effectively guarantees the quality of the reconstructed image block while maintaining low complexity, improves the encoding performance and decoding performance, and reduces the complexity. For the encoding technology based on the neural network, a low-complexity context model scheme is proposed, which is applied to the coding and decoding process of the related neural network. By optimizing the overall framework of the mean prediction network, such as adjusting the position of the internal subnetwork and canceling part of the subnetwork, the model complexity is effectively reduced. An optimization scheme with lower complexity is proposed for each subnetwork, which reduces the complexity while maintaining less performance degradation. Two optimization schemes of feature separation and feature aggregation are proposed, which fully utilize the information between channels and improve the performance.

[0294] For example, each of the above embodiments can be implemented alone or in combination, such as each of embodiments 1-17, which can be implemented alone, and at least two of embodiments 1-17, which can be implemented in combination.

[0295] For example, the content of the encoding end in each of the above embodiments can also be applied to the decoding end, i.e., the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, i.e., the encoding end can be processed in the same way.

[0296] Based on the same application concept as the above method, the embodiments of the present application also propose a decoding device, which is applied to the decoding end, and the device comprises: a memory configured to store video data; and a decoder configured to implement the decoding method in embodiments 1-17 above, i.e., the processing flow of the decoding end.

[0297] For example, in a possible implementation, the decoder is configured to implement:

[0298] decode the first code stream of the current image block to obtain the coefficient hyperparameter feature of the current image block;

[0299] For each stage sub-block of the current image block, determine the probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature, and decode the second code stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block;

[0300] determine the mean feature of the stage sub-block based on a mean prediction network, wherein the mean prediction network comprises a first prediction network and a second prediction network, the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature, or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature;

[0301] determine the reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block;

[0302] perform feature aggregation on the reconstructed feature of each stage sub-block to obtain the reconstructed feature of the current image block;

[0303] determine the reconstructed image block of the current image block based on the reconstructed feature of the current image block.

[0304] Based on the same application concept as the above method, the embodiment of the present application also proposes an encoding device, which is applied to an encoding end, and comprises a memory configured to store video data, and an encoder configured to implement the encoding method in the above embodiments 1-17, i.e., the processing flow of the encoding end.

[0305] For example, in a possible implementation, the encoder is configured to implement:

[0306] input the current image block into an analysis transformation network to obtain a feature block corresponding to the current image block;

[0307] split the feature block into to-be-encoded features of multiple stage sub-blocks;

[0308] obtain a coefficient hyperparameter feature of the current image block, encode the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determine a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature;

[0309] determine the mean feature of the stage sub-block based on a mean prediction network, wherein the mean prediction network comprises a first prediction network and a second prediction network, the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature, or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature;

[0310] determine the residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block;

[0311] For each stage sub-block, residual features of the stage sub-block are encoded based on probability distribution parameters of the stage sub-block to obtain the second code stream of the current image block.

[0312] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided by the embodiments of the present application can be specifically seen from the hardware architecture diagram of the decoding end device as shown in the figure. Figure 14A The decoding method provided by the embodiments of the present application can be implemented by the decoding end device. As shown in the figure, the decoding end device includes a processor 1401 and a machine readable storage medium 1402. The machine readable storage medium 1402 stores machine executable instructions which can be executed by the processor 1401. The processor 1401 is configured to execute the machine executable instructions to implement the decoding method of the above embodiments 1-17 of the present application. For example, in a possible implementation, when the processor 1401 executes the machine executable instructions, the following steps are executed:

[0313] Decoding the first code stream of the current image block to obtain the coefficient hyperparameter feature of the current image block;

[0314] For each stage sub-block of the current image block, determining the probability distribution parameters of the stage sub-block based on the coefficient hyperparameter feature, and decoding the second code stream of the current image block based on the probability distribution parameters to obtain the residual features of the stage sub-block;

[0315] Determining the mean feature of the stage sub-block based on the mean prediction network. The mean prediction network includes a first prediction network and a second prediction network. The output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature. Alternatively, the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature;

[0316] Determining the reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual features of the stage sub-block;

[0317] Performing feature aggregation on the reconstructed features of each stage sub-block to obtain the reconstructed feature of the current image block;

[0318] Determining the reconstructed image block of the current image block based on the reconstructed feature of the current image block.

[0319] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided by the embodiments of the present application can be specifically seen from the hardware architecture diagram of the decoding end device as shown in the figure. Figure 14BThe apparatus is shown. It includes: a processor 1411 and a machine readable storage medium 1412, the machine readable storage medium 1412 stores machine executable instructions capable of being executed by the processor 1411; the processor 1411 is used to execute the machine executable instructions to implement the encoding method of the above-mentioned embodiments 1-17 of the present application. For example, in a possible implementation, the processor 1411 executes the following steps when executing the machine executable instructions:

[0320] Input the current image block into the analysis transformation network to obtain a feature block corresponding to the current image block;

[0321] Split the feature block into a plurality of stage sub-blocks of to-be-encoded features;

[0322] Obtain a coefficient hyperparameter feature of the current image block, encode the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determine a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature;

[0323] For each stage sub-block, determine a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or, the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature;

[0324] Determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block;

[0325] For each stage sub-block, encode the residual feature of the stage sub-block based on the probability distribution parameter of the stage sub-block to obtain a second code stream of the current image block.

[0326] Based on the same application concept as the above method, the embodiments of the present application provide an electronic device. It includes: a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions capable of being executed by the processor; the processor is used to execute the machine executable instructions to implement the decoding method or the encoding method of the above-mentioned embodiments 1-17 of the present application.

[0327] Based on the same application concept as the above method, the embodiments of the present application also provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by the processor, and the method disclosed in the above examples of the present application can be implemented, such as the decoding method or the encoding method in the above-mentioned embodiments.

[0328] Based on the same application concept as the above method, the embodiments of the present application also provide a computer application program, which, when executed by a processor, can implement the decoding method or the encoding method disclosed in the above examples of the present application.

[0329] Based on the same application concept as the above method, the embodiments of the present application also provide a decoding device, which can be applied to a decoding end, and includes: a decoding module, configured to decode a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block; for each stage sub-block of the current image block, determine a probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature, and decode a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; a determining module, configured to determine a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network includes a first prediction network and a second prediction network, and the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or, the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature; determine a reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block; a processing module, configured to aggregate the reconstructed features of each stage sub-block to obtain a reconstructed feature of the current image block; and determine a reconstructed image block of the current image block based on the reconstructed feature of the current image block.

[0330] For example, for each stage sub-block of the current image block, the decoding module is specifically configured to: input the coefficient hyperparameter feature into a probability hyperparameter decoding network to obtain the probability distribution parameter of the current image block, the probability distribution parameter being the probability distribution parameter of each stage sub-block of the current image block; or, for the first stage sub-block, determine the probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature; for the i-th stage sub-block, the i being greater than 1, input the coefficient hyperparameter feature and the reference feature of the i-th stage sub-block into a mean prediction network to obtain the probability distribution parameter of the stage sub-block.

[0331] For example, when the determining module determines the mean feature of the stage sub-block based on the mean prediction network, the determining module is specifically configured to: for the first stage sub-block, input the coefficient hyper-parameter feature into the mean prediction network to obtain the mean feature of the stage sub-block; for the i-th stage sub-block, i is greater than 1, input the coefficient hyper-parameter feature and the reference feature of the i-th stage sub-block into the mean prediction network to obtain the mean feature of the stage sub-block; wherein the reference feature comprises all reconstructed features of the previous i-1 stage sub-blocks, or part of the reconstructed features of the previous i-1 stage sub-blocks, or the reconstructed feature of the i-1-th stage sub-block.

[0332] For example, when the determining module inputs the coefficient hyper-parameter feature and the reference feature into the mean prediction network to obtain the mean feature of the stage sub-block, the determining module is specifically configured to: input the coefficient hyper-parameter feature into the first prediction network to obtain a first prediction feature; input the reference feature into the second prediction network to obtain a second prediction feature; perform feature separation on the first prediction feature to obtain a third prediction feature; and generate the mean feature of the stage sub-block based on the third prediction feature and the second prediction feature. When the determining module generates the mean feature of the stage sub-block based on the third prediction feature and the second prediction feature, the determining module is specifically configured to: if the mean prediction network further comprises a prediction fusion network, perform feature splicing on the third prediction feature and the second prediction feature, input the spliced feature into the prediction fusion network, and perform fusion processing on the spliced feature by the prediction fusion network to obtain the mean feature of the stage sub-block; or if the mean prediction network further comprises an addition layer, input the third prediction feature and the second prediction feature into the addition layer, and perform addition processing on the third prediction feature and the second prediction feature by the addition layer to obtain the mean feature of the stage sub-block.

[0333] For example, when the determining module inputs the coefficient hyper-parameter feature and the reference feature into the mean prediction network to obtain the mean feature of the stage sub-block, the determining module is specifically configured to: when obtaining the reconstructed feature of each stage sub-block, input the reconstructed feature of the stage sub-block into the second prediction network to obtain the second prediction feature of the stage sub-block, and store the second prediction feature of the stage sub-block; input the coefficient hyper-parameter feature into the first prediction network to obtain a first prediction feature; perform feature separation on the first prediction feature to obtain a third prediction feature; select the second prediction feature corresponding to the reference feature from all stored second prediction features; perform feature splicing on the third prediction feature and the selected second prediction feature; input the spliced feature into the prediction fusion network, and perform fusion processing on the spliced feature by the prediction fusion network to obtain the mean feature of the stage sub-block.

[0334] The mean prediction network further includes a feature fusion network and an addition layer, and the determining module is specifically configured to: when obtaining the reconstructed feature of each stage sub-block, input the reconstructed feature of the stage sub-block into the second prediction network to obtain the second prediction feature of the stage sub-block, and store the second prediction feature of the stage sub-block; input the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature; perform feature separation on the first prediction feature to obtain a third prediction feature; select the second prediction feature corresponding to the reference feature from all stored second prediction features, and input the selected second prediction feature into the feature fusion network to obtain a fused feature; and input the third prediction feature and the fused feature into the addition layer, and perform addition processing on the third prediction feature and the fused feature through the addition layer to obtain the mean feature of the stage sub-block.

[0335] The determining module is specifically configured to: when inputting the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature, if the first prediction network includes at least one enhancement network and at least one up-sampling network, input the coefficient hyperparameter feature into the first prediction network, and perform enhancement operation and up-sampling operation on the coefficient hyperparameter feature through the enhancement network and the up-sampling network to obtain the first prediction feature; or if the first prediction network includes at least one linear layer, at least one enhancement network and at least one up-sampling network, input the coefficient hyperparameter feature into the first prediction network, and perform linear operation, enhancement operation and up-sampling operation on the coefficient hyperparameter feature through the linear layer, the enhancement network and the up-sampling network to obtain the first prediction feature.

[0336] The determining module is specifically configured to: when inputting the reference feature into the second prediction network to obtain the second prediction feature, if the second prediction network includes at least one linear layer and at least one enhancement network, input the reference feature into the second prediction network, and perform linear operation and enhancement operation on the reference feature through the linear layer and the enhancement network to obtain the second prediction feature; or if the second prediction network includes at least one enhancement network and at least one activation layer, input the reference feature into the second prediction network, and perform enhancement operation and activation operation on the reference feature through the enhancement network and the activation layer to obtain the second prediction feature.

[0337] The determining module specifically inputs the reconstructed feature of the stage sub-block into the second prediction network to obtain the second prediction feature of the stage sub-block in the following manner: if the second prediction network comprises at least one linear layer and at least one enhancement network, the determining module inputs the reconstructed feature of the stage sub-block into the second prediction network, performs linear operation and enhancement operation on the reconstructed feature of the stage sub-block through the linear layer and the enhancement network, and obtains the second prediction feature of the stage sub-block; or if the second prediction network comprises at least one enhancement network and at least one activation layer, the determining module inputs the reconstructed feature of the stage sub-block into the second prediction network, performs enhancement operation and activation operation on the reconstructed feature of the stage sub-block through the enhancement network and the activation layer, and obtains the second prediction feature.

[0338] The determining module specifically inputs the spliced feature into the prediction fusion network to obtain the mean feature of the stage sub-block in the following manner: if the prediction fusion network comprises at least one linear layer and at least one enhancement network, the determining module inputs the spliced feature into the prediction fusion network, performs linear operation and enhancement operation on the spliced feature through the linear layer and the enhancement network, and obtains the mean feature of the stage sub-block.

[0339] The determining module specifically inputs the selected second prediction feature into the feature fusion network to obtain the fused feature in the following manner: if the feature fusion network comprises at least one linear layer and at least one enhancement network, the determining module inputs the selected second prediction feature into the feature fusion network, performs linear operation and enhancement operation on the selected second prediction feature through the linear layer and the enhancement network, and obtains the fused feature.

[0340] The determining module specifically performs feature separation on the first prediction feature to obtain the third prediction feature in the following manner: the first prediction feature is evenly divided into K parts along the channel dimension to obtain K prediction group features; each prediction group feature comprises N spatial domain feature maps, and each spatial domain feature map comprises a plurality of spatial domain feature points; K is a positive integer greater than 1, and N is a positive integer greater than or equal to 1; a feature separation manner corresponding to each prediction group feature is determined, and the feature separation manners corresponding to different prediction group features are different; for each prediction group feature, the spatial domain feature points of the prediction group feature are uniformly sampled at intervals based on the feature separation manner corresponding to the prediction group feature to obtain K sampling result groups; and the third prediction feature is generated based on the K sampling result groups; different sampling starting points or sampling step lengths are regarded as different feature separation manners.

[0341] Illustratively, when the determining module obtains K sampling result groups by uniformly sampling the spatial domain feature points of the prediction group feature based on the feature separation manner corresponding to the prediction group feature, the method specifically comprises: dividing each spatial domain feature map in the prediction group feature into a plurality of sub-blocks in the spatial domain, each of the sub-blocks comprising N*2*2 spatial domain feature points; wherein, the K is 2, or the K is 4; determining, for each sub-block of the prediction group feature, a sampling result group to which each spatial domain feature point in the sub-block belongs based on the feature separation manner corresponding to the prediction group feature; wherein, the feature separation manner is used to indicate the correspondence between the spatial domain feature points and the sampling result groups, and the spatial domain feature points in the sub-block correspond to K sampling result groups; the feature separation manners corresponding to different prediction group features are different, which means that the same spatial domain feature point of the spatial domain feature maps in different prediction group features corresponds to different sampling result groups.

[0342] Illustratively, when the processing module performs feature aggregation on the reconstructed features of each stage sub-block to obtain the reconstructed features of the current image block, the method specifically comprises: dividing the reconstructed features of each stage sub-block into K grouped reconstructed features; based on the K grouped reconstructed features corresponding to each stage sub-block, splicing the grouped reconstructed features with the same index in the channel dimension to obtain spliced reconstructed features corresponding to K channel groups; for each channel group, obtaining a feature aggregation manner corresponding to the feature separation manner of the channel group, the feature aggregation manner being used to indicate the correspondence between the spatial domain feature points in the spliced reconstructed features and the spatial domain feature points in the aggregated reconstructed features; based on the feature aggregation manner corresponding to the channel group, performing feature rearrangement on the spatial domain feature points in the spliced reconstructed features corresponding to the channel group to obtain the aggregated reconstructed features corresponding to the channel group; and splicing the aggregated reconstructed features corresponding to the K channel groups in the channel dimension to obtain the reconstructed features of the current image block.

[0343] Based on the same application concept as the above method, an encoding device is also provided in the embodiments of the present application. The device is applied to an encoding end. The device comprises: a processing module, configured to input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block; and split the feature block into to-be-encoded features of a plurality of stage sub-blocks; an encoding module, configured to obtain a coefficient hyperparameter feature of the current image block, encode the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determine a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature; and a determination module, configured to, for each stage sub-block, determine a mean feature of the stage sub-block based on a mean prediction network. The mean prediction network comprises a first prediction network and a second prediction network. The output feature of the stage sub-block by the first prediction network and the output feature of the stage sub-block by the second prediction network are used to determine the mean feature. Alternatively, the output feature of the stage sub-block by the first prediction network and the output feature of a previous stage sub-block by the second prediction network are used to determine the mean feature. The residual feature of the stage sub-block is determined based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block. The encoding module is further configured to, for each stage sub-block, encode the residual feature of the stage sub-block based on the probability distribution parameter of the stage sub-block to obtain a second code stream of the current image block.

[0344] Those skilled in the art will understand that the embodiments of the present application can be provided as methods, systems, or computer program products. The present application can be in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. The embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code. The above is only an embodiment of the present application and is not intended to limit the present application.

[0345] The present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A decoding method, comprising: Applied to a decoding end, the method comprises: Decoding a first code stream of a current image block to obtain a coefficient hyperparameter feature of the current image block; For each stage sub-block of the current image block, determining a probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature, and decoding a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; Determining a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, and the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or, the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature; Determining a reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block; Performing feature aggregation on the reconstructed feature of each stage sub-block to obtain a reconstructed feature of the current image block; Determining a reconstructed image block of the current image block based on the reconstructed feature of the current image block.

2. The method of claim 1, wherein, The method further comprises: Inputting the coefficient hyperparameter feature into a probability hyperparameter decoding network to obtain the probability distribution parameter of the current image block, the probability distribution parameter being a probability distribution parameter of each stage sub-block of the current image block; or, For the first stage sub-block, determining the probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature; For the i-th stage sub-block, i being greater than 1, inputting the coefficient hyperparameter feature and a reference feature of the i-th stage sub-block into a mean prediction network to obtain the probability distribution parameter of the stage sub-block.

3. The method of claim 1, wherein: The method further comprises: For the first stage sub-block, inputting the coefficient hyperparameter feature into the mean prediction network to obtain the mean feature of the stage sub-block; 4. The method of claim 3, wherein, For the i-th stage sub-block, i being greater than 1, inputting the coefficient hyperparameter feature and a reference feature of the i-th stage sub-block into the mean prediction network to obtain the mean feature of the stage sub-block; wherein the reference feature comprises all reconstructed features of the previous i-1 stage sub-blocks, or part of the reconstructed features of the previous i-1 stage sub-blocks, or the reconstructed feature of the i-1-th stage sub-block. The method further comprises: Inputting the coefficient hyperparameter feature into the first prediction network to obtain a first prediction feature; Inputting the reference feature into the second prediction network to obtain a second prediction feature; Performing feature separation on the first prediction feature to obtain a third prediction feature; Generating the mean feature of the stage sub-block based on the third prediction feature and the second prediction feature.

5. The method of claim 4, wherein: The generating the mean feature of the stage sub-block based on the third prediction feature and the second prediction feature comprises: If the mean prediction network further comprises a prediction fusion network, the third prediction feature and the second prediction feature are spliced, and the spliced feature is input into the prediction fusion network, the spliced feature is processed by the prediction fusion network, and the mean feature of the stage sub-block is obtained; or, if the mean prediction network further comprises an addition layer, the third prediction feature and the second prediction feature are input into the addition layer, the third prediction feature and the second prediction feature are processed by the addition layer, and the mean feature of the stage sub-block is obtained.

6. The method of claim 4, wherein, The mean prediction network further comprises a prediction fusion network, and the inputting the coefficient hyperparameter feature and the reference feature into the mean prediction network to obtain the mean feature of the stage sub-block comprises: When the reconstruction feature of each stage sub-block is obtained, the reconstruction feature of the stage sub-block is input into the second prediction network to obtain the second prediction feature of the stage sub-block, and the second prediction feature of the stage sub-block is stored; The coefficient hyperparameter feature is input into the first prediction network to obtain the first prediction feature; The first prediction feature is separated to obtain the third prediction feature; The second prediction feature corresponding to the reference feature is selected from all the stored second prediction features, and the third prediction feature and the selected second prediction feature are spliced; the spliced feature is input into the prediction fusion network, and the spliced feature is processed by the prediction fusion network to obtain the mean feature of the stage sub-block.

7. The method of claim 4, wherein, The mean prediction network further comprises a feature fusion network and an addition layer, and the inputting the coefficient hyperparameter feature and the reference feature of the i-th stage sub-block into the mean prediction network to obtain the mean feature of the stage sub-block comprises: When the reconstruction feature of each stage sub-block is obtained, the reconstruction feature of the stage sub-block is input into the second prediction network to obtain the second prediction feature of the stage sub-block, and the second prediction feature of the stage sub-block is stored; The coefficient hyperparameter feature is input into the first prediction network to obtain the first prediction feature; The first prediction feature is separated to obtain the third prediction feature; The second prediction feature corresponding to the reference feature is selected from all the stored second prediction features, and the selected second prediction feature is input into the feature fusion network to obtain a fused feature; The third prediction feature and the fused feature are input into the addition layer, and the third prediction feature and the fused feature are processed by the addition layer to obtain the mean feature of the stage sub-block.

8. The method of any one of claims 4-7, wherein, The inputting the coefficient hyperparameter feature into the first prediction network to obtain the first prediction feature comprises: If the first prediction network comprises at least one enhancement network and at least one up-sampling network, the coefficient hyper-parameter feature is input into the first prediction network, the coefficient hyper-parameter feature is subjected to enhancement operation and up-sampling operation by the enhancement network and the up-sampling network, and the first prediction feature is obtained; or, If the first prediction network comprises at least one linear layer, at least one enhancement network and at least one up-sampling network, the coefficient hyper-parameter feature is input into the first prediction network, the coefficient hyper-parameter feature is subjected to linear operation, enhancement operation and up-sampling operation by the linear layer, the enhancement network and the up-sampling network, and the first prediction feature is obtained.

9. The method of claim 6 or 7, wherein the inputting of the reference feature into the second prediction network to obtain a second prediction feature comprises: if the second prediction network comprises at least one linear layer and at least one enhancement network, inputting the reference feature into the second prediction network, and subjecting the reference feature to linear operation and enhancement operation by the linear layer and the enhancement network to obtain the second prediction feature; or if the second prediction network comprises at least one enhancement network and at least one activation layer, inputting the reference feature into the second prediction network, and subjecting the reference feature to enhancement operation and activation operation by the enhancement network and the activation layer to obtain the second prediction feature. the inputting of the reconstructed feature of the stage sub-block into the second prediction network to obtain a second prediction feature of the stage sub-block comprises: if the second prediction network comprises at least one linear layer and at least one enhancement network, inputting the reconstructed feature of the stage sub-block into the second prediction network, and subjecting the reconstructed feature of the stage sub-block to linear operation and enhancement operation by the linear layer and the enhancement network to obtain the second prediction feature of the stage sub-block; or if the second prediction network comprises at least one enhancement network and at least one activation layer, inputting the reconstructed feature of the stage sub-block into the second prediction network, and subjecting the reconstructed feature of the stage sub-block to enhancement operation and activation operation by the enhancement network and the activation layer to obtain the second prediction feature.

10. The method of claim 5 or 6, wherein the inputting of the spliced feature into the prediction fusion network, and subjecting the spliced feature to fusion processing by the prediction fusion network to obtain a mean feature of the stage sub-block comprises: if the prediction fusion network comprises at least one linear layer and at least one enhancement network, inputting the spliced feature into the prediction fusion network, and subjecting the spliced feature to linear operation and enhancement operation by the linear layer and the enhancement network to obtain the mean feature of the stage sub-block.

11. The method of claim 7, wherein the inputting of the selected second prediction feature into the feature fusion network to obtain a fused feature comprises: ​ ​ ​ ​ If the feature fusion network comprises at least one linear layer and at least one enhancement network, the selected second prediction feature is input into the feature fusion network, and the selected second prediction feature is subjected to linear operation and enhancement operation through the linear layer and the enhancement network to obtain the fused feature.

12. The method of claim 8, wherein, The enhancement network comprises a convolution layer; or the enhancement network comprises, in sequence, a convolution layer, an activation layer, a convolution layer, an activation layer, and a convolution layer; or the enhancement network comprises, in sequence, a convolution layer, an activation layer, and a convolution layer; or the enhancement network comprises, in sequence, a convolution layer, an activation layer, a convolution layer, and an addition layer; or the enhancement network comprises, in sequence, a convolution layer, an activation layer, a convolution layer, and a multiplication layer. The up-sampling network comprises, in sequence, an up-sampling layer, a crop layer, and an activation layer; or the up-sampling network comprises, in sequence, an up-sampling layer and a crop layer; or the up-sampling network comprises, in sequence, an up-sampling layer, a crop layer, an activation layer, and a convolution layer. The linear layer comprises N convolution layers, and N is a non-negative integer. The up-sampling layer is an inverse convolution layer, a Pixshuffle layer, an Unpooling layer, a Nearest layer, a Bilinear layer, or a Bicubic layer. The activation layer is a relu activation layer, a leaky relu activation layer, a sigmoid activation layer, a tanh activation layer, or a gelu activation layer.

13. The method of any one of claims 4-7, wherein the feature separation of the first prediction feature to obtain a third prediction feature comprises: uniformly dividing the first prediction feature into K parts along a channel dimension to obtain K group prediction features; each group prediction feature comprises N spatial feature maps, and each spatial feature map comprises a plurality of spatial feature points; K is a positive integer greater than 1, and N is a positive integer greater than or equal to 1; determining a feature separation mode corresponding to each group prediction feature, wherein different group prediction features correspond to different feature separation modes; for each group prediction feature, performing uniform point sampling on the spatial feature points of the group prediction feature in the spatial domain based on the feature separation mode corresponding to the group prediction feature to obtain K sampling result groups; and generating the third prediction feature based on the K sampling result groups. Different sampling starting points or sampling step lengths for uniform sampling are regarded as different feature separation modes. The uniform point sampling on the spatial feature points of the group prediction feature in the spatial domain based on the feature separation mode corresponding to the group prediction feature to obtain K sampling result groups comprises:

14. The method of claim 13, wherein, for each spatial feature map in the group prediction feature, dividing the spatial feature map into a plurality of sub-blocks in the spatial domain, and each sub-block comprises N*2*2 spatial feature points; K is 2, or K is 4. ​ For each sub-block of the prediction group feature, determine the sampling result group to which each spatial feature point in the sub-block belongs based on the feature separation mode corresponding to the prediction group feature; wherein the feature separation mode is used to indicate the correspondence between the spatial feature point and the sampling result group, and the spatial feature point in the sub-block corresponds to K sampling result groups; different prediction group features correspond to different feature separation modes, that is, the same spatial feature point of the spatial feature map in different prediction group features corresponds to different sampling result groups.

15. The method of claim 13, wherein, the feature aggregation of the reconstructed features of each stage sub-block to obtain the reconstructed features of the current image block comprises: for the reconstructed features of each stage sub-block, dividing the reconstructed features of the stage sub-block into K grouped reconstructed features; based on the K grouped reconstructed features corresponding to each stage sub-block, channel dimension splicing is performed on the grouped reconstructed features with the same index to obtain spliced reconstructed features corresponding to K channel groups; for each channel group, obtaining a feature aggregation mode corresponding to the feature separation mode of the channel group, the feature aggregation mode being used to indicate the correspondence between the spatial feature points in the spliced reconstructed features and the spatial feature points in the aggregated reconstructed features; based on the feature aggregation mode corresponding to the channel group, rearranging the spatial feature points in the spliced reconstructed features corresponding to the channel group to obtain the aggregated reconstructed features corresponding to the channel group; channel dimension splicing is performed on the aggregated reconstructed features corresponding to the K channel groups to obtain the reconstructed features of the current image block.

16. An encoding method, characterized by, Applied to the encoding end, the method comprises: inputting a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block; splitting the feature block into to-be-encoded features of multiple stage sub-blocks; obtaining a coefficient hyperparameter feature of the current image block, encoding the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determining a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature; for each stage sub-block, determining a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, and the output features of the stage sub-block for the first prediction network and the output features of the stage sub-block for the second prediction network are used to determine the mean feature; or, the output features of the stage sub-block for the first prediction network and the output features of the previous stage sub-block for the second prediction network are used to determine the mean feature; determining a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block; for each stage sub-block, encoding the residual feature of the stage sub-block based on the probability distribution parameter of the stage sub-block to obtain a second code stream of the current image block.

17. A decoding apparatus, comprising: Applied to the decoding end, the device comprises: The decoding module is configured to decode a first code stream of the current image block to obtain a coefficient hyperparameter feature of the current image block; for each stage sub-block of the current image block, determine a probability distribution parameter of the stage sub-block based on the coefficient hyperparameter feature, and decode a second code stream of the current image block based on the probability distribution parameter to obtain a residual feature of the stage sub-block; The determining module is configured to determine a mean feature of the stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, and the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature; and determine a reconstructed feature of the stage sub-block based on the mean feature of the stage sub-block and the residual feature of the stage sub-block; The processing module is configured to aggregate the reconstructed features of each stage sub-block to obtain a reconstructed feature of the current image block, and determine a reconstructed image block of the current image block based on the reconstructed feature of the current image block.

18. An encoding apparatus, comprising: Applied to an encoding end, the apparatus comprises: The processing module is configured to input a current image block into an analysis transformation network to obtain a feature block corresponding to the current image block, and split the feature block into to-be-encoded features of multiple stage sub-blocks; The encoding module is configured to obtain a coefficient hyperparameter feature of the current image block, encode the coefficient hyperparameter feature to obtain a first code stream of the current image block, and determine a probability distribution parameter of each stage sub-block based on the coefficient hyperparameter feature; The determining module is configured to determine a mean feature of each stage sub-block based on a mean prediction network; wherein the mean prediction network comprises a first prediction network and a second prediction network, and the output feature of the stage sub-block for the first prediction network and the output feature of the stage sub-block for the second prediction network are used to determine the mean feature; or the output feature of the stage sub-block for the first prediction network and the output feature of the previous stage sub-block for the second prediction network are used to determine the mean feature; and determine a residual feature of the stage sub-block based on the to-be-encoded feature of the stage sub-block and the mean feature of the stage sub-block; The encoding module is further configured to encode the residual feature of each stage sub-block based on the probability distribution parameter of the stage sub-block to obtain a second code stream of the current image block.

19. A decoding-side device, comprising: comprise: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method in any of claims 1-15.

20. An encoding end device, comprising: comprise: a processor and a machine readable storage medium storing machine executable instructions executable by the processor; the processor is configured to execute the machine executable instructions to implement the method in claim 16.

21. A machine-readable storage medium, characterized in that, The machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-15, or the computer instructions are executed by the processor to implement the method in claim 16.

Images