Decoding, encoding method, device and the device and media thereof

Abstract

The present application provides a decoding method, a coding method, an apparatus, and a device and medium thereof, wherein the decoding method comprises: a step of performing decoding on a first bit stream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; a step of determining a probability distribution parameter for each stage sub-block based on the coefficient hyperparameter features of the stage sub-block, and performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain residual features of the stage sub-block; a step of determining reconstruction features of the stage sub-block based on the residual features of the stage sub-block and the mean value features of the stage sub-block; and a step of determining a reconstruction image block corresponding to the current image block based on the reconstruction features of each stage sub-block. Through the technical solution of the present application, coding performance and decoding performance can be improved.

Description

Technology Field

[0001] This application relates to the field of coding and decoding technology, and in particular to decoding, coding methods, devices, and devices and media thereof. Background Technology

[0002] To achieve the goal of saving space, both video and images are transmitted after undergoing coding. A complete video coding method may include processes such as prediction, transformation, quantization, entropy encoding, and filtering. Regarding the prediction process, it may include intra-prediction and inter-prediction. Inter-prediction utilizes video temporal domain correlations and uses pixels from adjacent already coded pictures to predict the current pixel, thereby achieving the goal of effectively eliminating video temporal redundancy. Intra-prediction utilizes video spatial domain correlations and uses pixels from already coded blocks of the current frame picture to predict the current pixel, thereby achieving the goal of effectively eliminating video spatial redundancy.

[0003] As deep learning has rapidly advanced, it has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning is also gradually beginning to be applied to the fields of coding and decoding, meaning that coding and decoding of images can be performed using neural networks. Although neural network-based coding and decoding methods show enormous potential, problems still exist, such as relatively poor coding performance, relatively poor decoding performance, and relatively high complexity. The problem to be solved

[0004] In light of this, the present application provides a decoding, a coding method, a device, and a device and medium thereof, thereby improving coding performance and decoding performance. means of solving the problem

[0005] This application provides a decoding method, which is applied to the decoding unit, and the above method is:

[0006] A step of performing decoding on the first bit stream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;

[0007] For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the stage sub-block, and decoding is performed on the second bit stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block;

[0008] A step of determining a reconstructed feature of a stage subblock based on the residual feature of the stage subblock and the mean value feature of the stage subblock;

[0009] It includes a step of determining a reconstructed image block corresponding to the current image block based on the reconstruction characteristics of each stage sub-block.

[0010] This application provides a coding method, which is applied to the coding unit, and the above method is:

[0011] A step of inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block;

[0012] A step of splitting the above feature block into features to be coded into a plurality of stage sub-blocks;

[0013] For each stage sub-block corresponding to the current image block, a step of obtaining coefficient hyperparameter features of said stage sub-block and also coding the coefficient hyperparameter features of said stage sub-block into a first bit stream of said current image block;

[0014] A step of determining the residual features of the stage sub-block based on the features to be coded of the stage sub-block and the average value features of the stage sub-block;

[0015] The method includes the step of determining a probability distribution parameter based on the coefficient hyperparameter features of the stage sub-block and also coding the residual features of the stage sub-block based on the probability distribution parameter into the second bit stream of the current image block.

[0016] This application provides a decoding method, which is applied to the decoding unit, and the above method is:

[0017] A step of performing decoding on the first bit stream of the current image block to obtain the coefficient hyperparameter features of the current image block;

[0018] A step of determining a probability distribution parameter based on the coefficient hyperparameter feature, performing decoding on a second bit stream of a current image block based on the probability distribution parameter to obtain a residual feature of the current image block, and determining a reconstruction feature of the current image block based on the residual feature;

[0019] The method includes the step of performing decoding on an auxiliary bit stream corresponding to the current image block to obtain a bit rate control parameter corresponding to the current image block, and also inputting the reconstruction feature and the bit rate control parameter into a synthetic transformation network to obtain a reconstruction image block corresponding to the current image block.

[0020] This application provides a decoding device, which is applied to the decoding stage, and said device:

[0021] A decoding module for performing decoding on a first bit stream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter features of the stage sub-block, and performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain residual features of the stage sub-block;

[0022] It includes a determination module for determining the reconstruction features of a stage sub-block based on the residual features and average value features of the stage sub-block; and determining a reconstruction image block corresponding to the current image block based on the reconstruction features of each stage sub-block.

[0023] This application provides a coding device, which is applied to the coding unit, and the said device:

[0024] Inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block; and an acquisition module for dividing the feature block into features to be coded into a plurality of stage sub-blocks;

[0025] For each stage sub-block corresponding to the current image block, a coding module for acquiring coefficient hyperparameter features of said stage sub-block and also coding coefficient hyperparameter features of said stage sub-block into a first bit stream of said current image block;

[0026] It includes a determination module for determining the residual features of the stage sub-block based on the features to be coded of the stage sub-block and the average value features of the stage sub-block;

[0027] The above coding module is further used to determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block and to code the residual feature of the stage sub-block based on the probability distribution parameter into a second bit stream of the current image block. The present application provides a decoding method applied to a decoding unit, wherein the method comprises: a step of performing decoding on a bit stream of the current image block to obtain the coefficient hyperparameter feature of each stage sub-block of the current image block; and a step of, for any one stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block and performing decoding on a bit stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block. For a first stage sub-block, the method comprises the steps of obtaining an average value feature of the stage sub-block through an average value prediction network based on coefficient hyperparameter features of the stage sub-block, and / or, for an i-th stage sub-block, obtaining a reference feature of the i-th stage sub-block based on reconstruction features of i-1 stage sub-blocks preceding the i-th stage sub-block, where i is greater than 1; obtaining an average value feature of the stage sub-block through an average value prediction network based on coefficient hyperparameter features of the stage sub-block and the reference feature; for any one stage sub-block, determining a reconstruction feature of the stage sub-block based on residual features of the stage sub-block and an average value feature of the stage sub-block; and determining a reconstruction image block corresponding to the current image block based on reconstruction features of a plurality of stage sub-blocks.In a possible implementation, the reference feature comprises all reconstruction features of the preceding i-1 stage sub-blocks; or partial reconstruction features of the preceding i-1 stage sub-blocks; or reconstruction features of the i-1th stage sub-block. In a possible implementation, the mean value prediction network comprises a second prediction network, and the step of obtaining a mean value feature of the stage sub-block through the mean value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the reference feature comprises: a step of obtaining a second prediction feature corresponding to the reference feature through the second prediction network; and a step of determining a mean value feature of the stage sub-block based on the second prediction network and the coefficient hyperparameter feature of the stage sub-block. In a possible implementation, the step of obtaining a second prediction feature corresponding to the reference feature through the second prediction network comprises: if the reference feature includes at least two reconstruction features, performing feature concatenation according to the channel dimension for all reconstruction features among the reference features; The method includes the step of obtaining the second predicted feature by performing a convolution operation on the features after concatenation; or, if the reference feature includes one reconstructed feature, the method includes the step of obtaining the second predicted feature by performing a convolution operation on the reconstructed feature.In a possible implementation, the mean value prediction network includes a first prediction network, and the step of determining the mean value feature of the stage sub-block based on the second prediction network and the coefficient hyperparameter feature of the stage sub-block comprises: performing feature concatenation on the second prediction feature and the coefficient hyperparameter feature of the stage sub-block to obtain a feature after concatenation, and inputting the feature after concatenation into the first prediction network; obtaining a first prediction feature corresponding to the feature after concatenation through the first prediction network; and determining the mean value feature of the stage sub-block based on the first prediction feature. In a possible implementation, the step of obtaining a first prediction feature corresponding to the coefficient hyperparameter feature or the feature after concatenation through the first prediction network comprises: performing a feature augmentation operation on the coefficient hyperparameter feature or the feature after concatenation through the first prediction network to obtain a first prediction feature - wherein the feature augmentation operation includes a convolution operation and an activation operation -; wherein the activation operation includes a Relu activation operation In a possible implementation, the step of obtaining a first prediction feature corresponding to the coefficient hyperparameter feature or the post-concatenation feature through a first prediction network comprises: a step of obtaining the first prediction feature by performing a convolution operation, an activation operation, a convolution operation, an activation operation, and a convolution operation in sequence on the coefficient hyperparameter feature or the post-concatenation feature through the first prediction network; wherein the activation operation includes a Relu activation operation.In a possible implementation method, after determining a reconstruction feature of a stage subblock based on a residual feature of the stage subblock and an average value feature of the stage subblock, the method comprises the step of: performing feature augmentation on the reconstruction feature of the stage subblock to obtain a reconstruction feature after augmentation; wherein the reconstruction feature after augmentation is used to determine an average value feature of the stage subblock, and the reconstruction feature after augmentation is used to determine a reconstruction image block.The present application provides a coding method applied to the coding stage, wherein the method comprises: a step of inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block; a step of dividing the feature block into features to be coded for a plurality of stage sub-blocks; for any one stage sub-block corresponding to the current image block, a step of obtaining a coefficient hyperparameter feature of said stage sub-block and also coding the coefficient hyperparameter feature of said stage sub-block into a bit stream of said stage sub-block; for a first stage sub-block, a step of obtaining an average value feature of said stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of said stage sub-block, or obtaining an already set default reference feature, and obtaining an average value feature of said stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of said stage sub-block and said default reference feature. and / or, for the i-th stage sub-block, i is greater than 1, and a reference feature of the i-th stage sub-block is obtained based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; and a step of obtaining an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the reference feature; for any one stage sub-block, a residual feature of the stage sub-block is determined based on the feature to be coded of the stage sub-block and the average value feature of the stage sub-block; a probability distribution parameter is determined based on the coefficient hyperparameter feature of the stage sub-block, and also a step of coding the residual feature of the stage sub-block into the bit stream of the current image block based on the probability distribution parameter.The present application provides a decoding method applied to a decoding unit, wherein the method comprises: a step of performing decoding on a bitstream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; a step of determining a probability distribution parameter for any one stage sub-block based on the coefficient hyperparameter features of said stage sub-block, and performing decoding on the bitstream of the current image block based on said probability distribution parameter to obtain residual features of said stage sub-block; a step of determining reconstruction features of said stage sub-block based on the residual features of said stage sub-block and the mean value features of said stage sub-block; a step of performing feature aggregation on reconstruction features of a plurality of stage sub-blocks to obtain a feature after aggregation; and a step of determining a reconstruction image block corresponding to said current image block based on said feature after aggregation; or, inputting reconstruction features of a plurality of stage sub-blocks into a synthetic transformation network to obtain a segmented reconstruction image block corresponding to said stage sub-block; The method includes the step of performing a merge on the divided reconstructed image blocks corresponding to all stage sub-blocks to obtain a reconstructed image block corresponding to the current image block. In one possible implementation, the step of determining the reconstructed image block corresponding to the current image block based on the features after aggregation comprises: inputting the features after aggregation into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; or, performing a division on the features after aggregation to obtain a plurality of divided features; inputting the plurality of divided features into a synthetic transformation network to obtain a divided reconstructed image block corresponding to said divided features; and performing a merge on the divided reconstructed image blocks corresponding to the plurality of divided features to obtain a reconstructed image block corresponding to the current image block.In a possible implementation, the step of performing feature aggregation on the reconstruction features of the plurality of stage sub-blocks to obtain a feature after aggregation comprises: a step of performing aggregation according to the phase on the reconstruction features of the plurality of stage sub-blocks to obtain the feature after aggregation; or a step of performing aggregation according to the phase of the reconstruction features of the plurality of stage sub-blocks to obtain a feature after aggregation of the plurality of phases, and also performing concatenation according to the channel on the feature after aggregation of the plurality of phases to obtain the feature after aggregation. In a possible implementation, the step of performing division on the feature after aggregation to obtain a plurality of divided features comprises: determining a target size of the divided features; and based on the target size, dividing the feature after aggregation into a plurality of divided features such that they are averaged in order from top to bottom and from left to right, and also having the size of any one divided feature be the target size. Alternatively, the method includes the step of determining the actual partition size and overlap size of the divided features, wherein the actual partition size is the partition size after removing the overlap portion among the divided features, and the overlap size is the size of the overlap portion between adjacent divided features; and dividing the feature after aggregation into a plurality of divided features based on the actual partition size and the overlap size, and wherein the size of any one divided feature is the target size.In a possible implementation method, the step of performing a merge on the divided reconstructed image blocks corresponding to the plurality of divided features to obtain a reconstructed image block corresponding to the current image block comprises: performing an arrangement on the plurality of divided reconstructed image blocks corresponding to the plurality of divided features in order from top to bottom and from left to right, and concatenating the plurality of divided reconstructed image blocks after arrangement in order to obtain a reconstructed image block corresponding to the current image block; or, determining the actual partition size and overlap size of the plurality of divided reconstructed image blocks; and performing an arrangement on the plurality of divided reconstructed image blocks corresponding to the plurality of divided features in order from top to bottom and from left to right; Based on the actual partition size and the overlap size, a step of sequentially concatenating a plurality of divided reconstructed image blocks after arrangement to obtain a reconstructed image block corresponding to the current image block - wherein the size of the non-overlapping portion of the divided reconstructed image block is the actual partition size of the divided reconstructed image block, and the size of the overlapping portion of the divided reconstructed image block adjacent to the divided reconstructed image block is the overlapping size of the divided reconstructed image block -; comprising; and for the overlapping portion of the left divided reconstructed image block and the right divided reconstructed image block, the value of the overlapping portion is the value of the left divided reconstructed image block, or, the value of the overlapping portion is the average value of the left divided reconstructed image block and the right divided reconstructed image block; For the overlapping portion of the upper divided reconstructed image block and the lower divided reconstructed image block, the value of the said overlapping portion is the value of the upper divided reconstructed image block, or the value of the said overlapping portion is the average value of the upper divided reconstructed image block and the lower divided reconstructed image block.The present application provides a coding method applied to the coding stage, wherein the method comprises: a step of inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block; a step of splitting the feature block into coding features of a plurality of stage sub-blocks; for any one stage sub-block corresponding to the current image block, a step of obtaining a coefficient hyperparameter feature of said stage sub-block and also coding the coefficient hyperparameter feature of said stage sub-block into the bit stream of said stage sub-block; for any one stage sub-block, a step of determining a residual feature of said stage sub-block based on the coding feature of said stage sub-block and the mean value feature of said stage sub-block; a step of determining a probability distribution parameter based on the coefficient hyperparameter feature of said stage sub-block and also coding the residual feature of said stage sub-block into the bit stream of said current image block based on said probability distribution parameter; and a step of determining a reconstruction feature of said stage sub-block based on the residual feature of said stage sub-block and the mean value feature of said stage sub-block. The method comprises the steps of: performing feature aggregation on the reconstruction features of a plurality of stage sub-blocks to obtain a feature after aggregation; determining a reconstruction image block corresponding to the current image block based on the feature after aggregation; or inputting the reconstruction features of a plurality of stage sub-blocks into a synthetic transformation network to obtain a divided reconstruction image block corresponding to the stage sub-block; and performing merging on the divided reconstruction image blocks corresponding to all stage sub-blocks to obtain a reconstruction image block corresponding to the current image block.The present application provides a decoding method applied to a decoding unit, wherein the method comprises: a step of performing decoding on a bitstream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; a step of, for any one stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter features of said stage sub-block, and performing decoding on a bitstream of the current image block based on said probability distribution parameter to obtain residual features of said stage sub-block; a step of determining reconstruction features of said stage sub-block based on the residual features of said stage sub-block and the average value features of said stage sub-block; and a step of performing decoding on a bitstream corresponding to the current image block to obtain bit rate control parameters of said current image block. The method includes the step of determining a reconstructed image block corresponding to the current image block based on the reconstruction features of a plurality of stage sub-blocks and the bit rate control parameter. In one possible implementation, the step of determining a reconstructed image block corresponding to the current image block based on the reconstruction features of a plurality of stage sub-blocks and the bit rate control parameter comprises: a step of performing feature aggregation on the reconstruction features of a plurality of stage sub-blocks to obtain a feature after aggregation; a step of inputting the bit rate control parameter and the feature after aggregation into a synthetic transformation network and performing processing on the feature after aggregation through the synthetic transformation network to obtain a first feature; a step of performing processing on the bit rate control parameter through the synthetic transformation network to obtain a second feature; a step of generating a third feature based on the first feature and the second feature; and a step of determining a reconstructed image block corresponding to the current image block based on the third feature.In a possible implementation method, the step of determining a reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the bit rate control parameter comprises: performing feature aggregation on the reconstruction features of the plurality of stage sub-blocks to obtain a feature after aggregation; performing division on the feature after aggregation to obtain a plurality of divided features, and the bit rate control parameter of the current image block includes the bit rate control parameter of the plurality of divided features; for any one divided feature, input said divided feature and the bit rate control parameter of said divided feature into a synthetic transformation network, and perform processing on said divided feature through said synthetic transformation network to obtain a first feature; and perform processing on said divided feature's bit rate control parameter through said synthetic transformation network to obtain a second feature; The method includes the step of generating a third feature based on the first feature and the second feature, and also determining a divided reconstructed image block corresponding to the divided feature based on the third feature; and the step of performing a merger on the divided reconstructed image blocks corresponding to a plurality of divided features to obtain a reconstructed image block corresponding to the current image block.In a possible implementation method, the step of determining a reconstructed image block corresponding to the current image block based on the reconstruction features of the plurality of stage sub-blocks and the bit rate control parameters comprises: the bit rate control parameters of the current image block include the bit rate control parameters of the plurality of stage sub-blocks, and for any one stage sub-block, input the reconstruction features of the stage sub-block and the bit rate control parameters of the stage sub-block into a synthetic transformation network, and perform processing on the reconstruction features of the stage sub-block through the synthetic transformation network to obtain a first feature; and perform processing on the bit rate control parameters of the stage sub-block through the synthetic transformation network to obtain a second feature; The method includes the step of generating a third feature based on the first feature and the second feature, and determining a divided reconstructed image block corresponding to the stage sub-block based on the third feature; and the step of performing a merge on the divided reconstructed image blocks corresponding to all stage sub-blocks to obtain a reconstructed image block corresponding to the current image block. In one possible implementation, the step of determining a reconstructed feature of the stage sub-block based on the residual feature of the stage sub-block and the average value feature of the stage sub-block comprises: for the first stage sub-block, obtaining an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block, or obtaining a previously set default reference feature, and obtaining an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature.In a possible implementation, the step of determining a reconstruction feature of a stage subblock based on residual features of the stage subblock and average features of the stage subblock comprises: for the i-th stage subblock, i is greater than 1, and obtaining a reference feature of the i-th stage subblock based on reconstruction features of the i-1 stage subblocks preceding the i-th stage subblock; obtaining an average feature of the stage subblock through an average feature prediction network based on coefficient hyperparameter features of the stage subblock and the reference feature; and for any one stage subblock, determining a reconstruction feature of the stage subblock based on residual features of the stage subblock and average features of the stage subblock. In a possible implementation, the reference feature may include all reconstruction features of the preceding i-1 stage subblocks; or, may include partial reconstruction features of the preceding i-1 stage subblocks; Alternatively, it includes a reconstruction feature of the i-1th stage sub-block. In one possible implementation, the mean value prediction network includes a second prediction network, and the step of obtaining a mean value feature of the stage sub-block through the mean value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the reference feature comprises: a step of obtaining a second prediction feature corresponding to the reference feature through the second prediction network; and a step of determining a mean value feature of the stage sub-block based on the second prediction network and the coefficient hyperparameter feature of the stage sub-block.The present application provides a coding method applied to the coding stage, wherein the method comprises: inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block; dividing the feature block into features to be coded for a plurality of stage sub-blocks; for any one stage sub-block corresponding to the current image block, obtaining a coefficient hyperparameter feature of said stage sub-block and also coding the coefficient hyperparameter feature of said stage sub-block into the bit stream of said stage sub-block; for any one stage sub-block, determining a residual feature of said stage sub-block based on the features to be coded of said stage sub-block and the average value feature of said stage sub-block; determining a probability distribution parameter based on the coefficient hyperparameter feature of said stage sub-block and also coding the residual feature of said stage sub-block into the bit stream of said current image block based on said probability distribution parameter; determining a reconstruction feature of said stage sub-block based on the residual feature of said stage sub-block and the average value feature of said stage sub-block; The method includes the step of obtaining bit rate control parameters of the current image block; determining a reconstructed image block corresponding to the current image block based on the reconstruction features of a plurality of stage sub-blocks and the bit rate control parameters; and coding the bit rate control parameters into a bit stream corresponding to the current image block.

[0028] The present application provides a decoding device comprising a processor and a machine-readable storage medium, wherein machine-executable instructions that can be executed by said processor are stored in said machine-readable storage medium;

[0029] The above processor executes machine-executable instructions and is used to implement the decoding method described above.

[0030] The present application provides a coding device comprising a processor and a machine-readable storage medium, wherein machine-executable instructions that can be executed by said processor are stored in said machine-readable storage medium;

[0031] The above processor executes machine-executable instructions and is used to implement the coding method described above.

[0032] The present application provides an electronic device comprising a processor and a machine-readable storage medium, wherein a machine-executable instruction capable of being executed by the processor is stored in the machine-readable storage medium; wherein the processor executes the machine-executable instruction and is used to implement the decoding method described above; or wherein the processor executes the machine-executable instruction and is used to implement the coding method described above.

[0033] The present application provides a machine-readable storage medium, wherein multiple computer instructions are stored in the machine-readable storage medium, and when the computer instructions are executed by a processor, the decoding method described above is implemented; or, the coding method described above is implemented.

[0034] The present application provides a computer application, wherein when said computer application is executed by a processor, the decoding method described above is implemented; or, when said computer application is executed by a processor, the coding method described above is implemented.

[0035] As can be seen from the technical solution described above, among the embodiments of this application, regarding the neural network-based coding and decoding technology, a bit rate variable and adjustable coding and decoding solution is provided to improve concurrency, effectively save the feature storage cache, achieve higher bit rate control precision, have less loss in coding performance, and possess better coding performance and bit rate control precision. By using block-based coding and decoding, peak value memory occupancy is lower, single-block decoding time is shorter, and high-speed parallel decoding capability is provided, thereby enabling the neural network to maintain low complexity while effectively ensuring the quality of the reconstructed image block, improving coding performance and decoding performance, and also reducing complexity. Brief explanation of the drawing

[0036] FIG. 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present application; FIG. 2 is a flowchart of a decoding method in one implementation method of the present application; FIG. 3 is a flowchart of a coding method in one implementation method of the present application; FIG. 4 is a schematic diagram of the processing process of the coding unit during one implementation method of the present application; FIG. 5 is a schematic diagram of the processing process of the decoding unit in one implementation method of the present application; FIG. 6 is a schematic diagram of the processing process of the coding unit in one embodiment of the present application; FIG. 7 is a schematic diagram of the processing process of the decoding unit in one embodiment of the present application; FIGS. 8a to 8h are schematic diagrams of the structure of an average value prediction network in one embodiment of the present application; FIGS. 9a to 9f are schematic diagrams of assembly and division in one embodiment of the present application; FIGS. 10a to 10c are schematic diagrams of the structure of a synthetic transformation network in one embodiment of the present application; FIGS. 11a and FIGS. 11b are schematic diagrams of the processing of a convolution layer in one embodiment of the present application; FIG. 12 is a schematic diagram of the determination of the λ parameter in one embodiment of the present application; FIG. 13a is a hardware structure diagram of a decoding device in one embodiment of the present application; FIG. 13b is a hardware structure diagram of a coding device in one embodiment of the present application. Specific details for implementing the invention

[0037] The terms used in the embodiments of this application are merely for describing specific embodiments and are not intended to limit this application. The singular forms ('a', 'said', and 'the') used in this disclosure and the appended claims are also intended to include the majority of forms unless the context clearly indicates other meanings. Additionally, the term "and / or" as used herein should be understood to refer to and include any or all possible combinations of one or more associated listed items. Although various information may be described using terms such as "first," "second," "third," etc. in this disclosure, it should be understood that such information is not limited to these terms. These terms are merely for distinguishing information of the same type from one another. For example, without departing from the scope of this disclosure, first information may be referred to as second information, and similarly, second information may be referred to as first information. Additionally, depending on the context, the word "if" as used in this specification may be interpreted as "when," "upon," or "in response to a decision."

[0038] In the embodiments of this application, a decoding method and a coding method are provided, which may be subject to the following concepts:

[0039] Entropy Encoding: Entropy encoding is a coding method that does not lose any information according to the entropy principle during the coding process, and the information entropy is the average amount of information (an uncertain measure) of the source. Entropy encoding methods may include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0040] Neural Network (NN): The term "neural network" refers to an artificial neural network. As a computational model, a neural network is composed of interconnected nodes (referred to as neurons). In a neural network, neurons can represent different objects, such as features, letters, concepts, or some meaningful abstract mode. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive information data from the external world; output units produce the output of processing results; and hidden units are located between input and output units and are units that cannot be observed from outside the system. Connection weights between neurons reflect the strength of the connections between units, and the display and processing of information are realized through the connection relationships of the processing units. Neural networks are a type of non-programmed, brain-like style of information processing. The essence of a neural network is to achieve parallel, distributed information processing capabilities through the network's transformations and dynamic behavior, and to mimic the information processing functions of the human neural system at different degrees and levels. In the field of video processing, commonly used neural networks may include, but are not limited to, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Fully Connected Networks.

[0041] Convolutional Neural Network (CNN): The Convolutional Neural Network is a feedforward neural network and is one of the most representative network structures in deep learning technology. Its artificial neurons can respond to neighboring units within a certain range, and it offers excellent representation in large-picture processing. The basic structure of a Convolutional Neural Network includes two layers: one is the feature extraction layer (also referred to as the convolution layer), where the input of each neuron is connected to the local receptive field of the previous layer and extracts the corresponding local feature. Once the local feature is extracted, the positional relationship between it and other features is determined accordingly. The other is the feature mapping layer (also referred to as the activation layer), where each computational layer of the neural network is mapped to multiple features. Each feature mapping is a plane, and the weights of all neurons on that plane are identical. For feature mapping structures, functions such as the Sigmoid function (S-type function), ReLU function (modified linear unit function), Leaky-ReLU function, PReLU function (linear unit function with leakage correction), and GDN (Generalized Difference Network) function can be used as activation functions for convolutional networks. Additionally, since neurons on a single mapping plane share weights, the number of network free parameters can be reduced.

[0042] For example, when comparing a convolutional neural network to a picture processing algorithm, its advantage is that it can avoid complex pre-processing of the picture (such as artificial feature extraction) and perform end-to-end learning by directly inputting the original image. When comparing a convolutional neural network to a general neural network, general neural networks all use a fully connected method, meaning that all neurons from the input layer to the hidden layer are fully connected. This method results in a massive amount of parameters, which consumes time and makes training the network difficult. However, convolutional neural networks have the advantage of avoiding this difficulty through methods such as local connections and weight sharing.

[0043] Deconvolution: The deconvolution layer is also referred to as the transposed convolution layer. The operation of the deconvolution layer and the convolution layer is very similar, but the main difference is that the inverse convolution layer makes the input larger than the output through padding (it may also maintain the same size). If the stride is 1, it indicates that the output size is equal to the input size; if the stride is N, it indicates that the width of the output feature is N times the width of the input feature and the height of the output feature is N times the height of the input feature.

[0044] Generalization Ability: This refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the rules hidden (included) behind the data. For data other than the training set that possesses certain rules, a trained network can also provide appropriate output, and this ability can be referred to as generalization ability.

[0045] Feature: The feature of this application is a 3D feature matrix or tensor of C*W*H, and as illustrated in FIG. 1, this is a schematic diagram of a 3D feature matrix, in which C represents the number of channels, H represents the feature height, and W represents the feature width. The 3D feature matrix may be the input of a neural network or the output of a neural network.

[0046] Rate-Distortion Optimized: The two main metrics for evaluating coding efficiency are bit rate and PSNR (Peak Signal to Noise Ratio); a lower bit rate results in greater compression, while a higher PSNR leads to better quality of the reconstructed picture. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of both. For example, the costs corresponding to the modes are as follows: J (mode) = D +λ* R , here, D α represents distortion, which can generally be measured using the Sum of the Squared Errors (SSE) metric, where SSE refers to the sum of squared differences between the reconstructed image block and the source picture block; to implement cost considerations, the Sum of Absolute Differences (SAD) metric may also be used, where SAD refers to the sum of absolute differences between the reconstructed picture block and the source picture; λ is the Lagrange multiplier; R ... is the actual number of bits required for picture block coding under the corresponding mode, and includes the total number of bits required, such as coding mode information, motion information, and residuals. When selecting a mode, if the coding mode is determined by comparing it using the rate-distortion optimization principle, optimization of coding performance can generally be achieved.

[0047] A large number of coding tools are submitted for each module of the coding unit, and each tool often features multiple modes. For different video sequences, the coding tool capable of achieving optimal coding performance often varies. Therefore, during the coding process, Rate-Distortion Optimizer (RDO) is typically used to compare the coding performance of different tools or modes and select the optimal mode. After determining the optimal tool or mode, the decision information is transmitted by coding flag information within the bitstream. Although this method results in relatively high coding complexity, it can achieve optimal coding performance by adaptively selecting the optimal combination of modes for different content. The decoding unit can obtain relevant mode information by directly interpreting flag information, and the impact on decoding complexity is relatively small.

[0048] The decoding method and coding method in the embodiments of this application will be described in detail below in conjunction with several specific embodiments.

[0049] Example 1: An embodiment of the present application provides a decoding method, as illustrated in FIG. 2, which is a schematic flowchart of the decoding method, and the method may be applied to a decoding unit (also referred to as a video decoder), and the method may include the following steps:

[0050] Step 201, decoding is performed on the first bit stream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block. Here, the current image block is divided into a plurality of stage sub-blocks, that is, the current image block includes a plurality of stage sub-blocks.

[0051] Step 202, for each stage sub-block, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the stage sub-block, and decoding is performed on the second bit stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block.

[0052] Step 203, a reconstruction feature of the stage subblock is determined based on the residual feature of the stage subblock and the average feature of the stage subblock. Here, the average feature of the stage subblock is obtained based on the coefficient hyperparameter feature of the stage subblock and / or the reference feature of the stage subblock, such as the coefficient hyperparameter feature, or the coefficient hyperparameter feature and the reference feature.

[0053] Step 204, based on the reconstruction characteristics of each stage sub-block, determine the reconstruction image block corresponding to the current image block.

[0054] For example, for the first stage sub-block, the average value feature of the stage sub-block is obtained through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block, or a default reference feature that has already been set is obtained, and the average value feature of the stage sub-block is obtained through an average value prediction network based on the coefficient hyperparameter feature and the default reference feature of the stage sub-block.

[0055] For the i-th stage sub-block, i is greater than 1, and a reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block; and an average value feature of the stage sub-block is obtained through an average value prediction network based on the coefficient hyperparameter feature and the reference feature of the stage sub-block; wherein the reference feature includes all reconstruction features of the preceding i-1 stage sub-blocks; or partial reconstruction features of the preceding i-1 stage sub-blocks; or reconstruction features of the i-1 stage sub-blocks.

[0056] For example, the average value prediction network may include a first prediction network, a second prediction network, and a prediction fusion network, and the step of obtaining an average value feature of the stage sub-block through the average value prediction network based on coefficient hyperparameter features and reference features of the stage sub-block may include: a step of obtaining a first prediction feature corresponding to the coefficient hyperparameter features through the first prediction network; a step of obtaining a second prediction feature corresponding to the reference feature through the second prediction network; a step of performing feature concatenation on the first prediction feature and the second prediction feature to obtain a concatenated feature, inputting the concatenated feature into the prediction fusion network, and performing processing on the concatenated feature through the prediction fusion network to obtain an average value feature of the stage sub-block.

[0057] For example, the step of obtaining a first prediction feature corresponding to a coefficient hyperparameter feature through a first prediction network may include: a step of obtaining the first prediction feature by performing a feature augmentation operation and an upsampling operation on the coefficient hyperparameter feature through the first prediction network. Here, the feature augmentation operation may include: a convolution operation, or a convolution operation and an activation operation; and the upsampling operation may include: a deconvolution operation, a cropping operation and an activation operation, or a deconvolution operation, a cropping operation, an activation operation and a convolution operation. For example, the activation operation is used as a ReLU operation.

[0058] For example, the step of obtaining a second predicted feature corresponding to a reference feature through a second prediction network may include: performing feature concatenation according to the channel dimension for all reconstructed features among the reference features; and performing a convolution operation on the features after concatenation to obtain the second predicted feature; or, performing feature addition for all reconstructed features among the reference features, and also performing a convolution operation on the features after addition to obtain the second predicted feature. Here, the reference feature includes all reconstructed features of the preceding i-1 stage sub-blocks; or partial reconstructed features of the preceding i-1 stage sub-blocks; or reconstructed features of the i-1th stage sub-block.

[0059] For example, the step of obtaining an average value feature of a stage sub-block by performing processing on features after concatenation through a predictive fusion network may include, but is not limited to, the step of obtaining an average value feature of a stage sub-block by performing a convolution operation and at least one fusion operation on features after concatenation through a predictive fusion network; wherein, the fusion operation includes an activation operation and a convolution operation. For example, the activation operation is used as a ReLU operation.

[0060] For example, after determining the reconstruction features of the stage subblock based on the residual features and average value features of the stage subblock, feature augmentation may be performed on the reconstruction features of the stage subblock to obtain the reconstruction features after augmentation; wherein the reconstruction features after augmentation are used to determine the average value features of the stage subblock, and the reconstruction features after augmentation are used to determine the reconstruction image block.

[0061] For example, the step of determining a reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block comprises: a step of performing feature aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features, and inputting the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; or a step of performing feature aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features, and performing partitioning on the aggregated features to obtain a plurality of partitioned features; inputting each partitioned feature into a synthetic transformation network to obtain a partitioned reconstructed image block corresponding to the partitioned features, and performing merging on the partitioned reconstructed image blocks corresponding to the plurality of partitioned features to obtain a reconstructed image block corresponding to the current image block; Alternatively, the method may include, but is not limited to, a step of inputting the reconstruction features of each stage sub-block into a synthetic transformation network to obtain a divided reconstructed image block corresponding to the stage sub-block, and performing a merge on the divided reconstructed image blocks corresponding to all stage sub-blocks to obtain a reconstructed image block corresponding to the current image block.

[0062] For example, the step of performing feature aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features may include: a step of performing aggregation according to the phase of the reconstructed features of each stage sub-block to obtain the aggregated features; or a step of performing aggregation according to the phase of the reconstructed features of each stage sub-block to obtain a plurality of phase aggregated features, and then performing concatenation according to a channel on the plurality of phase aggregated features to obtain the aggregated features, but is not limited thereto.

[0063] For example, the step of performing partitioning on a feature after aggregation to obtain a plurality of partitioned features may include: determining a target size of the partitioned features; dividing the feature after aggregation into a plurality of partitioned features in an order from top to bottom and from left to right based on the target size, and also dividing the size of each partitioned feature into a target size; or, determining an actual partition size and an overlap size of the partitioned features, wherein the actual partition size is a partition size with the overlap portion removed from the partitioned features, and the overlap size is the size of the overlap portion between adjacent partitioned features; dividing the feature after aggregation into a plurality of partitioned features, and also dividing the size of each partitioned feature into a target size; wherein, if the actual partition size is tile_c * tile_w * tile_h and the overlap size is tile_c * padding_w * padding_h, the target size is tile_c * (tile_w + 2 * padding_w) * (tile_h + 2 * padding_h).

[0064] For example, the step of performing a merge on a divided reconstructed image block corresponding to a plurality of divided features to obtain a reconstructed image block corresponding to the current image block may include, but is not limited to, the step of performing an arrangement on the divided reconstructed image blocks corresponding to a plurality of divided features in order from top to bottom and from left to right, and sequentially concatenating the divided reconstructed image blocks corresponding to a plurality of divided features after arrangement to obtain a reconstructed image block corresponding to the current image block; or, determining the actual partition size and overlap size of each divided reconstructed image block; performing an arrangement on the divided reconstructed image blocks corresponding to a plurality of divided features in order from top to bottom and from left to right; and sequentially concatenating the divided reconstructed image blocks corresponding to a plurality of divided features after arrangement based on the actual partition size and overlap size to obtain a reconstructed image block corresponding to the current image block. Here, the reconstructed image block includes a plurality of divided reconstructed image blocks, the size of the non-overlapping portion of the divided reconstructed image block is the actual partition size of the divided reconstructed image block, and the size of the overlapping portion of the divided reconstructed image block adjacent to the divided reconstructed image block is the overlapping size of the divided reconstructed image block. Here, for the overlapping portion of the left divided reconstructed image block and the right divided reconstructed image block, the value of the overlapping portion is the value of the left divided reconstructed image block, or the value of the overlapping portion is the average value of two divided reconstructed image blocks; and for the overlapping portion of the upper divided reconstructed image block and the lower divided reconstructed image block, the value of the overlapping portion is the value of the upper divided reconstructed image block, or the value of the overlapping portion is the average value of two divided reconstructed image blocks.

[0065] For example, the step of inputting features after aggregation into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block may include: performing decoding on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters of the current image block, and inputting the bit rate control parameters and features after aggregation into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block.

[0066] The step of inputting each segmented feature into a synthetic transformation network to obtain a segmented reconstructed image block corresponding to said segmented feature may include: performing decoding on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters for each segmented feature, and inputting the segmented feature and the bit rate control parameters for said segmented feature into a synthetic transformation network to obtain a segmented reconstructed image block corresponding to said segmented feature.

[0067] The step of inputting the reconstruction features of each stage sub-block into a synthetic transformation network to obtain a divided reconstructed image block corresponding to the stage sub-block may include: performing decoding on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters of each stage sub-block, and inputting the reconstruction features of the stage sub-block and the bit rate control parameters of the stage sub-block into a synthetic transformation network to obtain a divided reconstructed image block corresponding to the stage sub-block.

[0068] For example, the step of inputting bit rate control parameters and features after aggregation into a synthetic transformation network to obtain a reconstructed image block corresponding to a current image block may include: processing the features after aggregation through the synthetic transformation network to obtain a first feature; processing the bit rate control parameters through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and also determining a reconstructed image block corresponding to a current image block based on the third feature.

[0069] The step of obtaining a divided reconstructed image block corresponding to the divided feature by inputting a divided feature and a bit rate control parameter of the divided feature into a synthetic transformation network may include: performing processing on the divided feature through the synthetic transformation network to obtain a first feature; performing processing on the bit rate control parameter of the divided feature through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and also determining a divided reconstructed image block corresponding to the divided feature based on the third feature.

[0070] The step of obtaining a divided reconstructed image block corresponding to the stage subblock by inputting the reconstruction features of the stage subblock and the bit rate control parameters of the stage subblock into a synthetic transformation network may include: performing processing on the reconstruction features of the stage subblock through the synthetic transformation network to obtain a first feature; performing processing on the bit rate control parameters of the stage subblock through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and determining a divided reconstructed image block corresponding to the stage subblock based on the third feature.

[0071] For example, the execution order described above is merely an example for convenience of explanation, and in actual application, the execution order between steps may be changed and is not limited thereto. Additionally, in other embodiments, steps of the corresponding method may not be performed in the order shown and described herein, and the number of steps included in the method may be more or fewer than that described herein. Furthermore, a single step described herein may be divided into multiple steps in other embodiments; and all multiple steps described herein may be merged into a single step in other embodiments.

[0072] As can be seen from the technical solution described above, among the embodiments of this application, regarding the neural network-based coding and decoding technology, a bit rate variable and adjustable coding and decoding solution is provided to improve concurrency, effectively save feature storage cache, achieve higher bit rate control precision, have less loss in coding performance, and possess better coding performance and bit rate control precision. By using block-based coding and decoding, peak value memory occupancy is lower, single-block decoding time is shorter, and high-speed parallel decoding capability is provided, so that the neural network maintains low complexity while effectively ensuring the quality of the reconstructed image block, improving coding performance and decoding performance, and also reducing complexity.

[0073] Example 2: An example of the present application provides a coding method, as illustrated in FIG. 3, which is a schematic flowchart of the coding method, and the method may be applied to a coding unit (also referred to as a video encoder), and the method may include the following steps:

[0074] Step 301, input the current image block into an analysis transformation network to obtain a feature block corresponding to the current image block.

[0075] Step 302, divide the feature block into features to be coded into multiple stage sub-blocks.

[0076] Step 303, for each stage sub-block corresponding to the current image block, the coefficient hyperparameter feature of the stage sub-block is obtained, and the coefficient hyperparameter feature of the stage sub-block is also coded into the first bit stream of the current image block.

[0077] Step 304, a residual feature of the stage subblock is determined based on the coded feature of the stage subblock and the average feature of the stage subblock. Here, the average feature of the stage subblock is obtained based on the coefficient hyperparameter feature of the stage subblock and / or the reference feature of the stage subblock, such as the coefficient hyperparameter feature, or the coefficient hyperparameter feature and the reference feature.

[0078] Step 305, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the stage sub-block, and the residual feature of the stage sub-block is coded into the second bit stream of the current image block based on the probability distribution parameter.

[0079] For example, for the first stage sub-block, the average value feature of the stage sub-block is obtained through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block, or a default reference feature that has already been set is obtained, and the average value feature of the stage sub-block is obtained through an average value prediction network based on the coefficient hyperparameter feature and the default reference feature of the stage sub-block.

[0080] For the i-th stage sub-block, i is greater than 1, and a reference feature of the i-th stage sub-block is obtained based on the reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block; and an average value feature of the stage sub-block is obtained through an average value prediction network based on the coefficient hyperparameter feature and the reference feature of the stage sub-block; wherein the reference feature includes all reconstruction features of the preceding i-1 stage sub-blocks; or partial reconstruction features of the preceding i-1 stage sub-blocks; or reconstruction features of the i-1 stage sub-blocks.

[0081] For example, the processing of the coding unit is similar to the processing of the decoding unit, and the identical parts are not explained further here.

[0082] For example, the execution order described above is merely an example for convenience of explanation, and in actual application, the execution order between steps may be changed and is not limited thereto. Additionally, in other embodiments, steps of the corresponding method may not be performed in the order shown and described herein, and the number of steps included in the method may be more or fewer than that described herein. Furthermore, a single step described herein may be divided into multiple steps in other embodiments; and all multiple steps described herein may be merged into a single step in other embodiments.

[0083] As can be seen from the technical solution described above, among the embodiments of this application, regarding the neural network-based coding and decoding technology, a bit rate variable and adjustable coding and decoding solution is provided to improve concurrency, effectively save feature storage cache, achieve higher bit rate control precision, have less loss in coding performance, and possess better coding performance and bit rate control precision. By using block-based coding and decoding, peak value memory occupancy is lower, single-block decoding time is shorter, and high-speed parallel decoding capability is provided, so that the neural network maintains low complexity while effectively ensuring the quality of the reconstructed image block, improving coding performance and decoding performance, and also reducing complexity.

[0084] Example 3: For Examples 1 and 2, the processing of the coding unit may be referenced in FIG. 4, and of course, FIG. 4 is merely one example of the processing of the coding unit and is not limited to this processing of the coding unit.

[0085] After the coding unit obtains the current image block x (the current image block x may be the original image block x, i.e., the input image block), it can perform an analysis transformation on the current image block x through an analysis transformation network (i.e., a neural network) to obtain image features y corresponding to the current image block x. Here, performing a feature transformation on the current image block x through an analysis transformation network is to: transform the current image block x into image features y in the latent domain, thereby making it convenient to perform operations in the latent domain in all subsequent processes.

[0086] An image can be divided into one image block or into multiple image blocks, and if the image is divided into one image block, the current image block x may be an image, that is, the coding and decoding process of the image block may be used directly on the image.

[0087] After the coding unit obtains image feature y, it performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. For example, image feature y is input into a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. Here, the hyperparameter coding network may be an already trained neural network, and there are no restrictions on the training process of this hyperparameter coding network; it is sufficient as long as it can perform a coefficient hyperparameter feature transformation on the image feature y. Here, after the image feature y in the latent domain passes through the hyperparameter coding network, hyperprior latent information z (i.e., coefficient hyperparameter feature z) is obtained.

[0088] After the coding unit obtains the coefficient hyperparameter feature z, it performs quantization on the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z; that is, the Q operation in FIG. 4 is the quantization process. After obtaining the hyperparameter quantization feature corresponding to this coefficient hyperparameter feature z, it performs coding on the hyperparameter quantization feature to obtain Bitstream #1 (i.e., the first bit stream) corresponding to the current image block; that is, the AE operation in FIG. 4 represents the coding process, for example, the entropy encoding process. Alternatively, the coding unit may directly perform coding on the coefficient hyperparameter feature z to obtain Bitstream #1 corresponding to the current image block. Here, the hyperparameter quantization feature or the coefficient hyperparameter feature z carried in Bitstream #1 is mainly used to obtain the mean value and the parameters of the probability distribution model (i.e., the probability distribution parameters).

[0089] After obtaining Bitstream #1 corresponding to the current image block, the coding unit may transmit Bitstream #1 corresponding to the current image block to the decoding unit, and the processing process for Bitstream #1 corresponding to the current image block by the decoding unit refers to a subsequent embodiment.

[0090] After the coding unit obtains Bitstream #1 corresponding to the current image block, it may perform decoding on Bitstream #1 to obtain hyperparameter quantization features, that is, AD in FIG. 4 represents the decoding process, and then, inverse quantization is performed on the hyperparameter quantization features to obtain coefficient hyperparameter feature z_hat, and coefficient hyperparameter feature z_hat may be the same as or different from coefficient hyperparameter feature z, and IQ operation in FIG. 4 is the inverse quantization process. Alternatively, after the coding unit obtains Bitstream #1 corresponding to the current image block, it may perform decoding on Bitstream #1 to obtain coefficient hyperparameter feature z_hat, and may not perform the inverse quantization process of coefficient hyperparameter feature z_hat.

[0091] For the coding process of Bitstream #1, a coding method of a fixed probability density model may be used, and for the decoding process of Bitstream #1, a decoding method of a fixed probability density model may be used, and there are no restrictions on these coding and decoding processes.

[0092] After the coding unit obtains the coefficient hyperparameter feature z_hat, context-based prediction is performed based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (refer to subsequent examples for the process of determining the image feature y_hat) to obtain a prediction value mu (i.e., mean value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input into a mean value prediction network, and the mean value prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat, and this prediction process is not limited. Here, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the already decoded image feature y_hat, and a more accurate prediction value mu is obtained through quantum federation input, and the prediction value mu is used to obtain a residual by performing subtraction with the original feature, and to obtain a reconstructed feature y_hat by adding it to the decoded residual.

[0093] It should be noted that the mean value prediction network is a selectable neural network, meaning there may be no mean value prediction network, that is, there is no need to determine the predicted value mu through the mean value prediction network, and the dotted box in Figure 4 indicates that the mean value prediction network is selectable.

[0094] After the coding unit obtains image feature y, it can determine residual feature r based on image feature y and predicted value mu; for example, the difference between image feature y and predicted value mu is considered as residual feature r. Next, feature processing is performed on residual feature r to obtain image feature s, and this feature processing method is not restricted and can be any feature processing method. In this case, it is necessary to deploy an average value prediction network and provide the predicted value mu by the average value prediction network. Alternatively, after the coding unit obtains image feature y, it can perform feature processing on image feature y to obtain image feature s, and this feature processing method is not restricted and can be any feature processing method. In this case, without the need to deploy an average value prediction network, the residual process is indicated as an optional process through a dotted box.

[0095] After the coding unit obtains an image feature s, it can perform quantization on the image feature s to obtain an image quantization feature corresponding to the image feature s, that is, the Q operation in FIG. 4 is the quantization process. After the coding unit obtains an image quantization feature corresponding to the image feature s, the coding unit can perform coding on the image quantization feature to obtain Bitstream #2 (i.e., the second bit stream) corresponding to the current image block, that is, the AE operation in FIG. 4 represents the coding process, for example, the entropy encoding process. Alternatively, the coding unit may directly perform coding on the image feature s to obtain Bitstream #2 corresponding to the current image block, and may not perform the quantization process of the image feature s.

[0096] After the coding unit obtains Bitstream #2 corresponding to the current image block, the coding unit may transmit Bitstream #2 corresponding to the current image block to the decoding unit, and the processing process for Bitstream #2 corresponding to the current image block by the decoding unit refers to a subsequent embodiment.

[0097] After the coding unit obtains Bitstream #2 corresponding to the current image block, it may perform decoding on Bitstream #2 to obtain image quantization features, that is, AD in FIG. 4 represents the decoding process, and then the coding unit may perform inverse quantization on the image quantization features to obtain image features s', and image features s' may be the same as or different from image features s, and IQ operation in FIG. 4 is the inverse quantization process. Alternatively, after the coding unit obtains Bitstream #2 corresponding to the current image block, it may perform decoding on Bitstream #2 to obtain image features s', and may not perform an inverse quantization process of the image quantization features.

[0098] After the coding unit obtains image feature s', it may perform feature recovery (i.e., the inverse process of feature processing) on ​​image feature s', and this feature recovery process is not restricted and may be any feature recovery method, and obtain residual feature r_hat, and residual feature r_hat may be the same as or different from residual feature r. After the coding unit obtains residual feature r_hat, it determines image feature y_hat based on residual feature r_hat and predicted value mu, and image feature y_hat may be the same as or different from image feature y, for example, the sum of residual feature r_hat and predicted value mu is considered as image feature y_hat. Under this case, it is necessary to deploy a mean value prediction network and provide a predicted value mu by the mean value prediction network. Alternatively, after the coding unit obtains image feature s', it can perform feature recovery (i.e., the inverse process of feature processing) on ​​image feature s' to obtain image feature y_hat, which may be identical to or different from image feature y. In this case, without the need to deploy the mean prediction network, the dashed box indicates that the residual process is an optional process.

[0099] After the coding unit obtains image feature y_hat, it can perform a composite transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into a composite transformation network, and a composite transformation is performed on the image feature y_hat by the composite transformation network to obtain a reconstructed image block x_hat, and up to this point, the image reconstruction process is completed.

[0100] Among possible embodiments, when a coding unit performs coding on an image quantization feature or image feature s to obtain Bitstream #2 corresponding to the current image block, the coding unit needs to first determine a probability distribution model and then perform coding on the image quantization feature or image feature s based on said probability distribution model. Additionally, when a coding unit performs decoding on Bitstream #2, it needs to first determine a probability distribution model and then perform decoding on Bitstream #2 based on said probability distribution model.

[0101] To obtain a probability distribution model, referring further to FIG. 4, the coding unit obtains the coefficient hyperparameter feature z_hat and then performs the inverse transformation of the coefficient hyperparameter feature on z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into a probability hyperparameter decoding network, and the probability hyperparameter decoding network performs the inverse transformation of the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Here, the probability hyperparameter decoding network may be an already trained neural network, and there are no restrictions on the training process of this probability hyperparameter decoding network, as long as it is capable of performing the inverse transformation of the coefficient hyperparameter feature z_hat.

[0102] Among the possible implementation methods, the processing of the coding unit described above can be performed by a deep learning model or a neural network model to implement an end-to-end image compression and coding process, and is not limited to this coding process.

[0103] Example 4: For Examples 1 and 2, the processing of the decoding unit may be referenced in FIG. 5, and of course, FIG. 5 is merely one example of the processing of the decoding unit and is not limited to the processing of the decoding unit.

[0104] After the decoding unit obtains Bitstream #1 corresponding to the current image block, it may perform decoding on Bitstream #1 to obtain hyperparameter quantization features, that is, AD in FIG. 5 represents the decoding process, and then, inverse quantization is performed on the hyperparameter quantization features to obtain coefficient hyperparameter feature z_hat, and coefficient hyperparameter feature z_hat may be the same as or different from coefficient hyperparameter feature z, and IQ operation in FIG. 5 is the inverse quantization process. Alternatively, after the decoding unit obtains Bitstream #1 corresponding to the current image block, it may perform decoding on Bitstream #1 to obtain coefficient hyperparameter feature z_hat, and may not perform the inverse quantization process of coefficient hyperparameter feature z_hat.

[0105] For the decoding process of Bitstream #1, a decoding method of a fixed probability density model may be used, and there are no restrictions on this.

[0106] An image can be divided into one image block or into multiple image blocks, and if the image is divided into one image block, the current image block x may be an image, that is, the decoding process of the image block may be used directly on the image.

[0107] After the decoding unit obtains the coefficient hyperparameter feature z_hat, context-based prediction is performed based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block (for the process of determining the image feature y_hat, refer to subsequent examples) to obtain a prediction value mu (i.e., mean value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the image feature y_hat are input into a mean value prediction network, and the mean value prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the image feature y_hat, and this prediction process is not limited. Here, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the already decoded image feature y_hat, and a more accurate prediction value mu is obtained through quantum federated input.

[0108] It should be noted that the mean value prediction network is a selectable neural network, meaning there may be no mean value prediction network, that is, there is no need to determine the predicted value mu through the mean value prediction network, and the dotted box in Fig. 5 indicates that the mean value prediction network is selectable.

[0109] After the decoding unit obtains Bitstream #2 corresponding to the current image block, it may perform decoding on Bitstream #2 to obtain image quantization features, that is, AD in FIG. 5 represents the decoding process, and then the decoding unit may perform inverse quantization on the image quantization features to obtain image features s', and image features s' may be the same as or different from image features s, and IQ operation in FIG. 5 is the inverse quantization process. Alternatively, after the decoding unit obtains Bitstream #2 corresponding to the current image block, it may perform decoding on Bitstream #2 to obtain image features s', and may not perform an inverse quantization process of the image quantization features.

[0110] After the decoding unit obtains image feature s', it may perform feature recovery (i.e., the inverse process of feature processing) on ​​image feature s' to obtain residual feature r_hat, and the residual feature r_hat may be the same as or different from residual feature r. After the decoding unit obtains residual feature r_hat, it may determine image feature y_hat based on residual feature r_hat and predicted value mu, and image feature y_hat may be the same as or different from image feature y, for example, the sum of residual feature r_hat and predicted value mu is considered as image feature y_hat. In this case, it is necessary to deploy a mean prediction network and provide the predicted value mu by the mean prediction network. Alternatively, after the decoding unit obtains image feature s', it may perform feature recovery on image feature s' to obtain image feature y_hat, and image feature y_hat may be the same as or different from image feature y. In this case, the dashed box indicates that the residual process is an optional process, without the need to deploy the mean prediction network.

[0111] After the decoding unit obtains image feature y_hat, it can perform a composite transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, image feature y_hat is input into a composite transformation network, and a composite transformation is performed on the image feature y_hat by the composite transformation network to obtain a reconstructed image block x_hat, and up to this point, the image reconstruction process is completed.

[0112] Among the possible embodiments, when the decoding unit performs decoding on Bitstream #2, it first determines a probability distribution model and then performs decoding on Bitstream #2 based on said probability distribution model. To obtain the probability distribution model, referring to FIG. 5, the decoding unit obtains a coefficient hyperparameter feature z_hat and then performs an inverse transformation of the coefficient hyperparameter feature on the coefficient hyperparameter feature z_hat to obtain a probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into a probability hyperparameter decoding network, and the coefficient hyperparameter feature inverse transformation of the coefficient hyperparameter feature z_hat is performed by the probability hyperparameter decoding network to obtain a probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Here, the probabilistic hyperparameter decoding network may be an already trained neural network, and there are no restrictions on the training process of this probabilistic hyperparameter decoding network; it is sufficient to obtain the probability distribution parameter p by performing an inverse transform of the coefficient hyperparameter feature z_hat.

[0113] Among the possible implementation methods, the processing of the decoding unit described above can be performed by a deep learning model or a neural network model to implement an end-to-end image compression and coding process, and is not limited to this decoding process.

[0114] Example 5: For Examples 1, 2, 3, and 4, a mean value prediction network may be deployed or may not be deployed, and to improve feature coding performance, a mean value prediction network is deployed as an example. When a mean value prediction network is deployed, in order to obtain accurate prediction values ​​of image features, it is necessary to use the reconstruction values ​​of adjacent features (i.e., by performing context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the image feature y_hat of the previous image block to obtain the prediction value mu corresponding to the current image block). Therefore, the coding process of the current feature can only be performed after the feature value on the right (considered as a reference using the reconstruction values ​​of the left or upper-left feature) has completed the coding reconstruction of the feature value on the left or upper-left. This dependency relationship causes the coding between features to be performed serially and cannot be performed in parallel, and thus results in relatively high complexity.

[0115] Regarding the above-described findings, by using block-based coding and decoding in the present embodiment, it is possible to improve concurrency, effectively save feature storage cache, achieve higher bit rate control precision, have less coding performance loss, provide better coding performance and bit rate control precision, have lower peak value memory occupancy, have shorter single-block decoding time, and provide high-speed parallel decoding capability.

[0116] Refer to Fig. 6 for the processing of the coding unit, and the processing of this coding unit is not limited.

[0117] 1. After obtaining the original image x, it is determined whether to perform segmentation on the original image x based on the image resolution of the segmentation unit. For example, if the image resolution is greater than a threshold, segmentation is performed on the original image x; otherwise, segmentation is not performed on the original image x. If yes, the original image x is segmented into multiple image blocks x_p, and there is an overlap between adjacent image blocks; if no, image block x_p is the original image x, meaning there is only one image block in the original image x, and each image block x_p can be denoted as the current image block. For the convenience of explanation, the processing of a single current image block x_p is subsequently explained as an example.

[0118] 2. After obtaining the current image block x_p, an analysis transformation is performed on the current image block x_p through an analysis transformation network to obtain image features y_p (feature block y_p) corresponding to the current image block x_p. For example, by transforming the current image block x_p into image features y_p in the latent domain through an analysis transformation network, the manipulation in the latent domain for all subsequent processes is made convenient.

[0119] 3. A merge classification unit performs merge classification on image feature y_p to obtain coding feature y_i (i=1⪋N) for N stage sub-blocks, that is, splits image feature y_p into coding feature y_i for N stage sub-blocks.

[0120] 4. For each stage sub-block corresponding to the current image block, a coefficient hyperparameter feature corresponding to the stage sub-block is obtained. For example, a coefficient hyperparameter feature transformation is performed on the feature y_i to be coded in stage sub-block i to obtain a coefficient hyperparameter feature z_i corresponding to stage sub-block i. For example, the feature y_i to be coded is input into a hyperparameter coding network (i.e., a neural network), and a coefficient hyperparameter feature transformation is performed on the feature y_i to be coded by the hyperparameter coding network to obtain a coefficient hyperparameter feature z_i.

[0121] 5. For each stage sub-block corresponding to the current image block, the coefficient hyperparameter feature of the said stage sub-block is coded into the first bitstream (Bitstream#1) of the current image block. For example, quantization is performed on the coefficient hyperparameter feature z_i corresponding to stage sub-block i to obtain a hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z_i, and coding is performed on the said hyperparameter quantization feature to obtain the first bitstream corresponding to the current image block. Alternatively, coding is performed directly on the coefficient hyperparameter feature z_i corresponding to stage sub-block i to obtain the first bitstream corresponding to the current image block. After obtaining the first bitstream corresponding to the current image block, the first bitstream corresponding to the current image block is transmitted to the decoding unit, and the processing process for the first bitstream corresponding to the current image block at the decoding unit is described in subsequent embodiments.

[0122] 6. After obtaining a first bit stream corresponding to the current image block, decoding may be performed on the first bit stream to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block, for example, a coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i (i=1…N).

[0123] For example, decoding can be performed on the first bit stream to obtain the hyperparameter quantization feature of stage sub-block i, and then inverse quantization can be performed on the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i. Alternatively, decoding can be performed on the first bit stream to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i, and the inverse quantization process may be omitted.

[0124] For the coding process of the first bit stream, a coding method of a fixed probability density model may be used, and for the decoding process of the first bit stream, a decoding method of a fixed probability density model may be used, and there are no restrictions on these coding and decoding processes.

[0125] 7. For each stage sub-block corresponding to the current image block, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the said stage sub-block. For example, a probability distribution parameter p_i corresponding to the stage sub-block i can be obtained by performing an inverse transformation of the coefficient hyperparameter feature z_hat_i corresponding to the stage sub-block i. For example, the coefficient hyperparameter feature z_hat_i is input into a probability hyperparameter decoding network, and an inverse transformation of the coefficient hyperparameter feature z_hat_i is performed by the probability hyperparameter decoding network to obtain a probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, a probability distribution model can be generated based on the probability distribution parameter p_i.

[0126] 8. After obtaining the coding features of N stage sub-blocks, for the coding feature y_1 of the first stage sub-block, the coefficient hyperparameter feature z_hat_1 of the stage sub-block is input into the mean value prediction network to obtain the mean value feature of the stage sub-block (i.e., predicted value mu_1, mean value mu_1). Alternatively, a default reference feature that has already been set is obtained, and the coefficient hyperparameter feature z_hat_1 of the stage sub-block and the default reference feature are also input into the mean value prediction network to obtain the mean value feature of the stage sub-block.

[0127] A residual feature r_1 is determined based on the coding feature y_1 and mean value feature mu_1 of the first stage sub-block, for example, the difference between the coding feature y_1 and the mean value feature mu_1 is considered as the residual feature r_1. Next, feature processing is performed on the residual feature r_1 to obtain the residual feature r_1 after feature processing. The method of this feature processing is not restricted and can be any feature processing method. Of course, feature processing may be an optional step, and feature processing may not be performed on the residual feature r_1.

[0128] Quantization is performed on residual feature r_1 (or residual feature r_1 after feature processing) to obtain image quantization features, and coding is performed on said image quantization features to obtain a second bitstream (Bitstream#2) corresponding to the current image block. Alternatively, coding can be performed directly on residual feature r_1 (or residual feature r_1 after feature processing) to obtain a second bitstream corresponding to the current image block. After obtaining the second bitstream corresponding to the current image block, the second bitstream corresponding to the current image block is transmitted to a decoding unit, and the processing process for the second bitstream corresponding to the current image block by the decoding unit refers to a subsequent embodiment. Here, when coding is performed on image quantization features or residual feature r_1, coding is performed on image quantization features or residual feature r_1 using a probability distribution model corresponding to the probability distribution parameter p_1 of the first stage sub-block to obtain the second bitstream.

[0129] After obtaining the second bit stream, decoding is performed on the second bit stream using a probability distribution model corresponding to the probability distribution parameter p_1 of the first stage sub-block to obtain image quantization features, inverse quantization is performed on the image quantization features, and feature recovery is performed on the features after inverse quantization to obtain residual features r_hat_1, or, after performing inverse quantization on the image quantization features, residual features r_hat_1 are obtained directly and the feature recovery process is omitted. Alternatively, after performing decoding on the second bit stream, feature recovery is performed on the features after decoding to obtain residual features r_hat_1, or, after performing decoding on the second bit stream, residual features r_hat_1 are obtained directly.

[0130] After obtaining residual feature r_hat_1, the reconstruction feature y_hat_1 of the first stage subblock is determined based on residual feature r_hat_1 and mean feature mu_1, for example, the sum of residual feature r_hat_1 and mean feature mu_1 is considered as the reconstruction feature y_hat_1.

[0131] 9. After obtaining the coding features of N stage sub-blocks, for the coding feature y_i of the i-th stage sub-block (i > 1 and i is less than or equal to N), the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i are input into the mean value prediction network to obtain the mean value feature of stage sub-block i (i.e., predicted value mu_i, mean value mu_i).

[0132] Here, for the reference feature of stage sub-block i (i.e., the i-th stage sub-block), it may be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, based on already reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature of stage sub-block i is obtained based on all already reconstructed features among y_hat_1 to y_hat_i-1, i.e., the reference feature includes y_hat_1 to y_hat_i-1. Or, the reference feature of stage sub-block i is obtained based on some already reconstructed features among y_hat_1 to y_hat_i-1, i.e., the reference feature includes some already reconstructed features among y_hat_1 to y_hat_i-1. Alternatively, a reference feature of stage sub-block i is obtained based on the reconstruction feature of the i-1th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.

[0133] A residual feature r_i is determined based on the coding feature y_i and mean feature mu_i of the i-th stage sub-block, for example, the difference between the coding feature y_i and the mean feature mu_i is considered as the residual feature r_i. Feature processing is performed on the residual feature r_i to obtain the residual feature r_i after feature processing. Of course, feature processing may not be performed on the residual feature r_i.

[0134] Quantization is performed on residual feature r_i (or residual feature r_i after feature processing) to obtain image quantization features, and coding is performed on the image quantization features to obtain a second bit stream corresponding to the current image block. Alternatively, coding is performed directly on residual feature r_i (or residual feature r_i after feature processing) to obtain a second bit stream corresponding to the current image block. The second bit stream corresponding to the current image block is transmitted to a decoding unit, and the processing process for the second bit stream corresponding to the current image block at the decoding unit refers to subsequent embodiments. Here, when coding is performed on image quantization features or residual feature r_i, coding is performed on image quantization features or residual feature r_i using a probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block to obtain the second bit stream.

[0135] After obtaining the second bit stream, decoding is performed on the second bit stream using a probability distribution model corresponding to the probability distribution parameter p_i of the i-th stage sub-block to obtain image quantization features, inverse quantization is performed on the image quantization features, and feature recovery is performed on the features after inverse quantization to obtain residual features r_hat_i, or, after performing inverse quantization on the image quantization features, residual features r_hat_i can be obtained directly. Alternatively, after performing decoding on the second bit stream, feature recovery is performed on the features after decoding to obtain residual features r_hat_i, or, after performing decoding on the second bit stream, residual features r_hat_i can be obtained directly.

[0136] After obtaining residual feature r_hat_i, the reconstruction feature y_hat_i of the i-th stage subblock is obtained based on residual feature r_hat_i and mean feature mu_i, for example, the sum of residual feature r_hat_i and mean feature mu_i is considered as the reconstruction feature y_hat_i.

[0137] 10. A clustering and partitioning unit obtains reconstructed features y_hat_i (i=1…N) of N stage sub-blocks and performs feature aggregation on the reconstructed features y_hat_i of N stage sub-blocks to obtain features after aggregation. Next, the clustering and partitioning unit performs partitioning on the features after aggregation to obtain at least one partitioned feature y_hat_p, i.e., a feature block y_hat_p.

[0138] 11, for each segmented feature y_hat_p, the clustering and segmentation unit inputs the segmented feature y_hat_p into a synthetic transformation network and outputs a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p by the synthetic transformation network.

[0139] 12. A merging unit obtains a divided reconstructed image block x_hat_p corresponding to a plurality of divided features y_hat_p, and also performs merging on this divided reconstructed image block x_hat_p to obtain a reconstructed image block x_hat corresponding to a current image block. Clearly, if the original image x is divided into a plurality of current image blocks x_p, the reconstructed image block x_hat corresponding to all current image blocks x_p can constitute the reconstructed image. If the original image x corresponds to only one current image block x_p, the reconstructed image block x_hat is the reconstructed image.

[0140] During the process described above, the coding unit may assign different bit rate control parameters to different stage sub-blocks of different image blocks based on region of interest or bit rate information, thereby causing different stage sub-blocks of different image blocks to consume different bit rates.

[0141] For example, if it is necessary to avoid uniform quantization distortion of each image block by applying special processing to the region of interest (e.g., non-uniform processing, ensuring higher quality of the region of interest), the bit rate control parameter may be a quantization parameter (the bit rate control parameter needs to be transmitted to the decoding unit via the bit stream), and with reference to FIG. 6, the bit rate control parameter affects the quantization process (Q) and inverse quantization process (IQ) of the first bit stream, and also affects the quantization process (Q) and inverse quantization process (IQ) of the second bit stream.

[0142] For example, when performing quantization on a coefficient hyperparameter feature z_i corresponding to a stage sub-block i, if the stage sub-block i is located within the region of interest, quantization is performed on the coefficient hyperparameter feature z_i corresponding to the stage sub-block i using the bit rate control parameter, and the bit rate control parameter is used to ensure that the image quality corresponding to the stage sub-block i is higher. When performing inverse quantization on a hyperparameter quantization feature corresponding to the stage sub-block i, it is also necessary to perform inverse quantization on a hyperparameter quantization feature corresponding to the stage sub-block i using the bit rate control parameter.

[0143] When performing quantization on residual feature r_i (or residual feature r_i after feature processing) corresponding to stage sub-block i, if stage sub-block i is located within the region of interest, quantization is performed on residual feature r_i corresponding to stage sub-block i using the bit rate control parameter, and the bit rate control parameter is used to ensure that the image quality corresponding to stage sub-block i is higher. When performing inverse quantization on image quantization features corresponding to stage sub-block i, it is also necessary to perform inverse quantization on image quantization features corresponding to stage sub-block i using the bit rate control parameter.

[0144] For example, to perform accurate bit rate control, the bit rate control parameter may be a control parameter of the synthetic transformation network (the bit rate control parameter needs to be transmitted to the decoding unit via the bit stream), and with reference to FIG. 6, the bit rate control parameter affects the analysis transformation network and the synthetic transformation network. For example, when performing an analysis transformation on the current image block x_p through the analysis transformation network, the bit rate control parameter and the current image block x_p are input to the analysis transformation network, and the analysis transformation network outputs an image feature y_p corresponding to the current image block x_p. When inputting the segmented feature y_hat_p to the synthetic transformation network, the segmented feature y_hat_p and the bit rate control parameter are input to the synthetic transformation network, and the synthetic transformation network outputs a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p.

[0145] Based on a previously set target bit rate (where each image block corresponds to one target bit rate, or each image corresponds to one target bit rate, or multiple images correspond to one target bit rate), the coding unit may implement different bit rate control parameters, and after using a specific bit rate control parameter, if the bit rate corresponding to the bit stream of the current image block (e.g., a first bit stream and a second bit stream) is the target bit rate or approaches the target bit rate (i.e., the difference between the two is less than a threshold value), this bit rate control parameter is the bit rate control parameter that is finally used, and the coding unit may transmit the bit rate control parameter to the decoding unit.

[0146] Example 6: Refer to FIG. 7 for the processing process of the decoding unit, and the processing process of this decoding unit is not limited.

[0147] 1. After obtaining a first bitstream (Bitstream#1) corresponding to the current image block, decoding is performed on the first bitstream to obtain a coefficient hyperparameter feature of each stage sub-block of the current image block, for example, a coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i (i=1…N).

[0148] For example, decoding can be performed on the first bit stream to obtain the hyperparameter quantization feature of stage sub-block i, and then inverse quantization can be performed on the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i. Alternatively, decoding can be performed on the first bit stream to obtain the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i, and the inverse quantization process may be omitted. For the decoding process of the first bit stream, a decoding method of a fixed probability density model may be used, and this decoding process is not limited.

[0149] 2. For each stage sub-block corresponding to the current image block, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the said stage sub-block. For example, a probability distribution parameter p_i corresponding to stage sub-block i can be obtained by performing an inverse transformation of the coefficient hyperparameter feature z_hat_i corresponding to stage sub-block i. For example, the coefficient hyperparameter feature z_hat_i is input into a probability hyperparameter decoding network, and an inverse transformation of the coefficient hyperparameter feature z_hat_i is performed by the probability hyperparameter decoding network to obtain a probability distribution parameter p_i. After obtaining the probability distribution parameter p_i, a probability distribution model can be generated based on the probability distribution parameter p_i.

[0150] 3. For each stage sub-block corresponding to the current image block, decoding is performed on the second bit stream (Bitstream#2) of the current image block based on the probability distribution parameter (probability distribution model) of the stage sub-block to obtain residual features of the stage sub-block.

[0151] For example, decoding can be performed on the second bit stream using a probability distribution model corresponding to the probability distribution parameter p_i of the i-th (i=1…N)-th stage sub-block to obtain image quantization features, inverse quantization can be performed on the image quantization features, and feature recovery can be performed on the features after inverse quantization to obtain residual features r_hat_i, or, after performing inverse quantization on the image quantization features, the residual features r_hat_i can be obtained directly. Alternatively, after performing decoding on the second bit stream, feature recovery can be performed on the features after decoding to obtain residual features r_hat_i, or after performing decoding on the second bit stream, the residual features r_hat_i can be obtained directly.

[0152] 4. After obtaining residual features of N stage sub-blocks, for the residual feature r_hat_1 of the first stage sub-block, the coefficient hyperparameter feature z_hat_1 of the corresponding stage sub-block is input into the mean value prediction network to obtain the mean value feature of the corresponding stage sub-block (i.e., predicted value mu_1, mean value mu_1). Alternatively, a default reference feature that has already been set is obtained, and the coefficient hyperparameter feature z_hat_1 of the corresponding stage sub-block and the default reference feature are also input into the mean value prediction network to obtain the mean value feature of the corresponding stage sub-block.

[0153] Next, the reconstruction feature y_hat_1 of the first stage subblock is determined based on the residual feature r_hat_1 and the mean feature mu_1. For example, the sum of the residual feature r_hat_1 and the mean feature mu_1 can be considered as the reconstruction feature y_hat_1.

[0154] After obtaining residual features of 5, N stage sub-blocks, for the residual feature r_hat_i of the i-th stage sub-block (i > 1 and i is less than or equal to N), the coefficient hyperparameter feature z_hat_i of stage sub-block i and the reference feature of stage sub-block i are input into the mean prediction network to obtain the mean feature of stage sub-block i (i.e., predicted value mu_i, mean value mu_i).

[0155] Here, for the reference feature of stage sub-block i (i.e., the i-th stage sub-block), it may be obtained based on the reconstructed features of the i-1 stage sub-blocks preceding stage sub-block i, that is, based on already reconstructed features y_hat_1 to y_hat_i-1. For example, the reference feature of stage sub-block i is obtained based on all already reconstructed features among y_hat_1 to y_hat_i-1, i.e., the reference feature includes y_hat_1 to y_hat_i-1. Or, the reference feature of stage sub-block i is obtained based on some already reconstructed features among y_hat_1 to y_hat_i-1, i.e., the reference feature includes some already reconstructed features among y_hat_1 to y_hat_i-1. Alternatively, a reference feature of stage sub-block i is obtained based on the reconstruction feature of the i-1th stage sub-block, that is, the reference feature of stage sub-block i is y_hat_i-1.

[0156] Here, to control the size of the cache, the number of already reconstructed features included in the reference feature must not exceed a preset amount, and assuming the preset amount is M, if the number of already reconstructed features is less than or equal to M, the reference feature includes all already reconstructed features among y_hat_1 to y_hat_i-1, and if the number of already reconstructed features is greater than M, the reference feature includes M already reconstructed features (i.e., some already reconstructed features) among y_hat_1 to y_hat_i-1. To select M already reconstructed features from y_hat_1 to y_hat_i-1, M already reconstructed features may be selected based on default strategies until M already reconstructed features are selected, for example, M already reconstructed features whose decoding order is closest to the current stage sub-block, such as y_hat_i-1, y_hat_i-2, y_hat_i-3, etc., and of course, M already reconstructed features may also be selected using other methods, and there are no restrictions on this. Alternatively, index values ​​of M already reconstructed features may be obtained by performing decoding on the bit stream, and M already reconstructed features may be selected based on the index values ​​obtained by decoding.

[0157] Next, the reconstruction feature y_hat_i of the i-th stage subblock is obtained based on the residual feature r_hat_i and the mean feature mu_i, for example, the sum of the residual feature r_hat_i and the mean feature mu_i is considered as the reconstruction feature y_hat_i.

[0158] 6. A clustering and partitioning unit obtains reconstructed features y_hat_i (i=1…N) of N stage sub-blocks, and also performs feature aggregation on the reconstructed features y_hat_i of N stage sub-blocks to obtain a feature after aggregation. Next, the clustering and partitioning unit performs partitioning on the feature after aggregation to obtain at least one partitioned feature y_hat_p, i.e., a feature block y_hat_p.

[0159] 7. For each segmented feature y_hat_p, the clustering and segmentation unit inputs the segmented feature y_hat_p into a synthetic transformation network and outputs a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p by the synthetic transformation network.

[0160] 8. A merging unit obtains a divided reconstructed image block x_hat_p corresponding to a plurality of divided features y_hat_p, and also performs merging on this divided reconstructed image block x_hat_p to obtain a reconstructed image block x_hat corresponding to a current image block. Clearly, if the original image x is divided into a plurality of current image blocks x_p, the reconstructed image block x_hat corresponding to all current image blocks x_p can constitute the reconstructed image. If the original image x corresponds to only one current image block x_p, the reconstructed image block x_hat is the reconstructed image.

[0161] During the process described above, based on the region of interest or bit rate information, different bit rate control parameters may be assigned to different stage sub-blocks of different image blocks so that different stage sub-blocks of different image blocks consume different bit rates.

[0162] For example, if it is necessary to perform special processing on a region of interest (e.g., non-uniform processing, ensuring higher quality of the region of interest), the bit rate control parameter may be a quantization parameter (the bit rate control parameter needs to be transmitted to the decoding unit through the bit stream), and with reference to FIG. 7, the bit rate control parameter affects the inverse quantization process (IQ) of the first bit stream and the inverse quantization process (IQ) of the second bit stream.

[0163] For example, when performing inverse quantization on a hyperparameter quantization feature corresponding to stage sub-block i, if a bit rate control parameter corresponding to stage sub-block i is already decoded from a bit stream and said bit rate control parameter is used to ensure that the image quality corresponding to stage sub-block i is higher, then inverse quantization on a hyperparameter quantization feature corresponding to stage sub-block i can be performed using said bit rate control parameter.

[0164] For example, when performing inverse quantization on an image quantization feature corresponding to stage sub-block i, if a bit rate control parameter corresponding to stage sub-block i is already decoded from a bit stream and said bit rate control parameter is used to ensure that the image quality corresponding to stage sub-block i is higher, then inverse quantization can be performed on an image quantization feature corresponding to stage sub-block i using said bit rate control parameter.

[0165] For example, to perform accurate bit rate control, the bit rate control parameter may be a control parameter of the synthetic transformation network (indicating that the bit rate control parameter needs to be transmitted to the decoding unit via the bit stream, and that the control parameter of the synthetic transformation network can determine the synthetic transformation network based on the bit rate control parameter), and with reference to FIG. 7, the bit rate control parameter influences the synthetic transformation network. For example, when a segmented feature y_hat_p is input to the synthetic transformation network, the segmented feature y_hat_p and the bit rate control parameter are input to the synthetic transformation network, and the synthetic transformation network outputs a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p.

[0166] Example 7: For the i-th (i=1…N)-th stage sub-block in Examples 5 and 6, a reconstructed feature y_hat_i of the i-th stage sub-block (i.e., the sum of residual feature r_hat_i and mean feature mu_i) is obtained, and the reconstructed feature y_hat_i is used to determine the mean feature of a subsequent stage sub-block, and / or, the reconstructed feature y_hat_i is used to determine a reconstructed image block. Here, the use of the reconstructed feature y_hat_i to determine the mean feature of a subsequent stage sub-block refers to determining the mean feature of a subsequent stage sub-block by considering the reconstructed feature y_hat_i as a reference feature of a subsequent stage sub-block. The use of reconstruction feature y_hat_i in determining the reconstruction image block refers to: determining the reconstruction image block corresponding to the current image block by inputting the reconstruction feature y_hat_i into the clustering and partitioning unit.

[0167] For the i-th (i=1…N)-th stage sub-block, after obtaining the reconstructed feature y_hat_i of the i-th stage sub-block, feature augmentation may be performed on the reconstructed feature y_hat_i to obtain the reconstructed feature y_hat_i' after augmentation, and the reconstructed feature y_hat_i' after augmentation is used to determine the average feature of the subsequent stage sub-block, and / or, the reconstructed feature y_hat_i' after augmentation is used to determine the reconstructed image block. Here, the use of the reconstructed feature y_hat_i' after augmentation to determine the average feature of the subsequent stage sub-block refers to determining the average feature of the subsequent stage sub-block by considering the reconstructed feature y_hat_i' after augmentation as a reference feature of the subsequent stage sub-block. The use of the reconstructed feature y_hat_i' after augmentation to determine the reconstructed image block refers to determining the reconstructed image block corresponding to the current image block by inputting the reconstructed feature y_hat_i' after augmentation into a clustering and partitioning unit. For example, decoding can be performed on a bit stream to obtain augmentation information, and augmentation can be performed on the reconstructed feature y_hat_i based on the augmentation information to obtain a new reconstructed feature y_hat_i', and the new reconstructed feature y_hat_i' may be used for the prediction of subsequent y_hat_i+1 to y_hat_N, and may also be used in the subsequent clustering and partitioning process.

[0168] Among the possible implementation methods, regarding the augmentation process of the reconstructed feature y_hat, the following method may be used:

[0169] Method 1, the inputs are reconstruction features y_hat, Threshold[idx], GreaterFlag[idx], Scale1[idx], and Scale2[idx] before augmentation, and the output is reconstruction feature y_hat_en after augmentation. Here, Threshold represents a threshold value, and different idx may correspond to the same or different threshold values, and idx represents an index, e.g., 0, 1, 2, 3, etc. GreaterFlag represents a flag bit, and different idx may correspond to the same or different flag bits. Scale1 represents a scaling factor, and different idx may correspond to the same or different Scale1. Scale2 represents a scaling factor, and different idx may correspond to the same or different Scale2.

[0170] Step 1: Determine mask[c,i,j] (mask) under filter (filter, index is idx) for each set. For example, mask[c,i,j] can be determined using the following mathematical formula, and of course, the following mathematical formula is only one example and is not limited thereto.

[0171]

[0172] go Represents the probability distribution parameter value, where c represents the channel, i represents the horizontal coordinate, and j represents the vertical coordinate; if the probability distribution parameter value is greater than Threshold[idx], GreaterFlag[idx] is true, and If is 1, and if the probability distribution parameter value is less than Threshold[idx], GreaterFlag[idx] is false, and If is 1, otherwise ga is 0.

[0173] Step 2: Execute each set of filters (index is idx) in order, and if mask[idx, c, i, j] is 1, then y_hat_en[c, i, j] = y_hat[c, i, j] + mean_hat[c, i, j] * Scale2[idx] + residual_hat[c, i, j] * Scale1[idx]; y_hat[c, i, j] = y_hat_en[c, i, j]. If multiple sets of filters exist, y_hat is compensated multiple times. Here, mean_hat represents the mean feature, i.e., the aforementioned mu_i; and residual_hat represents the residual feature, i.e., the aforementioned r_hat.

[0174] Among method 2 and method 1, the scaling factor of each channel c is the same, and among method 2, different channels c are allowed to have different scaling factors, that is, the scaling factors are Scale1[c,idx] and Scale2[c,idx], and a control switch can be used to determine whether to scale the current feature, or to turn off scaling by modifying the channel's scaling factor to 0. The difference between method 2 and method 1 is the augmentation method: y_hat_en[c,i,j]=y_hat[c,i,j]+mean_hat [c,i,j]*Scale2[c,idx]+residual_hat[c,i,j]*Scale1[c,idx].

[0175] Method 3, compared to Method 1, the implementation methods of both are similar, and the difference lies in the augmentation method: y_hat_en[c,i,j]=y_hat[c,i,j]*Scale1[idx]+mean_hat [c,i,j]*Scale2[idx].

[0176] Method 4, compared to Method 1, has a similar implementation method, but the difference lies in the augmentation method: y_hat_en[c,i,j]=y_hat[c,i,j]*Scale1[idx]+residual_hat[c,i,j]*Scale2[idx]. In Method 4, Scale1[idx] may be 1 by default, and the adjustment method is affected simply by changing the Scale2[idx] value.

[0177] Example 8: Among Examples 5 and 6, the coefficient hyperparameter feature z_hat_1 of the first stage sub-block is input into the mean value prediction network, and the mean value feature of the first stage sub-block is output by the mean value prediction network, and the prediction process of this mean value prediction network is not restricted. For the i-th (i=1…N)-th stage sub-block, the coefficient hyperparameter feature z_hat_i of the i-th stage sub-block and the reference feature of the i-th stage sub-block (which may be the default reference feature) are input into the mean value prediction network, and the mean value feature of the i-th stage sub-block can be output by the mean value prediction network, and the prediction process of the mean value prediction network is described below.

[0178] As shown in FIG. 8a, this is a schematic diagram of the structure of an average value prediction network, and the average value prediction network may include a first prediction network, a second prediction network, and a prediction fusion network, and of course, this is merely an example and is not limited to the structure of this average value prediction network.

[0179] The coefficient hyperparameter feature z_hat_i of the i-th stage sub-block is input into a first prediction network, and a first prediction feature corresponding to the coefficient hyperparameter feature z_hat_i can be obtained through the first prediction network. The reference feature of the i-th stage sub-block (e.g., already reconstructed features y_hat_1 to y_hat_i-1) is input into a second prediction network, and a second prediction feature corresponding to the reference feature can be obtained through the second prediction network. Next, feature concatenation is performed on the first prediction feature and the second prediction feature to obtain the concatenated feature, the concatenated feature is input into a prediction fusion network, and processing is performed on the concatenated feature through the prediction fusion network to obtain the average value feature mu_i of the i-th stage sub-block.

[0180] For the first prediction network: A first prediction feature is obtained by performing a feature augmentation operation and an upsampling operation on the coefficient hyperparameter feature z_hat_i through the first prediction network. Here, the feature augmentation operation may include: a convolution operation, or a convolution operation and an activation operation, but is not limited thereto, and of course, the foregoing is merely an example of a feature augmentation operation and is not limited to this feature augmentation operation. The upsampling operation may include: a deconvolution operation, a cropping operation and an activation operation, or a deconvolution operation, a cropping operation, an activation operation and a convolution operation, but is not limited thereto, and of course, the foregoing is merely an example of an upsampling operation and is not limited to this upsampling operation. For example, the activation operation is used as a Relu operation.

[0181] For example, as shown in FIG. 8b, this is a schematic diagram of the structure of a first prediction network, and the first prediction network is composed of three augmentation networks and two 2x upsampling networks. Of course, the number of augmentation networks may be more or less, the number of 2x upsampling networks may be more or less, and the 2x upsampling networks may be replaced with other multiples of upsampling networks.

[0182] After passing the coefficient hyperparameter feature z_hat_i through Augmentation Network 1 and 2x Upsampling Network 1, an Augmented Large Size Feature 1 is obtained, and the spatial size (width and height) of Augmented Large Size Feature 1 is twice that of the coefficient hyperparameter feature z_hat_i. After passing Augmented Large Size Feature 1 through Augmentation Network 2 and 2x Upsampling Network 2, an Augmented Large Size Feature 2 is obtained, and the spatial size (width and height) of Augmented Large Size Feature 2 is twice that of Augmented Large Size Feature 1. After passing Augmented Large Size Feature 2 through Augmentation Network 3, a first prediction feature is obtained.

[0183] For example, augmentation network 1 may be the same as augmentation network 2 or different, augmentation network 1 may be the same as augmentation network 3 or different, and augmentation network 2 may be the same as augmentation network 3 or different.

[0184] For example, one embodiment of an augmentation network (e.g., augmentation network 1, and / or augmentation network 2, and / or augmentation network 3) can be illustrated in FIG. 8c, wherein the augmentation network is composed of one convolutional layer, and said convolutional layer may be a 1x1 convolutional layer, a 3x3 convolutional layer, or a 5x5 convolutional layer, but is not limited thereto, for example, a 3x3 convolutional layer is selected.

[0185] For example, one embodiment of an augmentation network (e.g., augmentation network 1 and / or augmentation network 2 and / or augmentation network 3) can be illustrated with FIG. 8d, wherein the augmentation network may be composed of a convolution layer 1, an activation layer 1, a convolution layer 2, an activation layer 2, and a convolution layer 3. Here, the convolution layer (e.g., convolution layer 1 and / or convolution layer 2 and / or convolution layer 3) may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and is not limited thereto, for example, a 3x3 convolution layer is selected.

[0186] The activation layer (e.g., activation layer 1, and / or activation layer 2) may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh (hyperbolic tangent) layer, or a Gelu (Gaussian Error Linear Units) layer, and there are no restrictions on the type of activation layer, so, for example, a relu layer may be selected.

[0187] Of course, the above are merely two examples of augmented networks and are not limited to them; any network capable of implementing augmentation functions is sufficient.

[0188] For example, 2x upsampling network 1 may be the same as or different from 2x upsampling network 2.

[0189] For example, one embodiment of a 2x upsampling network (e.g., 2x upsampling network 1 and / or 2x upsampling network 2) can be illustrated with FIG. 8e, wherein the 2x upsampling network may be composed of a 2x deconvolution layer, a crop layer, and an activation layer. Here, the deconvolution layer may be a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer, and is not limited thereto; for example, a 4x4 deconvolution layer may be selected. The activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer, and is not limited to the type of the activation layer; for example, a relu layer may be selected. Additionally, the Crop layer is used to crop features, causing the spatial resolution of the features to change only slightly and the number of channels to remain constant.

[0190] For example, one embodiment of a 2x upsampling network (e.g., 2x upsampling network 1 and / or 2x upsampling network 2) can be illustrated with FIG. 8f, wherein the 2x upsampling network may be composed of a 2x deconvolution layer, a crop layer, an activation layer, and a convolution layer. Here, the deconvolution layer may be a 2x2 deconvolution layer, a 3x3 deconvolution layer, a 4x4 deconvolution layer, or a 5x5 deconvolution layer. The activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer. The crop layer is used to crop features so that the spatial resolution of the features changes only slightly and the number of channels remains constant. The convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and is not limited thereto; for example, a 3x3 convolution layer is selected.

[0191] Of course, the above is merely two examples of 2x upsampling networks, and is not limited to this; any network capable of implementing upsampling functionality is sufficient.

[0192] For example, regarding the convolution in the first prediction network, it may be a general 2D convolution or a group convolution, such as a double group convolution, and the implementation method of the group convolution may refer to subsequent embodiments and is not described further here.

[0193] Regarding the second prediction network: Through the second prediction network, feature concatenation is performed according to the channel dimension on all reconstructed features among the reference features of the i-th stage sub-block, and a convolution operation is performed on the features after concatenation to obtain the second prediction feature. Alternatively, through the second prediction network, feature addition is performed on all reconstructed features among the reference features of the i-th stage sub-block, and a convolution operation is performed on the features after addition to obtain the second prediction feature. Here, the reference features include all reconstructed features of the preceding i-1 stage sub-blocks; or partial reconstructed features of the preceding i-1 stage sub-blocks; or reconstructed features of the i-1 stage sub-blocks.

[0194] For example, as illustrated in FIG. 8g, this is a schematic diagram of the structure of a second prediction network, and the second prediction network may be composed of a feature concatenation layer and a convolution layer, and is not limited to the structure of the second prediction network. If the reference feature includes all reconstructed features of the preceding i-1 stage sub-blocks (e.g., y_hat_1 to y_hat_i-1), the feature concatenation layer performs feature concatenation on these reconstructed features according to the channel dimension to obtain the concatenated features. Or, if the reference feature includes partial reconstructed features of the preceding i-1 stage sub-blocks (e.g., parts of y_hat_1 to y_hat_i-1), the feature concatenation layer performs feature concatenation on these reconstructed features according to the channel dimension to obtain the concatenated features. Alternatively, if the reference feature includes the reconstruction feature of the i-1th stage sub-block, the feature junction layer may directly consider the reconstruction feature of the i-1th stage sub-block as a feature after junction.

[0195] Features after connecting the feature connecting layer are input to the convolution layer, and a second prediction feature can be obtained by performing a convolution operation on the features after connecting by the convolution layer. Here, the convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and is not limited thereto; for example, a 3x3 convolution layer may be selected. Regarding the convolution layer in the second prediction network, it may be a general 2D convolution or a group convolution, for example, a double group convolution, and the implementation method of the group convolution is described in subsequent embodiments.

[0196] For example, the second prediction network may be composed of a feature addition layer and a convolution layer. If the reference feature includes all reconstructed features of the preceding i-1 stage sub-blocks, the feature addition layer performs feature summing on these reconstructed features to obtain the feature after summing. Alternatively, if the reference feature includes partial reconstructed features of the preceding i-1 stage sub-blocks, the feature addition layer performs feature summing on these reconstructed features to obtain the feature after summing. Alternatively, if the reference feature includes reconstructed features of the i-1th stage sub-block, the feature addition layer may directly consider the reconstructed features of the i-1th stage sub-block as the feature after summing. The feature addition layer inputs the feature after summing to the convolution layer, and the convolution layer performs a convolution operation on the feature after summing to obtain the second prediction feature.

[0197] Here, to control the size of the cache, the quantity of already reconstructed features included in the reference feature must not exceed a preset quantity. Assuming the preset quantity is M, if the quantity of already reconstructed features is less than or equal to M, the reference feature includes all already reconstructed features among y_hat_1 to y_hat_i-1. Or, if the quantity of already reconstructed features is greater than M, the reference feature includes M already reconstructed features among y_hat_1 to y_hat_i-1. For example, M already reconstructed features can be selected based on a default strategy, for example, M already reconstructed features closest to the current stage sub-block in the decoding order can be selected. Alternatively, index values ​​of M already reconstructed features can be obtained by performing decoding on the bit stream, and M already reconstructed features can be selected based on the index values ​​obtained by decoding.

[0198] Regarding the predictive fusion network: A convolution operation and at least one fusion operation are performed on the post-concatenation feature (i.e., the post-concatenation feature of the first predictive feature and the second predictive feature) through the predictive fusion network to obtain the average value feature mu_i of the i-th stage sub-block; wherein, for each fusion operation, the fusion operation may include an activation operation and a convolution operation. For example, the activation operation is used as a Relu operation.

[0199] For example, as illustrated in FIG. 8h, this is a schematic diagram of the structure of a predictive fusion network, and the predictive fusion network may be composed of a convolutional layer and N fusion layers, where N is a positive integer, for example, N is 1 or 2, and each fusion layer may be composed of an activation layer and a convolutional layer, and there are no restrictions on the structure of this predictive fusion network. Clearly, after the input feature (i.e., the feature after convolution) passes through the convolutional layer and N "activation layer + convolutional layer," the output feature, i.e., the average value feature mu_i of the i-th stage sub-block, can be obtained.

[0200] Here, the convolution layer may be a 1x1 convolution layer, a 3x3 convolution layer, or a 5x5 convolution layer, and is not limited thereto; for example, a 1x1 convolution layer may be selected. Regarding the convolution layer in the predictive fusion network, it may be a general 2D convolution or a group convolution, for example, a double group convolution, and the implementation method of the group convolution is described in subsequent examples. The activation layer may be a relu layer, a leaky relu layer, a sigmoid layer, a tanh layer, or a Gelu layer; for example, a relu layer may be selected.

[0201] Example 9: For the coding and decoding units, it is necessary to determine a reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block. For example, in Examples 5 and 6, a clustering and segmenting unit may perform feature aggregation on the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks to obtain a feature after aggregation (full-size aggregated feature y_hat_all), and may also perform segmentation on the feature after aggregation y_hat_all to obtain at least one segmented feature y_hat_p, such as segmented feature y_hat_p_1 to segmented feature y_hat_p_K. For each segmented feature y_hat_p, the clustering and segmenting unit inputs the segmented feature y_hat_p into a synthetic transformation network, and the synthetic transformation network outputs a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p. A merging unit obtains a divided reconstructed image block x_hat_p corresponding to a plurality of divided features y_hat_p, and also performs merging on this divided reconstructed image block x_hat_p to obtain a reconstructed image block x_hat corresponding to the current image block.

[0202] First, the clustering and partitioning unit performs feature aggregation on the reconstruction features of N stage sub-blocks to obtain the aggregated features.

[0203] For example, referring to FIG. 9a, reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks are aggregated according to phase to obtain the aggregated feature y_hat_all. This is explained using N=4 in FIG. 9a as an example. For example, one can refer to the content below for an example of performing aggregate according to phase, and this y_hat_group1 After obtaining, y_hat_group1 It can be considered as the feature y_hat_all after aggregation.

[0204]

[0205] For example, referring to FIG. 9b, reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks are aggregated according to phase to obtain multiple features after phase aggregation, and concatenation is performed according to channels on the multiple features after phase aggregation to obtain the aggregated features y_hat_all. For example, an example of performing aggregation according to phase refers to the mathematical formula described above, and according to phase y_hat_group1 and y_hat_group2 It is obtained by aggregating, and if there are relatively many reconstruction features, y_hat_group1 and y_hat_group2 After obtaining the group, based on the multiple y_hat_groups after aggregation, concatenation can be performed again according to the channel to obtain the feature y_hat_all after aggregation.

[0206] For example, before aggregating the reconstructed features of N stage sub-blocks, meaningless feature values ​​may be removed. For instance, if the height or width of a full-size feature is odd and it is divided into 4 parts at the coding stage, padding must be performed on the reconstructed features of a specific stage sub-block to ensure that the size of the reconstructed features of each stage sub-block is the same. The padding method may involve adding one column to the right or one row to the bottom of the full-size feature. Based on this, after aggregating the reconstructed features of N stage sub-blocks, it is necessary to remove meaningless features. Referring to Fig. 9c, an example is provided where the width is even. If the height is even, one row at the bottom is removed. Of course, the above description applies to the case where it is divided into 4 parts; if there is a case where it is divided into 16 parts, the removed features may not be limited to just one row or one column.

[0207] Second, the clustering and partitioning unit performs partitioning on the feature y_hat_all after aggregation to obtain K partitioned features y_hat_p.

[0208] For example, partitioning can be performed on the feature y_hat_all after aggregation to reduce subsequent memory cost overhead (where the synthesis transform operation requires multiple upsamplings on the feature, causing the feature data size to exceed the size allowed by memory). Of course, regarding processes where the feature partitioning process is unnecessary, if it is determined that the current feature size is relatively small or that partitioning is not necessary based on bitstream information, the partitioning process below can be skipped directly, and y_hat_p is equal to y_hat_all, meaning only one partitioned feature exists.

[0209] To perform partitioning on the feature y_hat_all after aggregation, a feature partitioning method in which no overlapping parts exist can be used, and as shown in FIG. 9d, this is an example of partitioning the feature y_hat_all after aggregation into four partitioned features y_hat_p.

[0210] Among feature partitioning methods in which there is no overlap, the target size of the partitioned features is determined, and based on the target size, the aggregated feature y_hat_all can be partitioned into multiple partitioned features in order from top to bottom and left to right, and the size of each partitioned feature is the target size, for example, partitioned into partitioned features y_hat_p_1, y_hat_p_2, y_hat_p_3, and y_hat_p_4.

[0211] For example, the target size is the size of each segmented feature, and the target size may be the size read from the bit stream or may be determined based on the size of the feature currently to be segmented, without restriction thereon. If the number of channels of each segmented feature is tile_c, the width is tile_w, and the height is tile_h, the feature y_hat_all after aggregation is averaged in order from top to bottom and from left to right into multiple segmented features of tile_c*tile_w*tile_h, e.g., segmented features y_hat_p_1, y_hat_p_2, y_hat_p_3, y_hat_p_4.

[0212] To perform partitioning on the feature y_hat_all after aggregation, a feature partitioning method with overlapping parts can be used, as shown in FIG. 9e, which is an example of partitioning the feature y_hat_all after aggregation into four partitioned features y_hat_p.

[0213] Among the feature partitioning methods in which overlapping parts exist, the actual partition size and overlapping size of the partitioned features can be determined, the actual partition size may be the partition size with the overlapping part removed from the partitioned features, and the overlapping size may be the size of the overlapping part between adjacent partitioned features; based on the actual partition size and the overlapping size, the feature after aggregation is partitioned into multiple partitioned features, and the size of each partitioned feature is the target size, for example, partitioned into partitioned features y_hat_p_1, y_hat_p_2, y_hat_p_3, y_hat_p_4.

[0214] For example, the actual partition size and the overlap size are determined, where the actual partition size is the partition size with the overlap portion removed, and the overlap size is the size of the overlap portion between adjacent partitioned features. This size information may be read from a bit stream or determined based on the feature size currently to be partitioned, and is not limited thereto. Referring to FIG. 9e, if the actual partition size is tile_c * tile_w * tile_h and the overlap size is tile_c * padding_w * padding_h, then when overlapping vertically and horizontally, the size of each partitioned feature (i.e., target size) is tile_c * (tile_w + 2 * padding_w) * (tile_h + 2 * padding_h). Based on this, for the feature y_hat_all after aggregation, the features from the first row of the first column (tile_w + 2 * padding_w) to the first row (tile_h + 2 * padding_h) are partitioned into the first partitioned features. The features of the second tile_w+padding_w+1 column, the second tile_h+padding_h+1 row, and the third (2*tile_h+3*padding_h) row are divided into the second divided features, and inferred in this way.

[0215] Referring to Fig. 9e, the feature y_hat_all after assembly of 4x4x4 can be divided into 4x3x3 divided features, among which tile_c is 4, padding_w and padding_h are 1, and tile_w and tile_h are 1.

[0216] For example, for the last row or column partition, if the quantity is insufficient, an overlapping area can be added so that the final partition size meets the demand. Referring to Fig. 9f, one example is provided in which the size of y_hat_all is 11x13, padding_w and padding_h are 1, and tile_w and tile_h are 3. To ensure that the size of the first and second partitions of the first row is 5x5, and the size of the third partition, for example, is also 5x5, an overlapping area of ​​2 columns is expanded to the left.

[0217] Third, the clustering and partitioning unit inputs each partitioned feature y_hat_p (e.g., K partitioned features y_hat_p) into a synthetic transformation network, and the synthetic transformation network outputs a partitioned reconstructed image block x_hat_p corresponding to each partitioned feature y_hat_p.

[0218] For each segmented feature y_hat_p, the segmented feature y_hat_p is input into a synthetic transformation network, and then a synthetic transformation is performed on the segmented feature y_hat_p through the synthetic transformation network to obtain a segmented reconstructed image block x_hat_p, and is not restricted therefrom.

[0219] Fourth, the merging unit performs merging on multiple divided reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.

[0220] For example, a merging unit can obtain K divided reconstructed image blocks x_hat_p corresponding to K divided features y_hat_p, and also perform merging on the K divided reconstructed image blocks x_hat_p to obtain reconstructed image blocks x_hat.

[0221] For example, if segmentation is performed on the aggregated feature y_hat_all using a feature segmentation method that has no overlap, the merging unit performs merging on the K segmented reconstructed image blocks x_hat_p using a feature merging method that has no overlap. For example, the segmented reconstructed image blocks x_hat_p corresponding to the K segmented features are arranged in order from top to bottom and left to right, and the segmented reconstructed image blocks x_hat_p corresponding to the K segmented features are concatenated in order after arrangement to obtain the reconstructed image block x_hat corresponding to the current image block. Here, the concatenation process may be the inverse process of FIG. 9d, which is not described further here.

[0222] For example, if a feature y_hat_all after aggregation is segmented using a feature segmentation method that has overlapping parts, a merging unit performs merging on K segmented reconstructed image blocks x_hat_p using a feature merging method that has overlapping parts. For example, the merging unit can determine the actual partition size and overlap size of each segmented reconstructed image block x_hat_p. The merging unit performs arrangement on the segmented reconstructed image blocks x_hat_p corresponding to the K segmented features y_hat_p in order from top to bottom and left to right; based on the actual partition size and overlap size, the segmented reconstructed image blocks x_hat_p corresponding to the K segmented features y_hat_p after arrangement are concatenated in order to obtain the reconstructed image block x_hat corresponding to the current image block. Here, the reconstructed image block may include K divided reconstructed image blocks, the size of the non-overlapping portion of the divided reconstructed image block may be the actual partition size of the divided reconstructed image block, and the size of the overlapping portion of the divided reconstructed image block adjacent to the divided reconstructed image block may be the overlapping size of the divided reconstructed image block. Here, the concatenation process may be the inverse process of FIG. 9e, which is not described further here.

[0223] For example, the actual partition size and the overlap size are determined, where the actual partition size is the partition size with the overlap portion removed, and the overlap size is the size of the overlap portion between adjacent partitioned features; this size information may be read from a bit stream or determined based on the size of the feature currently to be partitioned, and there are no restrictions on this. Assuming that the actual partition size is x_c * xtile_w * xtile_h, the overlap width within the overlap size is xpadding_w, and the overlap height within the overlap size is xpadding_h, multiple partitioned reconstructed image blocks x_hat_p are obtained by concatenating them sequentially according to the reception order of multiple partitioned reconstructed image blocks x_hat_p.

[0224] For the overlapping portion of the left split reconstruction image block and the right split reconstruction image block, the value of the overlapping portion is the value of the left split reconstruction image block, and the value of the right split reconstruction image block is discarded, that is, the value of the initial (left) split reconstruction image block is used, and for the subsequent sub-block overlapping portion, the value of the overlapping portion is discarded. Or, the value of the overlapping portion is the average value of the two split reconstruction image blocks.

[0225] For the overlapping portion of the upper partitioned reconstructed image block and the lower partitioned reconstructed image block, the value of the overlapping portion is the value of the upper partitioned reconstructed image block, and the value of the lower partitioned reconstructed image block is discarded, that is, the value of the initial (upper) partitioned reconstructed image block is used, and for the subsequent sub-block overlapping portion, the value of the overlapping portion is directly discarded. Or, the value of the overlapping portion is the average value of the two partitioned reconstructed image blocks.

[0226] Example 10: For the coding and decoding units, it is necessary to determine a reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block. For example, a clustering and partitioning unit may perform feature aggregation on the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks to obtain a feature after aggregation (which may also be referred to as a full-size aggregated feature y_hat_all), and the clustering and partitioning unit inputs the aggregated feature y_hat_all into a synthetic transformation network, and the synthetic transformation network outputs a reconstructed image block corresponding to the aggregated feature y_hat_all, and said reconstructed image block is the reconstructed image block x_hat corresponding to the current image block. Compared to Example 9, it does not involve performing partitioning on the feature y_hat_all after aggregation, nor does it involve performing merging on a plurality of partitioned reconstructed image blocks x_hat_p.

[0227] First, the clustering and partitioning unit performs feature aggregation on the reconstruction features of N stage sub-blocks to obtain the aggregated features.

[0228] For example, reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks are aggregated according to phase to obtain aggregated features y_hat_all. Alternatively, reconstruction features (y_hat_1 to y_hat_N) of N stage sub-blocks are aggregated according to phase to obtain multiple features after phase aggregation, and concatenation is performed according to channels on the multiple features after phase aggregation to obtain aggregated features y_hat_all. Here, the feature aggregation can refer to Example 9 and is not described further here.

[0229] Second, the clustering and partitioning unit inputs the aggregated features y_hat_all into a synthetic transformation network, performs a synthetic transformation on the aggregated features y_hat_all through the synthetic transformation network, and is not restricted to this synthetic transformation method, and outputs a reconstructed image block corresponding to the aggregated features y_hat_all, that is, a reconstructed image block x_hat corresponding to the current image block, by the synthetic transformation network.

[0230] Example 11: For the coding and decoding units, it is necessary to determine a reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block. For example, after the clustering and splitting unit obtains the reconstructed features (y_hat_1 to y_hat_N) of N stage sub-blocks, it does not perform feature aggregation on the reconstructed features, nor perform splitting on the features after aggregation, but directly inputs the reconstructed features of each stage sub-block into a synthetic transformation network, and outputs a split reconstructed image block x_hat_p corresponding to each stage sub-block by the synthetic transformation network. A merging unit obtains a split reconstructed image block x_hat_p corresponding to a plurality of stage sub-blocks, and also performs merging on this split reconstructed image block x_hat_p to obtain a reconstructed image block x_hat corresponding to the current image block.

[0231] First, after the clustering and partitioning unit obtains the reconstruction features of N stage sub-blocks, the reconstruction features of each stage sub-block are input into a synthetic transformation network, and a synthetic transformation is performed on the reconstruction features through the synthetic transformation network, without restricting the method of this synthetic transformation, and the synthetic transformation network outputs a partitioned reconstruction image block x_hat_p corresponding to each stage sub-block.

[0232] Second, the merging unit performs merging on multiple divided reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.

[0233] For example, a merging unit can obtain N divided reconstructed image blocks x_hat_p corresponding to N stage sub-blocks, and also perform merging on the N divided reconstructed image blocks x_hat_p to obtain a reconstructed image block x_hat.

[0234] For example, a merging unit performs merging on N partitioned reconstructed image blocks x_hat_p using a feature merging method where no overlapping parts exist. For example, arraying is performed on the N partitioned reconstructed image blocks x_hat_p in order from top to bottom and from left to right, and the N partitioned reconstructed image blocks x_hat_p after arraying are concatenated in order to obtain the reconstructed image block x_hat corresponding to the current image block. Alternatively, a merging unit performs merging on N partitioned reconstructed image blocks x_hat_p using a feature merging method where overlapping parts exist. For example, the actual partition size and overlap size of each partitioned reconstructed image block x_hat_p are determined, and arraying is performed on the N partitioned reconstructed image blocks x_hat_p in order from top to bottom and from left to right; Based on the actual partition size and overlap size, N partitioned reconstructed image blocks x_hat_p after array are sequentially concatenated to obtain the reconstructed image block x_hat corresponding to the current image block. The reconstructed image block contains N partitioned reconstructed image blocks, the size of the non-overlapping portion of the partitioned reconstructed image block is the actual partition size of the partitioned reconstructed image block, and the size of the overlapping portion of the partitioned reconstructed image block adjacent to the partitioned reconstructed image block is the overlapping size of the partitioned reconstructed image block. For the overlapping portion of the left partitioned reconstructed image block and the right partitioned reconstructed image block, the value of the overlapping portion is the value of the left partitioned reconstructed image block, and the value of the right partitioned reconstructed image block is discarded. Or the value of the overlapping portion is the average value of two partitioned reconstructed image blocks.For the overlapping portion of the upper partitioned reconstructed image block and the lower partitioned reconstructed image block, the value of the overlapping portion is the value of the upper partitioned reconstructed image block, and the value of the lower partitioned reconstructed image block is discarded. Or the value of the overlapping portion is the average value of the two partitioned reconstructed image blocks.

[0235] Example 12: In Examples 1 to 11, regarding the synthetic conversion network, in order to perform accurate bit rate control, the coding unit may transmit bit rate control parameters for the synthetic conversion network to the decoding unit. For example, the coding unit may code the bit rate control parameters into an auxiliary bit stream corresponding to the current image block, and the decoding unit may perform decoding on the auxiliary bit stream corresponding to the current image block to obtain the bit rate control parameters and determine the synthetic conversion network based on the bit rate control parameters. That is, the synthetic conversion network may be determined based on bit stream information, particularly based on bit rate-related information. Here, the synthetic transform network may be composed of at least one doubling deconvolution layer (e.g., a 2x2, 3x3, 4x4, or 5x5 doubling deconvolution layer, e.g., a 4x4 doubling deconvolution layer, and of course, other multiple deconvolution layers), at least one activation layer (e.g., a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, e.g., a relu layer), at least one convolution layer (e.g., a 1x1, 3x3, or 5x5 convolution layer, e.g., a 3x3 convolution layer, and for the convolution layer in the synthetic transform network, it may be a general 2D convolution or a group convolution, e.g., a doubling group convolution), and the output feature space resolution of the synthetic transform network may be a positive integer multiple of the input feature space resolution, e.g. It can be 16 times (both length and width are 16 times), and the number of output feature channels is 1 or 3. Based on the synthetic transform network, the synthetic transform network can be controlled based on bit rate control parameters, that is, the bit rate control parameters are considered as inputs to the synthetic transform network.

[0236] Among the possible embodiments, for Example 9, the clustering and partitioning unit inputs K partitioned features y_hat_p into a synthetic transformation network, and thus the coding unit codes K bit rate control parameters corresponding to each of the K partitioned features y_hat_p into an auxiliary bit stream corresponding to the current image block, and is not restricted to the source of the K bit rate control parameters, and when the coding unit controls the analysis transformation network and the synthetic transformation network using these K bit rate control parameters (i.e., the K bit rate control parameters are considered as inputs to the analysis transformation network and the synthetic transformation network, for example, the partitioned feature y_hat_p and the bit rate control parameters corresponding to the partitioned feature y_hat_p are input together into the synthetic transformation network and considered as input features), the bit rate corresponding to the bit stream of the current image block can be made to be the target bit rate or close to the target bit rate (i.e., the difference between the two is smaller than the threshold value). The decoding unit performs decoding on the auxiliary bit stream corresponding to the current image block to obtain K bit rate control parameters corresponding to K divided features y_hat_p.

[0237] Based on this, a clustering and segmentation unit can obtain K segmented features y_hat_p and K bit rate control parameters corresponding to the K segmented features y_hat_p, and for each segmented feature y_hat_p, the segmented feature y_hat_p and the bit rate control parameters corresponding to the segmented feature y_hat_p are input together into a synthetic transformation network, and a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p is generated by the synthetic transformation network based on the segmented feature y_hat_p and the bit rate control parameters corresponding to the segmented feature y_hat_p.

[0238] For example, the step of generating a segmented reconstructed image block x_hat_p corresponding to a segmented feature y_hat_p based on a segmented feature y_hat_p and a bit rate control parameter corresponding to said segmented feature y_hat_p based on a synthetic transform network may include, but is not limited to: a step of performing processing on the segmented feature y_hat_p through a synthetic transform network to obtain a first feature; a step of performing processing on the bit rate control parameter corresponding to said segmented feature y_hat_p through a synthetic transform network to obtain a second feature; a step of generating a third feature based on the first feature and the second feature, and also determining a segmented reconstructed image block x_hat_p corresponding to the segmented feature y_hat_p based on the third feature.

[0239] Among the possible embodiments, for Example 10, the clustering and partitioning unit inputs the feature y_hat_all after aggregation into a synthetic transformation network, and thus the coding unit codes one bit rate control parameter corresponding to the feature y_hat_all after aggregation, i.e., the bit rate control parameter corresponding to the current image block, into an auxiliary bit stream corresponding to the current image block, and the source of said bit rate control parameter is not restricted, and when the coding unit controls the analysis transformation network and the synthetic transformation network using said bit rate control parameter (i.e., considering said bit rate control parameter as an input to the analysis transformation network and the synthetic transformation network, for example, the feature y_hat_all after aggregation is input to the synthetic transformation network together with said bit rate control parameter, and the current image block is input to the analysis transformation network together with said bit rate control parameter and considered as an input feature), the bit rate corresponding to the bit stream of the current image block can be made to be a target bit rate or close to the target bit rate. The decoding unit can perform decoding on the auxiliary bit stream corresponding to the current image block to obtain the bit rate control parameter corresponding to the current image block.

[0240] Based on this, a clustering and partitioning unit obtains the features y_hat_all after aggregation and bit rate control parameters, and also inputs the features y_hat_all after aggregation and bit rate control parameters together into a synthetic transformation network, and the synthetic transformation network can generate a reconstructed image block x_hat corresponding to the features y_hat_all after aggregation based on the features y_hat_all after aggregation and bit rate control parameters.

[0241] A step of generating a reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the aggregated feature y_hat_all and bit rate control parameters by a synthetic transform network may include: a step of obtaining a first feature by performing processing on the aggregated feature y_hat_all through a synthetic transform network; a step of obtaining a second feature by performing processing on the bit rate control parameters through a synthetic transform network; a step of generating a third feature based on the first feature and the second feature, and also determining a reconstructed image block x_hat corresponding to the aggregated feature y_hat_all based on the third feature.

[0242] Among the possible embodiments, for Example 11, the clustering and partitioning unit inputs the reconstruction features of N stage sub-blocks into a synthetic transformation network. Accordingly, the coding unit codes N bit rate control parameters corresponding to each of the N stage sub-blocks into an auxiliary bit stream corresponding to the current image block, and does not restrict the source of the N bit rate control parameters. When the coding unit controls the analysis transformation network and the synthetic transformation network using these N bit rate control parameters (i.e., considering the N bit rate control parameters as inputs to the analysis transformation network and the synthetic transformation network, for example, inputting the reconstruction features of the stage sub-blocks and the bit rate control parameters corresponding to the stage sub-blocks together into the synthetic transformation network), the bit rate corresponding to the bit stream of the current image block can be made to be the target bit rate or close to the target bit rate. The decoding unit performs decoding on the auxiliary bit stream corresponding to the current image block to obtain N bit rate control parameters corresponding to the N stage sub-blocks.

[0243] Based on this, a clustering and partitioning unit can obtain reconstruction features of N stage sub-blocks and N bit rate control parameters corresponding to the N stage sub-blocks, and for each stage sub-block, the reconstruction features of the stage sub-block and the bit rate control parameters corresponding to the stage sub-block are input together into a synthetic transformation network, and a partitioned reconstruction image block x_hat_p corresponding to the stage sub-block is generated by the synthetic transformation network based on the reconstruction features of the stage sub-block and the bit rate control parameters corresponding to the stage sub-block.

[0244] For example, the step of generating a segmented reconstructed image block x_hat_p corresponding to the stage subblock based on a reconstructed feature of the stage subblock and a bit rate control parameter corresponding to the stage subblock by a synthetic transform network may include, but is not limited to: a step of obtaining a first feature by performing processing on the reconstructed feature of the stage subblock through a synthetic transform network; a step of obtaining a second feature by performing processing on the bit rate control parameter corresponding to the stage subblock through a synthetic transform network; and a step of generating a third feature based on the first feature and the second feature, and also determining a segmented reconstructed image block x_hat_p corresponding to the stage subblock based on the third feature.

[0245] Example 13: Based on Example 12, a synthetic transform network can be referred to as a synthetic transform network controlled by a bit rate control parameter, denoted as the bit rate control parameter λ, and the λ parameter is obtained by decoding from an auxiliary bit stream corresponding to the current image block, and as shown in FIG. 10a, this is a schematic diagram of the structure of a synthetic transform network controlled by the λ parameter, and said synthetic transform network is a synthetic transform network of variable bit rates, and based on said synthetic transform network, output features with a number of channels of 1 or 3 can be obtained. Of course, FIG. 10a is merely one example of a synthetic transform network and is not limited to the structure of said synthetic transform network.

[0246] Referring to FIG. 10a, a synthetic transform network controlled by a λ parameter may include at least one Conv (convolution layer) and at least one λ-RSTB (Residual Swin Transformer Block). Taking the four convolution layers and four λ-RSTBs in FIG. 10a as examples, the number of convolution layers may be greater or less, and the number of λ-RSTBs may be greater or less. The convolution layers may be, for example, 1x1, 3x3, or 5x5 convolution layers, and for example, a 3x3 convolution layer may be selected. The convolution layers in the synthetic transform network may be general 2D convolutions or group convolutions, for example, double group convolutions, and the structure of these convolution layers is not limited. λ-RSTB may be an RSTB network layer controlled by a λ parameter, and the structure of the RSTB network layer controlled by the λ parameter can be referenced in FIG. 10b.

[0247] Referring to FIG. 10b, the λ-RSTB may include network layers such as FE (Feature Embedding), λ-STB (Swin Transformer Block), and FU (Feature Un-embedding), and is not limited thereto. Of course, this is merely an example, and the structure of this λ-RSTB is not limited. FE is the feature embedding layer and is used to implement feature embedding processing, and FU is the feature un-embedding layer and is used to implement feature un-embedding processing.

[0248] A λ-STB represents an STB network layer controlled by a parameter λ, and the structure of the STB network layer controlled by the parameter λ is illustrated in FIG. 10b. The λ-STB may include a linear layer (LN), a weighted average layer (λ-WA), a stochastic weighted average layer (λ-SWA), a multilayer perceptron (MLP), etc., and is not limited thereto. Of course, this is merely an example, and the structure of the λ-STB is not restricted. The LN is a linear layer, and the λ-STB may include at least one LN, and the number of LNs is not restricted; here, four LNs are used as an example. The MLP is a multilayer perceptron, and the λ-STB may include at least one MLP, and the number of MLPs is not restricted; here, two MLPs are used as an example.

[0249] The structure of λ-SWA is identical to the structure of λ-WA. Here, λ-WA is used as an example. λ-WA represents a WA network layer controlled by a λ parameter, and the structure of the WA network layer controlled by the λ parameter can be referenced in FIG. 10c. λ-WA may include an FC (Full Connection) layer, MatMul (Matrix Multiplication), Scale, SoftMax (Logical Regression, also referred to as Normalization), MatMul, etc., and is not limited thereto. Of course, this is merely an example, and the structure of this λ-WA is not limited thereto.

[0250] The parameter λ can be mapped to one scaling factor through the FC layer, that is, processing is performed on the bit rate control parameter through the FC layer of the synthetic transformation network to obtain a second feature, and said second feature may be a scaling factor.

[0251] Processing is performed on input features (e.g., partitioned features y_hat_p, features after aggregation y_hat_all, or reconstructed features of stage sub-blocks) through a transformer network to obtain Q features, K features, and V features, all of which originate from the input features themselves and are feature vectors generated by multiplying by corresponding weights based on the input features, without being restricted to this process, that is, the composite transformation network may include a transformer network, and processing is performed on input features (e.g., partitioned features, features after aggregation, or reconstructed features) through the transformer network of the composite transformation network to obtain a first feature.

[0252] After obtaining Q features, K features, and V features, matrix multiplication, scaling, and logical regression operations are performed sequentially based on Q features and K features to obtain intermediate feature 1. Intermediate feature 2 can be obtained by multiplying the scaling coefficient (obtained by mapping the parameter λ through the FC layer) by V feature. The output feature of λ-WA is obtained by multiplying the matrix of intermediate feature 1 and intermediate feature 2. In summary, the scaling coefficient is the second feature, and Q features, K features, and V features are the first features; therefore, a third feature can be generated based on the first and second features, and the third feature is the output feature of λ-WA.

[0253] After obtaining the output features of λ-WA, in conjunction with FIG. 10a and FIG. 10b, based on the output features of λ-WA (i.e., the third feature), the output features of the synthetic transformation network, such as the divided reconstructed image block x_hat_p corresponding to the divided feature y_hat_p, the reconstructed image block x_hat corresponding to the feature y_hat_all after aggregation, and the divided reconstructed image block x_hat_p corresponding to the stage sub-block can be determined.

[0254] Referring to the λ-WA illustrated in FIG. 10c, the calculation method of λ-WA may be as described below, and through this method, the image encoder and decoder outputs can be controlled using the λ parameter, thereby obtaining bit streams of different sizes and reconstructed images of different qualities.

[0255]

[0256] Example 14: Among Examples 1 to 13, the convolution layer may be a general 2D convolution or a group convolution, such as a double group convolution. For example, the step of a general 2D convolution may be referenced in FIG. 11a, where the input feature is (H×W×C), and then C' filters (each filter having a size of (h×w×C) and the filter size C being the same as the input feature C) are applied, and the input layer is converted to an output feature having a size of (H'×W'×C').

[0257] Refer to FIG. 11b for the group convolution step, and for each filter group, a quantity of C' / g (e.g., g is 2) filters may be included, and the number of channels in each filter is half that of a typical 2D convolution filter. Each filter group acts on 1 / g of the number of channels corresponding to the existing W×H×C, i.e., W×H×C / g. Therefore, the output corresponding to each filter group is characterized by C' / g channels. Finally, by stacking the channels to obtain a final C' channel, the same effect as the aforementioned typical 2D convolution can be achieved. Clearly, through group convolution, the amount of parameters and computation can be reduced.

[0258] Example 15: Among Examples 1 to 14, in order to perform accurate bit rate control, the coding unit needs to transmit bit rate control parameters to the decoding unit, for example, the bit rate control parameters are carried and transmitted through an auxiliary bit stream corresponding to the current image block, and at the coding unit, the bit rate control parameters affect the analysis transform network and the synthesis transform network. In order to make the bit rate corresponding to the bit stream of the current image block equal to the target bit rate or close to the target bit rate (pre-set bit rate), the coding unit can implement different bit rate control parameters to find the target bit rate control parameters, and under the target bit rate control parameters, the bit rate corresponding to the bit stream of the current image block equals the target bit rate or closes to the target bit rate, and the coding unit codes the target bit rate control parameters into the auxiliary bit stream corresponding to the current image block.

[0259] If the current image block corresponds to only one bit rate control parameter, for example, if the feature y_hat_all after aggregation corresponds to one bit rate control parameter, then each bit rate control parameter is executed sequentially from a plurality of bit rate control parameters until the bit rate corresponding to the bit stream of the current image block is the target bit rate or approaches the target bit rate, and the target bit rate control parameter is obtained.

[0260] If the current image block corresponds to multiple bit rate control parameters, for example, K bit rate control parameters corresponding to K segmented features, and N stage sub-blocks correspond to N bit rate control parameters, the initial values ​​of the multiple bit rate control parameters are first set. If the bit rate corresponding to the bit stream of the current image block does not meet the demand, one candidate bit rate control parameter is selected from the multiple bit rate control parameters and the value of the candidate bit rate control parameter is lowered. If the bit rate corresponding to the bit stream of the current image block does not meet the demand, one candidate bit rate control parameter (which may be the same as or different from the previous candidate bit rate control parameter) is selected from the multiple bit rate control parameters and the value of the candidate bit rate control parameter is lowered. This process is continued until the bit rate corresponding to the bit stream of the current image block meets the demand (i.e., the bit rate corresponding to the bit stream of the current image block is the target bit rate or is close to the target bit rate) and the target value of the multiple bit rate control parameters is obtained.

[0261] When selecting one candidate bit rate control parameter from multiple bit rate control parameters, one bit rate control parameter may be randomly selected and considered as a candidate bit rate control parameter, the largest bit rate control parameter may be selected and considered as a candidate bit rate control parameter, or a candidate bit rate control parameter may be selected based on the bit rate distortion relationship. For example, when bit rate control parameter A is lowered from the current level to the next level, the distortion is A1, and when bit rate control parameter B is lowered from the current level to the next level, the distortion is B1, and if distortion A1 is smaller than distortion B1, that is, if distortion A1 is the minimum distortion, then bit rate control parameter A is considered as a candidate bit rate control parameter.

[0262] Among the possible implementation methods, how the coding unit of each (feature or image) block The method of implementing the task of setting parameters is as follows: each block For (e.g., divided features or stage sub-blocks), different Coding is performed using [the method]. Before coding, several discrete λ values ​​are selected for each block, and these discrete λ values ​​are defined as quality classes. Additionally, the bit rate and distortion of each block and each quality class are statistically analyzed before coding. For each frame of an image, this process needs to be performed only once before the first coding.

[0263] During the coding process, coding is performed for each block by selecting λ from the quality grades. During the coding process, first, coding is performed for all blocks using the same quality grade, and since the said quality grade is the minimum quality grade that makes the total bit rate greater than the target bit rate, that is, satisfying the following mathematical formula, is the target bit rate, and represents the sum of the bit rates of all blocks under quality grade q.

[0264]

[0265] Subsequently, the quality level of each block is gradually lowered until the total bit rate is less than the target bit rate. For each block, the slopes of the current quality level and the previous quality level on the rate-distortion curve can be calculated; that is, the slope can be calculated using the following mathematical formula:

[0266]

[0267] Here, If α is the reconstruction loss under quality class q of block i, and a loss function that gradually decreases with bit rate (e.g., Mean Squared Error) is considered as a metric, then the minimum Select block i (i.e., candidate bit rate control parameter for minimum distortion) that has the above, and also lower it by one quality level. Infer in this way until the total bit rate is less than the target bit rate.

[0268] As shown in Fig. 12, this is a schematic diagram of the determination of the λ parameter, designing seven quality grades (i.e., seven values ​​of the λ parameter) on the rate-distortion curve, and at quality grade 3, As can be calculated, that is, at quality level 3, reducing the quality level of Block 2 results in a greater loss of coding performance, and therefore, reduces the quality level of Block 1.

[0269] As can be seen from the technical methods of each of the embodiments described above, among the embodiments, regarding the neural network-based coding and decoding technology, a bit rate variable and adjustable coding and decoding method is provided to improve concurrency, effectively save the feature storage cache, achieve higher bit rate control precision, have smaller coding performance loss, and possess better coding performance and bit rate control precision. By utilizing block-based coding and decoding, peak value memory occupancy is lower, single-block decoding time is shorter, and high-speed parallel decoding capability is provided, so that the neural network maintains low complexity while effectively securing the quality of reconstructed image blocks, improving coding performance and decoding performance, and also reducing complexity. For example, the bit rate control algorithm of the present method can achieve a bit rate control precision of around 99%, and at the same time, compared to no bit rate control, the coding performance loss is only around 1%. For example, when decoding a single 2k resolution image with a block size of 256*256, using block-based coding and decoding reduces peak value memory usage to 1 / 10 of that of the entire image decoding, while simultaneously decoding a single block for less than 0.1s and providing the capability for high-speed parallel decoding. It also provides better coding and decoding performance when adding fewer parameters.

[0270] Example 16: An example of the present application provides a decoding method, wherein the method may be applied to a decoding unit (also referred to as a video decoder), and the method may include, but is not limited to: a step of performing decoding on a first bit stream of a current image block to obtain a coefficient hyperparameter feature of the current image block; a step of determining a probability distribution parameter based on the coefficient hyperparameter feature, performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain a residual feature of the current image block, and determining a reconstruction feature of the current image block based on the residual feature; a step of performing decoding on an auxiliary bit stream corresponding to the current image block to obtain a bit rate control parameter corresponding to the current image block, and also inputting the reconstruction feature and the bit rate control parameter into a synthetic transform network to obtain a reconstruction image block corresponding to the current image block.

[0271] For example, the step of inputting reconstruction features and bit rate control parameters into a synthetic transformation network to obtain a reconstruction image block corresponding to a current image block may include, but is not limited to: a step of processing the reconstruction features through the synthetic transformation network to obtain a first feature; a step of processing the bit rate control parameters through the synthetic transformation network to obtain a second feature; a step of generating a third feature based on the first feature and the second feature, and also determining a reconstruction image block corresponding to a current image block based on the third feature.

[0272] For example, based on Examples 3 and 4, adjustments can be made to the synthetic transformation network, and in Example 16, only the adjustment process of the synthetic transformation network is described, and other processes can be referred to in Examples 3 and 4.

[0273] For example, accurate bit rate control can be performed through bit rate control parameters, and bit rate control parameters affect the analysis transformation network and the synthesis transformation network. For example, when the coding unit performs an analysis transformation on the current image block through the analysis transformation network, the bit rate control parameters and the current image block are input to the analysis transformation network, and the analysis transformation network outputs image features corresponding to the current image block. When the coding unit or the decoding unit inputs image features y_hat to the synthesis transformation network, the image features y_hat and the bit rate control parameters are input to the synthesis transformation network, and the synthesis transformation network outputs a reconstructed image block corresponding to the current image block.

[0274] Based on a target bit rate already set, the coding unit may implement different bit rate control parameters, and after using a specific bit rate control parameter, if the bit rate corresponding to the bit stream for the current image block is the target bit rate or approaches the target bit rate, this bit rate control parameter is the bit rate control parameter that is finally used, and the coding unit may transmit the bit rate control parameter to the decoding unit.

[0275] For example, the coding unit can transmit bit rate control parameters for the synthetic transformation network to the decoding unit. For instance, the coding unit can code bit rate control parameters corresponding to the current image block into an auxiliary bit stream corresponding to the current image block, and the decoding unit can perform decoding on the auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters corresponding to the current image block, and also determine the synthetic transformation network based on the bit rate control parameters. Here, the synthetic transform network may be composed of at least one doubling deconvolution layer (e.g., a 2x2, 3x3, 4x4, or 5x5 doubling deconvolution layer, e.g., a 4x4 doubling deconvolution layer, and of course, other multiple deconvolution layers), at least one activation layer (e.g., a relu layer, or a leaky relu layer, or a sigmoid layer, or a tanh layer, or a Gelu layer, e.g., a relu layer), at least one convolution layer (e.g., a 1x1, 3x3, or 5x5 convolution layer, e.g., a 3x3 convolution layer, and for the convolution layer in the synthetic transform network, it may be a general 2D convolution or a group convolution, e.g., a doubling group convolution), and the output feature space resolution of the synthetic transform network may be a positive integer multiple of the input feature space resolution, e.g. The output feature has 16 times (both length and width are 16 times), and the number of channels is 1 or 3. Based on the synthetic transform network, the synthetic transform network can be controlled based on bit rate control parameters, that is, the bit rate control parameters are considered as inputs to the synthetic transform network.

[0276] For example, the coding unit codes a bit rate control parameter corresponding to the current image block into an auxiliary bit stream corresponding to the current image block, and does not restrict the source of said bit rate control parameter. When the coding unit controls an analysis transformation network and a synthesis transformation network (i.e., considering said bit rate control parameter as an input to the analysis transformation network and the synthesis transformation network) using said bit rate control parameter, the bit rate corresponding to the bit stream of the current image block can be made to be a target bit rate or close to a target bit rate. The decoding unit can perform decoding on the auxiliary bit stream corresponding to the current image block to obtain a bit rate control parameter corresponding to the current image block. Based on this, the decoding unit inputs the image feature y_hat and the bit rate control parameter together into a synthesis transformation network, and the synthesis transformation network generates a reconstructed image block corresponding to the current image block based on the image feature y_hat and the bit rate control parameter, and can also output said reconstructed image block.

[0277] Here, the step of generating a reconstructed image block corresponding to the current image block based on image feature y_hat and bit rate control parameters by a synthetic transform network may include: a step of obtaining a first feature by performing processing on image feature y_hat (i.e., reconstructed feature y_hat) through a synthetic transform network; a step of obtaining a second feature by performing processing on bit rate control parameters through a synthetic transform network; and a step of generating a third feature based on the first feature and the second feature, and also determining a reconstructed image block corresponding to the current image block based on the third feature.

[0278] For example, referring to FIG. 10a, FIG. 10b and FIG. 10c, the synthetic transformation network may include at least one Conv (convolution layer) and at least one λ-RSTB, and the λ-RSTB may include network layers such as FE, λ-STB, FU, etc., but is not limited thereto, the λ-STB may include LN, λ-WA, λ-SWA, MLP, etc., but is not limited thereto, the structure of the λ-SWA is identical to the structure of the λ-WA, and the λ-WA may include an FC layer, MatMul, Scale, SoftMax, MatMul, etc., but is not limited thereto, and based on this, the parameter λ (i.e., bit rate control parameter) can be mapped to one scaling factor through the FC layer, that is, processing can be performed on the bit rate control parameter through the FC layer of the synthetic transformation network to obtain the second feature. By performing processing on image feature y_hat through a transformer network, Q feature, K feature, and V feature can be obtained; that is, by performing processing on image feature y_hat through the transformer network of the synthetic transformation network, a first feature can be obtained. After obtaining Q feature, K feature, and V feature, matrix multiplication, scaling, and logical regression operations are performed sequentially based on Q feature and K feature to obtain intermediate feature 1; the scaling factor is multiplied by feature V to obtain intermediate feature 2; and the intermediate feature 1 and the intermediate feature 2 matrix are multiplied to obtain the output feature of λ-WA, that is, a third feature can be generated based on the first feature and the second feature, and the third feature is the output feature of λ-WA. After obtaining the output feature of λ-WA, in conjunction with FIG. 10a and FIG. 10b, the output feature of the synthetic transformation network can be determined based on the output feature of λ-WA (i.e., the third feature), for example, the output feature may be a reconstructed image block corresponding to the current image block.

[0279] For example, each of the above-described embodiments may be implemented individually or in combination; for instance, each of embodiments 1 to 16 may be implemented individually, and at least two of embodiments 1 to 16 may be implemented in combination.

[0280] For example, among the embodiments described above, the contents of the coding unit can be applied to the decoding unit, that is, the decoding unit can process using the same method, and the contents of the decoding unit can be applied to the coding unit, that is, the coding unit can process using the same method.

[0281] For example, among the embodiments described above, the contents between different embodiments can be used with respect to each other, for instance, the contents of Embodiment 3 can be applied to Embodiment 16, the contents of Embodiment 4 can be applied to Embodiment 16, the contents of Embodiments 5 to 15 can be applied to Embodiment 16, and the contents between Embodiments 5 to 15 can be used with respect to each other, and are not limited thereto.

[0282] Based on the same application concept as the method described above, the present application further provides a decoding device among embodiments said device applied to a decoding unit, said device comprising: a memory that stores video data through configuration; and a decoder that implements the decoding method of embodiments 1 to 16 described above, i.e., the processing process of the decoding unit, through configuration.

[0283] For example, among possible implementations, the decoder undergoes configuration:

[0284] Performing decoding on the first bit stream of the current image block to obtain the coefficient hyperparameter features of each stage sub-block of the current image block;

[0285] For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the stage sub-block, and decoding is performed on the second bit stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block;

[0286] Determining the reconstruction characteristics of the stage subblock based on the residual characteristics of the stage subblock and the average value characteristics of the stage subblock;

[0287] Based on the reconstruction characteristics of each stage sub-block, a reconstruction image block corresponding to the current image block is determined.

[0288] Based on the same application concept as the method described above, the embodiment of the present application further provides a coding device, said device applied to a coding unit, said device comprising: a memory that stores video data through configuration; an encoder, and an encoder that implements the coding method of the embodiments 1 to 16 described above, that is, the processing process of the coding unit, through configuration.

[0289] For example, among possible implementations, the encoder undergoes configuration:

[0290] Inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block;

[0291] Dividing the above feature block into features to be coded into a plurality of stage sub-blocks;

[0292] For each stage sub-block corresponding to the above current image block, obtaining the coefficient hyperparameter feature of the said stage sub-block, and also coding the coefficient hyperparameter feature of the said stage sub-block into the first bit stream of the above current image block;

[0293] Determining the residual features of the stage sub-block based on the features to be coded of the stage sub-block and the average value features of the stage sub-block;

[0294] It includes determining a probability distribution parameter based on the coefficient hyperparameter features of the stage sub-block and also coding the residual features of the stage sub-block based on the probability distribution parameter into the second bit stream of the current image block.

[0295] Based on the same application concept as the method described above, the decoding device (which may also be referred to as a video decoder) provided in the embodiment of the present application, in terms of hardware, may refer to the specific details of the schematic diagram of its hardware framework as illustrated in FIG. 13a. The decoding device comprises a processor (1301) and a machine-readable storage medium (1302), and a machine-executable instruction that can be executed by the processor (1301) is stored in the machine-readable storage medium (1302); and the processor (1301) executes the machine-executable instruction and is used to implement the decoding method of the embodiments 1 through 16 described above of the present application. For example, in one possible embodiment, when the processor (1301) executes the machine-executable instruction, the following steps are performed:

[0296] A step of performing decoding on the first bit stream of the current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block;

[0297] For each stage sub-block, a probability distribution parameter is determined based on the coefficient hyperparameter feature of the stage sub-block, and decoding is performed on the second bit stream of the current image block based on the probability distribution parameter to obtain the residual feature of the stage sub-block;

[0298] A step of determining the reconstruction features of the stage subblock based on the residual features of the stage subblock and the average value features of the stage subblock;

[0299] A step is performed to determine a reconstructed image block corresponding to the current image block based on the reconstruction characteristics of each stage sub-block.

[0300] Based on the same application concept as the method described above, a coding device (which may also be referred to as a video encoder) provided in an embodiment of the present application may, in terms of hardware, refer specifically to the schematic diagram of its hardware framework shown in FIG. 13b. The coding device comprises: a processor (1311) and a machine-readable storage medium (1312), wherein a machine-executable instruction that can be performed by the processor (1311) is stored in the machine-readable storage medium (1312); and the processor (1311) performs the machine-executable instruction and is used to implement the coding method of the embodiments 1 through 16 described above of the present application. For example, in one possible embodiment, when the processor (1311) performs the machine-executable instruction, the following steps are performed:

[0301] A step of inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block;

[0302] A step of dividing the above feature block into features to be coded into a plurality of stage sub-blocks;

[0303] For each stage sub-block corresponding to the current image block, a step of obtaining coefficient hyperparameter features of said stage sub-block and also coding the coefficient hyperparameter features of said stage sub-block into a first bit stream of said current image block;

[0304] A step of determining the residual features of the stage sub-block based on the features to be coded of the stage sub-block and the average value features of the stage sub-block;

[0305] The method includes the step of determining a probability distribution parameter based on the coefficient hyperparameter features of the stage sub-block and also coding the residual features of the stage sub-block based on the probability distribution parameter into the second bit stream of the current image block.

[0306] Based on the same application concept as the method described above, an embodiment of the present application provides an electronic device. The electronic device comprises: a processor and a machine-readable storage medium, wherein a machine-executable instruction that can be performed by the processor is stored in the machine-readable storage medium; and the processor performs the machine-executable instruction and is used to implement the decoding method or coding method of the embodiments 1 through 16 described above of the present application.

[0307] Based on the same application concept as the method described above, an embodiment of the present application further provides a machine-readable storage medium, and when several computer instructions are stored in the machine-readable storage medium and said computer instructions are executed by a processor, the method disclosed in the example described above of the present application, such as the decoding method or the coding method among each of the embodiments described above, can be implemented.

[0308] Based on the same application concept as the method described above, an embodiment of the present application further provides a computer application, and when said computer application is executed by a processor, the decoding method or coding method disclosed in the example described above of the present application may be implemented.

[0309] Based on the same application concept as the method described above, an embodiment of the present application further provides a decoding device that can be applied to a decoding unit, wherein the decoding device comprises: a decoding module for performing decoding on a first bit stream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; for each stage sub-block, determining a probability distribution parameter based on the coefficient hyperparameter features of the stage sub-block, and performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain residual features of the stage sub-block; and a determination module for determining reconstruction features of the stage sub-block based on the residual features of the stage sub-block and the average value features of the stage sub-block; and determining a reconstruction image block corresponding to the current image block based on the reconstruction features of each stage sub-block.

[0310] For example, the above decision module is further used to obtain, for a first stage sub-block, an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block, or, obtain a previously set default reference feature and obtain an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature; or, for an i-th stage sub-block, i is greater than 1 and obtain a reference feature of the i-th stage sub-block based on the reconstruction feature of the i-1 stage sub-blocks preceding the i-th stage sub-block; and obtain an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the reference feature, wherein the reference feature includes all reconstruction features of the preceding i-1 stage sub-blocks; or partial reconstruction features of the preceding i-1 stage sub-blocks; or reconstruction features of the i-1 stage sub-block.

[0311] For example, the average value prediction network comprises a first prediction network, a second prediction network, and a prediction fusion network, and the decision module obtains the average value feature of the stage sub-block through the average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the reference feature, specifically: obtaining a first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network; obtaining a second prediction feature corresponding to the reference feature through the second prediction network; performing feature concatenation on the first prediction feature and the second prediction feature to obtain a concatenated feature, inputting the concatenated feature into the prediction fusion network, and performing processing on the concatenated feature through the prediction fusion network, and is used to obtain the average value feature of the stage sub-block.

[0312] For example, when the decision module obtains a first prediction feature corresponding to the coefficient hyperparameter feature through a first prediction network, specifically: a feature augmentation operation and an upsampling operation are performed on the coefficient hyperparameter feature through the first prediction network to obtain the first prediction feature; the feature augmentation operation includes: a convolution operation, or a convolution operation and an activation operation; and the upsampling operation includes: a deconvolution operation, a cropping operation and an activation operation, or a deconvolution operation, a cropping operation, an activation operation and a convolution operation. When the decision module obtains a second prediction feature corresponding to the reference feature through a second prediction network, specifically: feature concatenation is performed according to the channel dimension for all reconstructed features among the reference features; and a convolution operation is performed on the features after concatenation to obtain the second prediction feature; Alternatively, feature addition is performed on all reconstructed features among the reference features, and a convolution operation is also performed on the features after addition to obtain the second predicted feature. Specifically, when the decision module performs processing on the features after concatenation through the prediction fusion network to obtain the average value feature of the stage sub-block, the convolution operation and at least one fusion operation are performed on the features after concatenation through the prediction fusion network to obtain the average value feature of the stage sub-block; wherein the fusion operation includes an activation operation and a convolution operation.

[0313] For example, the above-mentioned decision module is further used to perform feature augmentation on the reconstruction feature of the stage sub-block to obtain the reconstruction feature after augmentation; wherein the reconstruction feature after augmentation is used to determine the average value feature of the stage sub-block, and the reconstruction feature after augmentation is used to determine the reconstruction image block.

[0314] For example, when the decision module determines a reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block, specifically: performing feature aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features; inputting the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; or, performing feature aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features; performing partitioning on the aggregated features to obtain a plurality of partitioned features; inputting each partitioned feature into a synthetic transformation network to obtain a partitioned reconstructed image block; performing merging on the partitioned reconstructed image blocks corresponding to the plurality of partitioned features to obtain a reconstructed image block corresponding to the current image block; or, inputting the reconstructed features of each stage sub-block into a synthetic transformation network to obtain a partitioned reconstructed image block; It is used to obtain a reconstructed image block corresponding to the current image block by performing a merge on the partitioned reconstructed image blocks corresponding to all stage sub-blocks.

[0315] For example, when the above decision module performs feature aggregation on the reconstructed features of each stage sub-block to obtain the features after aggregation, specifically: by performing aggregation according to the phase of the reconstructed features of each stage sub-block to obtain the features after aggregation; or by performing aggregation according to the phase of the reconstructed features of each stage sub-block to obtain features after aggregation of multiple phases, and also by performing concatenation according to a channel on the features after aggregation of multiple phases to obtain the features after aggregation.

[0316] For example, when the above-described decision module performs partitioning on the feature after aggregation to obtain a plurality of partitioned features, specifically: determining a target size of the partitioned feature; dividing the feature after aggregation into a plurality of partitioned features by averaging them in order from top to bottom and left to right based on the target size, and the size of each partitioned feature is the target size; or, determining an actual partition size and an overlap size of the partitioned feature, wherein the actual partition size is a partition size with the overlapping portion of the partitioned feature removed, and the overlap size is the size of the overlapping portion between adjacent partitioned features; and dividing the feature after aggregation into a plurality of partitioned features based on the actual partition size and the overlap size, and the size of each partitioned feature is the target size.

[0317] For example, when the above-described decision module performs merging on a plurality of divided reconstructed image blocks corresponding to a plurality of divided features to obtain a reconstructed image block corresponding to the current image block, specifically: performs arrangement on a plurality of divided reconstructed image blocks corresponding to a plurality of divided features in order from top to bottom and from left to right, and sequentially concatenates the plurality of divided reconstructed image blocks after arrangement to obtain a reconstructed image block corresponding to the current image block; or, determines the actual partition size and overlap size of each divided reconstructed image block; performs arrangement on a plurality of divided reconstructed image blocks corresponding to a plurality of divided features in order from top to bottom and from left to right; and based on the actual partition size and the overlap size, sequentially concatenates the plurality of divided reconstructed image blocks after arrangement to obtain a reconstructed image block corresponding to the current image block; Here, the size of the non-overlapping portion of the divided reconstructed image block is the actual partition size of the divided reconstructed image block, and the size of the overlapping portion of the divided reconstructed image block adjacent to the divided reconstructed image block is the overlapping size of the divided reconstructed image block; for the overlapping portion of the left divided reconstructed image block and the right divided reconstructed image block, the value of the overlapping portion is the value of the left divided reconstructed image block, or the value of the overlapping portion is the average value of the two divided reconstructed image blocks; for the overlapping portion of the upper divided reconstructed image block and the lower divided reconstructed image block, the value of the overlapping portion is the value of the upper divided reconstructed image block, or the value of the overlapping portion is the average value of the two divided reconstructed image blocks.

[0318] For example, when the decision module inputs the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block, specifically: decoding is performed on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters of the current image block, and the bit rate control parameters and the aggregated features are input into a synthetic transformation network to be used to obtain a reconstructed image block corresponding to the current image block; or, when the decision module inputs each divided feature into a synthetic transformation network to obtain a divided reconstructed image block, specifically: decoding is performed on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters of each divided feature, and the divided feature and the bit rate control parameters of said divided feature are input into a synthetic transformation network to be used to obtain a divided reconstructed image block corresponding to said divided feature; Alternatively, when the above decision module inputs the reconstruction features of each stage sub-block into a synthetic transformation network to obtain a divided reconstructed image block, specifically: decoding is performed on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters of each stage sub-block, and the reconstruction features of the stage sub-block and the bit rate control parameters of the stage sub-block are input into a synthetic transformation network to obtain a divided reconstructed image block.

[0319] For example, if the decision module inputs bit rate control parameters and the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block, specifically: performing processing on the aggregated features through the synthetic transformation network to obtain a first feature; performing processing on the bit rate control parameters through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and also using the third feature to determine a reconstructed image block corresponding to the current image block; or, if the decision module inputs a segmented feature and a bit rate control parameter of said segmented feature into a synthetic transformation network to obtain a segmented reconstructed image block, specifically: performing processing on said segmented feature through the synthetic transformation network to obtain a first feature; performing processing on said segmented feature's bit rate control parameter through the synthetic transformation network to obtain a second feature; It is used to generate a third feature based on a first feature and a second feature, and also to determine a divided reconstructed image block corresponding to the divided feature based on the third feature; or, specifically when the determination module inputs the reconstructed feature of the stage sub-block and the bit rate control parameter of the stage sub-block into a synthetic transformation network to obtain a divided reconstructed image block: processing is performed on the reconstructed feature of the stage sub-block through the synthetic transformation network to obtain a first feature; processing is performed on the bit rate control parameter of the stage sub-block through the synthetic transformation network to obtain a second feature; and a third feature is generated based on the first feature and the second feature, and is used to determine a divided reconstructed image block corresponding to the stage sub-block based on the third feature.

[0320] Based on the same application concept as the method described above, an embodiment of the present application further provides a coding device, said device applied to a coding unit, said device comprising: an acquisition module for inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to said current image block; and dividing said feature block into coding features of a plurality of stage sub-blocks; a coding module for, for each stage sub-block corresponding to said current image block, acquiring a coefficient hyperparameter feature of said stage sub-block and also coding the coefficient hyperparameter feature of said stage sub-block into a first bit stream of said current image block; and a determination module for determining a residual feature of said stage sub-block based on the coding feature of said stage sub-block and the average value feature of said stage sub-block; said coding module further used to determine a probability distribution parameter based on the coefficient hyperparameter feature of said stage sub-block and also to code the residual feature of said stage sub-block into a second bit stream of said current image block based on said probability distribution parameter.

[0321] For example, the decision module obtains an average value feature of a stage sub-block through an average value prediction network based on coefficient hyperparameter features of the stage sub-block for the first stage sub-block, or obtains a default reference feature that has already been set, and obtains an average value feature of the stage sub-block through an average value prediction network based on coefficient hyperparameter features of the stage sub-block and the default reference feature; or, for the i-th stage sub-block, i is greater than 1, and obtains a reference feature of the i-th stage sub-block based on reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block; and is used to obtain an average value feature of the stage sub-block through an average value prediction network based on coefficient hyperparameter features of the stage sub-block and the reference feature, wherein the reference feature includes all reconstruction features of the preceding i-1 stage sub-blocks; or partial reconstruction features of the preceding i-1 stage sub-blocks; or reconstruction features of the i-1 stage sub-block.

[0322] A person skilled in the art should understand that one or more examples of this application may be provided as a method, system, or computer program product. Accordingly, one or more examples of this application may take the form of a complete hardware example, a complete software example, or an example combining software and hardware. One or more examples of this application may take the form of a computer program product having one or more computer-usable storage media (which may include, but are not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The foregoing is merely an example of this application and is not intended to limit this application.

[0323] To those skilled in the art, the present invention may be subject to various changes and modifications. Any modification, equivalent substitution, improvement, etc. within the spirit and principles of the present invention will be included within the scope of protection of the present invention.

Claims

Claim 1 A decoding method comprising: a step of performing decoding on a first bit stream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; a step of determining a probability distribution parameter for each stage sub-block based on the coefficient hyperparameter features of the stage sub-block, and performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain residual features of the stage sub-block; a step of determining a reconstructed feature of the stage sub-block based on the residual feature of the stage sub-block and the mean value feature of the stage sub-block; and a step of determining a reconstructed image block corresponding to the current image block based on the reconstructed feature of each stage sub-block. A decoding method characterized by further including the step of, before determining the reconstruction feature of the stage subblock based on the residual feature of the stage subblock and the average value feature of the stage subblock, the method: for the first stage subblock, obtaining the average value feature of the stage subblock through an average value prediction network based on the coefficient hyperparameter feature of the stage subblock, or obtaining a previously set default reference feature, and obtaining the average value feature of the stage subblock through the average value prediction network based on the coefficient hyperparameter feature of the stage subblock and the default reference feature. Claim 2 A decoding method according to claim 1, wherein, before determining a reconstruction feature of a stage sub-block based on a residual feature of the stage sub-block and an average value feature of the stage sub-block, the method further comprises: a step of obtaining a reference feature of the i-th stage sub-block based on the reconstruction features of the i-1 stage sub-blocks preceding the i-th stage sub-block, wherein i is greater than 1 for the i-th stage sub-block; a step of obtaining an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the i-th stage sub-block and the reference feature, wherein the reference feature includes all reconstruction features of the preceding i-1 stage sub-blocks; or partial reconstruction features of the preceding i-1 stage sub-blocks; or reconstruction features of the i-1 stage sub-block. Claim 3 A decoding method according to claim 2, wherein the average value prediction network comprises a first prediction network, a second prediction network, and a prediction fusion network, and the step of obtaining an average value feature of a stage sub-block through the average value prediction network based on a coefficient hyperparameter feature of the i-th stage sub-block and a reference feature comprises: a step of obtaining a first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network; a step of obtaining a second prediction feature corresponding to the reference feature through the second prediction network; and a step of performing feature concatenation on the first prediction feature and the second prediction feature to obtain a concatenated feature, inputting the concatenated feature into the prediction fusion network, and performing processing on the concatenated feature through the prediction fusion network to obtain an average value feature of the stage sub-block. Claim 4 A decoding method according to claim 3, wherein the step of obtaining a first prediction feature corresponding to the coefficient hyperparameter feature through the first prediction network comprises: a step of obtaining the first prediction feature by performing a feature enhancement operation and an upsampling operation on the coefficient hyperparameter feature through the first prediction network; wherein the feature enhancement operation comprises: a convolution operation, or a convolution operation and an activation operation; and wherein the upsampling operation comprises: a deconvolution operation, a cropping operation and an activation operation, or a deconvolution operation, a cropping operation, an activation operation and a convolution operation. Claim 5 A decoding method according to claim 4, characterized in that the above-mentioned activation operation is used as a modified linear unit function Relu operation. Claim 6 A decoding method according to claim 3, wherein the step of obtaining a second predicted feature corresponding to the reference feature through the second predicted network comprises: a step of performing feature concatenation according to the channel dimension for all reconstructed features among the reference features; and a step of performing a convolution operation on the features after concatenation to obtain the second predicted feature; or a step of performing feature addition for all reconstructed features among the reference features and also performing a convolution operation on the features after addition to obtain the second predicted feature. Claim 7 A decoding method according to claim 3, wherein the step of obtaining an average value feature of a stage sub-block by performing processing on the features after the connection through the predictive fusion network comprises: a step of obtaining an average value feature of a stage sub-block by performing a convolution operation and at least one fusion operation on the features after the connection through the predictive fusion network; wherein the fusion operation includes an activation operation and a convolution operation. Claim 8 A decoding method according to claim 7, characterized in that the above-mentioned activation operation is used as a modified linear unit function Relu operation. Claim 9 A decoding method according to claim 1, wherein, after determining a reconstruction feature of a stage sub-block based on a residual feature of the stage sub-block and an average value feature of the stage sub-block, the method comprises the step of: performing feature augmentation on the reconstruction feature of the stage sub-block to obtain a reconstruction feature after augmentation; wherein the reconstruction feature after augmentation is used to determine an average value feature of the stage sub-block and the reconstruction feature after augmentation is used to determine a reconstruction image block. Claim 10 In claim 1, the step of determining a reconstructed image block corresponding to the current image block based on the reconstructed features of each stage sub-block comprises: performing feature aggregation on the reconstructed features of each stage sub-block to obtain a feature after aggregation; inputting the feature after aggregation into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; or, performing feature aggregation on the reconstructed features of each stage sub-block to obtain a feature after aggregation; performing partitioning on the feature after aggregation to obtain a plurality of partitioned features; inputting each partitioned feature into the synthetic transformation network to obtain a partitioned reconstructed image block; and performing merging on the partitioned reconstructed image blocks corresponding to the plurality of partitioned features to obtain a reconstructed image block corresponding to the current image block; or, inputting the reconstructed features of each stage sub-block into the synthetic transformation network to obtain a partitioned reconstructed image block; A decoding method characterized by including the step of performing a merge on divided reconstructed image blocks corresponding to all stage sub-blocks to obtain a reconstructed image block corresponding to the current image block. Claim 11 A decoding method according to claim 10, wherein the step of performing feature aggregation on the reconstructed features of each stage sub-block to obtain the aggregated features comprises: a step of performing aggregation according to the phase on the reconstructed features of each stage sub-block to obtain the aggregated features; or a step of performing aggregation according to the phase of the reconstructed features of each stage sub-block to obtain a plurality of phase-aggregated features, and also performing concatenation according to the channel on the plurality of phase-aggregated features to obtain the aggregated features. Claim 12 A decoding method according to claim 10, wherein the step of performing division on the feature after aggregation to obtain a plurality of divided features comprises: determining a target size of the divided features; dividing the feature after aggregation into a plurality of divided features in an order from top to bottom and from left to right based on the target size, wherein the size of each divided feature is the target size; or, determining an actual partition size and an overlap size of the divided features, wherein the actual partition size is a partition size with the overlapping portion of the divided features removed, and the overlap size is the size of the overlapping portion between adjacent divided features; and dividing the feature after aggregation into a plurality of divided features based on the actual partition size and the overlap size, wherein the size of each divided feature is the target size. Claim 13 In claim 12, the step of performing a merger on the divided reconstructed image blocks corresponding to the plurality of divided features to obtain a reconstructed image block corresponding to the current image block comprises: performing an arrangement on the plurality of divided reconstructed image blocks corresponding to the plurality of divided features in order from top to bottom and from left to right, and concatenating the plurality of divided reconstructed image blocks in order after arrangement to obtain a reconstructed image block corresponding to the current image block; or, determining the actual partition size and overlap size of each divided reconstructed image block; and performing an arrangement on the plurality of divided reconstructed image blocks corresponding to the plurality of divided features in order from top to bottom and from left to right; Based on the actual partition size and the overlap size, a step of sequentially concatenating a plurality of divided reconstructed image blocks after arrangement to obtain a reconstructed image block corresponding to the current image block - wherein the size of the non-overlapping portion of the divided reconstructed image block is the actual partition size of the said divided reconstructed image block, and the size of the overlap portion of the divided reconstructed image block adjacent to the divided reconstructed image block is the overlap size of the said divided reconstructed image block -; comprising; and for the overlap portion of the left divided reconstructed image block and the right divided reconstructed image block, the value of said overlap portion is the value of the left divided reconstructed image block, or, the value of said overlap portion is the average value of said left divided reconstructed image block and said right divided reconstructed image block; A decoding method characterized by, for an overlapping portion of an upper divided reconstructed image block and a lower divided reconstructed image block, the value of said overlapping portion is the value of said upper divided reconstructed image block, or said overlapping portion is the average value of said upper divided reconstructed image block and said lower divided reconstructed image block. Claim 14 In claim 10, the step of inputting the aggregated features into a synthetic transformation network to obtain a reconstructed image block corresponding to the current image block comprises: performing decoding on an auxiliary bit stream corresponding to the current image block to obtain a bit rate control parameter of the current image block, and inputting the bit rate control parameter and the aggregated features into the synthetic transformation network to obtain a reconstructed image block corresponding to the current image block; or, the step of inputting each divided feature into the synthetic transformation network to obtain a divided reconstructed image block comprises: performing decoding on an auxiliary bit stream corresponding to the current image block to obtain a bit rate control parameter of each divided feature, and inputting the divided feature and the bit rate control parameter of said divided feature into the synthetic transformation network to obtain a divided reconstructed image block corresponding to said divided feature; Alternatively, a decoding method characterized by comprising the step of inputting the reconstruction features of each stage sub-block into the synthetic transformation network to obtain a divided reconstructed image block, wherein the step comprises: performing decoding on an auxiliary bit stream corresponding to the current image block to obtain a bit rate control parameter of each stage sub-block, and inputting the reconstruction features of the stage sub-block and the bit rate control parameter of the stage sub-block into the synthetic transformation network to obtain the divided reconstructed image block. Claim 15 In claim 14, the step of inputting the bit rate control parameter and the aggregated feature into the synthetic transformation network to obtain a reconstructed image block corresponding to the current image block comprises: performing processing on the aggregated feature through the synthetic transformation network to obtain a first feature; performing processing on the bit rate control parameter through the synthetic transformation network to obtain a second feature; generating a third feature based on the first feature and the second feature, and also determining a reconstructed image block corresponding to the current image block based on the third feature; or, the step of inputting the divided feature and the bit rate control parameter of said divided feature into the synthetic transformation network to obtain a divided reconstructed image block corresponding to said divided feature comprises: performing processing on said divided feature through the synthetic transformation network to obtain a first feature; and performing processing on the bit rate control parameter of said divided feature through the synthetic transformation network to obtain a second feature; The method comprises the step of generating a third feature based on the first feature and the second feature, and also determining a divided reconstructed image block corresponding to the divided feature based on the third feature; or, the step of obtaining the divided reconstructed image block by inputting the reconstructed feature of the stage sub-block and the bit rate control parameter of the stage sub-block into the synthetic transformation network, wherein the method comprises: performing processing on the reconstructed feature of the stage sub-block through the synthetic transformation network to obtain a first feature; and performing processing on the bit rate control parameter of the stage sub-block through the synthetic transformation network to obtain a second feature;A decoding method characterized by including the step of generating a third feature based on the first feature and the second feature, and determining a divided reconstructed image block corresponding to the stage sub-block based on the third feature. Claim 16 A coding method comprising: inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block; splitting the feature block into features to be coded for a plurality of stage sub-blocks; for each stage sub-block corresponding to the current image block, obtaining a coefficient hyperparameter feature of the stage sub-block and coding the coefficient hyperparameter feature of the stage sub-block into a first bit stream of the current image block; determining the difference between the feature to be coded of the stage sub-block and the mean value feature of the stage sub-block as the residual feature of the stage sub-block; determining a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block and coding the residual feature of the stage sub-block into a second bit stream of the current image block based on the probability distribution parameter; A coding method characterized by further comprising the step of, before determining the difference between the feature to be coded of the stage sub-block and the average value feature of the stage sub-block as the residual feature of the stage sub-block: for the first stage sub-block, obtaining the average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block, or obtaining a previously set default reference feature, and obtaining the average value feature of the stage sub-block through the average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature. Claim 17 A decoding method comprising: a step of performing decoding on a first bit stream of a current image block to obtain coefficient hyperparameter features of the current image block; a step of determining a probability distribution parameter based on the coefficient hyperparameter features, performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain residual features of the current image block, and determining reconstruction features of the current image block based on the residual features; and a step of performing decoding on an auxiliary bit stream corresponding to the current image block to obtain bit rate control parameters corresponding to the current image block, and inputting the reconstruction features and the bit rate control parameters into a synthetic transformation network to obtain a reconstruction image block corresponding to the current image block. Claim 18 A decoding method according to claim 17, wherein the step of inputting the reconstruction feature and the bit rate control parameter into a synthetic transformation network to obtain a reconstruction image block corresponding to the current image block comprises: a step of performing processing on the reconstruction feature through the synthetic transformation network to obtain a first feature; a step of performing processing on the bit rate control parameter through the synthetic transformation network to obtain a second feature; a step of generating a third feature based on the first feature and the second feature; and a step of determining a reconstruction image block corresponding to the current image block based on the third feature. Claim 19 A decoding device for performing decoding on a first bit stream of a current image block to obtain coefficient hyperparameter features of each stage sub-block of the current image block; a decoding module for determining a probability distribution parameter for each stage sub-block based on the coefficient hyperparameter features of the stage sub-block and performing decoding on a second bit stream of the current image block based on the probability distribution parameter to obtain residual features of the stage sub-block; and determining reconstruction features of the stage sub-block based on the residual features of the stage sub-block and the average value features of the stage sub-block; A decoding device comprising: a determination module for determining a reconstructed image block corresponding to the current image block based on the reconstruction features of each stage sub-block; wherein, for a first stage sub-block, the determination module is further used to obtain an average value feature of the stage sub-block through an average value prediction network based on the coefficient hyperparameter feature of the stage sub-block, or to obtain a pre-set default reference feature, and to obtain an average value feature of the stage sub-block through the average value prediction network based on the coefficient hyperparameter feature of the stage sub-block and the default reference feature. Claim 20 A coding device comprising: inputting a current image block into an analysis and transformation network to obtain a feature block corresponding to the current image block; an acquisition module for dividing the feature block into features to be coded for a plurality of stage sub-blocks; a coding module for, for each stage sub-block corresponding to the current image block, acquiring a coefficient hyperparameter feature of the stage sub-block and coding the coefficient hyperparameter feature of the stage sub-block into a first bit stream of the current image block; and a determination module for determining the difference value between the feature to be coded for the stage sub-block and the mean value feature of the stage sub-block as a residual feature of the stage sub-block; wherein the coding module is further used to determine a probability distribution parameter based on the coefficient hyperparameter feature of the stage sub-block and to code the residual feature of the stage sub-block into a second bit stream of the current image block based on the probability distribution parameter; A coding device characterized in that the above-described decision module is further used to obtain an average value feature of a stage sub-block through an average value prediction network based on coefficient hyperparameter features of the stage sub-block for a first stage sub-block, or to obtain a pre-set default reference feature, and to obtain an average value feature of the stage sub-block through the average value prediction network based on coefficient hyperparameter features of the stage sub-block and the default reference feature. Claim 21 A decoding device comprising a processor and a machine-readable storage medium, wherein a machine-executable instruction that can be executed by the processor is stored in the machine-readable storage medium; and wherein the processor executes the machine-executable instruction and is used to implement a method according to any one of claims 1 to 15. Claim 22 A coding device comprising a processor and a machine-readable storage medium, wherein a machine-executable instruction that can be executed by the processor is stored in the machine-readable storage medium; and wherein the processor executes the machine-executable instruction and is used to implement the method according to claim 16. Claim 23 A machine-readable storage medium, wherein multiple computer instructions are stored in the machine-readable storage medium, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 15 is implemented, or when the computer instructions are executed by a processor, the method according to claim 16 is implemented. Claim 24 A computer application stored on a storage medium, characterized in that, when the computer application is executed by a processor, it implements a method according to any one of claims 1 to 15, or, when the computer application is executed by a processor, it implements a method according to claim 16. Claim 25 delete Claim 26 delete Claim 27 delete

Images