A decoding, encoding method, apparatus and device thereof

Abstract

The application provides a decoding method, an encoding method, a decoding device, an encoding device, and a decoding apparatus and an encoding apparatus. The decoding method comprises: decoding a first code stream corresponding to a current image block to obtain a hyperparameter quantization feature corresponding to the current image block; determining an original probability distribution parameter corresponding to the current image block by using a probability hyperparameter decoding network based on the hyperparameter quantization feature; obtaining an influence coefficient; scaling the original probability distribution parameter based on the influence coefficient to obtain a scaled probability distribution parameter; decoding a second code stream corresponding to the current image block based on the scaled probability distribution parameter to obtain a scaled residual feature; performing inverse scaling on the scaled residual feature based on the influence coefficient to obtain a quantized original residual feature; and determining a reconstructed image block corresponding to the current image block based on the quantized original residual feature and a mean value prediction feature. In this way, the encoding performance and the decoding performance can be improved. Through the above technical solution, the performance loss caused by the variable code rate is reduced.

Description

Technical Field

[0001] This application relates to the field of encoding and decoding technology, and in particular to a decoding and encoding method, apparatus and device thereof. Background Technology

[0002] To save space, images are compressed and encoded before transmission. With the rapid development of deep learning, it has gradually been applied to the field of image encoding and decoding, where several neural network models can be used in the encoding and decoding process.

[0003] A complete image encoding process can include transformation, prediction, quantization, entropy coding, and filtering. Before inputting the residual features between the image features and the image's mean prediction features into the quantization process, the residual features often need to be scaled to control the quantization error generated during quantization.

[0004] However, in scenarios involving variable bitrate compression of images, current scaling methods result in significant performance losses associated with variable bitrate compression. Summary of the Invention

[0005] In view of this, this application provides a decoding and encoding method, apparatus and device, to reduce the performance loss caused by variable bit rate compression.

[0006] This application provides a decoding method, including:

[0007] The first bitstream corresponding to the current image block is decoded to obtain the hyperparameter quantization feature corresponding to the current image block; based on the hyperparameter quantization feature, the original probability distribution parameters corresponding to the current image block are determined through a probabilistic hyperparameter decoding network; and the mean prediction feature of the current image block is obtained.

[0008] Obtain the influence coefficient, wherein the influence coefficient is generated based on the content-related features of the current image patch;

[0009] The original probability distribution parameters are scaled based on the influence coefficient to obtain scaled probability distribution parameters;

[0010] Based on the scaling probability distribution parameters, the second bitstream corresponding to the current image block is decoded to obtain the scaling residual features; based on the influence coefficient, the scaling residual features are inversely scaled to obtain the quantized original residual features.

[0011] Based on the quantized original residual features and the mean prediction features, the reconstructed image block corresponding to the current image block is determined by a synthetic transformation network.

[0012] This application provides an encoding method, including:

[0013] Based on the image features of the current image block, the hyperparameter quantization features corresponding to the current image block are determined through a hyperparameter coding network; based on the hyperparameter quantization features, a first bitstream is encoded.

[0014] An influence coefficient is generated based on the content-related features of the current image block. Based on the influence coefficient, the unquantized original residual features corresponding to the current image block are scaled and then quantized to obtain scaled residual features. Based on the scaled residual features, a second bitstream is encoded.

[0015] Based on the influence coefficient, the scaled residual features are inversely scaled to obtain the quantized original residual features;

[0016] The first bitstream and the second bitstream are sent to the decoding end.

[0017] This application provides a decoding device, including:

[0018] The determination module decodes the first bitstream corresponding to the current image block to obtain the hyperparameter quantization feature corresponding to the current image block; based on the hyperparameter quantization feature, it determines the original probability distribution parameter corresponding to the current image block through a probabilistic hyperparameter decoding network; it obtains the mean prediction feature of the current image block; it obtains the influence coefficient, wherein the influence coefficient is generated based on the content-related features of the current image block; and it scales the original probability distribution parameter based on the influence coefficient to obtain the scaled probability distribution parameter.

[0019] The processing module decodes the second bitstream corresponding to the current image block based on the scaling probability distribution parameters to obtain scaling residual features; and performs inverse scaling on the scaling residual features based on the influence coefficient to obtain quantized original residual features.

[0020] The reconstruction module obtains the mean prediction features of the current image block, and determines the reconstructed image block corresponding to the current image block through a synthetic transformation network based on the quantized original residual features and the mean prediction features.

[0021] This application provides an encoding device, comprising:

[0022] The first processing module determines the hyperparameter quantization features corresponding to the current image block through a hyperparameter coding network based on the image features of the current image block; and encodes the first bitstream based on the hyperparameter quantization features.

[0023] The second processing module generates an influence coefficient based on the content-related features of the current image block; based on the influence coefficient, it scales and quantizes the unquantized original residual features corresponding to the current image block to obtain scaled residual features; and based on the scaled residual features, it encodes a second bitstream.

[0024] The third processing module performs inverse scaling on the scaled residual features based on the influence coefficient to obtain the quantized original residual features;

[0025] The sending module sends the first bitstream and the second bitstream to the decoding end.

[0026] This application provides a decoding end device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0027] The processor is used to execute machine-executable instructions to implement the above-described decoding method.

[0028] This application provides an encoding terminal device, including: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0029] The processor is used to execute machine-executable instructions to implement the above-described encoding method.

[0030] This application provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is configured to execute the machine-executable instructions to implement the above-described decoding method; or, the processor is configured to execute the machine-executable instructions to implement the above-described encoding method.

[0031] This application provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, implement the above-described decoding method; or, implement the above-described encoding method.

[0032] In the above technical solution, under the variable bitrate encoding scenario, on the one hand, it is necessary to encode the image block with different compression bitrates, and on the other hand, the influence coefficient (used for scaling and descaling) used in the encoding and decoding process of the image block can determine the compression bitrate used for encoding the image block. Therefore, it is necessary to determine an adaptive influence coefficient for each image block.

[0033] Considering that the compression bitrate suitable for a particular image block is closely related to the image content of that block, an influence coefficient (used for scaling and descaling) can be determined based on the content-related characteristics of that image block, so that the bitrate determined by this influence coefficient is suitable for that image block. Specifically:

[0034] The encoder needs to scale the unquantized raw residual features between the image features and the mean prediction features based on the current image patch, then quantize them (to obtain scaled residual features), and then inverse scale the scaled residual features (to obtain quantized raw residual features). Therefore, the scaling and inverse scaling processes performed by the encoder will use the aforementioned influence coefficients.

[0035] The decoding end needs to scale the original probability distribution parameters obtained from the first bitstream and descale the scaled residual features obtained from the second bitstream. Therefore, the scaling and descaled processing performed by the decoding end will have the aforementioned influence coefficients.

[0036] Through the above technical solution, the influence coefficient used in the encoding and decoding process of each image block can take into account the importance of each channel of the image block (the importance of the same channel in different image blocks may be different in scaling and descaling processes). This is beneficial to control the quantization error according to the content characteristics of the image block itself, which can make the compression bitrate of the image block better adapt to the image block, thereby reducing the performance loss caused by compressing the image block based on the compression bitrate. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a three-dimensional matrix in one embodiment of this application;

[0038] Figure 2 This is a schematic diagram of the processing procedure at the decoding end in one embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the processing procedure at the encoding end in one embodiment of this application;

[0040] Figure 4 This is a flowchart illustrating a decoding method in one embodiment of this application;

[0041] Figure 5 This is a flowchart illustrating an encoding method in one embodiment of this application;

[0042] Figure 6 This is a flowchart illustrating the process of determining the influence coefficients acting on the channel domain dimension using a neural network model in one embodiment of this application.

[0043] Figure 7 This is a flowchart illustrating the process of determining the influence coefficients acting on the spatial dimension using a neural network model in one embodiment of this application.

[0044] Figure 8 This is a flowchart illustrating the process of determining the influence coefficients acting on the binding domain dimension using a neural network model in one embodiment of this application.

[0045] Figure 9 This is a flowchart illustrating the process of determining the influence coefficients acting on the binding domain dimension using a neural network model in another embodiment of this application.

[0046] Figure 10 This is a schematic diagram of the generation of influence coefficients at the encoding end in one embodiment of this application.

[0047] Figure 11A This is a schematic diagram of the generation of influence coefficients at the decoding end in one embodiment of this application.

[0048] Figure 11B This is a schematic diagram illustrating how the decoding end obtains the influence coefficient from the third bitstream sent by the encoding end in one embodiment of this application.

[0049] Figure 12A This is a hardware structure diagram of the decoding end device in one embodiment of this application;

[0050] Figure 12B This is a hardware structure diagram of the encoding end device in one embodiment of this application. Detailed Implementation

[0051] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” used in the embodiments and claims of this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any or all possible combinations including one or more of the associated listed items. It should be understood that although the terms first, second, third, etc., may be used to describe various information in the embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of this application, and similarly, second information may also be referred to as first information, depending on the context. Furthermore, the word “if” as used can be interpreted as “when,” “in response to a determination,” or “when…”.

[0052] This application proposes a decoding method and an encoding method, which may involve the following concepts:

[0053] Neural Networks (NNs): Neural networks are artificial neural networks, a computational model composed of numerous interconnected nodes (or neurons). In a neural network, neurons can represent different objects, such as features, letters, concepts, or meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units; the representation and processing of information are reflected in the connections between processing units. Neural networks are a non-programmed, brain-like information processing method. The essence of a neural network is to obtain a parallel and distributed information processing function through the transformations and dynamic behavior of the network, mimicking the information processing function of the human brain's nervous system to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and fully connected networks.

[0054] Convolutional Neural Networks (CNNs) are a type of feedforward neural network and one of the most representative network structures in deep learning. The artificial neurons in a CNN can respond to surrounding units within a certain coverage area, exhibiting excellent performance in large-scale image processing. The basic structure of a CNN consists of two layers: a feature extraction layer (also called a convolutional layer), where the input of each neuron is connected to the local receptive field of the previous layer, extracting features from that local area. Once these local features are extracted, their positional relationship with other features is determined. The second layer is a feature mapping layer (also called an activation layer). Each computational layer of the neural network consists of multiple feature maps, each a plane where all neurons have equal weights. Feature mapping structures can use functions such as the Sigmoid function, ReLU function, LReLU function, PReLU function, and GDN function as activation functions for the convolutional network. Furthermore, because neurons on a single feature map share weights, the number of free parameters in the network can be reduced.

[0055] For example, one advantage of convolutional neural networks (CNNs) over image processing algorithms is that they avoid complex preprocessing steps (such as extracting artificial features) and can directly input the original image for end-to-end learning. Another advantage of CNNs over ordinary neural networks is that ordinary neural networks use fully connected layers, meaning all neurons from the input layer to the hidden layer are connected. This results in a huge number of parameters, making network training time-consuming or even difficult. CNNs, however, avoid this difficulty through local connectivity and weight sharing.

[0056] Compression bitrate: also known as compression ratio, refers to the encoding length required to encode a unit pixel in image compression, and is generally measured in bpp (Bits Per Pixel).

[0057] Variable bitrate: Variable compression bitrate refers to the compression bitrate that is determined in real time based on the image that needs to be compressed, so as to balance the amount of data after image compression with the loss of too much image information.

[0058] Channel domain: A one-dimensional space representing channels in a three-dimensional image.

[0059] Spatial domain: The two-dimensional space representing width and height in a three-dimensional image.

[0060] Combined domain: The three-dimensional space representing all data in a three-dimensional image, which is a combination of the channel domain and the spatial domain.

[0061] 3D Tensor: The data shape of features or parameters involved in image compression is usually a 3D tensor, which can be understood as a C*W*H 3D matrix. See [link to documentation]. Figure 1 The diagram shows a schematic of a three-dimensional matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the matrix height, and W represents the matrix width. It should be noted that C corresponds to the channel domain dimension of the image, W*H corresponds to the spatial domain dimension, and C*W*H corresponds to the binding domain dimension. If the channel domain dimension of the feature or parameter is not considered, then the data shape of the feature or parameter can be transformed from C*W*H to W*H; if the spatial domain dimension of the feature or parameter is not considered, then the data shape of the feature or parameter can be transformed from C*W*H to C.

[0062] Image block: The basic object for image compression in this disclosure. An image block can be all or part of an image.

[0063] Quantization is the process of mapping a set of continuous numerical values ​​to a set of discrete numerical values, typically using rounding principles. In image coding, it is usually necessary to quantize the residual features between the image features and the mean prediction features.

[0064] Scale and inv-scale: Scale refers to scaling the values ​​of features or parameters involved in image compression, while inv-scale is the inverse operation of scale. It should be noted that before inputting the residual features between the image features and the image's mean prediction features into the quantization process, the residual features often need to be scaled to control quantization errors.

[0065] Taking the unquantized raw residual feature (denoted as Resi) as an example, the formula for scaling Resi is: r = ft * k * Resi, where ft is the scaling factor, k is the influence factor, and r is the scaled unquantized raw residual feature. After quantizing r, the scaled residual feature referred to in this disclosure is obtained (denoted as r_hat), i.e., r_hat = Q(r).

[0066] The formula for inverse scaling of r_hat is: Resi_hat=r_hat / (ft*k), where Resi_hat is the quantized original residual feature.

[0067] Taking the probability distribution parameter (denoted as p) corresponding to an image patch as an example, the formula for scaling p is: p_hat=ft*k*p, where ft is the scaling factor, k is the influence factor, and p_hat is the scaled p.

[0068] In some implementations, the influence coefficient k is a one-dimensional vector shape, and each coefficient value corresponds one-to-one with each channel of the feature or parameter (such as Resi mentioned above) involved in the image compression process. Values ​​within the same channel are scaled or descaled using the coefficient value corresponding to that channel. The value of the influence coefficient k is usually fixed in the prior art (learned during neural network model training and fixed during model application).

[0069] It's easy to understand that, on the one hand, the larger ft*k is, the smaller the quantization error generated during the quantization process; on the other hand, the larger ft*k is, the larger the amount of Resi data, which means a higher compression bitrate. Therefore, in order to control the quantization error and reduce the performance loss caused by compression (this requires balancing the amount of data after image compression with minimal loss of image information), a suitable bitrate needs to be determined for the current image block to be compressed. In some implementations, the bitrate is determined by adjusting the size of ft (k is fixed, so only ft can be adjusted).

[0070] In variable bitrate coding scenarios, different compression bitrates need to be used to encode image blocks. Although different ft can be set for different image blocks to determine different compression bitrates, since the existing k is fixed, the same channel of different image blocks has the same importance in scaling and descaling. This is not conducive to controlling the quantization error according to the content characteristics of the image block itself. Although the compression bitrate of the resulting image block is also variable, it is still not conducive to reducing the performance loss caused by the compression of each image block.

[0071] Based on this, this disclosure aims to provide a technical solution for deriving scaling coefficients for different scopes based on image block content features. In this technical solution, under variable bitrate encoding scenarios, on the one hand, it is necessary to encode the image block using a compression bitrate adapted to the image block; on the other hand, the influence coefficients used in the encoding and decoding process of the image block (used for scaling and descaling) can determine the compression bitrate used for encoding the image block. Therefore, it is necessary to determine an adaptive influence coefficient for each image block.

[0072] Considering that the compression bitrate adapted to a certain image block is closely related to the image content of that image block, the influence coefficient (used for scaling and descaling) used in the encoding and decoding process of that image block can be determined based on the content-related characteristics of that image block, so that the bitrate determined by the influence coefficient is adapted to that image block.

[0073] The encoder needs to scale the unquantized raw residual features between the image features and the mean prediction features based on the current image patch, then quantize them (to obtain scaled residual features), and then inverse scale the scaled residual features (to obtain quantized raw residual features). Therefore, the scaling and inverse scaling processes performed by the encoder will use the aforementioned influence coefficients.

[0074] The decoding end needs to scale the original probability distribution parameters obtained from the first bitstream and descale the scaled residual features obtained from the second bitstream. Therefore, the scaling and descaled processing performed by the decoding end will have the aforementioned influence coefficients.

[0075] Through the above technical solution, the influence coefficient used in the encoding and decoding process of each image block can take into account the importance of each channel of the image block (the importance of the same channel in different image blocks may be different in scaling and descaling processes), which can make the compression bitrate of the image block better adapt to the image block, thereby reducing the performance loss caused by compressing the image block based on the compression bitrate.

[0076] Furthermore, the influence coefficient determined by the above technical solution can also take into account the importance of each bit of the spatial domain and the importance of the binding domain of the image block. Therefore, by using this influence coefficient, the spatial domain dimension and the binding domain dimension of the image block can also be scaled and descaled, which can also make the compression bitrate of the image block better adapt to the image block, thereby reducing the performance loss caused by compressing the image block based on the compression bitrate.

[0077] The decoding and encoding methods in the embodiments of this application will be described in detail below with reference to several specific embodiments.

[0078] Example 1: For the processing procedure at the decoding end, please refer to... Figure 2 As shown, of course, Figure 2 This is just one example of the processing procedure at the decoding end, and no restrictions are imposed on the processing procedure at the decoding end.

[0079] An image can be divided into one image block or multiple image blocks. If the image is divided into one image block, then the current image block x can also be an image. That is, the encoding and decoding process for image blocks can also be directly applied to the image.

[0080] After receiving the first bitstream (i.e., Bitstream#1) corresponding to the current image block from the encoding end, the decoding end can decode Bitstream#1. Figure 2 In this context, AD represents decoding (e.g., entropy decoding), resulting in the hyperparameter quantization feature z_hat. The source of this hyperparameter quantization feature z_hat is:

[0081] After obtaining the current image patch x (which can be the original image patch x, i.e., the input image patch), the encoder can perform analysis and transformation on the current image patch x through an analysis and transformation network (a type of neural network) to obtain the corresponding image features y. The feature transformation of the current image patch x through the analysis and transformation network means transforming the current image patch x into image features y in the latent domain, thus facilitating all subsequent processes to operate in the latent domain. After obtaining the image features y, the encoder performs a coefficient hyperparameter feature transformation on the image features y to obtain coefficient hyperparameter features z. For example, the image features y can be input into a hyperparameter coding network (a type of neural network), which performs a coefficient hyperparameter feature transformation on the image features y to obtain coefficient hyperparameter features z. The hyperparameter coding network can be a trained neural network, and the training process of this hyperparameter coding network is not restricted, as long as it can perform coefficient hyperparameter feature transformation on the image features y. The image features y in the latent domain, after passing through the hyperparameter coding network, obtain the hyper-prior latent information z. After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature z_hat corresponding to the coefficient hyperparameter feature z.

[0082] For the decoding end, after obtaining z_hat, it inputs z_hat into the mean prediction network, which outputs the mean prediction feature mu of the current image patch based on the context prediction process. On the other hand, it inputs z_hat into the probability hyperparameter decoding network to obtain the original probability distribution parameter p corresponding to the current image patch.

[0083] After receiving the second bitstream (Bitstream #2) corresponding to the current image patch from the encoder, the decoder can decode Bitstream #2 to obtain the scaled residual feature r_hat. The scaled residual feature r_hat originates from the following: the encoder inputs z_hat to the mean prediction network, which, based on context, outputs the mean prediction feature mu for the current image patch. The difference between the image feature y of the current image patch and the mean prediction feature mu is obtained to get the unquantized raw residual feature Resi. Resi is then scaled to obtain r, and r is quantized to obtain the scaled residual feature r_hat.

[0084] The decoding end actually decodes the received Bitstream#2 based on the original probability distribution parameter p mentioned above, to obtain the scaled residual feature r_hat. Since r_hat in Bitstream#2 is a scaled feature, it is actually necessary to scale the original probability distribution parameter p to obtain the scaled probability distribution parameter p_hat, and then use p_hat to decode Bitstream#2 to obtain r_hat.

[0085] Since r_hat is a scaled feature and cannot be directly used to reconstruct image patches, it needs to be inversely scaled to obtain Resi_hat (note that the quantized original residual feature Resi_hat here is different from the unquantized original residual feature Resi; Resi_hat is obtained by directly inversely scaling r_hat without inverse quantization). Then, the decoder can add mu and Resi_hat to obtain the reconstructed image feature y_hat of the current image patch. y_hat is then input into the synthesis transform network for synthesis transform, outputting the reconstructed image patch x_hat corresponding to the current image patch x. At this point, the image patch reconstruction at the decoder is complete.

[0086] Furthermore, since the mean prediction network performs iterative prediction based on context, the input to each iteration includes not only the hyperparameter quantization feature z_hat but also the reconstructed image feature y_hat, with an output of mu. In fact, the processes of calculating the mean prediction feature of the current image patch, decoding the second bitstream, and calculating the reconstructed feature y_hat are all staged, iterative processes. The following provides a more detailed explanation of the flow executed at the decoding end:

[0087] The decoder decodes the first bitstream to obtain z_hat, and inputs z_hat into a probabilistic hyperparameter decoding network to obtain p. After scaling p using ft*k, p_hat is obtained, and p_hat is divided into N corresponding stages p_hat_i (i = 1….N). p_hat_i (i = 1….N) is then used to decode r_hat_i (i = 1….N) from the second bitstream.

[0088] The decoding process to obtain y_hat_1 is as follows:

[0089] Based on the second bitstream decoding, r_hat_1 is obtained. R_hat_1 is inversely scaled using 1 / k*ft to obtain Resi_hat_1. The mean value mu_1 is then added to obtain the reconstructed feature y_hat_1.

[0090] The process of decoding to obtain y_i (i>1) is as follows:

[0091] Based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stage, the mean prediction network is used to obtain the mean mu_i. Based on the second bitstream decoding, r_hat_i is obtained, and r_hat_i is inversely scaled using 1 / k*ft to obtain Resi_hat_i. The mean mu_i is then added to obtain the reconstructed feature y_hat_i.

[0092] The y_hat_i (i = 1…N) is processed through clustering blocks to obtain y_hat. After y_hat is processed through a synthesis transform network, the reconstructed image block x_hat is obtained.

[0093] Example 2: For the processing procedure at the encoding end, please refer to... Figure 3 As shown, of course, Figure 3 This is merely an example of the encoding process, and no restrictions are placed on it. The preceding text introduced some operations related to the encoding side during the decoding process; these will not be repeated here.

[0094] After obtaining the current image block x (which can be the original image block x, i.e. the input image block), the encoder can analyze and transform the current image block x through an analysis and transformation network (a type of neural network) to obtain the image features y corresponding to the current image block x.

[0095] After obtaining the image feature y, the encoder performs a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z. For example, the image feature y can be input into the hyperparameter coding network, and the hyperparameter coding network can perform a coefficient hyperparameter feature transformation on the image feature y to obtain the coefficient hyperparameter feature z.

[0096] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantized feature z_hat corresponding to the coefficient hyperparameter feature z, i.e. Figure 3 The Q operation in the code is the quantization process. After z_hat, z_hat is encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image patch. Figure 3 The AE operation in Bitstream#1 represents the encoding process, such as entropy encoding. Specifically, z_hat carried in Bitstream#1 is primarily used as input to the mean prediction network (a type of neural network) to obtain the mean prediction features (denoted as mu) of the current image patch, and as input to the probability hyperparameter decoding network to obtain the original probability distribution parameters p.

[0097] After obtaining Bitstream #1 corresponding to the current image block, the encoder can send Bitstream #1 to the decoder. For the decoding process of Bitstream #1 corresponding to the current image block, please refer to the previous embodiment. After obtaining Bitstream #1, the encoder can also decode Bitstream #1 to obtain the hyperparameter quantization feature z_hat, i.e. Figure 3 In the diagram, AD represents the decoding process, resulting in z_hat. On one hand, z_hat is input into the mean prediction network to obtain the mean prediction feature mu of the current image patch. On the other hand, z_hat is input into the probability hyperparameter decoding network to obtain the original probability distribution parameter p corresponding to the current image patch.

[0098] For the encoding process of Bitstream#1, a fixed probability density model encoding method can be used; for the decoding process of Bitstream#1, a fixed probability density model decoding method can be used. There are no restrictions on the encoding and decoding processes.

[0099] After obtaining z_hat, the encoder can input z_hat into the mean prediction network. The mean prediction network, based on context, outputs the mean prediction feature mu for the current image patch. The image feature y of the current image patch is subtracted from the mean prediction feature mu to obtain the unquantized raw residual feature Resi. Resi is scaled to obtain r, and r is quantized to obtain the scaled residual feature r_hat. r_hat can then be encoded to obtain Bitstream #2 (i.e., the second bitstream) corresponding to the current image patch. Figure 3 In this context, the AE operation represents an encoding process, such as entropy encoding.

[0100] After obtaining the scaled residual feature r_hat, the encoder can also perform inverse scaling (inv-Scale) on r_hat to obtain the quantized original residual feature Resi_hat. Resi_hat is then added to the mean prediction feature mu output by the mean prediction network to obtain the reconstructed image feature y_hat of the current image patch. y_hat is then input back into the mean prediction network.

[0101] Furthermore, the encoder can decode Bitstream#2 based on the original probability distribution parameter p mentioned above to obtain the scaled residual feature r_hat. Since r_hat in Bitstream#2 is a scaled feature, it is actually necessary to scale the original probability distribution parameter p to obtain the scaled probability distribution parameter p_hat, and then use p_hat to decode Bitstream#2 to obtain r_hat. R_hat is then inversely scaled to obtain Resi_hat. The encoder can then add mu and Resi_hat to obtain the reconstructed image feature y_hat of the current image patch. y_hat is input into the synthesis transform network for synthesis transform, outputting the reconstructed image patch x_hat corresponding to the current image patch x. The encoder can optimize the encoding operation for the next current image patch based on the difference between x and x_hat.

[0102] Example 3: This application proposes a decoding method, see [link to example]. Figure 4 The diagram shown is a flowchart of the decoding method. This decoding method is performed in... Figure 2 Based on the decoding process shown, this paper focuses on the influence coefficients used in the scaling and descaling processes performed by the decoding end. Figure 4 The decoding method shown can be applied to the decoding end, and the method may include:

[0103] S400: Decode the first bitstream corresponding to the current image block to obtain the hyperparameter quantization feature corresponding to the current image block; based on the hyperparameter quantization feature, determine the original probability distribution parameter corresponding to the current image block through a probabilistic hyperparameter decoding network.

[0104] S402: Obtain the mean prediction feature of the current image patch.

[0105] S404: Obtain the influence coefficient, and scale the original probability distribution parameters based on the influence coefficient to obtain scaled probability distribution parameters.

[0106] S406: Based on the scaling probability distribution parameters, decode the second bitstream corresponding to the current image block to obtain the scaling residual features; based on the influence coefficient, inversely scale the scaling residual features to obtain the quantized original residual features.

[0107] S408: Based on the quantized original residual features and the mean prediction features, the reconstructed image block corresponding to the current image block is determined by a synthetic transformation network.

[0108] See also Figure 2 Understanding the decoding process shown Figure 4 The decoding method shown. Figure 4The key to the decoding method shown lies in determining the influence coefficients. That is, in Figure 4 In the encoding method shown, Figure 2 The influence coefficient k used in all scaling and descaling modules (Scale and invScale) is not a fixed value, but an adaptive value determined based on the content-related features of the current image patch.

[0109] This means that at the decoding end, the original probability distribution parameter p needs to be scaled based on an adaptive influence coefficient k to obtain the scaled probability distribution parameter p_hat. The scaled residual feature r_hat also needs to be inversely scaled based on the adaptive influence coefficient k to obtain the quantized original residual feature Resi_hat.

[0110] The adaptive influence coefficient k is determined based on the content-related features of the current image patch. These content-related features can be any feature related to the content of the current image patch. For example, they could include the current image patch itself, its image features, the mean prediction feature of the current image patch, the unquantized raw residual feature Resi, the raw probability distribution parameter p, and so on.

[0111] If the decoding end receives a third bitstream, it decodes k or ft*k from the third bitstream; if it does not receive a third bitstream, it generates k based on the content-related features of the current image block agreed upon with the encoding end.

[0112] For the decoding end, the content-related features of the image block can be obtained (wherein, the decoding end usually cannot obtain the current image block and the image features of the current image block), and then the influence coefficient is generated based on the content-related features.

[0113] As an example, a neural network model can be used to map the content-related features of the current image patch into mapped feature values, and then the influence coefficient can be determined based on the mapped feature values.

[0114] The decoder can receive a third bitstream sent by the encoder and decode it to obtain the influence coefficients. Of course, if the decoder is capable of generating the content-related features used by the encoder, it can generate influence coefficients that are the same as those used by the encoder, thus avoiding the bitrate consumption caused by the encoder transmitting the influence coefficients.

[0115] The influence coefficient can act not only on the channel domain dimension of the features (such as the unquantized original residual feature Resi, the scaled residual feature r_hat) or parameters (such as the probability distribution parameter p) corresponding to the current image patch, but also on the spatial domain dimension and the combined domain dimension of the features or parameters corresponding to the current image patch.

[0116] In other words, the influence coefficient is used to perform inverse scaling on the channel domain dimension of the scaled residual feature, or the influence coefficient is used to perform inverse scaling on the spatial domain dimension of the scaled residual feature, or the influence coefficient is used to perform inverse scaling on the combination domain dimension of the scaled residual feature.

[0117] Specifically, when performing inverse scaling on the channel domain dimension of the scaled residual feature based on the influence coefficient, the influence coefficient is a one-dimensional vector shape, and each coefficient value of the influence coefficient corresponds one-to-one with each channel of the scaled residual feature. Each feature value within the same channel of the scaled residual feature is inversely scaled using the coefficient value corresponding to that channel.

[0118] When inverse scaling is performed on the spatial dimension of the scaled residual feature based on the influence coefficient, the influence coefficient is a two-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature position point of the scaled residual feature on the horizontal and vertical planes. The feature values ​​in different channels corresponding to the same feature position point of the scaled residual feature are inversely scaled using the coefficient value corresponding to that feature position point.

[0119] When inverse scaling is performed on the combined domain dimension of the scaling residual feature based on the influence coefficient, the influence coefficient is a three-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature value of the scaling residual feature. Each feature value of the scaling residual feature is inversely scaled using the corresponding coefficient value.

[0120] Example 4: An encoding method is proposed in this application embodiment, see [link to example]. Figure 5 The diagram shown is a flowchart of the encoding method, which is performed in... Figure 3 Based on the encoding process shown, this paper focuses on the influence coefficients used in the scaling and descaling processes performed at the encoding end. Figure 5 The encoding method shown can be applied to the encoding end, and the method may include:

[0121] S500: Based on the image features of the current image block, determine the hyperparameter quantization features corresponding to the current image block through a hyperparameter coding network, and encode the first bitstream based on the hyperparameter quantization features.

[0122] S502: Generate an influence coefficient based on the content-related features of the current image block.

[0123] S504: Based on the influence coefficient, the unquantized original residual features corresponding to the current image block are scaled and then quantized to obtain scaled residual features;

[0124] S506: Based on the influence coefficient, the scaled residual features are inversely scaled to obtain the quantized original residual features.

[0125] S508: Send the first bitstream and the second bitstream to the decoding end.

[0126] See also Figure 3 Understanding the encoding process shown Figure 5 The encoding method shown. Figure 5 The key to the coding method shown lies in determining the influence coefficient. That is, in Figure 5 In the encoding method shown, Figure 3 The influence coefficient k used in all scaling and descaling modules (Scale and inv-Scale) is not a fixed value, but an adaptive value determined based on the content-related features of the current image patch.

[0127] This means that, at the encoding end, the unquantized original residual feature Resi corresponding to the current image patch needs to be scaled and then quantized based on an adaptive influence coefficient k to obtain the scaled residual feature r_hat. The original probability distribution parameter p needs to be scaled to p_hat based on an adaptive influence coefficient k. Finally, r_hat needs to be inversely scaled to the quantized original residual feature Resi_hat based on an adaptive influence coefficient k.

[0128] The adaptive influence coefficient k is determined based on the content-related features of the current image patch. These content-related features can be any feature related to the content of the current image patch. For example, they could include the current image patch itself, its image features, the mean prediction feature of the current image patch, the unquantized raw residual feature Resi, the raw probability distribution parameter p, and so on.

[0129] For the encoding end, the content-related features of the image patch can be obtained, and then the influence coefficient can be generated based on the content-related features.

[0130] As an example, a neural network model can be used to map the content-related features of the current image patch into mapped feature values, and then the influence coefficient can be determined based on the mapped feature values.

[0131] The encoder can encode the determined influence coefficients into a third bitstream and send it to the decoder, allowing the decoder to use it during corresponding scaling and descaling processes. If the decoder is capable of generating the content-related features used by the encoder, it can generate influence coefficients identical to those used by the encoder, thus avoiding the bitrate overhead caused by transmitting influence coefficients from the encoder.

[0132] Specifically, the encoding end can encode k with a certain precision using a fixed length and write it into the third bitstream to ensure that the decoding end can obtain the influence coefficient K.

[0133] The influence coefficient can act not only on the channel domain dimension of the features (such as the unquantized original residual feature Resi, the scaled residual feature r_hat) or parameters (such as the probability distribution parameter p) corresponding to the current image patch, but also on the spatial domain dimension and the combined domain dimension of the features or parameters corresponding to the current image patch.

[0134] In other words, the influence coefficient can be used to perform scaling on the channel domain dimension of the scaled residual feature, or on the spatial domain dimension of the scaled residual feature, or on the combined domain dimension of the scaled residual feature.

[0135] Specifically, when scaling is performed on the channel domain dimension of the unquantized original residual features based on the influence coefficient, the influence coefficient is a one-dimensional vector shape, and each coefficient value of the influence coefficient corresponds one-to-one with each channel of the unquantized original residual features. Each feature value within the same channel of the unquantized original residual features is scaled using the coefficient value corresponding to that channel.

[0136] When scaling is performed on the spatial dimension of the unquantized original residual feature based on the influence coefficient, the influence coefficient is a two-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature position point of the unquantized original residual feature on the horizontal and vertical planes. The feature values ​​in different channels corresponding to the same feature position point of the unquantized original residual feature are scaled using the coefficient value corresponding to that feature position point.

[0137] When scaling is performed on the combined domain dimension of the unquantized original residual features based on the influence coefficient, the influence coefficient is a three-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature value of the unquantized original residual features. Each feature value of the unquantized original residual features is scaled using the corresponding coefficient value.

[0138] Example 5: In this embodiment of the application, taking the image content features including the original probability distribution parameter p as an example, it is introduced how to use a neural network model to determine the influence coefficient k acting on the channel domain dimension.

[0139] The original probability distribution parameters can be input into a first neural network mapping model for processing, outputting a first mapping feature value. Then, based on the first mapping feature value, the influence coefficient is determined.

[0140] The original probability distribution parameter has a shape of a three-dimensional tensor C*W*H. During the process of the first neural network mapping model processing the original probability distribution parameter, the shape of the original probability distribution parameter is converted from C*W*H to a one-dimensional vector C that matches the channel domain dimension.

[0141] As a concrete example, see Figure 6 The first neural network mapping model comprises a spatial pooling layer (Avgpooling), a dimensionality culling layer (Squeeze), a fully connected layer (FC), an activation function layer (RELU), and another fully connected layer (FC) connected in sequence. The shape C*W*H of the original probability distribution parameters is first input into the spatial pooling layer for spatial pooling (averaging the width and height dimensions to obtain the average value of the data within each channel), outputting C*1*1. Then, C*1*1 is input into the dimensionality culling layer to reduce the dimensionality of C*1*1 to C. The original probability distribution parameters with the reduced shape are then input into the fully connected layer-activation function layer (for nonlinear transformation)-fully connected layer C for processing, obtaining the model output of the first neural network model. Then, the absolute values ​​(Abs) of each coefficient value of the model output with shape C are taken to obtain k_mid. A bias value (bias) with shape C is used to bias k_mid, resulting in k. Clearly, the shape of k is C. The initial value of bias can be 1.

[0142] Example 6: In this embodiment of the application, taking the image content features including the original probability distribution parameter p as an example, it is introduced how to use a neural network model to determine the influence coefficient k acting on the spatial dimension.

[0143] The original probability distribution parameters are input into a second neural network mapping model for processing, and a second mapping feature value is output. Then, the influence coefficient is determined based on the second mapping feature value.

[0144] The original probability distribution parameters are in the shape of a three-dimensional tensor C*W*H. During the processing of the original probability distribution parameters by the second neural network mapping model, the shape of the original probability distribution parameters is converted from C*W*H to a two-dimensional tensor W*H that matches the spatial dimension.

[0145] As a concrete example, see Figure 7The second neural network mapping model consists of a channel pooling layer (Avgpooling), a convolutional layer (Conv), an activation function layer (ReLU), and another convolutional layer (Conv) connected sequentially. The shape C*W*H of the original probability distribution parameters is first input into the channel pooling layer for channel pooling (averaging the channel dimensions to obtain the average value of data with the same width and height; the average value of each channel represents the importance of that channel), outputting 1*W*H. This 1*W*H is then processed by the convolutional layer-activation function layer-convolutional layer C to obtain the model output of the second neural network model. Next, the absolute values ​​(Abs) of each coefficient value in the 1*W*H model output are taken to obtain k_mid. A bias value (bias) of shape 1*W*H is then used to bias k_mid, resulting in k. Clearly, k has a shape of 1*W*H. The convolutional kernel size used in the convolutional layer can be 3, the stride can be 1, and the initial bias value can be 1.

[0146] Example 6: In this embodiment of the application, taking the image content features including the original probability distribution parameter p as an example, it is introduced how to use a neural network model to determine the influence coefficient k acting on the binding domain dimension.

[0147] The original probability distribution parameters are input into a third neural network mapping model for processing, and the third mapping feature value is output. Then, the influence coefficient is determined based on the third mapping feature value. The shape C*W*H of the original probability distribution parameters is preserved during the processing of the third neural network mapping model, thus ensuring that the determined influence coefficient k also has the shape C*W*H.

[0148] As a concrete example, see Figure 8 The third neural network mapping model consists of a series of convolutional layers, activation function layers, and another convolutional layer connected sequentially. The shape C*W*H of the original probability distribution parameters is input into the convolutional layer-activation function layer-convolutional layer C for processing, yielding the model output of the third neural network. Then, the absolute values ​​(Abs) of each coefficient value in the C*W*H shape of the model output are taken to obtain k_mid. A bias value (bias) of shape C*W*H is then used to bias k_mid, resulting in k. Clearly, k has a shape of C*W*H. The convolutional kernel size used in the convolutional layers can be 3, the stride can be 1, and the initial bias value can be 1.

[0149] Alternatively, the original probability distribution parameters can be input into a first neural network mapping model for processing to output a first mapping feature value; the original probability distribution parameters can be input into a second neural network mapping model for processing to output a second mapping feature value; and the influence coefficient can be determined based on the first mapping feature value and the second mapping feature value.

[0150] As another concrete example, see Figure 9 The shapes C*W*H of the original probability distribution parameters will be input respectively. Figure 6 The first and second neural network models shown (the structures can be as follows) Figure 7 As shown, it may also be less than Figure 7 As shown, Figure 9 The second neural network model shown in the figure is similar to Figure 7 (Different from the previous steps). For the first neural network model, the absolute value (Abs) of each coefficient value output by the model with shape C is taken to obtain k1_mid. The bias value (bias) of shape C is then applied to k1 to obtain k1. The absolute value (Abs) of each coefficient value output by the model with shape 1*W*H is taken to obtain k2_mid. The bias value (bias) of shape 1*W*H is then applied to k2 to obtain k. The convolutional kernel size used in the convolutional layer can be 3, the stride can be 1, and the initial bias value can be 1.

[0151] Furthermore, this disclosure provides a more detailed description of the execution process on the encoding end as follows:

[0152] Image x is processed by an analysis and transformation network to obtain image features y. y is then processed by a hyperparameter coding network to obtain z. z is encoded to obtain the first bitstream. Based on the first bitstream, z_hat is decoded, and z_hat is processed by a probabilistic hyperparameter decoding network to obtain p. The influence coefficient k is calculated based on content-related features, and combined with the scaling factor ft, a scaling factor k*ft is obtained. p is scaled using k*ft to obtain p_hat. The encoding of y includes the following steps: y is merged and classified to obtain N stages of features to be encoded, y_i (i = 1…N). p_hat is also divided into N corresponding stages, p_hat_i (i = 1…N). The encoding process of feature y_1 is as follows:

[0153] For the first-stage feature y_1, y is processed by a hyperparameter coding network to obtain z, which is then encoded into the first bitstream. Based on the first bitstream, z_hat is decoded, and based on z_hat (features from other stages cannot be used in the first stage), a mean prediction network is used to obtain the mean prediction feature mu_1. Based on mu_1, the residual feature Resi_1 of feature y_1 is obtained as Resi_1 = y_1 – mu_1. Resi_1 is scaled using k*ft to obtain r_1, and r_1 is rounded and quantized to obtain r_hat_1. Then, a probabilistic model is constructed using p_hat_1 to entropy-encode r_hat_1, which is then written into the second bitstream. Next, based on the second bitstream, r_hat_1 is decoded again, and 1 / k*ft is used to inversely scale r_hat_1 to obtain Resi_hat_1. The mean mu_1 is then added to obtain the reconstructed feature y_hat_1.

[0154] For the i-th stage feature y_i (i>1), the encoding process is as follows:

[0155] Based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stages, the mean mu_i is obtained through a mean prediction network. Based on mu_i, the residual Resi_i of feature y_i is obtained. Resi_i is scaled using k*ft to obtain r_i, which is then rounded and quantized to obtain r_hat_i. A probabilistic model using p_hat_i is then used to entropy-encode r_hat_i, and this entropy is written into the second bitstream. Based on the second bitstream, r_hat_i is decoded and inversely scaled using 1 / k*ft to obtain Resi_hat_i. The mean mu_i is then added to obtain the reconstructed feature y_hat_i.

[0156] y_hat_i (i = 1…N) is processed through clustering and segmentation units to obtain y_hat. After passing through a synthesis transform network, y_hat yields the reconstructed image block x_hat. Based on the content-related features upon which k is generated, it is determined whether the decoding end can acquire these content-related features. If the decoding end cannot acquire them, k or ft*k is quantized, encoded with a certain precision using fixed-length encoding, and written into a third bitstream. This third bitstream is then transmitted to the decoding end. In this case, the k or ft*k used by the encoding end is also quantized.

[0157] This disclosure further details the process executed on the decoding end as follows:

[0158] If the decoding end receives a third bitstream, it decodes k or ft*k from the third bitstream; if it does not receive a third bitstream, it generates k based on the content-related features of the current image block agreed upon with the encoding end.

[0159] The decoder decodes the first bitstream and then obtains p through a probabilistic hyperparameter decoding network. After scaling p using ft*k, p_hat is obtained, and p_hat is divided into N corresponding stages p_hat_i (i = 1….N). r_hat_i (i = 1….N) is then decoded from the second bitstream using p_hat_i (i = 1….N).

[0160] The decoding process to obtain y_hat_1 is as follows:

[0161] Based on the second bitstream decoding, r_hat_1 is obtained. R_hat_1 is inversely scaled using 1 / k*ft to obtain Resi_hat_1. The mean value mu_1 is then added to obtain the reconstructed feature y_hat_1.

[0162] The process of decoding to obtain y_i (i>1) is as follows:

[0163] Based on z_hat and the reconstructed features y_hat_1 to y_hat_i-1 from the previous stage, the mean prediction network is used to obtain the mean mu_i. Based on the second bitstream decoding, r_hat_i is obtained, and r_hat_i is inversely scaled using 1 / k*ft to obtain Resi_hat_i. The mean mu_i is then added to obtain the reconstructed feature y_hat_i.

[0164] The y_hat_i (i = 1…N) is processed through clustering blocks to obtain y_hat. After y_hat is processed through a synthesis transform network, the reconstructed image block x_hat is obtained.

[0165] In addition, see Figure 10 As shown, the encoding end can input the content-related features of the current image block into the influence coefficient determination module (including at least a neural network model) to obtain the influence coefficient k. Then, ft*k is used to scale the unquantized original residual features before quantization to obtain the scaled residual features.

[0166] See Figure 11A As shown, if the content-related features used by the encoder to generate the influence coefficient k (which are the same as those used by the encoder) can be obtained by the decoder, then the decoder can obtain the influence coefficient k from the content-related determination module (which includes at least a neural network model) of the current image block, and use ft*k to inversely scale the scaled residual features to obtain the quantized original residual features.

[0167] See Figure 11BAs shown, if the content-related features used by the encoder to generate the influence coefficient k cannot be obtained by the decoder (for example, the content-related feature is the image feature y of the current image block), then the encoder needs to encode the influence coefficient k into a third bitstream and send the third bitstream to the decoder. The decoder decodes the third bitstream to obtain the influence coefficient k, and uses ft*k to inversely scale the scaled residual features to obtain the quantized original residual features.

[0168] This disclosure also provides a decoding apparatus, including:

[0169] The determination module decodes the first bitstream corresponding to the current image block to obtain the hyperparameter quantization feature corresponding to the current image block; based on the hyperparameter quantization feature, it determines the original probability distribution parameter corresponding to the current image block through a probabilistic hyperparameter decoding network; it obtains the mean prediction feature of the current image block; it obtains the influence coefficient, wherein the influence coefficient is generated based on the content-related features of the current image block; and it scales the original probability distribution parameter based on the influence coefficient to obtain the scaled probability distribution parameter.

[0170] The processing module decodes the second bitstream corresponding to the current image block based on the scaling probability distribution parameters to obtain scaling residual features; and performs inverse scaling on the scaling residual features based on the influence coefficient to obtain quantized original residual features.

[0171] The reconstruction module obtains the mean prediction features of the current image block, and determines the reconstructed image block corresponding to the current image block through a synthetic transformation network based on the quantized original residual features and the mean prediction features.

[0172] The acquisition of the influence coefficient includes:

[0173] Obtain the content-related features of the current image patch;

[0174] The influence coefficient is generated based on the content-related characteristics.

[0175] The determining module obtains the influence coefficient based on the received third bitstream; wherein, the third bitstream is the bitstream obtained by the encoding end after generating the influence coefficient based on the content-related features of the current image block, and then encoding based on the influence coefficient.

[0176] The content-related features include at least one of the following:

[0177] The original probability distribution parameters;

[0178] Unquantified raw residual characteristics;

[0179] The mean prediction feature.

[0180] The step of generating the influence coefficient based on the content-related features of the current image patch includes:

[0181] The content-related features of the current image patch are input into a neural network mapping model, and the mapped feature values ​​are output.

[0182] The influence coefficient is determined based on the mapping feature value.

[0183] The influence coefficient is used to perform inverse scaling on the channel domain dimension of the scaled residual feature;

[0184] Alternatively, the influence coefficient is used to perform inverse scaling on the spatial dimension of the scaled residual feature;

[0185] Alternatively, the influence coefficient can be used to perform inverse scaling on the combined domain dimension of the scaled residual feature.

[0186] Specifically, when inverse scaling is performed on the channel domain dimension of the scaled residual feature based on the influence coefficient, the influence coefficient is a one-dimensional vector shape, and each coefficient value of the influence coefficient corresponds one-to-one with each channel of the scaled residual feature. Each feature value within the same channel of the scaled residual feature is inversely scaled using the coefficient value corresponding to that channel.

[0187] Wherein, when the content-related features include the probability distribution parameters, the step of generating the influence coefficient based on the content-related features of the current image patch includes:

[0188] The original probability distribution parameters are input into the first neural network mapping model for processing, and the first mapping feature value is output. The original probability distribution parameters are in the shape of a three-dimensional tensor C*W*H. During the processing of the original probability distribution parameters, the first neural network mapping model converts the shape of the original probability distribution parameters from C*W*H into a one-dimensional vector C that matches the channel domain dimension.

[0189] The influence coefficient is determined based on the first mapping feature value.

[0190] In the case where inverse scaling is performed on the spatial dimension of the scaled residual feature based on the influence coefficient, the influence coefficient is a two-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature position point of the scaled residual feature on the horizontal and vertical planes. The feature values ​​in different channels corresponding to the same feature position point of the scaled residual feature are inversely scaled using the coefficient value corresponding to that feature position point.

[0191] Wherein, when the content-related features include the probability distribution parameters, the step of generating the influence coefficient based on the content-related features of the current image patch includes:

[0192] The original probability distribution parameters are input into the second neural network mapping model for processing, and the second mapping feature value is output. The original probability distribution parameters are in the shape of a three-dimensional tensor C*W*H. During the processing of the original probability distribution parameters, the second neural network mapping model converts the shape of the original probability distribution parameters from C*W*H into a two-dimensional tensor W*H that matches the spatial dimension.

[0193] The influence coefficient is determined based on the second mapping feature value.

[0194] Specifically, when inverse scaling is performed on the combined domain dimension of the scaling residual feature based on the influence coefficient, the influence coefficient is a three-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature value of the scaling residual feature. Each feature value of the scaling residual feature is inversely scaled using the corresponding coefficient value.

[0195] Wherein, when the content-related features include the probability distribution parameters, the step of generating the influence coefficient based on the content-related features of the current image patch includes:

[0196] The original probability distribution parameters are input into the first neural network mapping model for processing, and the first mapping feature value is output. The original probability distribution parameters are in the shape of a three-dimensional tensor C*W*H. During the processing of the original probability distribution parameters, the first neural network mapping model converts the shape of the original probability distribution parameters from C*W*H into a one-dimensional vector C that matches the channel domain dimension.

[0197] The original probability distribution parameters are input into the second neural network mapping model for processing, and the second mapping feature value is output; wherein, in the process of processing the original probability distribution parameters, the shape of the original probability distribution parameters is converted from C*W*H to a two-dimensional tensor W*H that matches the spatial dimension;

[0198] The influence coefficient is determined based on the first mapping feature value and the second mapping feature value.

[0199] An encoding device, comprising:

[0200] The first processing module determines the hyperparameter quantization features corresponding to the current image block through a hyperparameter coding network based on the image features of the current image block; and encodes the first bitstream based on the hyperparameter quantization features.

[0201] The second processing module generates an influence coefficient based on the content-related features of the current image block; based on the influence coefficient, it scales and quantizes the unquantized original residual features corresponding to the current image block to obtain scaled residual features; and based on the scaled residual features, it encodes a second bitstream.

[0202] The third processing module performs inverse scaling on the scaled residual features based on the influence coefficient to obtain the quantized original residual features;

[0203] The sending module sends the first bitstream and the second bitstream to the decoding end.

[0204] The content-related features include at least one of the following:

[0205] The current image block;

[0206] The image features of the current image block;

[0207] The mean prediction feature of the current image patch;

[0208] The unquantified original residual characteristics;

[0209] Original probability distribution parameters.

[0210] The sending module encodes a third bitstream based on the influence coefficient and sends the third bitstream to the decoding end.

[0211] Specifically, when scaling is performed on the channel domain dimension of the unquantized original residual features based on the influence coefficient, the influence coefficient is a one-dimensional vector shape, and each coefficient value of the influence coefficient corresponds one-to-one with each channel of the unquantized original residual features. Each feature value within the same channel of the unquantized original residual features is scaled using the coefficient value corresponding to that channel.

[0212] Specifically, when scaling is performed on the spatial dimension of the unquantized original residual features based on the influence coefficient, the influence coefficient is a two-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature position point of the unquantized original residual features on the horizontal and vertical planes. The feature values ​​in different channels corresponding to the same feature position point of the unquantized original residual features are scaled using the coefficient value corresponding to that feature position point.

[0213] Specifically, when scaling is performed on the combined domain dimension of the unquantized original residual features based on the influence coefficient, the influence coefficient is a three-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature value of the unquantized original residual features. Each feature value of the unquantized original residual features is scaled using the corresponding coefficient value.

[0214] See Figure 12A A decoding end device is provided, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0215] The processor is used to execute machine-executable instructions to implement the methods of the above embodiments of this application.

[0216] See Figure 12B An encoding end device is provided, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor;

[0217] The processor is used to execute machine-executable instructions to implement the methods of the above embodiments of this application.

[0218] Based on the same concept as the methods described above, this application provides an electronic device. It includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor; the processor executes the machine-executable instructions to implement the methods of the embodiments described above.

[0219] Based on the same application concept as the above methods, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the methods disclosed in the above examples of this application, such as the methods in the above embodiments.

[0220] Based on the same concept as the above-described method, this application also provides a computer application that, when executed by a processor, can implement the method disclosed in the above-described examples of this application.

[0221] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above descriptions are merely embodiments of this application and are not intended to limit this application.

[0222] Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A decoding method, characterized in that, include: Decode the first bitstream corresponding to the current image block to obtain the hyperparameter quantization features corresponding to the current image block; Based on the hyperparameter quantization features, the original probability distribution parameters corresponding to the current image patch are determined through a probabilistic hyperparameter decoding network; the mean prediction features of the current image patch are then obtained. An influence coefficient is obtained, wherein the influence coefficient is generated based on the content-related features of the current image patch; wherein the content-related features include at least one of the following: the original probability distribution parameter; the mean prediction feature; wherein, if the content-related features include the original probability distribution parameter, the decoding end automatically generates the influence coefficient based on the original probability distribution parameter; if the content-related features include the mean prediction feature, the decoding end automatically generates the influence coefficient based on the mean prediction feature. The original probability distribution parameters are scaled based on the influence coefficient to obtain scaled probability distribution parameters; Based on the scaling probability distribution parameters, the second bitstream corresponding to the current image block is decoded to obtain the scaling residual features; based on the influence coefficient, the scaling residual features are inversely scaled to obtain the quantized original residual features. Based on the quantized original residual features and the mean prediction features, the reconstructed image block corresponding to the current image block is determined by a synthetic transformation network.

2. The method as described in claim 1, characterized in that, The step of generating the influence coefficient based on the content-related features of the current image patch includes: The content-related features of the current image patch are input into a neural network mapping model, and the mapped feature values ​​are output. The influence coefficient is determined based on the mapping feature value.

3. The method as described in claim 1, characterized in that, The influence coefficient is used to perform inverse scaling on the channel domain dimension of the scaled residual feature; Alternatively, the influence coefficient is used to perform inverse scaling on the spatial dimension of the scaled residual feature; Alternatively, the influence coefficient can be used to perform inverse scaling on the combined domain dimension of the scaled residual feature.

4. The method as described in claim 3, characterized in that, When inverse scaling is performed on the channel domain dimension of the scaled residual feature based on the influence coefficient, the influence coefficient is a one-dimensional vector shape, and each coefficient value of the influence coefficient corresponds one-to-one with each channel of the scaled residual feature. Each feature value in the same channel of the scaled residual feature is inversely scaled using the coefficient value corresponding to that channel.

5. The method as described in claim 4, characterized in that, When the content-related features include the probability distribution parameters, the step of generating the influence coefficient based on the content-related features of the current image patch includes: The original probability distribution parameters are input into the first neural network mapping model for processing, and the first mapping feature value is output. The original probability distribution parameters are in the shape of a three-dimensional tensor C×W×H. During the processing of the original probability distribution parameters, the first neural network mapping model converts the shape of the original probability distribution parameters from C×W×H into a one-dimensional vector C that matches the channel domain dimension. The influence coefficient is determined based on the first mapping feature value.

6. The method as described in claim 3, characterized in that, When inverse scaling is performed on the spatial dimension of the scaled residual feature based on the influence coefficient, the influence coefficient is a two-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature position point of the scaled residual feature on the horizontal and vertical planes. The feature values ​​in different channels corresponding to the same feature position point of the scaled residual feature are inversely scaled using the coefficient value corresponding to that feature position point.

7. The method as described in claim 6, characterized in that, When the content-related features include the probability distribution parameters, the step of generating the influence coefficient based on the content-related features of the current image patch includes: The original probability distribution parameters are input into the second neural network mapping model for processing, and the second mapping feature value is output. The original probability distribution parameters are in the shape of a three-dimensional tensor C×W×H. During the processing of the original probability distribution parameters, the second neural network mapping model converts the shape of the original probability distribution parameters from C×W×H into a two-dimensional tensor W×H that matches the spatial dimension. The influence coefficient is determined based on the second mapping feature value.

8. The method as described in claim 3, characterized in that, When inverse scaling is performed on the combined domain dimension of the scaling residual feature based on the influence coefficient, the influence coefficient is a three-dimensional tensor shape, and each coefficient value of the influence coefficient corresponds one-to-one with each feature value of the scaling residual feature. Each feature value of the scaling residual feature is inversely scaled using the corresponding coefficient value.

9. The method as described in claim 8, characterized in that, When the content-related features include the probability distribution parameters, the step of generating the influence coefficient based on the content-related features of the current image patch includes: The original probability distribution parameters are input into the first neural network mapping model for processing, and the first mapping feature value is output. The original probability distribution parameters are in the shape of a three-dimensional tensor C×W×H. During the processing of the original probability distribution parameters, the first neural network mapping model converts the shape of the original probability distribution parameters from C×W×H into a one-dimensional vector C that matches the channel domain dimension. The original probability distribution parameters are input into the second neural network mapping model for processing, and the second mapping feature value is output; wherein, in the process of processing the original probability distribution parameters, the shape of the original probability distribution parameters is converted from C×W×H to a two-dimensional tensor W×H that matches the spatial dimension; The influence coefficient is determined based on the first mapping feature value and the second mapping feature value.

10. A decoding device, characterized in that, include: The determination module decodes the first bitstream corresponding to the current image block to obtain the hyperparameter quantization features corresponding to the current image block; Based on the hyperparameter quantization features, the original probability distribution parameters corresponding to the current image patch are determined through a probabilistic hyperparameter decoding network; the mean prediction features of the current image patch are then obtained. An influence coefficient is obtained, wherein the influence coefficient is generated based on the content-related features of the current image patch; the original probability distribution parameters are scaled based on the influence coefficient to obtain scaled probability distribution parameters; wherein the content-related features include at least one of the following: the original probability distribution parameters; the mean prediction feature; wherein, if the content-related features include the original probability distribution parameters, the decoding end automatically generates the influence coefficient based on the original probability distribution parameters; if the content-related features include the mean prediction feature, the decoding end automatically generates the influence coefficient based on the mean prediction feature; The processing module decodes the second bitstream corresponding to the current image block based on the scaling probability distribution parameters to obtain scaling residual features; and performs inverse scaling on the scaling residual features based on the influence coefficient to obtain quantized original residual features. The reconstruction module obtains the mean prediction features of the current image block, and determines the reconstructed image block corresponding to the current image block through a synthetic transformation network based on the quantized original residual features and the mean prediction features.

11. A decoding device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1-9.

12. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1-9.

13. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores a plurality of computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-9.

Images