Image encoding method, image encoding method, apparatus, and computer storage medium

Abstract

The application discloses an image coding method, an image decoding method, an image coding device, an image decoding device and a computer storage medium. The image coding method comprises the following steps: inputting an image to be coded, and extracting a feature to be coded of the image to be coded; inputting the feature to be coded into a main coding network to perform feature transformation, and obtaining a first coding feature; inputting the first coding feature into an entropy model network to perform feature analysis, and obtaining context parameters and probability estimation parameters of the first coding feature; obtaining distribution information of the first coding feature according to the context parameters and the probability estimation parameters; and using the main coding network to code the first coding feature according to the distribution information, and obtaining a feature code stream of the feature to be coded. The image coding method can more accurately estimate the distribution of the feature to be coded of the main coding network by combining the context parameters and the probability estimation parameters of the feature.

Description

Technical Field

[0001] This application relates to the field of feature coding technology, and in particular to an image coding method, an image coding apparatus, and a computer storage medium. Background Technology

[0002] Traditional image coding techniques are designed for human vision. However, with the superior performance of deep neural networks in various machine vision tasks, such as image classification, object detection, and semantic segmentation, numerous AI applications based on machine vision have emerged. To ensure that the performance of machine vision tasks is not compromised by the image coding process, a "analysis-then-encode" approach is adopted to address machine vision requirements. This involves directly extracting features from the lossless image using a neural network at the image acquisition end, then encoding and transmitting the extracted features. The decoding end directly uses the decoded features as input to subsequent network structures to complete different machine vision tasks. Therefore, to conserve transmission bandwidth resources, it is necessary to research image coding methods specifically for machine vision.

[0003] Current feature encoding algorithms mainly fall into two categories: traditional encoding methods and learning-based schemes. Traditional encoding methods include the following: First, replacing high-precision data types with low-precision data types reduces the space occupied by the original feature data; however, this is not true encoding of the feature data but rather an improvement from a computer storage perspective. Second, using dimensionality reduction methods, such as PCA (Principal Component Analysis), to extract the main data components of the original feature data, thus using low-dimensional data to roughly represent the information of the original data; this is a lossy encoding method. Third, hybrid encoder schemes, which first quantize deep features and then use encoders such as High Efficiency Video Coding (HEVC) or H.266 / VVC to perform lossy encoding on the quantized features. The drawback of this scheme is that the quality degradation of the lossy encoding by the hybrid encoder is inconsistent with the performance degradation of the features during visual analysis tasks, resulting in the features failing to provide the important information needed for visual analysis. Summary of the Invention

[0004] This application provides an image encoding method, an image encoding apparatus, and a computer storage medium.

[0005] One technical solution adopted in this application is to provide an image encoding method, the image encoding method comprising:

[0006] Input the image to be encoded, and extract the features to be encoded from the image;

[0007] The feature to be encoded is input into the main encoding network for feature transformation to obtain the first encoded feature;

[0008] The first encoded feature is input into the entropy model network for feature analysis to obtain the context parameters and probability estimation parameters of the first encoded feature;

[0009] Based on the context parameters and the probability estimation parameters, the distribution information of the first encoded feature is obtained;

[0010] The first coding feature is encoded using the main coding network according to the distribution information to obtain the feature bitstream of the feature to be encoded.

[0011] The step of inputting the first encoded feature into an entropy model network for feature analysis to obtain the context parameters and probability estimation parameters of the first encoded feature includes:

[0012] The first encoded feature is quantized to obtain the second encoded feature;

[0013] The second encoded feature is then entropy encoded and entropy decoded sequentially to obtain the third encoded feature;

[0014] Based on the third coding feature, probability estimation is performed to obtain the probability estimation parameters of the first coding feature;

[0015] Obtain the contextual relevance between the second encoded feature and adjacent encoded features, and obtain the contextual parameters of the first encoded feature based on the contextual relevance.

[0016] Before quantizing the first encoded feature to obtain the second encoded feature, the image encoding method further includes:

[0017] The first encoded feature is subjected to feature transformation to obtain the feature to be encoded of the entropy model network, wherein the feature dimension of the feature to be encoded is smaller than the feature dimension of the first encoded feature.

[0018] The step of obtaining the distribution information of the first encoded feature based on the context parameters and the probability estimation parameters includes:

[0019] The context parameters and the probability estimation parameters are concatenated along the channel dimension;

[0020] The concatenated result is subjected to feature transformation to obtain the distribution information of the first encoded feature.

[0021] Wherein, after the main coding network encodes the first coding feature according to the distribution information to obtain the feature bitstream of the feature to be encoded, the image coding method further includes:

[0022] The feature code stream is decoded to obtain the reconstructed image;

[0023] Obtain evaluation metrics in multiple dimensions for the reconstructed image and the image to be encoded;

[0024] The evaluation metrics are used to train the main encoding network and the entropy model network.

[0025] The evaluation metrics include at least one of perceptual similarity and perceptual distance.

[0026] The step of inputting the feature to be encoded into the main coding network for feature transformation to obtain the first encoded feature includes:

[0027] The feature to be encoded is input into the main encoding network and processed sequentially through one or more of the following sub-networks: preprocessing sub-network, feature enhancement sub-network, and feature pruning sub-network, to obtain the first encoded feature.

[0028] The preprocessing subnetwork implements preprocessing functions by employing several convolutional layers and / or several residual blocks. The preprocessing functions increase the nonlinearity of the input features to be encoded and output the preprocessed features to be encoded.

[0029] The feature enhancement subnetwork includes one or more of the following: spatial attention subnetwork, invertible subnetwork, channel attention subnetwork, and multi-scale residual subnetwork.

[0030] The feature enhancement subnetwork is used to input the feature to be encoded, or the preprocessed feature to be encoded output by the preprocessing subnetwork, and to sequentially input the feature to be encoded or the preprocessed feature to be encoded into one or more of the spatial attention subnetwork, the invertible subnetwork, the channel attention subnetwork, and the multi-scale residual subnetwork to enhance the feature to be encoded or the preprocessed feature to be encoded, and output the feature-enhanced feature to be encoded.

[0031] The reversible subnetwork is composed of at least one reversible unit, wherein the reversible unit includes a feature rearrangement layer and a feature coupling layer; the feature rearrangement layer is used to change the feature shape of the feature to be encoded, and the feature coupling layer is used to combine the features of the previous layer with the features of the current layer.

[0032] The feature pruning subnetwork is used to rearrange the input features to be encoded, or the preprocessed features to be encoded output by the preprocessing subnetwork, or the enhanced features to be encoded output by the feature enhancement subnetwork, to obtain rearranged features. The average feature channel dimension of the rearranged features is calculated using a preset pruning coefficient, so as to obtain the pruning features of the rearranged features according to the average feature channel dimension, thereby realizing the feature pruning function and outputting the first encoded feature.

[0033] Another technical solution adopted in this application is to provide an image decoding method, the image decoding method comprising:

[0034] Input feature bitstream;

[0035] The main coding network is used to decode the feature bitstream according to preset distribution information to obtain the first decoded feature of the feature bitstream, wherein the preset distribution information is determined by the entropy model network during the encoding process;

[0036] The first decoding feature is input into the main coding network for feature reconstruction to obtain the second decoding feature;

[0037] The decoded image is obtained by reconstructing according to the second decoding feature.

[0038] Another technical solution adopted in this application is to provide an image encoding device, which includes a memory and a processor coupled to the memory;

[0039] The memory is used to store program data, and the processor is used to execute the program data to implement the image encoding method described above.

[0040] Another technical solution adopted in this application is to provide an image decoding device, which includes a memory and a processor coupled to the memory;

[0041] The memory is used to store program data, and the processor is used to execute the program data to implement the image decoding method described above.

[0042] Another technical solution adopted in this application is to provide a computer storage medium for storing program data, which, when executed by a computer, is used to implement the image encoding method and / or image decoding method as described above.

[0043] The beneficial effects of this application are as follows: The image encoding device inputs an image to be encoded and extracts the features to be encoded from the image; the features to be encoded are input into a main encoding network for feature transformation to obtain first encoded features; the first encoded features are input into an entropy model network for feature analysis to obtain context parameters and probability estimation parameters of the first encoded features; based on the context parameters and the probability estimation parameters, the distribution information of the first encoded features is obtained; the main encoding network encodes the first encoded features according to the distribution information to obtain the feature bitstream of the features to be encoded. The image encoding method of this application can more accurately estimate the distribution of the features to be encoded in the main encoding network by combining the context parameters and probability estimation parameters of the features. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating an embodiment of the image encoding method provided in this application;

[0046] Figure 2 This is a schematic diagram of the network structure of the end-to-end image encoding and decoding method provided in this application;

[0047] Figure 3 This is a schematic diagram of an embodiment of the dense block structure provided in this application;

[0048] Figure 4 This is a schematic diagram of the LeakyReLU activation function provided in this application;

[0049] Figure 5 This is a schematic diagram of the preprocessing and postprocessing unit containing dense blocks provided in this application;

[0050] Figure 6 This is a schematic diagram of the structure of the reversible enhancement subnetwork of the multi-attention mechanism provided in this application;

[0051] Figure 7 This is a schematic diagram of the spatial attention subnetwork provided in this application;

[0052] Figure 8 This is a schematic diagram of the feature with shape n*h*w provided in this application;

[0053] Figure 9 This is a schematic diagram of the structure of the reversible subnetwork provided in this application;

[0054] Figure 10 This is a schematic diagram of the channel attention subnetwork provided in this application;

[0055] Figure 11 This is a schematic diagram of the structure of the multi-scale residual subnetwork provided in this application;

[0056] Figure 12 This is a schematic diagram of the spatial attention layer provided in this application;

[0057] Figure 13 This is a schematic diagram of the structure of the reversible subnetwork unit provided in this application;

[0058] Figure 14This is a schematic diagram of the channel attention and multi-scale residual layer structure provided in this application;

[0059] Figure 15 This is a schematic diagram of the probability density curves of the standard t-distribution with different degrees of freedom provided in this application;

[0060] Figure 16 This is a schematic diagram of the entropy model network provided in this application;

[0061] Figure 17 This is a schematic diagram of the feature transformation module of the entropy model network provided in this application;

[0062] Figure 18 This is a schematic diagram of the probability estimation module of the entropy model network provided in this application;

[0063] Figure 19 This is a schematic diagram of the context module structure of the entropy model network provided in this application;

[0064] Figure 20 This is a schematic diagram of the structure for obtaining the final distribution parameters of feature y provided in this application;

[0065] Figure 21 This is a schematic diagram of the MS-SSIM calculation process provided in this application;

[0066] Figure 22 This is a flowchart illustrating an embodiment of the image decoding method provided in this application;

[0067] Figure 23 This is a schematic diagram of an embodiment of the image encoding device provided in this application;

[0068] Figure 24 This is a schematic diagram of an embodiment of the image decoding apparatus provided in this application;

[0069] Figure 25 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0071] Please refer to details. Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating an embodiment of the image encoding method provided in this application. Figure 2 This is a schematic diagram of the network structure of the end-to-end image encoding and decoding method provided in this application.

[0072] like Figure 2 As shown, Figure 2 This represents the overall network framework of the image encoding and decoding methods proposed in this application. Figure 2 The quantization method uses rounding to quantize floating-point data to integer form, and the entropy coding uses traditional entropy coding methods, such as Huffman coding and arithmetic coding.

[0073] exist Figure 2 Based on the overall network framework shown, the main improvements of this application are as follows:

[0074] 1) In the feature transformation and feature reconstruction process of the main coding network, a more efficient feature transformation and reconstruction is achieved by using three parts: preprocessing / postprocessing, feature enhancement, and feature pruning / copying.

[0075] 2) In the entropy model network, a mixed probability model with multiple t-distributions is proposed to improve the accuracy of probability estimation.

[0076] 3) During training, subjective and objective image distortions are considered in the loss function.

[0077] The main improvements mentioned above will be elaborated in detail below with reference to subsequent image coding method embodiments:

[0078] like Figure 1 As shown, the image encoding method of this application embodiment includes the following steps:

[0079] Step S11: Input the image to be encoded and extract the features to be encoded from the image.

[0080] In this embodiment, the image encoding device acquires the image to be encoded and then extracts the features to be encoded from the image. Feature extraction can be performed by a feature extraction layer in the main encoding network or by other feature extraction networks; no limitation is imposed here.

[0081] Step S12: Input the feature to be encoded into the main encoding network for feature transformation to obtain the first encoded feature.

[0082] In this embodiment of the application, the image encoding device inputs the features to be encoded into the main encoding network for feature transformation, so as to reduce the amount of data of the features to be encoded and facilitate subsequent encoding work.

[0083] Specifically, the main encoding network may include one or more of a preprocessing subnetwork, a feature enhancement subnetwork, and a feature trimming subnetwork. The features to be encoded are sequentially input into one or more of these subnetworks for feature transformation, thereby obtaining the first encoded features. The preprocessing subnetwork implements feature preprocessing, the feature enhancement subnetwork implements feature enhancement, and the feature trimming subnetwork implements feature trimming. The specific implementation processes of the feature preprocessing, feature enhancement, and feature trimming functions are described below:

[0084] like Figure 2 As shown, the main processes of the feature transformation stage (encoding) and feature reconstruction stage (decoding) of the main coding network include preprocessing / postprocessing, feature enhancement, and feature cropping / copying. The following section uses the feature transformation stage as an example to illustrate these feature processing steps:

[0085] In the end-to-end image coding of this application, preprocessing or postprocessing is used to increase the nonlinearity of the data during the feature transformation and reconstruction stages. The input and output of the preprocessing and postprocessing have the same size, including but not limited to the following methods:

[0086] (1) Stacking multiple convolutional layers increases the degree of nonlinearity of the data.

[0087] (2) Use multi-layer residual blocks to increase the degree of nonlinearity of the data.

[0088] (3) Use pre-trained neural networks to increase the nonlinearity of the data.

[0089] Therefore, the preprocessing subnetwork of this application embodiment can be configured with several convolutional layers and / or residual blocks to implement the preprocessing function, thereby increasing the nonlinearity of the input features to be encoded and outputting the preprocessed features to be encoded.

[0090] In one specific embodiment, the image encoding device may employ any one or a combination of the above methods to implement preprocessing or postprocessing functions.

[0091] For example, please see Figure 3 , Figure 3 This is a schematic diagram of an embodiment of the dense block structure provided in this application. This embodiment uses dense blocks to increase the nonlinearity of features. Dense blocks have the structure and properties of residual blocks.

[0092] in, Figure 3 The ⊕ symbol represents concatenation along the channel dimension. The activation function used is LeakyReLU, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of the LeakyReLU activation function provided in this application.

[0093] For example, for an input feature x of size n*h*w, the preprocessing and postprocessing are as follows:

[0094] A. Use as follows Figure 5 The network shown, which contains units with Dense Blocks, processes the input features, where... Figure 5 This is a schematic diagram of the preprocessing and postprocessing unit containing dense blocks provided in this application.

[0095] B. Multiply the output of process A by 0.2.

[0096] C. Add the result of process B directly to the original input features to obtain the final features.

[0097] Furthermore, end-to-end image coding based on neural networks extracts image features and encodes these features to achieve the coding objective. Therefore, the quality of the extracted features directly affects the coding efficiency and reconstruction effect.

[0098] The human eye tends to focus on a specific area of ​​an image rather than the entire image. Furthermore, during the encoding process, it is more reasonable to allocate more bits to the areas that the human eye focuses on, and to allocate fewer bits to areas of the image that the human eye does not focus on.

[0099] The encoding stage requires downsampling the width and height of the input image features, while the decoding stage requires upsampling to reduce the amount of data and facilitate encoding. However, downsampling using conventional pooling methods inevitably leads to irreversible information loss. Therefore, reducing information loss during the encoding stage can improve the quality of the reconstructed image.

[0100] For the above reasons, this application proposes a reversible subnetwork with a multi-attention mechanism for feature enhancement. After passing through this module, the width and height are downsampled with minimal data loss. The structure of the reversible enhancement subnetwork with the multi-attention mechanism is as follows: Figure 6 As shown, Figure 6 This is a schematic diagram of the reversible enhancement subnetwork of the multi-attention mechanism provided in this application. (See diagram below.) Figure 6 As shown, the reversible enhancement subnetwork sequentially inputs the features to be encoded into the spatial attention subnetwork, the reversible subnetwork, the channel attention subnetwork, and the multi-scale residual subnetwork to enhance the features to be encoded.

[0101] Specifically, Figure 6 For details on the structure of the spatial attention subnetwork, please refer to [link / reference]. Figure 7 , Figure 7 This is a schematic diagram of the spatial attention subnetwork provided in this application. (See diagram below.) Figure 7 As shown, the spatial attention structure consists of a feature extraction layer and several feature transformation layers. By fusing the input features with the feature transformation results of the feature transformation layers, attention to the features of key regions is improved. Here, ⊙ represents the Hada code product, i.e., element-wise dot product.

[0102] The spatial attention subnetwork first needs to extract spatial features along the channel dimension, with a schematic diagram of the features of shape n*h*w as shown below. Figure 8 As shown.

[0103] Assuming the input feature has a shape of n*h*w, where n represents the number of channels, h represents the height, and w represents the width, and the mean of the feature is calculated along the channel dimension, then the output feature has a shape of 1*h*w, where the value in the h-th row and w-th column represents the mean of all elements in all channels. Figure 8 The mean value of medium-dark gray elements.

[0104] The features extracted along the channel dimension include, but are not limited to, the following features: (1) mean; (2) maximum value; (3) minimum value. Feature transformation includes, but is not limited to, the following methods: using a single-layer or multi-layer convolutional neural network.

[0105] Specifically, Figure 6 For details on the structure of the reversible subnetwork, please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of the reversible subnetwork provided in this application. (Example:) Figure 9 As shown, the reversible unit is the basic unit of the reversible subnetwork. Multiple reversible units can be stacked. After passing through the reversible unit, the output features and input features have the same amount of data, only the shape has changed.

[0106] The reversible unit consists of a feature rearrangement layer and a feature coupling layer. Feature rearrangement transforms a feature of shape n*h*w into a feature of shape n`*h`*w`, where n*h*w = n`*h`*w`. Feature coupling combines the features of the previous layer with the features of the current layer.

[0107] Feature rearrangement methods may include, but are not limited to, the following: (1) grouping features along a certain dimension and then rearranging them; (2) unfolding the entire feature into a 1-dimensional form and then rearranging it. Feature coupling methods may include, but are not limited to, the following: using affine transformation (y = a * x + b) for feature coupling.

[0108] Specifically, Figure 6 For details on the structure of the channel attention subnetwork, please refer to [link / reference]. Figure 10 , Figure 10 This is a schematic diagram of the channel attention subnetwork provided in this application. This represents multiplication between channels.

[0109] The channel attention subnetwork first needs to acquire channel features along the width and height dimensions, in a manner similar to... Figure 8 As shown. Assume the shape of the input feature is n*h*w, where n represents the number of channels, h represents the height, and w represents the width. Calculate the mean of the feature along the width and height dimensions. First, calculate the mean along the width dimension, and the output shape is n*h*1. Then, calculate the mean of the feature with shape n*h*1 along the height dimension, and finally output the feature with shape n*1*1.

[0110] Features extracted along the width and height dimensions include, but are not limited to, the following features: (1) mean; (2) minimum; (3) maximum. Feature transformations include, but are not limited to, the following methods: using a single-layer or multi-layer convolutional neural network.

[0111] Specifically, Figure 6 For details on the structure of the multi-scale residual subnetwork, please refer to [link / reference]. Figure 11 , Figure 11 This is a schematic diagram of the structure of the multi-scale residual subnetwork provided in this application. Here, ⊕ represents splicing along the channel dimension.

[0112] Multi-scale residual subnetworks possess the structure of residual networks, where the final output feature equals the input feature plus the output feature of the multi-scale module. Furthermore, they exhibit multi-scale feature characteristics because the input features are transformed at different scales within the multi-scale module and then concatenated into a single feature. Multi-scale methods in multi-scale residual networks include, but are not limited to, the following: the feature transformation module uses different convolutional kernels to perform feature transformations at different scales, and then concatenates these transformations to obtain multi-scale features.

[0113] The following is a specific example to further illustrate the concept. Figure 6 Reversible enhancement subnetworks with multi-attention mechanisms:

[0114] Let the shape of the input features be n*h*w, then the process of the reversible enhancement subnetwork of the multi-attention mechanism is as follows:

[0115] A. The input features are processed through a spatial attention layer to obtain features with spatial attention characteristics. The structure of the spatial attention layer in this embodiment is as follows: Figure 12 As shown.

[0116] B. Use reversible network layers to enhance features, and reduce width and height while increasing the number of feature channels;

[0117] The reversible subnetwork consists of four cascaded sets of reversible subnetwork (INN) units. Each set of reversible subnetwork units differs only in the size of its convolutional kernel. For INN_1, INN_2, INN_3, and INN_4, the kernel sizes are set to 5, 5, 3, and 3, respectively. The structure of the reversible subnetwork unit in this embodiment is as follows: Figure 13 As shown.

[0118] Specifically, the reversible subnetwork unit consists of one feature rearrangement layer and three feature coupling layers, with the three coupling layers using convolutional kernels of the same size. Assuming the input feature shape of the reversible subnetwork unit is n*h*w, the output feature shape after passing through the reversible subnetwork unit is... The process is as follows:

[0119] a) Feature rearrangement

[0120] The width and height of the input feature are halved, but the number of channels is quadrupled. A 1x1 convolution is then performed, and the output feature is denoted as u, with the shape of u as follows:

[0121] b) Feature Coupling

[0122] 1. The feature is divided into two parts, u1 and u2, along the channel dimension.

[0123] Where u1 is the first 1 / 4 channel of u, and u2 is the last 3 / 4 channel of u, that is, the shape of u1 is as follows: The shape of u2 is

[0124] 2. Perform affine coupling based on u1 and u2 to obtain v1 and v2 respectively.

[0125] The affine coupling method is as follows:

[0126]

[0127] Where g2, h2, g1, and h1 represent multiple convolution operations, and the shape after convolution of g2 and h2 is as follows: The shape after convolution of g1 and h1 is: σ c This represents the central sigmoid function. exp(·) represents the exponential function. ⊙ represents the Hada code product (element-wise dot product).

[0128] After affine coupling, the shapes of v1 and v2 are respectively as follows: and

[0129] 3. Concatenate features v1 and v2 along the channel dimension to obtain a new feature, denoted as u.

[0130] 4. Repeat the process more than 3 times.

[0131] After processes a) and b) above, the INN unit transforms the original feature of shape n*h*w into a feature of shape n*h*w. The new feature, therefore, after 4 INN units, the shape of the feature becomes That is, the amount of data did not change, but the width and height were downsampled four times, and the number of channels increased by 256 times.

[0132] C. Channel attention and multi-scale residual layers combine multi-channel features at different scales. The structure of the channel attention and multi-scale residual layers is as follows: Figure 14 As shown.

[0133] The process of channel attention and multi-scale residuals is as follows:

[0134] a) Obtain channel attention weights

[0135] The channel attention weights are of shape n·4 4 *1*1 represents the importance of each channel.

[0136] b) Obtain new channel attention features based on the importance weights of each channel. Multiply the channel attention features by the input features of the multi-scale residual layer and the channel attention weights to obtain new features with channel attention, the shape of which is:

[0137] c) Multi-scale residual blocks combine channel attention features at different scales to shape... By feeding the channel attention features into a multi-scale residual block, channel attention features combining different scales can be obtained.

[0138] d) Number of convolutional recovery channels

[0139] The shape of the feature after the multi-scale residual block is: Then, another 3x3 convolution transforms the shape of the output features into...

[0140] Furthermore, to facilitate feature encoding, a feature cropping / copying process is performed during both the encoding and decoding processes to crop / copy the number of channels. Assuming different quality levels represent end-to-end image encoding at different bitrates, the number of feature channels after feature cropping differs at different quality levels, i.e., the feature cropping coefficient is different, denoted as α. For a feature x input to this module, its shape is n*h*w. The cropped feature y has the following shape:

[0141] The feature trimming module in the encoding process mainly includes two steps:

[0142] a. Rearrange the input features to change their shape.

[0143] b. Perform inter-channel averaging on the rearranged features.

[0144] Similarly, a feature augmentation module is used to recover features during the decoding process. The feature recovery module in the decoding process mainly includes two steps:

[0145] a. Copying the input features.

[0146] b. Rearrange the copied features to restore them to their original shape.

[0147] Examples of trimming the number of channels at the encoding end and increasing the number of channels at the decoding end are detailed in the following embodiments:

[0148] A. Feature trimming during the encoding process

[0149] Assuming the feature clipping factor is α = 6 at a certain quality level, then features with an input shape of n*h*w will be clipped to a shape of... The process of defining the characteristics is as follows:

[0150] a) Rearrangement of input features

[0151] Rearrange the input features into That is, divide the channel into α groups along the channel dimension to obtain the feature x`.

[0152] b) Perform inter-channel averaging on the rearranged features.

[0153] The average value of feature x' along the dimensions of the group is calculated to obtain feature y, and the shape of feature y is then given. This completes the feature trimming.

[0154] B. Feature recovery during the decoding process

[0155] Assuming the feature clipping factor is α = 6 at a certain quality level, the process of restoring a feature x with shape m*h*w to a feature y with shape α·m*h*w is as follows:

[0156] a) Copying input features

[0157] The input feature x is directly copied into a feature x` of shape α·m*h*w.

[0158] b) Rearrange the features after copying

[0159] The feature x` is rearranged to obtain the feature y, and the shape of the feature y is α·n*h*w, which completes the addition of features.

[0160] Step S13: Input the first encoded feature into the entropy model network for feature analysis to obtain the context parameters and probability estimation parameters of the first encoded feature.

[0161] Step S14: Obtain the distribution information of the first encoded feature based on the context parameters and probability estimation parameters.

[0162] In the embodiments of this application, such as Figure 2 As shown, after the feature to be encoded undergoes feature transformation, the image encoding device inputs the obtained first encoded feature into the entropy model network for subsequent feature analysis to obtain the context parameters and probability estimation parameters of the first encoded feature, which are used to predict and estimate the feature distribution, thereby improving the accuracy of feature encoding.

[0163] Specifically, this application proposes a multi-t-distribution mixture probability model (TMM) in the entropy model network. The probability density function of the standard t-distribution is as follows:

[0164]

[0165] The standard t-distribution has an expected value of 0 and a variance of . The distribution, and the probability density curves for different degrees of freedom n are as follows: Figure 15 As shown.

[0166] For the standard t-distribution, the larger the degrees of freedom n, the closer the standard t-distribution is to the standard normal distribution. When the degrees of freedom n = 1, the standard t-distribution degenerates into the standard normal distribution (when the degrees of freedom n = 1, the standard t-distribution degenerates into the Cauchy distribution).

[0167] Since the Gaussian distribution is a special form of the t-distribution, the formula for the Gaussian Mixture Model (GMM) is as follows:

[0168]

[0169] Therefore, the t-distribution mixture model (TMM) can also approximate other distributions. The TMM takes the following form:

[0170]

[0171] Therefore, the distribution of the features to be encoded in the main coding network is estimated using TMM as follows:

[0172]

[0173] Where z represents the feature to be encoded in the entropy model network.

[0174] Furthermore, the specific structure of the entropy model network is as follows: Figure 16 As shown. Among them, This represents splicing along the channel dimension.

[0175] The context module and probability estimation module estimate the distribution of the features to be encoded in the main coding network. The estimation model is the TMM model. Compared with the GMM (Gaussian Mixture Model) model, the TMM model has more parameters, so the estimation effect is more accurate.

[0176] The advantages of using the TMM model compared to the GMM model in this application are:

[0177] (1) The TMM model is more accurate than the GMM in estimating the actual distribution of samples when the sample size is small.

[0178] (2) The TMM model has more parameters and better generalization ability than the GMM model, and the Gaussian distribution itself is a special form of the t distribution.

[0179] like Figure 16 As shown, the process by which the entropy model network estimates the distribution of the features to be encoded in the main coding network is as follows:

[0180] A. In the entropy model network, feature transformation is performed on the features (the transformation process reduces dimensions) to obtain the feature z to be encoded in the entropy model network.

[0181] B. Quantize and entropy encode the feature z.

[0182] Since the entropy encoding of feature z also requires knowledge of the feature distribution, and no other entropy model network estimates the distribution of feature z, it is necessary to make prior assumptions about the distribution of the feature. Therefore, the prior assumptions about the distribution of feature z include, but are not limited to, the following distributions: (1) Gaussian distribution; (2) uniform distribution; (3) t distribution.

[0183] The methods for quantizing features include, but are not limited to, the following: (1) quantization using traditional methods, such as uniform quantization with rounding; (2) quantization using neural network-based methods.

[0184] Entropy coding methods for features include, but are not limited to, the following: (1) arithmetic coding and its variants; (2) Huffman coding.

[0185] C. Decoding the bitstream entropy of feature z and estimating the probability after decoding.

[0186] The features after entropy decoding are called features z`, which, after passing through the probability estimation module, yield the parameters of the TMM model:

[0187] θ1=[ω (k) n (k) ,μ (k) ,σ 2(k) ], k = 1, 2, 3, ..., K

[0188] Where k represents the k-th t-distribution, n (k) ,μ (k) ,σ 2(k) The parameter represents the k-th t-distribution. The TMM model has a total of K t-distributions, ω (k) This represents the weight corresponding to each t-distribution.

[0189] D. The context module calculates the context parameters for the quantized features.

[0190] Since the distribution of each element of feature z directly uses a predefined distribution (i.e., the prior distribution of feature z, such as Gaussian distribution, uniform distribution, t distribution, etc.) when encoding feature z, the distribution of feature y estimated by the C process still does not have the contextual correlation between the elements of feature y (i.e., the context parameter θ2 of feature y, which represents the correlation between the elements in feature y and the surrounding elements, including the spatial correlation within each channel and the correlation between channels).

[0191] The purpose of the context module is to obtain the contextual relevance of the feature, i.e., the context parameter θ2. Based on the context parameter θ2, the parameters obtained in process C can be fine-tuned to obtain better distribution parameters of feature y, which is more conducive to encoding feature y.

[0192] E. Concatenate the results of process C and process D along the channel dimension, and then perform feature transformation to obtain the distribution parameters of the final feature y.

[0193] The formula for calculating the final distribution (parameter θ) of the feature from the results of process C (parameter θ1) and process D (parameter θ2) is as follows:

[0194]

[0195] Where f(·) represents the feature transformation, and k represents the k-th t-distribution. Let represent the parameters of the i-th t-distribution. The TMM model has a total of K t-distributions. This represents the weight corresponding to each t-distribution.

[0196] Therefore, the distribution of features can be represented in the following form:

[0197]

[0198] The entropy model network uses a mixture probability model with multiple t-distributions to estimate the distribution of the features to be encoded in the main coding network, and examples of feature encoding are detailed in the following implementation examples:

[0199] Assuming the shape of the feature y to be encoded in the main coding network is n*h*w, and the number of t-distributions in the mixture t-distribution model is K, the distribution estimation process of the feature y to be encoded in the main coding network using the mixture t-distribution model of the entropy model network is as follows:

[0200] A. The feature transformation module of the entropy model network further reduces the dimensionality of feature y to obtain feature z. The structure of the feature transformation module of the entropy model network is as follows: Figure 17 As shown, Figure 17 In the diagram, `Convs3x3_1` and `Convs3x3_1` represent 3x3 convolutions with strides of 1 and 3, respectively. `ReakyReLU` is the activation function. The shape of the feature z is...

[0201] B. Quantize and entropy encode the feature z.

[0202] The quantization method uses rounding, and the entropy encoding uses a variant of arithmetic encoding: rAns (Range Asymmetric Numeral Systems) encoding, to obtain the feature bitstream.

[0203] C. Obtain the parameter θ1 by decoding the bitstream entropy and estimating the probability of feature z.

[0204] The feature after entropy decoding is feature z`, and its shape is The structure of the probability estimation module of the entropy model network is as follows: Figure 18 As shown in the diagram. Here, Convs3x3 represents a 3x3 convolution with a stride of 1, LeakyReLU represents the activation function, and PixelShuffle represents upsampling.

[0205] The input feature z` is the output feature, which is the parameter θ1, and its shape is 4·K·n*h*w.

[0206] D. The context module calculates the context parameters for the quantized features.

[0207] The network structure of the context module is as follows: Figure 19 As shown, Figure 19 As shown, mask convolution is used to obtain the context parameter θ2 of feature y. The input is the quantized feature y (the quantization method is rounding), and the output feature is the parameter θ2, which has the shape 4·K·n*h*w.

[0208] E. Feature splicing and transformation: After feature transformation, the final distribution parameters θ of the feature y are obtained.

[0209] The network structure of this process is as follows: Figure 20As shown, Convs1x1 represents a 1x1 convolution, and LeakyReLU represents the activation function. Input feature 1 and input feature 2 are the parameters θ1 and θ2 obtained from processes C and D, respectively. The output feature of the network is the distribution parameters θ of the final feature y, with a shape of 4·K·n*h*w.

[0210] F. Encode the quantized feature entropy based on the distribution parameter θ.

[0211] The desired distribution parameter θ has a shape of 4·K·n*h*w, which includes the parameters of the TMM model. The parameter θ is split into n groups along the channel dimension, each group corresponding to a shape of 4·h*w, where K is the number of t-distributions in the TMM model. Therefore, the probability estimate for the feature y to be encoded is as follows:

[0212]

[0213] Among them, y c,i,j θ represents the element in the c-th channel, i-th row, and j-th column of feature y, respectively. 4*c+0,i,j This represents the element in the 4*c+0th channel, in the i-th row and j-th column.

[0214] Next, the rAns encoder is used to encode the features to obtain the feature bitstream.

[0215] Step S15: The first coding feature is encoded using the main coding network according to the distribution information to obtain the feature bitstream of the feature to be encoded.

[0216] In this embodiment, the main coding network uses the distribution information output by the entropy model network to encode the first coding feature, which can improve coding accuracy and coding efficiency according to the distribution of the feature.

[0217] Furthermore, typical end-to-end image coding networks use a single-dimensional evaluation metric to assess the distortion of the reconstructed image, and this metric is usually an objective one. Objective evaluation metrics include, but are not limited to, the following methods:

[0218] (1) MSE (Mean Square Error).

[0219]

[0220] (2) MS-SSIM (Multi-Scale Structural Similarity Index Metric) multi-scale structural similarity. The specific calculation process of MS-SSIM is as follows: Figure 21 As shown.

[0221] This application uses multiple evaluation metrics to assess the quality of reconstructed images, such as the following two types: objective evaluation metrics and subjective evaluation metrics. Subjective evaluation metrics involve designing experiments based on a complete set of rules and assembling a team of testers to score the video quality. Objective evaluation metrics, on the other hand, are often obtained through mathematical formulas to quantify the video quality.

[0222] Subjective evaluation indicators include, but are not limited to, the following methods:

[0223] (1) LPIPS (Learned Perceptual Image Patch Similarity) is a perceptual similarity method that feeds the reconstructed image and the original image into a pre-trained LPIPS evaluation index network for quality evaluation.

[0224] (2) FID (Fréchet Inception Distance), sensing distance:

[0225]

[0226] Where μ x and Σ x Tr(·) represents the mean and covariance matrix of the n-dimensional features of the real image set output by the Inception v3 network, and Tr(·) represents the rank of the matrix.

[0227] The subjective and objective evaluation indicators are balanced using a weight 'a', as shown in the following formula:

[0228]

[0229] Among them, metic a (·) represents objective evaluation indicators, metic b (·) represents subjective evaluation indicators.

[0230] based on Figure 2 The end-to-end image coding network shown below details the specific loss design during training, as illustrated in the following example:

[0231] like Figure 2 The network shown uses LPIPS distortion and MSE distortion as measures of distortion in the reconstructed image during training, and the loss is as follows:

[0232]

[0233] in, This represents the code rate of the main coding network and the entropy model network for feature encoding. Measure the distortion of the reconstructed image compared to the original image. It includes two types of distortion, MSE distortion and LPIPS distortion. By balancing MSE distortion and LPIPS distortion through hyperparameters, the trained neural network model can reduce both MSE distortion and perceptual distortion when reconstructing images.

[0234] This application proposes a reversible enhancement subnetwork with a multi-attention mechanism, which reduces information loss during the encoding process and the encoded features have multi-attention and multi-scale characteristics; it proposes a multi-t distribution hybrid model (TMM), which theoretically can more accurately estimate the distribution of the features to be encoded in the main encoding network; and it adds LPIPS distortion metric to the training loss, combining objective and subjective distortion to improve the quality of the reconstructed image.

[0235] In contrast, please continue reading Figure 22 , Figure 22 This is a flowchart illustrating an embodiment of the image decoding method provided in this application.

[0236] like Figure 22 As shown, the image decoding method of this application embodiment includes the following steps:

[0237] Step S21: Input the feature code stream.

[0238] Step S22: The main coding network decodes the feature bitstream according to the preset distribution information to obtain the first decoded feature of the feature bitstream. The preset distribution information is determined by the entropy model network during the encoding process.

[0239] Step S23: Input the first decoded feature into the main coding network for feature reconstruction to obtain the second decoded feature.

[0240] Step S24: Reconstruct the image according to the second decoding feature to obtain the decoded image.

[0241] It should be noted that the image decoding method of this application embodiment is essentially the reverse process of the image encoding method of the above embodiment. Therefore, all the technical solutions of the image encoding method can be applied to the image decoding method of this embodiment without any doubt. The relevant technical solutions can be derived through simple reverse engineering, which will not be elaborated here.

[0242] The above embodiments are merely one common example of this application and do not constitute any limitation on the technical scope of this application. Therefore, any minor modifications, equivalent changes, or alterations made to the above content based on the substance of the solution of this application shall still fall within the scope of the technical solution of this application.

[0243] Please continue reading Figure 23 , Figure 23This is a schematic diagram of an embodiment of the image encoding apparatus provided in this application. The image encoding apparatus 500 of this application embodiment includes a processor 51, a memory 52, an input / output device 53, and a bus 54.

[0244] The processor 51, memory 52, and input / output device 53 are respectively connected to the bus 54. The memory 52 stores program data, and the processor 51 is used to execute the program data to implement the image encoding method described in the above embodiments.

[0245] In this embodiment, processor 51 can also be referred to as a CPU (Central Processing Unit). Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 51 can be any conventional processor.

[0246] Please continue reading Figure 24 , Figure 24 This is a schematic diagram of an embodiment of the image decoding apparatus provided in this application. The image encoding apparatus 600 of this application embodiment includes a processor 61, a memory 62, an input / output device 63, and a bus 64.

[0247] The processor 61, memory 62, and input / output device 63 are respectively connected to the bus 64. The memory 62 stores program data, and the processor 61 is used to execute the program data to implement the image decoding method described in the above embodiments.

[0248] This application also provides a computer storage medium; please refer to the following: Figure 25 , Figure 25 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 700 stores program data 71, which, when executed by a processor, is used to implement the image encoding method and / or image decoding method of the above embodiments.

[0249] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0250] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image encoding method, characterized in that, The image encoding method includes: Input the image to be encoded, and extract the features to be encoded from the image; The feature to be encoded is input into the main encoding network for feature transformation to obtain the first encoded feature; The first encoded feature is input into the entropy model network for feature analysis to obtain the context parameters and probability estimation parameters of the first encoded feature; Based on the context parameters and the probability estimation parameters, the distribution information of the first encoded feature is obtained; The first coding feature is encoded using the main coding network according to the distribution information to obtain the feature bitstream of the feature to be encoded; The step of inputting the first encoded feature into an entropy model network for feature analysis to obtain the context parameters and probability estimation parameters of the first encoded feature includes: quantizing the first encoded feature to obtain a second encoded feature; sequentially entropy encoding and entropy decoding the second encoded feature to obtain a third encoded feature; performing probability estimation based on the third encoded feature through a probability estimation module to obtain the probability estimation parameters of the first encoded feature; obtaining the contextual correlation between the second encoded feature and adjacent encoded features, and obtaining the context parameters of the first encoded feature based on the contextual correlation. The step of obtaining the distribution information of the first encoded feature based on the context parameter and the probability estimation parameter includes: concatenating the context parameter and the probability estimation parameter along the channel dimension; and performing feature transformation on the concatenation result to obtain the distribution information of the first encoded feature.

2. The image encoding method according to claim 1, characterized in that, Before quantizing the first encoded feature to obtain the second encoded feature, the image encoding method further includes: The first encoded feature is subjected to feature transformation to obtain the feature to be encoded of the entropy model network, wherein the feature dimension of the feature to be encoded is smaller than the feature dimension of the first encoded feature.

3. The image encoding method according to claim 1, characterized in that, After the image encoding method further includes encoding the first encoding feature according to the distribution information using the main coding network to obtain the feature bitstream of the feature to be encoded, the method further includes: The feature code stream is decoded to obtain the reconstructed image; Obtain evaluation metrics in multiple dimensions for the reconstructed image and the image to be encoded; The evaluation metrics are used to train the main encoding network and the entropy model network.

4. The image encoding method according to claim 3, characterized in that, The evaluation metrics include at least one of the perceptual similarity method and the perceptual distance method.

5. The image encoding method according to claim 1, characterized in that, The step of inputting the feature to be encoded into the main coding network for feature transformation to obtain the first encoded feature includes: The feature to be encoded is input into the main encoding network and processed sequentially through one or more of the following sub-networks: preprocessing sub-network, feature enhancement sub-network, and feature pruning sub-network, to obtain the first encoded feature.

6. The image encoding method according to claim 5, characterized in that, The preprocessing subnetwork implements preprocessing functions by employing several convolutional layers and / or several residual blocks. The preprocessing functions increase the nonlinearity of the input features to be encoded and output preprocessed features to be encoded.

7. The image encoding method according to claim 5, characterized in that, The feature enhancement subnetwork includes one or more of the following: spatial attention subnetwork, invertible subnetwork, channel attention subnetwork, and multi-scale residual subnetwork. The feature enhancement subnetwork is used to input the feature to be encoded, or the preprocessed feature to be encoded output by the preprocessing subnetwork, and to sequentially input the feature to be encoded or the preprocessed feature to be encoded into one or more of the spatial attention subnetwork, the invertible subnetwork, the channel attention subnetwork, and the multi-scale residual subnetwork to enhance the feature to be encoded or the preprocessed feature to be encoded, and output the feature-enhanced feature to be encoded.

8. The image encoding method according to claim 7, characterized in that, The reversible subnetwork consists of at least one reversible unit, wherein the reversible unit includes a feature rearrangement layer and a feature coupling layer; the feature rearrangement layer is used to change the feature shape of the feature to be encoded, and the feature coupling layer is used to combine the features of the previous layer with the features of the current layer.

9. The image encoding method according to claim 5, characterized in that, The feature pruning subnetwork is used to rearrange the input features to be encoded, or the preprocessed features to be encoded output by the preprocessing subnetwork, or the enhanced features to be encoded output by the feature enhancement subnetwork, to obtain rearranged features. The average feature channel dimension of the rearranged features is calculated using a preset pruning coefficient, so as to obtain the pruning features of the rearranged features according to the average feature channel dimension, thereby realizing the feature pruning function and outputting the first encoded feature.

10. An image decoding method, characterized in that, The image decoding method includes: Input feature bitstream; The main coding network decodes the feature bitstream according to preset distribution information to obtain the first decoded feature of the feature bitstream. The preset distribution information is determined by the entropy model network during the encoding process. The entropy model network is used to perform feature analysis on the features input to the entropy model network during the encoding process to obtain context parameters and probability estimation parameters. The preset distribution information is obtained based on the context parameters and the probability estimation parameters. The first decoding feature is input into the main coding network for feature reconstruction to obtain the second decoding feature; The decoded image is obtained by reconstructing according to the second decoding feature; The entropy model network is used to quantize the features input to the entropy model network during the encoding process to obtain a second encoded feature; the second encoded feature is then entropy encoded and entropy decoded sequentially to obtain a third encoded feature; a probability estimation module performs probability estimation based on the third encoded feature to obtain probability estimation parameters for the first encoded feature; the contextual relevance between the second encoded feature and adjacent encoded features is obtained, and the contextual parameters of the first encoded feature are obtained based on the contextual relevance; the contextual parameters and the probability estimation parameters are concatenated along the channel dimension; and the concatenation result is subjected to feature transformation to obtain the distribution information of the first encoded feature.

11. An image encoding device, characterized in that, The image encoding device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image encoding method as described in any one of claims 1 to 9.

12. An image decoding device, characterized in that, The image decoding device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image decoding method as described in claim 10.

13. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the image encoding method as described in any one of claims 1 to 9 and / or the image decoding method as described in claim 10.

Images