Image encoding method, image decoding method, apparatus, and computer storage medium
Patent Information
- Application Number
- CN202211598629.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-12-09
- Publication Date
- 2026-10-27
- Estimated Expiration
- 2042-12-09
AI Technical Summary
[0003]然而，目前的特征编码算法缺乏对全局信息的提取和利用，导致无法提取紧凑的特征表示
[0067]本申请的有益效果是：图像编码装置输入待编码图像，并利用主编码网络中变换模块的第一分支提取待编码图像的待编码图像特征和第一编码特征；利用所述变换模块的第二分支提取待编码图像特征的全局信息，基于全局信息获取第一编码特征的全局相关性因子；利用全局相关性因子对第一编码特征进行调制，获取调制后的第二编码特征；将第二编码特征输入第一熵模型网络进行特征分析，获取第二编码特征的分布信息；采用主编码网络的编码模块根据分布信息对第二编码特征进行编码，得到待编码图像的特征码流。本申请的图像编码方法通过采用基于全局信息指导的变换网络，该网络采用双分支结构充分利用输入特征的全局相关性，提取紧凑的特征表示。
Smart Images
Figure CN116016954B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of feature encoding technology, and in particular to an image encoding method, an image decoding method, an apparatus, and a computer storage medium. Background Technology
[0002] Traditional image coding techniques are designed for human vision. However, with the superior performance of deep neural networks in various machine vision tasks, such as image classification, object detection, and semantic segmentation, numerous AI applications based on machine vision have emerged. To ensure that the performance of machine vision tasks is not compromised by the image coding process, a "analysis-then-encode" approach is adopted to address machine vision requirements. This involves directly extracting features from the lossless image using a neural network at the image acquisition end, then encoding and transmitting the extracted features. The decoding end directly uses the decoded features as input to subsequent network structures to complete different machine vision tasks. Therefore, to conserve transmission bandwidth resources, it is necessary to research image coding methods specifically for machine vision.
[0003] However, current feature encoding algorithms lack the ability to extract and utilize global information, resulting in the inability to extract compact feature representations. Summary of the Invention
[0004] This application provides an image encoding method, an image decoding method, an apparatus, and a computer storage medium.
[0005] One technical solution adopted in this application is to provide an image encoding method, the image encoding method comprising:
[0006] Input the image to be encoded, and use the first branch of the transform module in the main coding network to extract the image features to be encoded and the first coding features of the image to be encoded;
[0007] The global information of the features of the image to be encoded is extracted using the second branch of the transformation module, and the global correlation factor of the first encoded feature is obtained based on the global information;
[0008] The first encoded feature is modulated using the global correlation factor to obtain the modulated second encoded feature;
[0009] The second encoded feature is input into the first entropy model network for feature analysis to obtain the distribution information of the second encoded feature;
[0010] The encoding module of the main encoding network encodes the second encoding feature according to the distribution information to obtain the feature bitstream of the image to be encoded.
[0011] The step of extracting global information of the features of the image to be encoded using the second branch of the transformation module includes:
[0012] The spatial attention weights of the image features to be encoded and the channel dimension attention weights of the image features to be encoded are obtained using the second branch of the transformation module.
[0013] Using the spatial attention weights, the third encoded features of the image to be encoded are obtained by non-local computation in the spatial domain;
[0014] Using the channel-dimensional attention weights, the fourth encoded feature of the image to be encoded is obtained by non-local computation in the channel dimension;
[0015] The third and fourth coding features are fused to extract global information of the features of the image to be encoded.
[0016] The step of obtaining the spatial attention weights of the features of the image to be encoded includes:
[0017] The first feature representation of the image to be encoded is obtained by using the first convolution;
[0018] The second feature representation of the first feature representation is obtained by using the second convolution;
[0019] By applying different dimensionality adjustments to the second feature representation, we obtain the third and fourth feature representations;
[0020] Perform matrix operations on the third feature representation and the fourth feature representation to obtain the spatial attention weights of the features of the image to be encoded;
[0021] The step of using the spatial attention weights to obtain the third encoded feature, which is obtained by non-local computation of the features of the image to be encoded in the spatial domain, includes:
[0022] Perform matrix operations on the image features to be encoded and the spatial attention weights to obtain a first weighted feature representation;
[0023] The first weighted feature representation is fused with the features of the image to be encoded to obtain the third encoded feature.
[0024] The step of obtaining the channel dimension attention weights of the features to be encoded includes:
[0025] The fifth feature representation of the first image feature to be encoded is obtained using the third convolution;
[0026] By applying different dimensional adjustment methods to the fifth feature representation, the sixth and seventh feature representations are obtained;
[0027] Perform matrix operations on the sixth feature representation and the seventh feature representation to obtain the channel dimension attention weights of the image features to be encoded;
[0028] The step of obtaining the fourth encoded feature, obtained by non-local computation of the image features to be encoded in the channel dimension using the channel-dimensional attention weights, includes:
[0029] The eighth feature representation of the image to be encoded is obtained using the fourth convolution;
[0030] Perform a matrix operation on the eighth feature representation and the channel dimension attention weight to obtain the second weighted feature representation;
[0031] The second weighted feature representation is fused with the features of the image to be encoded to obtain the fourth encoded feature.
[0032] The global relevance factor includes a masking factor and an offset factor;
[0033] The step of obtaining the global relevance factor of the first encoded feature based on the global information includes:
[0034] The global information is convolved using the fifth convolution to obtain the mask factor of the first encoded feature;
[0035] The global information is convolved using the sixth convolution to obtain the offset factor of the first encoded feature;
[0036] The step of modulating the first encoded feature using the global correlation factor to obtain the modulated second encoded feature includes:
[0037] The result of multiplying the first coding feature with the feature of the masking factor is added to the feature of the offset factor to obtain the modulated second coding feature.
[0038] The step of inputting the second encoded feature into the first entropy model network for feature analysis to obtain the distribution information of the second encoded feature includes:
[0039] The second encoded feature and the first auxiliary inverse transform result of the first entropy model network on the second encoded feature are input into the context predictor of the first entropy model network to obtain the context information of the second encoded feature;
[0040] The context information is input into the first probability model of the first entropy model network to obtain the distribution information output by the first probability model.
[0041] The step of inputting the second encoded feature and the first auxiliary inverse transform result of the first entropy model network on the second encoded feature into the context predictor of the first entropy model network to obtain the context information of the second encoded feature includes:
[0042] Obtain the encoded feature map of the second encoded feature;
[0043] The first estimation result of the encoded feature map is extracted by activating the convolution in the first half.
[0044] The prediction network in the context predictor is used to extract the first prediction result of the second encoded feature;
[0045] The second estimation result of the first prediction result is extracted by activating the convolution in the latter half.
[0046] The first estimation result and the second estimation result are fused to obtain the neighborhood range result of the second encoded feature;
[0047] The first auxiliary inverse transform result is fused with the neighborhood range result to obtain the context information of the second encoded feature.
[0048] Specifically, the weights of the first half of the convolution center point of the first half activation convolution are not 0, and the weights of the second half are 0; the weights of the first half of the convolution center point of the second half activation convolution are 0, and the weights of the second half are not 0.
[0049] The image encoding method further includes:
[0050] The second encoded feature is input into the auxiliary transform module of the first entropy model network to obtain the fifth encoded feature;
[0051] The predictor of the second entropy model network is used to obtain the prediction information of the fifth encoded feature;
[0052] The predicted information is input into the second probability model of the second entropy model network to obtain the distribution information of the fifth encoded feature;
[0053] The distribution information of the fifth coding feature is used to encode and decode the fifth coding feature to obtain the first auxiliary inverse transform result.
[0054] The step of using the predictor of the second entropy model network to obtain the prediction information of the fifth encoded feature includes:
[0055] The prediction network in the predictor of the second entropy model network is used to extract the second prediction result of the fifth encoded feature;
[0056] The second prediction result is fused with the second auxiliary inverse transform result of the second entropy model network on the fifth coding feature to obtain the prediction information of the fifth coding feature.
[0057] Another technical solution adopted in this application is to provide an image decoding method, the image decoding method comprising:
[0058] Input feature bitstream;
[0059] The feature bitstream is decoded using a master coding network based on distribution information to obtain the first decoded feature of the feature bitstream;
[0060] The main coding network is used to demodulate the first decoding feature according to a preset global correlation factor to obtain the demodulated second decoding feature;
[0061] The decoded image is obtained by reconstructing according to the second decoding feature.
[0062] Another technical solution adopted in this application is to provide an image encoding device, which includes a memory and a processor coupled to the memory;
[0063] The memory is used to store program data, and the processor is used to execute the program data to implement the image encoding method described above.
[0064] Another technical solution adopted in this application is to provide an image decoding device, which includes a memory and a processor coupled to the memory;
[0065] The memory is used to store program data, and the processor is used to execute the program data to implement the image decoding method described above.
[0066] Another technical solution adopted in this application is to provide a computer storage medium for storing program data, which, when executed by a computer, is used to implement the image encoding method and / or image decoding method as described above.
[0067] The beneficial effects of this application are as follows: The image encoding device inputs an image to be encoded and extracts the image features to be encoded and the first encoding features of the image using the first branch of the transform module in the main encoding network; it extracts the global information of the image features to be encoded using the second branch of the transform module, and obtains the global correlation factor of the first encoding features based on the global information; it modulates the first encoding features using the global correlation factor to obtain the modulated second encoding features; it inputs the second encoding features into the first entropy model network for feature analysis to obtain the distribution information of the second encoding features; and it uses the encoding module of the main encoding network to encode the second encoding features according to the distribution information to obtain the feature bitstream of the image to be encoded. The image encoding method of this application adopts a transform network guided by global information. This network uses a dual-branch structure to fully utilize the global correlation of the input features and extract a compact feature representation. Attached Figure Description
[0068] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0069] Figure 1 This is a schematic diagram of the basic structure of an end-to-end image codec;
[0070] Figure 2 This is a schematic diagram of the basic structure of a neural image codec;
[0071] Figure 3 This is a flowchart illustrating an embodiment of the image encoding method provided in this application;
[0072] Figure 4 This is a simplified structural diagram of the image end-to-end codec provided in this application;
[0073] Figure 5 yes Figure 3 The diagram shows the specific flow chart of step S12 in the image encoding method shown.
[0074] Figure 6 This application provides a two-branch transformation structure guided by global information.
[0075] Figure 7 This is a schematic diagram of the nonlocal computational structure of the spatial domain provided in this application;
[0076] Figure 8 This is a schematic diagram of the nonlocal computation structure of the channel dimension provided in this application;
[0077] Figure 9This is a schematic diagram of the fusion structure of spatial domain information and channel dimension information provided in this application;
[0078] Figure 10 yes Figure 3 The diagram shows the specific flow chart of step S14 in the image encoding method shown.
[0079] Figure 11 This is a schematic diagram of the structure of the context predictor that combines predicted values provided in this application;
[0080] Figure 12 This is the 3D mask convolution unroll diagram of the first half of the activated part provided in this application;
[0081] Figure 13 This is the 3D mask convolution unfolded diagram of the activated latter half (including the center point) provided in this application;
[0082] Figure 14 This is a schematic diagram of the predictor that combines predicted values provided in this application;
[0083] Figure 15 This is a flowchart illustrating an embodiment of the image decoding method provided in this application;
[0084] Figure 16 This is a schematic diagram of an embodiment of the image encoding device provided in this application;
[0085] Figure 17 This is a schematic diagram of an embodiment of the image decoding device provided in this application;
[0086] Figure 18 This is a schematic diagram of the structure of an embodiment of the computer storage medium provided in this application. Detailed Implementation
[0087] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0088] The Future-Oriented Video Coding Learning Group (MEG) is an organization under the IEEE Data Compression Standards Committee (DCSC) dedicated to developing neural network-based image coding standards. Currently, the organization has released reference software for an image codec that demonstrates superior performance compared to current state-of-the-art image codec standards, including JPEG, BPG, and VVC. This image codec, also known as the Neural Image Caption (NIC), is essentially an end-to-end image codec entirely constructed from neural networks. Its key feature is the joint optimization of the various modules within the codec. The basic structure of an end-to-end image codec is as follows: Figure 1 As shown, it includes a main encoder network and an entropy model network. Under the main encoder, there are transform and inverse transform, quantization and dequantization, entropy coding and entropy decoding.
[0089] (1) The transformation mainly uses convolutional neural networks for nonlinear downsampling, which is to use a more compact expression to represent the main features of the original image and reduce the image dimension and data volume. The inverse transformation is to recover the original image from the compact expression.
[0090] (2) Quantization is one of the lossy coding steps. It converts the data into integers to improve the compression rate. Dequantization (optional) is the opposite operation, but it can also be omitted because the powerful nonlinear ability of neural networks can include the effect of dequantization.
[0091] (3) Entropy coding is a lossless process. It calculates the probability of the sign bit in each feature by constructing a probability model and encodes it into a binary representation and writes it into the code stream. Entropy decoding is the reverse process.
[0092] The entropy model network includes auxiliary transforms and inverse auxiliary transforms, quantization and dequantization, entropy encoding and entropy decoding, and the construction of a probabilistic model. Except for the probabilistic model construction, the functions of the other modules are the same as the main encoding network type. The probabilistic model construction mainly involves learning model parameters through a neural network to calculate the probability of the features to be encoded by the main encoding network.
[0093] Currently, with the continuous advancement of the standard, the NIC has further developed its basic structure. The current standard has improved upon existing modules and added new modules including multi-entropy model networks, context models, preprocessing, entropy coding acceleration, and post-processing enhancements. A schematic diagram is shown below. Figure 2 As shown, specifically:
[0094] (1) Preprocessing: The image is divided into 512*512 blocks, and the input image is vertically flipped and rotated to generate 8 copies. Each copy is input into the network framework, and its rate distortion RD is calculated. The index with the smallest RD is recorded.
[0095] (2) Entropy coding accelerates the encoding process by encoding only the valid channels, i.e. the feature channels that are not all zeros, and encoding them by indexing them.
[0096] (3) Multi-entropy model network: Add an entropy model network 2 to fit the probability model of entropy model network 1.
[0097] (4) Context Predictor. This part contains a context model and a prediction network:
[0098] a) Context model. Since each feature point to be encoded depends on the previous encoded feature point, the context model can learn their correlations and reduce redundancy.
[0099] b) Prediction Network. Let the output of the transform network in the master encoder-decoder network be the main latent feature representation. The prediction network aims to estimate the predicted value of this latent representation and calculate the difference between it and the predicted value. It then encodes and decodes the residual.
[0100] (5) Predictor. It only contains the prediction network, which aims to estimate the latent feature representation of the auxiliary transformation network in order to encode the residuals. It does not contain the context model.
[0101] (6) Post-processing: The block effect, artifacts and ringing effect after reconstruction are eliminated by pre-trained neural network-based enhancement modules.
[0102] The loss function used in the joint optimization of NICs is as follows:
[0103]
[0104] in and These represent the main encoding network, entropy model network 1, and entropy model network 2, respectively, representing the features to be encoded. and Bitrate estimation, Used to measure the original image x and the reconstructed image Distortion is typically addressed using mean squared error or structural similarity, with λ used to balance bit rate and distortion.
[0105] This application is in Figure 1 The basic structure of the image end-to-end codec shown is as follows: Figure 2Based on the neural image codec shown, this paper proposes to adopt a dual-branch network structure in the main encoding / decoding network of the NIC, which adaptively utilizes spatial correlation in a local-to-global manner to remove redundant information, obtain compact feature representations, and improve encoding / decoding performance. It also proposes to utilize information within the surrounding neighborhood of the point to be encoded / decoded as much as possible in the context predictor of the NIC to predict the information of the point at that location, thereby improving encoding / decoding performance.
[0106] Please refer to the details. Figure 3 and Figure 4 , Figure 3 This is a flowchart illustrating an embodiment of the image encoding method provided in this application. Figure 4 This is a simplified structural diagram of the image end-to-end codec provided in this application.
[0107] This application proposes an efficient end-to-end image compression method, which, in the above... Figure 2 Based on the NIC shown, the main improvements include transform / inverse transform, auxiliary transform / inverse auxiliary transform, and context model. Specifically, as follows... Figure 4 As shown, Figure 4 This describes the overall framework of the image end-to-end codec proposed in this application:
[0108] (1) Transformation / inverse transformation: A dual-branch structure is adopted to adaptively learn the correlation of the neighborhood from local to global.
[0109] (2) Context predictor: Combines the context model of the predicted value, so that the point to be encoded / decoded can simultaneously refer to the context information before and after its position, thereby improving the encoding performance.
[0110] (3) Predictor. The output of the joint prediction network and the result of the auxiliary inverse transformation generate the parameters of the probability model.
[0111] The following is combined with Figure 3 The image encoding method embodiment shown above elaborates on the main improvements mentioned above:
[0112] like Figure 3 As shown, the image encoding method of this application embodiment includes the following steps:
[0113] Step S11: Input the image to be encoded, and use the first branch of the transform module in the main coding network to extract the image features to be encoded and the first coding features of the image to be encoded.
[0114] In this embodiment of the application, the image encoding device acquires the image to be encoded, and then extracts the image features to be encoded and the first encoding features of the image.
[0115] Step S12: Use the second branch of the transform module to extract global information of the features of the image to be encoded, and obtain the global correlation factor of the first encoded feature based on the global information.
[0116] In this embodiment, the image encoding device extracts global information of the features of the image to be encoded. The extraction methods or structures for this global information include, but are not limited to, the Transformer structure and other mature global feature extraction networks. The image encoding device uses this global information to guide the subsequent encoding process, thereby obtaining a compact feature representation and achieving a lower encoded bitstream size.
[0117] Specifically, such as Figure 4 As shown, this application employs a parallel dual-branch structure to extract global information in the NIC's transform / inverse transform module. Please continue reading. Figure 5 and Figure 6 , Figure 5 yes Figure 3 The diagram shows the detailed process of step S12 in the image encoding method. Figure 6 This application provides a bi-branch transformation structure based on global information guidance.
[0118] like Figure 6 As shown, the bi-branch transform structure guided by global information includes a main branch and a Mask branch. The main branch mainly contains multiple downsampling stages, performing nonlinear transformation and dimensionality reduction on the input image. The Mask branch mainly utilizes global information in the spatial domain and global information in the channel dimension to modulate and compensate the encoded features of each downsampling stage in the main branch.
[0119] Specifically, Figure 6 The dual-branch structure shown mainly includes the following modules:
[0120] (1) The main branch contains multiple convolutions, residual blocks, and M sets of sampling modules, where each set of sampling modules includes:
[0121] (a) Sampling operations: These are either upsampling (dimensionality increase) or downsampling (dimensionality reduction) operations. Downsampling includes, but is not limited to, strided convolution, pooling, or pixel recombination, while upsampling includes, but is not limited to, deconvolution or subpixel convolution. In image encoding, sampling operations are downsampling operations; in image decoding, sampling operations are upsampling operations.
[0122] (b) Nonlinear transformation: After each sampling operation, a nonlinear transformation is performed. Nonlinear transformation methods include, but are not limited to, ordinary residual blocks, dense residual blocks, and separable convolutional residual blocks.
[0123] (2) The Mask branch contains a joint global information extraction module and a downsampling module:
[0124] Global information extraction module: Obtains the global relevance of the input image, i.e. global information. Extraction methods include, but are not limited to, dual-attention-based bi-branch structures, Transformer structures, etc.
[0125] For example, in the dual-attention-based dual-branch structure of the global information extraction module, one branch mainly performs non-local computation in the spatial domain, while the other branch performs non-local computation in the channel dimension. Let the input dimension be H×W×C. Then, the global information extraction module consists of three parts: a non-local computation module in the spatial domain, a non-local computation module in the channel dimension, and a fusion module. The specific working process of the global information extraction module will be further described below, using step S12 as an example:
[0126] Step S121: Obtain the spatial attention weights of the features of the image to be encoded, and obtain the channel dimension attention weights of the features of the image to be encoded.
[0127] In this embodiment, please refer to the nonlocal computational structure diagram of the spatial domain. Figure 7 ,like Figure 7 As shown, the image encoding device acquires an input with dimensions H×W×C.
[0128] Then, the image encoding device first performs spatial or channel-wise dimensionality reduction or nonlinear mapping on the input using convolutions with different receptive fields (K×K and K0×K0) to obtain three feature representations.
[0129] Specifically, such as Figure 7 As shown, the image encoding device performs convolution processing on the features of the image to be encoded using a first convolution (K×K, stride s) to obtain a first feature representation with a feature dimension of H / s×W / s×C. Then, the image encoding device performs convolution processing on the first feature representation using a second convolution (K0×K0, stride s1) to obtain a second feature representation with a feature dimension of H / s×W / s×C / s1. Here, s and s1 both represent dimensionality reduction factors.
[0130] Furthermore, for Figure 7 The first two feature representations, namely the second feature representation, are adjusted in different dimensions to obtain the third feature representation (H / s×W / s)×C / s1 and the fourth feature representation C / s1×(H / s×W / s). The image coding device performs matrix multiplication on the third and fourth feature representations to output the attention weight of each pixel in the first coding feature, with a dimension of (H / s×W / s)×(H / s×W / s).
[0131] Step S122: Using spatial attention weights, obtain the third encoded feature obtained by non-local computation of the features of the image to be encoded in the spatial domain.
[0132] In this embodiment of the application, the image encoding device performs matrix multiplication on the image features to be encoded in step S121 and the attention weights, that is, each new pixel of the image features to be encoded is generated by weighting all pixels in the spatial location to obtain the first weighted feature representation, whose feature dimension is: H / s×W / s×C.
[0133] Furthermore, the image encoding device further transforms the first weighted feature representation through a transformation operation (dimensional transformation or nonlinear mapping) to obtain a feature (H×W×C) with the same dimension as the input image feature to be encoded, and adds the feature to the image feature to be encoded to obtain the third encoded feature obtained by nonlocal computation of the image feature in the spatial domain.
[0134] Step S123: Utilize channel-dimensional attention weights to obtain the fourth encoded feature obtained by non-local computation of the image features to be encoded in the channel dimension.
[0135] In this embodiment, please refer to the nonlocal computational structure diagram of the channel dimension. Figure 8 ,like Figure 8 As shown, the image encoding device acquires an input with dimensions H×W×C.
[0136] Then, the image encoding device first uses two third convolutions (K1×K1, K1>1) to perform spatial dimensionality reduction or nonlinear transformation on the input to obtain two feature representations, namely the fifth feature representations, each with a feature dimension of H / s2×W / s2×C, where s2 represents the dimensionality reduction factor.
[0137] In another branch, the image coding device uses a fourth convolution (K2×K2) to perform a nonlinear transformation on the input to obtain an eighth feature representation (H×W×C), while keeping the feature dimensions unchanged.
[0138] It is C×C.
[0139] In this embodiment, the image encoding device performs matrix multiplication of the eighth feature representation with the attention weights of the channel dimension, that is, each new pixel of the first encoding feature is generated by weighting all pixels in its channel to obtain the second weighted feature representation, whose feature dimension is: H×W×C.
[0140] Furthermore, the image encoding device adds the first weighted feature to the image feature to be encoded to obtain the fourth encoded feature obtained by non-local computation of the image feature in the channel dimension.
[0141] Step S124: Perform feature fusion on the third and fourth coding features to extract global information of the features of the image to be encoded.
[0142] In the embodiments of this application, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the fusion structure of spatial and channel dimension information provided in this application. The image encoding device concatenates the nonlocal computation results of the spatial domain and the nonlocal computation results of the channel dimension from the above steps at the channel level, and fuses them through convolution. It then adds the input to construct residual blocks, promoting information flow. This is equivalent to each output pixel being generated by a weighted average of all pixels in its spatial domain and channel, fully utilizing the global correlation between pixels.
[0143] In this embodiment of the application, a parallel structure based on dual attention is proposed in the transform network to fully consider global information in both spatial and channel dimensions, thereby improving coding performance.
[0144] Step S13: Modulate the first coding feature using the global correlation factor to obtain the modulated second coding feature.
[0145] In this embodiment, the global information calculated in step S12 can be characterized by a global correlation factor to represent the modulation effect on the first coding feature in the main branch. Specifically, the global correlation factor in this application includes a mask factor and an offset factor.
[0146] Please continue reading. Figure 6 In the main branch, the image to be encoded undergoes convolution and several residual blocks. The features of this image are then input into the Mask branch to extract global information. The Mask branch processes this global information through its sampling module, followed by convolution processing by the modulation module to generate corresponding global correlation factors. Additionally, in the main branch, the image to be encoded undergoes convolution, several residual blocks, and a sampling module to obtain the first encoded features.
[0147] Specifically, the sampling modules in the Mask branch and the main branch are of similar type. Global information is sequentially processed through M sets of corresponding sampling modules for dimensionality reduction (or dimensionality increase) and nonlinear transformation. The modulation module provides two sets of convolutions. The global information output by each sampling module is processed through the two sets of convolutions to generate two factors: a mask factor and an offset factor. The mask factor is multiplied by the first encoded feature output by each sampling module in the main branch and modulated by the offset factor to obtain the modulated second encoded feature.
[0148] In one specific implementation, Figure 6 The application of a globally information-guided bi-branch transform structure to Figure 4 In the transformation network, the specific configuration is as follows:
[0149] (a) The input of the bi-branch transform structure is the image to be encoded, and the output is the latent feature representation.
[0150] (b) The sampling in the sampling module is a downsampling operation, which is a stride convolution with a stride of 2, that is, the spatial dimension is 1 / 2 of the original.
[0151] (c) In the nonlocal computation of the spatial domain, to control the complexity, K=5, K0=1, s=2, s1=2, that is, the spatial dimension and channels are reduced to 1 / 2 of the original. At the same time, it is transformed into deconvolution upsampling.
[0152] (d) In the nonlocal computation of the channel dimension, K1 = 5, K1 = 1, s2 = 2, that is, the spatial dimension is reduced to 1 / 2 of the original.
[0153] In one specific implementation, Figure 6 The application of a globally information-guided bi-branch transform structure to Figure 4 In the inverse transform network, which is the opposite of the transform network, the specific configuration is as follows:
[0154] (e) The input of the two branches is the latent feature representation of the decoded image, and the output is the decoded image.
[0155] (f) The sampling in the sampling module is an upsampling operation, which is a deconvolution with an upsampling factor of 2, that is, the spatial dimension is twice the original.
[0156] (g) In the nonlocal computation of the spatial domain, K=5, K0=1, s=1, s1=1, that is, the spatial dimension and channels remain unchanged (because the dimension of the input latent feature representation is already relatively small). At the same time, it is transformed into a normal convolution operation.
[0157] (h) In the nonlocal computation of the channel dimension, K1 = 5, K2 = 1, s2 = 1, that is, the spatial dimension remains unchanged.
[0158] In this embodiment of the application, a transformation network guided by global information is proposed to be used in the transformation / inverse transformation. This network adopts a dual-branch structure to make full use of the global correlation of the input features and extract a compact feature representation.
[0159] Step S14: Input the second encoded feature into the first entropy model network for feature analysis to obtain the distribution information of the second encoded feature.
[0160] In this embodiment, the image encoding device inputs the second encoded features obtained after processing by the transform network into a first entropy model network for feature analysis. The probability model in the first entropy model network outputs the distribution information of the second encoded features. The first entropy model network can be an existing first entropy model network used in image end-to-end codecs, such as... Figure 1 and Figure 2 Entropy model networks, etc.
[0161] To further improve encoding and decoding performance, this application proposes a contextual model incorporating predicted values to optimize and improve the first entropy model network. Please refer to the full text. Figure 10 and Figure 11 , Figure 10 yes Figure 3 The diagram shows the detailed process of step S14 in the image encoding method. Figure 11 This is a schematic diagram of the structure of the context predictor that combines predicted values provided in this application.
[0162] like Figure 11 As shown, the context model provided in this application can learn the current feature point by utilizing already encoded / decoded feature points during entropy encoding. However, due to the dependencies between sample points, it is far from sufficient to predict the current feature point using only already encoded / decoded points. If the pixels in the neighborhood range following the current feature point (i.e., unencoded / decoded points) can be used simultaneously to assist in predicting the current point, the encoding and decoding performance can be improved.
[0163] The following section, using step S14 as a reference, further describes the specific working process of the context predictor that combines the predicted values:
[0164] Step S141: Input the second encoded feature and the first auxiliary inverse transform result of the first entropy model network on the second encoded feature into the context predictor of the first entropy model network to obtain the context information of the second encoded feature.
[0165] In the embodiments of this application, such as Figure 4 As shown, the image coding device inputs the auxiliary inverse transform result of the first entropy model network and the transform result of the main coding network into the context predictor, which then extracts the context information of the second coding feature in the transform structure based on the auxiliary inverse transform result.
[0166] Specifically, please refer to Figure 12 and Figure 13 , Figure 12 This is the 3D mask convolution unrolling diagram of the first half of the activated portion provided in this application. Figure 13 This is the 3D mask convolution unfolded diagram of the activated latter half (including the center point) provided in this application.
[0167] Image encoding device utilizes Figure 12 The first half of the activated convolution is shown to extract the first estimate of the encoded feature map. Specifically, for the encoded / decoded points, the image encoding device uses a K×K×K 3D mask convolution with the first half activated, such as... Figure 12 As shown, only the first half of the convolution center point has weights (not 0), while the second half is 0, thus obtaining the preliminary estimate of the current point, which is the first estimate.
[0168] Image encoding device utilizes Figure 13 The second estimation result of the uncoded feature map is extracted by the latter half of the activation convolution shown. Specifically, for the prediction result (uncoded / decoded point) output by the prediction network in the context predictor, the image encoding device first determines the position of the current feature point in the feature map output by the prediction network, and then uses a K×K×K 3D mask convolution of the latter half of the feature point's neighborhood, such as... Figure 13 As shown above, Figure 12 In contrast to 3D mask convolution, only the latter half of the convolution center point and the center point itself have weights, while the first half is weighted. That is, only the latter half of the neighborhood of the current feature point is taken to obtain the second estimate of the current feature point.
[0169] Please continue reading. Figure 11 The image encoding device fuses the first and second estimation results obtained from the above calculations. The fusion method includes, but is not limited to, addition, convolutional fusion, and other methods. This is equivalent to the current feature point being predicted by the pixels in its neighborhood (the first half of these pixels comes from the encoded points, and the second half comes from the estimation results of the prediction network, i.e., the uncoded points).
[0170] Furthermore, the image encoding device fuses the neighborhood-wide fusion result with the auxiliary inverse transform result of the first entropy model network to obtain the parameters of the probability model. Additionally, the prediction result output by the prediction network in the context predictor is used for subsequent subtraction / summation operations, specifically including subtracting the prediction result from the transform module output during encoding, and adding the prediction result to the decoding result during decoding.
[0171] Specifically, the difference between the prediction result and the transformation result is calculated, and the encoding / decoding module encodes the difference. Similarly, the decoding result of the feature code stream by the encoding / decoding module is the difference, which needs to be added to the prediction result before performing an inverse transformation to obtain the decoded image.
[0172] In one specific implementation, Figure 11 The context predictor that combines the predicted values is applied to Figure 4 The specific configuration for the context predictor is as follows:
[0173] (a) 3D mask convolution kernel size for encoded / decoded points: K = 11.
[0174] (b) 3D mask convolution kernel size for the prediction results (unencoded / decoded points) of the prediction network output: K = 11.
[0175] (c) Figure 11 The neighborhood range fusion method is: point-to-point addition strategy; the second fusion method is: channel splicing and 1×1 convolution fusion.
[0176] In this application embodiment, a context model combining the predicted value is proposed, which simultaneously considers the pixel values in the surrounding neighborhood of the current point (i.e., encoded / decoded points and unencoded / decoded points) to more accurately predict the information of the current point and improve encoding / decoding performance.
[0177] Further introduction Figure 4 The specific computational process of the first entropy model network and the second entropy model network:
[0178] The image encoding device obtains the second encoded feature and inputs it into the auxiliary transform module of the first entropy model network to obtain the fifth encoded feature. The auxiliary transform module has the same structure as the aforementioned transform module and will not be described again here. The image encoding device uses the predictor of the second entropy model network to obtain the prediction information of the fifth encoded feature; it inputs the prediction information into the second probability model of the second entropy model network to obtain the distribution information of the fifth encoded feature; and it uses the distribution information of the fifth encoded feature to encode and decode the fifth encoded feature, obtaining the first auxiliary inverse transform result.
[0179] Specifically, the image encoding device utilizes the prediction network in the predictor of the second entropy model network to extract the second prediction result of the fifth encoding feature; the second prediction result is then fused with the second auxiliary inverse transform result of the fifth encoding feature by the second entropy model network to obtain the prediction information of the fifth encoding feature. Please refer to [link to details]. Figure 14 , Figure 14 This is a schematic diagram of the predictor that combines predicted values provided in this application.
[0180] Figure 14 The predictor is primarily used in second entropy model networks to estimate the parameter values of the probabilistic model of the second entropy model network. Similar to the context predictor that combines the predicted values, the estimation results and auxiliary transformation results of the prediction network are mapped using convolutions with kernel sizes of M×M and N×N (including but not limited to ordinary convolution, residual blocks, or dilated convolution), and then fused to obtain the parameters required by the probabilistic model.
[0181] Specifically, Figure 14 The prediction result is then compared with the auxiliary transform result, and the difference is encoded. Similarly, the decoding result is the difference, which needs to be added to the prediction result before performing an auxiliary inverse transform to obtain the decoded features.
[0182] In one specific implementation, Figure 14 The predictor that combines the predicted values is applied to Figure 4 The specific configuration for the predictor is as follows:
[0183] (a) Convolution 1: 3×3 convolution.
[0184] (b) Convolution 2: 3×3 convolution.
[0185] (c) The fusion method is: channel splicing and 1×1 convolution.
[0186] In this embodiment, a predictor that combines predicted values with the predicted values of latent feature representations and auxiliary transformation results is proposed to fully predict the information of the current point.
[0187] Step S142: Input the context information into the first probability model of the first entropy model network, and obtain the distribution information output by the first probability model.
[0188] In this embodiment of the application, the image encoding device inputs the context information output by the context predictor into the probability model, and the probability model predicts the distribution of features based on the context information and outputs the distribution information.
[0189] Step S15: The encoding module of the main encoding network encodes the second encoding feature according to the distribution information to obtain the feature bitstream of the image to be encoded.
[0190] In this embodiment, the main coding network uses the distribution information output by the first entropy model network to encode the second coding feature, thereby improving coding accuracy and coding efficiency according to the distribution of the feature.
[0191] In this embodiment, the image encoding device inputs an image to be encoded and extracts the image features to be encoded and the first encoding features using the first branch of the transform module in the main encoding network; it then extracts global information of the image features to be encoded using the second branch of the transform module, and obtains the global correlation factor of the first encoding features based on the global information; it modulates the first encoding features using the global correlation factor to obtain the modulated second encoding features; it inputs the second encoding features into a first entropy model network for feature analysis to obtain the distribution information of the second encoding features; and it uses the encoding module of the main encoding network to encode the second encoding features according to the distribution information to obtain the feature bitstream of the image to be encoded. The image encoding method of this application employs a transform network guided by global information, which uses a dual-branch structure to fully utilize the global correlation of the input features and extract a compact feature representation.
[0192] In contrast, please continue reading Figure 15 , Figure 15 This is a flowchart illustrating an embodiment of the image decoding method provided in this application.
[0193] like Figure 15 As shown, the image decoding method of this application embodiment includes the following steps:
[0194] Step S21: Input the feature code stream.
[0195] Step S22: The main coding network is used to decode the feature bitstream according to the distribution information to obtain the first decoded feature of the feature bitstream.
[0196] Step S23: The master coding network is used to demodulate the first decoding feature according to the preset global correlation factor to obtain the demodulated second decoding feature.
[0197] Step S24: Reconstruct the image according to the second decoding feature to obtain the decoded image.
[0198] It should be noted that the image decoding method of this application embodiment is essentially the reverse process of the image encoding method of the above embodiment. Therefore, all the technical solutions of the image encoding method can be applied to the image decoding method of this embodiment without any doubt. The relevant technical solutions can be derived through simple reverse engineering, which will not be elaborated here.
[0199] The above embodiments are merely one common example of this application and do not constitute any limitation on the technical scope of this application. Therefore, any minor modifications, equivalent changes, or alterations made to the above content based on the substance of the solution of this application shall still fall within the scope of the technical solution of this application.
[0200] Please continue reading Figure 16 , Figure 16 This is a schematic diagram of an embodiment of the image encoding apparatus provided in this application. The image encoding apparatus 500 of this application embodiment includes a processor 51, a memory 52, an input / output device 53, and a bus 54.
[0201] The processor 51, memory 52, and input / output device 53 are respectively connected to the bus 54. The memory 52 stores program data, and the processor 51 is used to execute the program data to implement the image encoding method described in the above embodiments.
[0202] In this embodiment, processor 51 can also be referred to as a CPU (Central Processing Unit). Processor 51 may be an integrated circuit chip with signal processing capabilities. Processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 51 can be any conventional processor.
[0203] Please continue reading Figure 17 , Figure 17 This is a schematic diagram of an embodiment of the image decoding device provided in this application. The image decoding device 600 of this application embodiment includes a processor 61, a memory 62, an input / output device 63, and a bus 64.
[0204] The processor 61, memory 62, and input / output device 63 are respectively connected to the bus 64. The memory 62 stores program data, and the processor 61 is used to execute the program data to implement the image decoding method described in the above embodiments.
[0205] This application also provides a computer storage medium; please refer to the following: Figure 18 , Figure 18 This is a schematic diagram of a computer storage medium according to an embodiment of the present application. The computer storage medium 700 stores program data 71, which, when executed by a processor, is used to implement the image encoding method and / or image decoding method of the above embodiments.
[0206] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0207] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An image encoding method, characterized in that, The image encoding method includes: Input the image to be encoded, and use the first branch of the transform module of the main coding network to extract the image features to be encoded and the first coding features of the image to be encoded; The global information of the features of the image to be encoded is extracted using the second branch of the transformation module, and the global correlation factor of the first encoded feature is obtained based on the global information; The first encoded feature is modulated using the global correlation factor to obtain the modulated second encoded feature; The second encoded feature is input into the first entropy model network for feature analysis to obtain the distribution information of the second encoded feature; The encoding module of the main encoding network encodes the second encoding feature according to the distribution information to obtain the feature bitstream of the image to be encoded.
2. The image encoding method according to claim 1, characterized in that, The step of extracting global information of the features of the image to be encoded using the second branch of the transformation module includes: The spatial attention weights of the image features to be encoded and the channel dimension attention weights of the image features to be encoded are obtained using the second branch of the transformation module. Using the spatial attention weights, the third encoded features of the image to be encoded are obtained by non-local computation in the spatial domain; Using the channel-dimensional attention weights, the fourth encoded feature of the image to be encoded is obtained by non-local computation in the channel dimension; The third and fourth coding features are fused to extract global information of the features of the image to be encoded.
3. The image encoding method according to claim 2, characterized in that, The step of obtaining the spatial attention weights of the features of the image to be encoded includes: The first feature representation of the image to be encoded is obtained by using the first convolution; The second feature representation of the first feature representation is obtained by using the second convolution; By applying different dimensionality adjustments to the second feature representation, we obtain the third and fourth feature representations; Perform matrix operations on the third feature representation and the fourth feature representation to obtain the spatial attention weights of the features of the image to be encoded; The step of using the spatial attention weights to obtain the third encoded feature, which is obtained by non-local computation of the features of the image to be encoded in the spatial domain, includes: Perform matrix operations on the image features to be encoded and the spatial attention weights to obtain a first weighted feature representation; The first weighted feature representation is fused with the features of the image to be encoded to obtain the third encoded feature.
4. The image encoding method according to claim 2, characterized in that, The process of obtaining the channel dimension attention weights of the features of the image to be encoded includes: The fifth feature representation of the image to be encoded is obtained using the third convolution; By applying different dimensional adjustment methods to the fifth feature representation, the sixth and seventh feature representations are obtained; Perform matrix operations on the sixth feature representation and the seventh feature representation to obtain the channel dimension attention weights of the image features to be encoded; The step of obtaining the fourth encoded feature, obtained by non-local computation of the image features to be encoded in the channel dimension using the channel-dimensional attention weights, includes: The eighth feature representation of the image to be encoded is obtained using the fourth convolution; Perform a matrix operation on the eighth feature representation and the channel dimension attention weight to obtain the second weighted feature representation; The second weighted feature representation is fused with the features of the image to be encoded to obtain the fourth encoded feature.
5. The image encoding method according to claim 1, characterized in that, The global correlation factor includes a masking factor and a shift factor; The step of obtaining the global relevance factor of the first encoded feature based on the global information includes: The global information is convolved using the fifth convolution to obtain the mask factor of the first encoded feature; The global information is convolved using the sixth convolution to obtain the offset factor of the first encoded feature; The step of modulating the first encoded feature using the global correlation factor to obtain the modulated second encoded feature includes: The result of multiplying the first coding feature with the feature of the masking factor is added to the feature of the offset factor to obtain the modulated second coding feature.
6. The image encoding method according to claim 1, characterized in that, The step of inputting the second encoded feature into the first entropy model network for feature analysis to obtain the distribution information of the second encoded feature includes: The second encoded feature and the first auxiliary inverse transform result of the first entropy model network on the second encoded feature are input into the context predictor of the first entropy model network to obtain the context information of the second encoded feature; The context information is input into the first probability model of the first entropy model network to obtain the distribution information output by the first probability model.
7. The image encoding method according to claim 6, characterized in that, The step of inputting the second encoded feature and the first auxiliary inverse transform result of the first entropy model network on the second encoded feature into the context predictor of the first entropy model network to obtain the context information of the second encoded feature includes: Obtain the encoded feature map of the second encoded feature; The first estimation result of the encoded feature map is extracted by activating the convolution in the first half. The prediction network in the context predictor is used to extract the first prediction result of the second encoded feature; The second estimation result of the first prediction result is extracted by activating the convolution in the latter half. The first estimation result and the second estimation result are fused to obtain the neighborhood range result of the second encoded feature; The first auxiliary inverse transform result is fused with the neighborhood range result to obtain the context information of the second encoded feature.
8. The image encoding method according to claim 7, characterized in that, The weights of the first half of the convolution center point of the first half activation convolution are not 0, and the weights of the second half are 0; the weights of the first half of the convolution center point of the second half activation convolution are 0, and the weights of the second half are not 0.
9. The image encoding method according to claim 6, characterized in that, The image encoding method further includes: The second encoded feature is input into the auxiliary transform module of the first entropy model network to obtain the fifth encoded feature; The predictor of the second entropy model network is used to obtain the prediction information of the fifth encoded feature; The predicted information is input into the second probability model of the second entropy model network to obtain the distribution information of the fifth encoded feature; The distribution information of the fifth coding feature is used to encode and decode the fifth coding feature to obtain the first auxiliary inverse transform result.
10. The image encoding method according to claim 9, characterized in that, The predictor using the second entropy model network to obtain the prediction information of the fifth encoded feature includes: The prediction network in the predictor of the second entropy model network is used to extract the second prediction result of the fifth encoded feature; The second prediction result is fused with the second auxiliary inverse transform result of the second entropy model network on the fifth coding feature to obtain the prediction information of the fifth coding feature.
11. An image decoding method, characterized in that, The image decoding method includes: Input feature bitstream; The feature bitstream is decoded using a master coding network based on distribution information to obtain the first decoded feature of the feature bitstream; The first branch of the inverse transform module of the main coding network is used to extract the image features to be decoded and the second decoding features of the first decoding features; The global information of the features of the image to be decoded is extracted using the second branch of the inverse transform module, and the global correlation factor of the second decoded feature is obtained based on the global information; The main coding network is used to demodulate the second decoding feature according to the global correlation factor to obtain the demodulated third decoding feature; The decoded image is obtained by reconstructing according to the third decoding feature.
12. An image encoding device, characterized in that, The image encoding device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image encoding method as described in any one of claims 1 to 10.
13. An image decoding device, characterized in that, The image decoding device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the image decoding method as described in claim 11.
14. A computer storage medium, characterized in that, The computer storage medium is used to store program data, which, when executed by the computer, is used to implement the image encoding method as described in any one of claims 1 to 10 and / or the image decoding method as described in claim 11.