Image coding method and device
Dynamically divide potential features through channel attention mechanism, solving the problem of inapplicable feature segmentation in the prior art, and improving image encoding and machine vision performance.
Patent Information
- Application Number
- CN202311575103.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-27
- Filing Date
- 2023-11-23
- Publication Date
- 2024-12-06
AI Technical Summary
The existing machine learning-based scalable image encoders, when segmenting potential features into base and enhancement layers, fixed position segmentation results in features not being suitable for downstream tasks, affecting encoding performance and machine vision performance.
The channel attention mechanism is used to obtain the channel attention mask of the latent features, dynamically divide the latent features into the base layer and the enhancement layer potential features, and generate the encoded data of the target image.
The encoding performance and machine vision performance of machine learning-based scalable image encoder is improved, the characteristics are divided more reasonably, and the encoding efficiency and accuracy are improved.
Smart Images

Figure CN119094762A_ABST
Abstract
Description
[0001] This application claims the priority of Chinese patent application with application number 2023106645409 filed on June 6, 2023, and the priority of Chinese patent application with application number 2023114114882 filed with the China Patent Office on October 27, 2023. The contents of the prior applications are incorporated into this application by reference. Technical Field
[0002] Some embodiments of the present application relate to the field of image coding and decoding technology, and more specifically, to an image coding method and device. Background Art
[0003] At present, image / video coding algorithms are generally optimized for the human visual system and suitable for human viewing applications, but they are not optimized for other application scenarios, such as the very common image processing and machine vision applications. For these applications, it is often necessary to fully decode the image to obtain the reconstructed image / video before the reconstructed image / video can be used as the input of these applications. This process is complex and time-consuming, and the lossy image after decoding often has a negative impact on the accuracy of machine vision tasks. Reducing the bit rate while maintaining the subjective quality of the image unchanged and providing a representation suitable for direct machine vision input data has become an urgent problem to be solved. Therefore, efficient video coding for machines (VCM) has become an important topic in academia and industry. VCM technology is committed to connecting the two independent research directions of image / video compression and feature compression to a certain extent, while jointly optimizing high-accuracy machine vision tasks and high-fidelity human vision reconstruction as much as possible. Based on this, the Moving Picture Expert Group (MPEG) established a group called VCM to study the needs of VCM technology standardization. JPEG AI is a scalable image coding standard based on machine learning created by the Joint Photographic Experts Group (JPEG). Compared with traditional image coding, JPEG AI achieves a lower bit rate at the same subjective quality based on human vision. At the same time, the JPEG AI model also puts forward corresponding requirements for VCM.
[0004] At present, in the scalable image encoder based on machine learning, the input image is processed by the analysis transformation network to obtain the latent features (latent represetation) of the input image, and then the first preset number of channels of the latent features are determined as the enhancement layer channels and the remaining channels are determined as the base layer channels. The base layer channels and the enhancement layer channels are further eliminated for redundancy to obtain the code stream. As mentioned above, in the current scalable image encoder based on machine learning, the latent features are divided into base layer latent features and enhancement layer latent features according to fixed positions. However, if the latent features are directly trimmed according to the positions, it is likely that the features of some channels in the base layer latent features are not suitable for downstream tasks, thereby affecting the encoding performance and machine vision performance. Summary of the invention
[0005] The exemplary embodiments of the present application provide an image encoding method and apparatus for improving the encoding performance and machine vision performance of a scalable image encoder based on machine learning.
[0006] Some embodiments of the present application provide the following technical solutions:
[0007] In a first aspect, some embodiments of the present application provide an image encoding method, including:
[0008] Obtaining latent features of the target image;
[0009] Quantizing the latent feature based on a preset quantization parameter to obtain a quantized latent feature;
[0010] Processing the quantized latent feature based on a channel attention mechanism to obtain a channel attention mask of the latent feature;
[0011] dividing the latent features into base layer latent features and enhancement layer latent features according to the channel attention masks of the latent features;
[0012] The encoded data of the target image is generated according to the base layer latent features and the enhancement layer latent features.
[0013] In a second aspect, some embodiments of the present application provide an image encoding device, including:
[0014] a memory configured to store a computer program;
[0015] The processor is configured to, when calling a computer program, enable the image encoding device to implement the image encoding method described in the first aspect.
[0016] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computing device, the computing device implements the image encoding method described in the first aspect.
[0017] In a fourth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image encoding method described in the first aspect.
[0018] It can be seen from the above technical scheme that when encoding the target image, the image encoding method provided by the embodiment of the present application first obtains the potential features of the target image, then quantizes the potential features based on the preset quantization parameters to obtain the quantized potential features, and then processes the quantized potential features based on the channel attention mechanism to obtain the channel attention mask of the potential features, and divides the potential features into the base layer potential features and the enhancement layer potential features according to the channel attention mask of the potential features, and finally generates the encoding data of the target image according to the base layer potential features and the enhancement layer potential features. Compared with the prior art that divides the potential features into the base layer potential features and the enhancement layer potential features according to the fixed position, the embodiment of the present application can obtain the channel attention mask of the potential features based on the channel attention mechanism, and then divide the potential features into the base layer potential features and the enhancement layer potential features according to the channel attention mask of the potential features. Therefore, the embodiment of the present application can more reasonably divide the potential features into the base layer potential features and the enhancement layer potential features, thereby improving the encoding performance and machine vision performance of the scalable image encoder based on machine learning. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.
[0020] Figure 1 A schematic diagram showing the structure of an image encoder in some embodiments of the present application is shown;
[0021] Figure 2 A schematic diagram showing the structure of an image decoder in some embodiments of the present application is shown;
[0022] Figure 3 A flowchart showing the steps of the image encoding method in some embodiments of the present application is shown;
[0023] Figure 4A flowchart showing the steps of the image encoding method in some other embodiments of the present application is shown;
[0024] Figure 5 A schematic diagram showing the structure of an analysis transformation network in some embodiments of the present application is shown;
[0025] Figure 6 A schematic diagram showing the structure of a channel attention network in some embodiments of the present application is shown;
[0026] Figure 7 A schematic diagram showing the structure of a super-a priori coding network in some embodiments of the present application is shown;
[0027] Figure 8 A schematic diagram showing the structure of a super-a priori decoding network in some embodiments of the present application is shown;
[0028] Fig. 9 A schematic diagram of a masked convolution kernel of a context network in some embodiments of the present application is shown;
[0029] Fig.10 A schematic diagram showing the structure of an enhancement layer context network in some embodiments of the present application is shown;
[0030] Fig.11 A schematic diagram showing the structure of a channel context module in some embodiments of the present application is shown;
[0031] Fig.12 A schematic diagram showing the structure of an entropy parameter module in some embodiments of the present application is shown;
[0032] Fig.13 A schematic diagram showing the structure of a base layer context network in some embodiments of the present application is shown;
[0033] Fig.14 A flowchart showing the steps of the image decoding method in some embodiments of the present application is shown;
[0034] Fig.15 A flowchart showing the steps of the image decoding method in some other embodiments of the present application is shown;
[0035] Fig.16 A schematic diagram showing the structure of a synthetic transformation network in some embodiments of the present application is shown;
[0036] Fig.17 A schematic diagram showing the structure of a potential space transformation network in some embodiments of the present application is shown;
[0037] Fig.18 A schematic diagram showing the structure of an image decoder in some other embodiments of the present application is shown;
[0038] Fig.19A schematic diagram showing the structure of a content adaptive filtering network in some embodiments of the present application is shown;
[0039] Fig. 20 A flowchart showing the steps of the image decoding method in some other embodiments of the present application is shown;
[0040] Fig.21 A schematic diagram showing the structure of ICCI in other embodiments of the present application is shown;
[0041] Fig. 22 A schematic diagram of the structure of a residual block in some embodiments of the present application is shown. DETAILED DESCRIPTION
[0042] It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below. In order to make the purpose and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.
[0043] It is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and customary meanings.
[0044] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.
[0045] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).
[0046] The image in the embodiment of the present application can be an independent image or a video frame in a video. Since a video can be regarded as a video frame sequence composed of multiple images, when encoding and decoding each video frame based on the image encoding method and the image decoding method, the video can be encoded and decoded based on the image encoding method and the image decoding method.
[0047] The embodiments of the present application relate to the technical field of image coding and decoding. The following first describes an image codec for executing the image coding method and image decoding method provided by the embodiments of the present application.
[0048] Reference Figure 1 As shown, an image encoder for executing the image encoding method provided by some embodiments of the present application includes: an analysis transformation network 101, a first quantization module 102, a mask generation network 103, a first channel selection module 104, a base layer super-prior encoding network 105, a second quantization module 106, a first encoding module 107, an enhancement layer super-prior encoding network 108, a third quantization module 109, a second encoding module 110, a second channel selection module 111, a first decoding module 112, a base layer super-prior decoding network 113, a base layer context network 114, a third encoding module 115, a second decoding module 116, an enhancement layer super-prior decoding network 117, an enhancement layer context network 118, and a fourth encoding module 119.
[0049] Figure 1 The process of image encoding by the image encoder shown includes the following steps: Step 1, the analysis transformation network 101 processes the target image (video frame) X to obtain the potential feature Y of the target image X. Step 2, the first quantization module 102 quantizes the potential feature Y to obtain the quantized potential feature y. Step 3, the quantized potential feature y is input into the mask generation network 103, and the channel attention mask Mask of the potential feature Y output by the mask generation network 103 is obtained. Step 4, the first channel selection module 104 performs channel selection on the potential feature Y based on the channel attention mask Mask to divide the potential feature Y into the base layer potential feature Y base and the enhancement layer latent feature Y enhance Step 5: The base layer super prior coding network 105 encodes the base layer potential feature Y base Processing is performed to obtain the base layer super prior feature Z base Step 6: The second quantization module 106 quantizes the base layer super-prior feature Z base Perform quantization processing to obtain the basic layer quantization super prior feature z base Step 7: The first encoding module 107 quantizes the super-prior feature z of the base layer base Perform entropy coding to obtain the first coded data b1. Step 8: The enhancement layer super-prior coding network 108 encodes the enhancement layer potential feature Y enhance Processing is performed to obtain the enhanced layer super prior feature Z enhance Step 9: The third quantization module 109 quantizes the enhancement layer super-prior feature Z enhance Perform quantization processing to obtain the enhanced layer quantization super prior feature z enhanceStep 10: The second encoding module 110 quantizes the super-prior feature z of the enhancement layer enhance Step 11: The second channel selection module 111 performs channel selection on the quantized potential feature y based on the channel attention mask Mask to divide the quantized potential feature y into the base layer quantized potential feature y base and the enhancement layer quantized latent features y enhance Step 12: The first decoding module 112 decodes the first coded data b1 to obtain the base layer reconstruction super prior feature Step 13: The base layer super-prior decoding network 113 reconstructs super-prior features of the base layer Processing is performed to obtain the underlying features for reconstruction Step 14: Base layer context network 114 reconstructs potential features based on the base layer and the encoded base layer quantized latent features y base Get the base layer context feature Φ base Step 15: The third encoding module 115 generates a base layer context feature Φ base Quantize the latent features y of the base layer base The current element is entropy encoded, and finally the base layer quantization potential feature y is completed. base The entropy encoding of all elements is performed to obtain the third coded data b3. Step 16: The second decoding module 116 decodes the second coded data b2 to obtain the enhanced layer reconstruction super prior feature Step 17: The enhancement layer super-prior decoding network 117 reconstructs the super-prior features of the enhancement layer Processing is performed to obtain the potential features of the enhanced layer reconstruction Step 18: Enhanced layer context network 118 reconstructs potential features based on the enhanced layer and the quantized latent feature y of the enhanced layer that has been coded enhance Get the enhancement layer context feature Φ enhance Step 19: The fourth encoding module 119 generates the enhanced layer context feature Φ enhance Quantize the latent features y for the enhancement layer enhance The current element is entropy coded, and the enhancement layer quantization potential feature y is finally completed. enhance The entropy coding of all elements of is performed to obtain the fourth coded data b4. Step 20: The first coded data b1, the second coded data b2, the third coded data b3 and the fourth coded data b4 are used as the coded data of the target image X and sent to the decoding end.
[0050] It should be noted that when the first encoding module 107 is a lossless encoding module and the first decoding module 112 is a lossless decoding module, due to the base layer quantization super prior feature zbase There is no information loss in the encoding and decoding process, and the base layer reconstructs the super prior features Quantized hyper-prior features z with base layer base are exactly the same, so Figure 1 The encoder shown in the figure can omit the decoding module 112 and the above step 12, and directly use the base layer quantized super prior feature z base Replace the base layer in the subsequent steps to reconstruct the hyper-prior features Similarly, when the second encoding module 110 is a lossless encoding module and the second decoding module 116 is a lossless decoding module, the decoding module 115 and the above step 16 can be omitted, and the enhancement layer quantization super prior feature z is directly used. enhance Replace the base layer in the subsequent steps to reconstruct the hyper-prior features
[0051] Reference Figure 2 As shown, an image decoder for executing the image decoding method provided by some embodiments of the present application includes: a first decoding module 201, a base layer super priori decoding network 202, a base layer context network 203, a second decoding module 204, a third decoding module 205, an enhancement layer super priori decoding network 206, an enhancement layer context network 207, a fourth decoding module 208, a splicing module 209, a synthesis transformation network 210, a latent space transformation network 211 and a machine vision task model 212.
[0052] Figure 2 The process of image decoding by the image decoder shown in the figure includes the following steps: Step 1: The first decoding module 201 performs entropy decoding on the first coded data b1 to obtain the super prior feature of the base layer reconstruction Step 2: The base layer super-prior decoding network 202 reconstructs the super-prior features of the base layer Processing is performed to obtain the underlying features for reconstruction Step 3: The base layer context network 203 reconstructs the potential features based on the base layer and the enhanced layer reconstructed quantitative latent features (Decoding result of the decoded third coded data b3), obtain the base layer context feature Φ base Step 4: The third decoding module 204 decodes the base layer context feature Φ base The current element of the third coded data b3 is entropy decoded, and finally the entropy decoding of all elements of the third coded data b3 is completed to obtain the base layer reconstruction quantization potential feature Step 5: The third decoding module 205 performs entropy decoding on the second coded data b2 to obtain the enhanced layer reconstruction super prior feature Step 6: The enhancement layer super-prior decoding network 206 reconstructs the super-prior features of the enhancement layer Processing is performed to obtain the potential features of the enhanced layer reconstruction Step 7: Enhanced layer context network 207 reconstructs latent features based on the enhanced layer and the enhanced layer reconstructed quantitative latent features (Decoding result of the fourth coded data b4 that has been decoded), obtain the enhancement layer context feature Φ enhance Step 8: The fourth decoding module 208 decodes the enhanced layer context feature Φ enhance The current element of the fourth coded data b4 is entropy decoded, and finally the entropy decoding of all elements of the fourth coded data b4 is completed to obtain the enhanced layer reconstruction quantization potential feature Step 9: The splicing module 209 splices the base layer to reconstruct the quantized potential features and enhancement layer reconstruction quantization latent features Obtaining reconstruction quantization potential features Step 10: Reconstructing the quantized latent features through the synthetic transformation network 210 Processing is performed to obtain the reconstructed target image Step 11: Reconstruct the quantized latent features of the base layer through the latent transformation network 211 Processing to obtain machine vision features Step 12: Machine vision features The machine vision task model 212 is input to obtain the machine vision task processing result M output by the machine vision task model 212 .
[0053] The present application embodiment provides an image encoding method, referring to Figure 3 As shown, the image encoding method includes the following steps S31 to S35:
[0054] S31, obtaining latent features (Latent Represetation) of the target image.
[0055] In some embodiments, the feature tensor of the potential features of the target image may be 192×H×W, where H and W are the length and width of the target image, respectively.
[0056] In some embodiments, the target image may be processed by an Analysis Transform Network (AnalysisTransformNet) to obtain potential features of the target image.
[0057] The target image in the embodiment of the present application may be an independent image or a video frame in a video.
[0058] S32: quantize the potential features based on preset quantization parameters to obtain quantized potential features.
[0059] In the embodiment of the present application, the value of the quantization step size (the preset quantization parameter) of the potential feature is not limited. Since the larger the quantization step size, the smaller the bit rate of the image, the worse the corresponding reconstruction quality, and vice versa, the smaller the quantization step size, the larger the bit rate of the image, the better the corresponding reconstruction quality, so in actual use, the quantization step size can be set by balancing the bit rate and reconstruction quality requirements.
[0060] S33. Processing the quantized latent feature based on a channel attention mechanism to obtain a channel attention mask of the latent feature.
[0061] The channel attention mechanism refers to a mechanism that pays more attention to the relationship between channels of features in computer vision. It can include: compression network and excitation network. Among them, the compression network is mainly used to compress the global spatial information of the feature, and then learn the features from the channel dimension to obtain the importance of each channel. The excitation network assigns different weight values to each channel according to the importance of each channel, and recalibrates the features according to the weight values of each channel.
[0062] In some embodiments, the channel attention mask is a string or feature of length C, where C is the number of channels of the potential feature.
[0063] S34. Divide the potential features into base layer potential features and enhanced layer potential features according to the channel attention mask of the potential features.
[0064] In some embodiments, the latent feature is divided into a base layer latent feature and an enhanced layer latent feature according to the channel attention mask of the latent feature, including: calculating the dot product of the channel attention mask and the latent feature to obtain the base layer latent feature, and calculating the difference between the latent feature and the base layer latent feature to obtain the enhanced layer latent feature.
[0065] In some embodiments, the base layer potential feature, the enhancement layer potential feature, and the feature tensor of the potential feature are the same. The base layer potential feature is different from the potential feature in that the channel in the potential feature that is classified as the enhancement layer potential feature has a feature value of zero in the base layer potential feature. Similarly, the enhancement layer potential feature is different from the potential feature in that the channel in the potential feature that is classified as the base layer potential feature has a feature value of zero in the enhancement layer potential feature.
[0066] S35. Generate encoding data of the target image according to the base layer latent features and the enhancement layer latent features.
[0067] The image encoding method provided in the embodiment of the present application first obtains the potential features of the target image when encoding the target image, then quantizes the potential features based on preset quantization parameters to obtain quantized potential features, then processes the quantized potential features based on the channel attention mechanism to obtain the channel attention mask of the potential features, and divides the potential features into base layer potential features and enhancement layer potential features according to the channel attention mask of the potential features, and finally generates the encoding data of the target image according to the base layer potential features and the enhancement layer potential features. Compared with the prior art that divides the potential features into base layer potential features and enhancement layer potential features according to fixed positions, the embodiment of the present application can obtain the channel attention mask of the potential features based on the channel attention mechanism, and then divide the potential features into base layer potential features and enhancement layer potential features according to the channel attention mask of the potential features. Therefore, the embodiment of the present application can more reasonably divide the potential features into base layer potential features and enhancement layer potential features, thereby improving the encoding performance and machine vision performance of the scalable image encoder based on machine learning.
[0068] As an extension and refinement of the above embodiment, the present application embodiment also provides another image encoding method, referring to Figure 4 As shown, the image encoding method comprises the following steps:
[0069] S401: Process the target image based on an analysis transformation network to obtain potential features of the target image.
[0070] Reference Figure 5 As shown, in some embodiments, the analysis transformation network includes: a plurality of downsampling convolutional layers 51 and a plurality of residual activation units 52 serially connected between adjacent downsampling convolutional layers 51, Figure 5 In the example, the analysis transformation network includes four downsampling convolutional layers 51 and three residual activation units 52 connected in series between adjacent downsampling convolutional layers 51. The residual activation unit 52 includes: a first convolutional layer 521, an activation function layer 522, a second convolutional layer 523 and a fusion layer 524. The first convolutional layer 521, the activation function layer 522 and the second convolutional layer 523 process the input feature I1 of the residual activation unit 52 in sequence, and the output feature of the second convolutional layer and the input feature I1 of the residual activation unit are added and fused through the fusion layer 524 to obtain the output feature O1 of the residual activation unit 52.
[0071] In some embodiments, each downsampling convolution layer 51 in the analysis transformation network is a downsampling convolution layer with a convolution kernel size of 3×3, the first convolution layer and the second convolution layer in each residual activation unit of the analysis transformation network are convolution layers with a convolution kernel size of 3×3, and the activation area function of the activation function layer in each residual activation unit of the analysis transformation network is the LeakyReLU activation function (a variant of the ReLU function, which introduces a small slope in the negative value interval, solving the problem that the ReLU function outputs zero in the negative value interval).
[0072] The target image is represented as X, the potential features of the target image are represented as Y, and the processing of the features by the analysis transformation network is represented as g a (), then:
[0073] Y=g a (X)
[0074] S402: quantize the latent features based on preset quantization parameters to obtain quantized latent features.
[0075] The potential feature of the target image is represented as Y, the quantized potential feature is represented as y, and the quantization process based on the preset quantization parameter is represented as Q(), then:
[0076] y=Q(Y)
[0077] S403: Perform adaptive pooling processing (Adaptive Pooling) on the quantized potential features to obtain a first feature.
[0078] The difference between adaptive pooling and maximum pooling (Max Pooling) is that the adaptive pooling layer automatically calculates the size of the pooling layer and the moving step size according to the set output signal size (output_size) and the size of the input signal, while the size of the pooling layer and the moving step size of the maximum pooling layer are preset values. Adaptive pooling can accurately generate the first feature of the preset feature tensor.
[0079] In some embodiments, the feature tensor of the first feature is C×1×1, where C is the number of channels of the potential feature and the quantized potential feature.
[0080] S404: sequentially perform convolution processing, normalization processing, and nonlinear activation processing on the first feature to obtain a second feature.
[0081] In some embodiments, a convolution kernel size of a convolution layer performing convolution processing on the first feature is 1×1.
[0082] In some embodiments, the normalization layer that performs convolution processing on the first feature may be a batch normalization layer.
[0083] In some embodiments, the activation function for performing nonlinear activation processing on the first feature may be a Sigmoid activation function, a ReLU activation function, a Tanh activation function, or the like.
[0084] S405: Perform convolution processing on the second feature to obtain a third feature and a fourth feature.
[0085] The feature tensors of the third feature and the fourth feature are both C×1×1, where C is the number of channels of the potential feature.
[0086] In some embodiments, a convolution kernel size of a convolution layer performing convolution processing on the second feature is 1×1.
[0087] It should be noted that the output of the convolution layer that performs convolution processing on the second feature can be a feature with a tensor of 2C×1×1, and the third feature and the fourth feature can be two features obtained by segmenting the feature output by the fourth convolution layer from the channel dimension.
[0088] S406: Perform Gumbel sampling on the third feature and the fourth feature to obtain a channel attention mask of the latent feature.
[0089] Reference Figure 6 As shown, in some embodiments, the channel attention network for performing the above steps S403 to S406 includes: an adaptive pooling layer 61, a first convolutional layer 62, a batch normalization layer 63, a nonlinear activation function layer 64, a second convolutional layer 65, and a sampling module 66. The process of the channel attention network processing the quantized potential feature y to obtain the channel attention mask Mask includes: first, the adaptive pooling layer 61 processes the quantized potential feature y to obtain the first feature F1; secondly, the first convolutional layer 62, the normalization layer 63, and the nonlinear activation function layer 64 process the first feature F1 in sequence to obtain the second feature F2; and the second convolutional layer 65 processes the second feature F2 to obtain the third feature F3 and the fourth feature F4. Finally, the sampling module 66 performs Gumbel sampling on the third feature F3 and the fourth feature F4 to obtain the channel attention mask Mask of the potential feature.
[0090] The quantized potential feature is represented as y, and the processing of the feature by the adaptive pooling layer 61, the first convolution layer 62, the normalization layer 63, the nonlinear activation function layer 64, and the second convolution layer 65 is represented as f c(), Gumbel sampling is represented by Gum(), and the channel attention mask of the latent feature is represented by Mask, then:
[0091] Mask=Gum(f c (y))
[0092] S407: Perform a dot product operation on the latent feature and the channel attention mask to obtain a base layer latent feature.
[0093] The channel attention mask of the potential feature is represented as Mask, and the dot product operation is represented as The base layer latent feature is represented as Y base , then:
[0094]
[0095] S408: Calculate the difference between the potential feature and the base layer potential feature to obtain the enhancement layer potential feature.
[0096] The channel attention mask of the potential feature is represented as Mask, and the dot product operation is represented as The enhancement layer latent feature is represented as Y enhance , then:
[0097]
[0098] S409: Adjust the channels with zero characteristic values in the base layer potential features to the first end of the base layer potential features, and adjust the channels with zero characteristic values in the enhancement layer potential features to the second end of the enhancement layer potential features.
[0099] The first end is the head end and the second end is the tail end; or the first end is the tail end and the second end is the head end.
[0100] That is, the inverse mask (Trans Mask) operation is used to extract the channel indexes with masks of 0 and 1, and through the torch splicing operation, according to the extracted indexes, the channels of the base layer and the enhancement layer that are all 0 are transformed to the front and end of the features respectively, so as to obtain continuous non-zero base layer potential features and continuous non-zero enhancement layer potential features to make the subsequent entropy coding bit rate smaller.
[0101] S410: Process the base layer potential features through a base layer super-prior coding network to obtain base layer super-prior features.
[0102] Reference Figure 7As shown, in some embodiments, the base layer super priori coding network includes: a first convolution layer 71, a second convolution layer 72, a first downsampling layer 73, a third convolution layer 74 and a second downsampling layer 75 connected in series. The base layer super priori coding network performs a computation on the base layer potential feature Y base Processing to obtain the super prior features of the base layer includes: sequentially processing the base layer potential feature Y through the first convolution layer 71, the second convolution layer 72, the first downsampling layer 73, the third convolution layer 74 and the second downsampling layer 75 base Processing is performed to obtain the basic layer super prior feature Z base .
[0103] In some embodiments, the first convolution layer 71, the second convolution layer 72, the first down-sampling layer 73, the third convolution layer 74 and the second down-sampling layer 75 are all convolution layers with a convolution kernel size of 3×3.
[0104] Denote the base layer latent features as Y base , the base layer hyper-prior feature is represented as Z base , the processing of features by the base layer super prior coding network is expressed as Ha1(), then:
[0105] Z base =Ha1(Y base )
[0106] S411. Perform quantization processing and entropy coding on the super-prior features of the base layer to obtain first coded data.
[0107] Denote the base layer hyper-prior feature as Z base , quantization processing is represented by Q(), entropy coding is represented by E(), and the first coded data is represented by b1, then:
[0108] b1=E(Q(Z base ))
[0109] S412: Process the enhancement layer potential features through an enhancement layer super a priori coding network to obtain enhancement layer super a priori features.
[0110] In some embodiments, the network structure of the enhancement layer super-a priori coding network can be Figure 7 The network structure of the base layer super prior coding network is the same as shown in the figure. To avoid redundancy, it will not be described in detail here. enhance to obtain the basic layer super prior feature Z enhance The enhancement layer potential feature Y is sequentially obtained by the first convolution layer 71, the second convolution layer 72, the first downsampling layer 73, the third convolution layer 74 and the second downsampling layer 75. enhanceProcessing is performed to obtain the basic layer super prior feature Z enhance .
[0111] Denote the base layer latent features as Y enhance , the base layer hyper-prior feature is represented as Z enhance , the processing of features by the base layer super prior coding network is expressed as Ha2(), then:
[0112] Z enhance =Ha2(Y enhance )
[0113] S413: quantize and entropy encode the enhancement layer super-prior features to obtain second encoded data.
[0114] Related technologies generally process the base layer potential features and the enhancement layer potential features through the same super-prior coding network. However, the same super-analysis network is often not suitable for the generation of independent base layer super-prior features and enhancement layer super-prior features, and cannot achieve the best coding effect. The above embodiment uses two independent super-prior coding networks (base layer super-prior coding network and enhancement layer super-prior coding network) to generate base layer super-prior features and enhancement layer super-prior features respectively. Therefore, the above embodiment can better generate base layer super-prior features and enhancement layer super-prior features, thereby further improving coding performance.
[0115] S414. Divide the quantized latent features into base layer quantized latent features and enhanced layer quantized latent features according to the channel attention mask of the latent features.
[0116] That is, according to the channel attention mask Mask obtained in step S406, the quantized potential feature y obtained in step S402 is divided into the base layer quantized potential feature y base and the enhancement layer quantized latent features y enhance .
[0117] In some embodiments, the above-mentioned step S414 (dividing the quantized latent feature into a base layer quantized latent feature and an enhanced layer quantized latent feature according to the channel attention mask of the latent feature) includes: performing a dot multiplication operation on the quantized latent feature and the channel attention mask to obtain the base layer quantized latent feature; calculating the difference between the quantized latent feature and the base layer quantized latent feature to obtain the enhanced layer quantized latent feature.
[0118] The quantized latent feature is denoted as y, and the base layer quantized latent feature is denoted as y base , the enhanced layer quantized latent features are represented as y enhance , the channel attention mask is represented as Mask, and the dot product operation is represented as Then we have:
[0119]
[0120]
[0121] S415: Decode the first encoded data to obtain super-prior features for base layer reconstruction.
[0122] In some embodiments, decoding the first encoded data includes: losslessly decoding the first encoded data.
[0123] The first coded data is represented as b1, the decoding operation is represented as D(), and the base layer reconstructed super prior features are represented as Then we have:
[0124]
[0125] S416: Process the base layer reconstruction super-prior features through a base layer super-prior decoding network to obtain base layer reconstruction potential features.
[0126] In some embodiments, reference Figure 8 As shown, the base layer super priori decoding network includes: a convolution layer 81, a first upsampling layer 82, a second upsampling layer 83, a third upsampling layer 84 and a fourth upsampling layer 85. The base layer super priori decoding network reconstructs the super prior features of the base layer Processing is performed to obtain the underlying features for reconstruction It includes: reconstructing the super prior features of the base layer through the convolution layer 81, the first upsampling layer 82, the second upsampling layer 83, the third upsampling layer 84 and the fourth upsampling layer 85 in sequence Processing is performed to obtain the underlying features for reconstruction
[0127] In some embodiments, the convolution layer 81, the first upsampling layer 82, the second upsampling layer 83, the third upsampling layer 84 and the fourth upsampling layer 85 are all convolution layers with a convolution kernel size of 3×3.
[0128] The base layer reconstruction hyper-prior feature is expressed as The base layer reconstructed latent features are expressed as The operation of the base layer super prior decoding network is expressed as Hs1(), then:
[0129]
[0130] S417: Decode the second coded data to obtain a super-prior feature for reconstructing the enhancement layer.
[0131] In some embodiments, decoding the second encoded data includes: losslessly decoding the second encoded data.
[0132] The first coded data is represented as b2, the decoding operation is represented as D(), and the base layer reconstructed super prior features are represented as Then we have:
[0133]
[0134] S418: Process the enhancement layer reconstruction super-prior features through an enhancement layer super-prior decoding network to obtain enhancement layer reconstruction potential features.
[0135] In some embodiments, the network structure of the enhancement layer super-a priori decoding network can be Figure 8 The network structure of the base layer super prior decoding network shown is the same, and to avoid redundancy, it will not be repeated here. The enhancement layer super prior decoding network reconstructs the super prior features of the enhancement layer The processing includes: reconstructing the super prior features of the base layer through the convolution layer 81, the first upsampling layer 82, the second upsampling layer 83, the third upsampling layer 84 and the fourth upsampling layer 85 in sequence Processing is performed to obtain the underlying features for reconstruction
[0136] The enhanced layer reconstruction super prior feature is expressed as The enhanced layer reconstructed latent features are expressed as The operation of the enhancement layer super-prior decoding network is expressed as Hs2(), then:
[0137]
[0138] S419. Perform entropy coding on the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain third coded data and fourth coded data.
[0139] In some embodiments, the above step S419 (entropy encoding the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain third encoded data and fourth encoded data) includes the following steps a to d:
[0140] Step a: acquiring enhancement layer context features according to the enhancement layer reconstructed potential features and the quantized potential features of the enhancement layer that have been encoded.
[0141] In some embodiments, obtaining enhancement layer context features based on the enhancement layer reconstructed potential features and the enhanced layer quantized potential features that have been encoded includes: inputting the enhancement layer reconstructed potential features and the enhanced layer quantized potential features that have been encoded into a spatial context network based on autoregressive convolution to obtain enhancement layer context features.
[0142] The essence of the spatial context network based on autoregressive convolution is a masked convolution layer. For example, refer to Fig. 9 As shown, Fig. 9 Schematic diagram of the mask convolution kernel used in the context network based on autoregressive convolution provided in the embodiment of the present application. Fig. 9 As shown, the spatial context network based on autoregressive convolution uses the encoded elements before the current encoding position 901 in the latent space to predict the context features corresponding to the current encoding position, and uses the context features to subsequently predict the probability distribution of the current encoding position. Among them, the encoded elements are Fig. 9 Elements at the encoding positions shown in the dashed box.
[0143] In some embodiments, the calculation formula of the spatial context network based on autoregressive convolution is:
[0144] g cm (x)=(M⊙W)x+b
[0145] Among them, x is the input feature of the spatial context network based on autoregressive convolution, M and W are the k×k mask and convolution weight respectively, ⊙ is the Hadamard operator, and b is the bias term.
[0146] Step b: entropy coding the next element of the enhancement layer quantized potential feature according to the enhancement layer context feature.
[0147] When entropy coding is performed on the enhancement layer quantized latent feature, each element of the enhancement layer quantized latent feature is entropy coded one by one in the order of the elements. When entropy coding is performed on the first element of the enhancement layer quantized latent feature, there is no element of the enhancement layer quantized latent feature that has been coded. Therefore, the input of the enhancement layer context network only includes the enhancement layer reconstruction latent feature. When entropy coding is performed on other subsequent elements of the enhancement layer quantized latent features, the elements of the enhancement layer quantized latent features that have been coded can be input into the enhancement layer context network.
[0148] The above steps a and b are repeatedly performed to complete the encoding of all elements of the enhanced layer quantized latent features, thereby obtaining the fourth encoded data.
[0149] Step c: obtaining base layer context features according to the base layer reconstructed potential features and the encoded base layer quantized potential features.
[0150] In some embodiments, base layer context features are obtained based on the base layer reconstructed potential features and the base layer quantized potential features that have been encoded, including: inputting the base layer reconstructed potential features and the base layer quantized potential features that have been encoded into a spatial context network based on autoregressive convolution to obtain base layer context features.
[0151] Step d: entropy encode the next element of the base layer quantized potential feature according to the base layer context feature.
[0152] Similarly, when entropy coding is performed on the base layer quantized latent features, each element of the base layer quantized latent features is entropy coded one by one in the order of the elements. When entropy coding is performed on the first element of the base layer quantized latent features, there is no element of the base layer quantized latent features that has been coded. Therefore, the input of the base layer context network only includes the base layer reconstructed latent features. When entropy coding is performed on other subsequent elements of the base layer quantized latent features, the elements of the base layer quantized latent features that have been coded can be input into the base layer context network.
[0153] Repeat the above steps c and d to complete the encoding of all elements of the base layer quantized potential features to obtain the third encoded data.
[0154] In some other embodiments, the above step S419 (entropy encoding the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain third coded data and fourth coded data) includes the following steps ① to ④:
[0155] Step ①, obtaining enhancement layer context features through the enhancement layer context network, the encoded enhancement layer quantized latent features and the enhancement layer reconstructed latent features.
[0156] Reference Fig.10 As shown, in some embodiments, the enhancement layer context network includes: a channel context module 11, a spatial context module 12, a splicing module 13 and an entropy parameter module 14. The enhancement layer context network quantizes the potential feature y according to the enhancement layer enhance and the enhancement layer to reconstruct the latent features Get the enhancement layer context feature Φ enhance The process includes the following steps ①a to ①d:
[0157] Step ①a: Reconstruct the potential features of the enhanced layer through the channel context module 11 Perform channel context feature extraction to obtain enhanced layer channel context features
[0158] The channel context module includes multiple convolutional layers and activation function layers connected in series between adjacent convolutional layers.
[0159] Reference Fig.11 As shown, in some embodiments, the channel context module 11 includes: a first convolution layer 1101, a first activation function layer 1102, a second convolution layer 1103, a second activation function layer 1104 and a third convolution layer 1105. The channel context module 11 is used to sequentially reconstruct the potential features of the enhancement layer through the first convolution layer 1101, the first activation function layer 1102, the second convolution layer 1103, the second activation function layer 1104 and the third convolution layer 1105. Processing to obtain the enhanced layer channel context features
[0160] In some embodiments, the first convolution layer 1101 and the second convolution layer 1103 are convolution layers with a convolution kernel size of 3×3; the third convolution layer 1105 is a convolution layer with a convolution kernel size of 5×5; the activation functions of the first activation function layer 1102 and the second activation function layer 1104 are Gaussian Error Linear Unit (GELU) activation functions.
[0161] Step ①b, the spatial context module 12 quantizes the potential feature y of the enhanced layer that has been encoded enhance Perform spatial context feature extraction to obtain enhanced layer spatial context features
[0162] In some embodiments, the spatial context module 12 may be an autoregressive convolution-based spatial context network.
[0163] The schematic diagram of the mask convolution kernel used by the spatial context network based on autoregressive convolution can be shown as follows Fig. 9 As shown, no detailed description is given here.
[0164] Step ①c: The splicing module 13 reconstructs the potential features of the enhancement layer Enhancement layer channel context features And the enhancement layer spatial context features Perform stitching to obtain the first stitching feature
[0165] In some embodiments, the stitching module 13 can reconstruct the potential features of the enhancement layer from the channel dimension. Channel context features And the enhancement layer spatial context features Perform stitching to obtain the first stitching feature
[0166] Step ①d, the entropy parameter module 14 performs the first splicing feature Processing is performed to obtain the enhanced layer context feature Φ enhance .
[0167] The entropy parameter module includes a plurality of convolutional layers connected in series.
[0168] Reference Fig.12 As shown, in some embodiments, the entropy parameter module 14 includes: a first convolution layer 1401, a second convolution layer 1402 and a third convolution layer 1403. The entropy parameter module 14 is used to sequentially convolute the first convolution layer 1401, the second convolution layer 1402 and the third convolution layer 1403 to obtain the first concatenated feature. Processing is performed to obtain the enhanced layer context feature Φ enhance .
[0169] Step ②: According to the enhancement layer context feature Φ enhance Quantize the latent feature y of the enhancement layer enhance The next element of is entropy encoded.
[0170] Repeat the above steps ① and ② to complete the enhancement layer quantization potential feature y enhance The fourth encoded data b4 can be obtained by entropy encoding all elements of .
[0171] Step 3: Quantify the potential feature y through the base layer context network and the enhancement layer enhance , base layer reconstruction potential features And the encoded base layer quantized potential features y base , get the base layer context features.
[0172] Reference Fig.13 As shown, in some embodiments, the base layer context network includes: a first splicing module 131, a channel context module 132, a spatial context module 133, a second splicing module 134 and an entropy parameter module 135. The enhancement layer context network quantizes the potential feature y according to the enhancement layer enhance , base layer reconstruction potential features And the encoded base layer quantized potential features y base The steps for obtaining the base layer context features include the following steps ③a to ③e:
[0173] Step ③a: The first splicing module 131 quantizes the potential features y of the enhancement layer enhance and the base layer to reconstruct the latent features Perform stitching to obtain the second stitching feature
[0174] In some embodiments, the first splicing module 131 can quantize the potential features y of the already encoded enhancement layer from the channel dimension. enhance and the base layer to reconstruct the latent features Perform stitching to obtain the second stitching feature
[0175] Step ③b: The channel context module 132 processes the second concatenated feature Perform channel context feature extraction to obtain the base layer channel context feature
[0176] The structure of the channel context module 132 can be similar to Fig.11 The structure of the channel context module shown is the same as that of the channel context module 132, except that the number of channels in the channel context module 132 is the same as that in the Fig.11 The number of channels of the channel context modules shown is different, and to avoid redundancy, they will not be described in detail here.
[0177] Step ③c, the spatial context module 133 quantizes the potential features y of the encoded base layer base Perform spatial context feature extraction to obtain the base layer spatial context features
[0178] In some embodiments, the spatial context module 133 may be an autoregressive convolution-based spatial context network.
[0179] The schematic diagram of the mask convolution kernel used by the spatial context network based on autoregressive convolution can be shown as follows Fig. 9 As shown, no detailed description is given here.
[0180] Step ③d: The second splicing module 134 reconstructs the potential features of the base layer Base layer channel context features And the base layer spatial context features Perform stitching to obtain the third stitching feature
[0181] In some embodiments, the second stitching module 134 can reconstruct the potential features of the base layer from the channel dimension. Base layer channel context features And the base layer spatial context features Perform stitching to obtain the third stitching feature
[0182] Step ③e, the entropy parameter module 135 performs the third splicing feature Processing is performed to obtain the base layer context feature Φ base .
[0183] The structure of the entropy parameter module 135 can be similar to Fig.12 The structures of the entropy parameter modules shown are the same, except that the number of channels of the entropy parameter module 125 is the same as that of Fig.12 The number of channels of the entropy parameter modules shown is different, and to avoid redundancy, they will not be described in detail here.
[0184] Step ④: According to the base layer context feature Φ base Quantize the latent features Φ of the base layer base The next element of is entropy encoded.
[0185] Repeat the above steps ③ and ④ to complete the base layer quantization potential feature y base The third encoded data b3 can be obtained by encoding all the elements of .
[0186] S420, generating encoding data of the target image according to the first encoding data, the second encoding data, the third encoding data and the fourth encoding data.
[0187] That is, the first encoded data b1, the second encoded data b2, the third encoded data b3 and the fourth encoded data b4 obtained in the above step are used as the encoded data of the target image, or the encoded data of the target image includes the first encoded data b1, the second encoded data b2, the third encoded data b3 and the fourth encoded data b4.
[0188] The present application also provides an image decoding method, referring to Fig.14 As shown, the image decoding method includes the following steps S141 to S146:
[0189] S141. Obtain the encoded data of the target image.
[0190] Among them, the encoding data of the target image includes: first encoding data, second encoding data, third encoding data and fourth encoding data, the first encoding data and the second encoding data are respectively encoding data obtained by entropy encoding the base layer super prior features and the enhancement layer super prior features of the target image, and the third encoding data and the fourth encoding data are respectively encoding data obtained by entropy encoding the base layer quantized latent features and the enhancement layer quantized latent features of the target image.
[0191] The implementation method of encoding the encoding data of the target image to obtain the first encoding data, the second encoding data, the third encoding data and the fourth encoding data can refer to the image encoding method provided in any of the above embodiments. To avoid repetition, it will not be described in detail here.
[0192] S142. Perform entropy decoding on the first coded data and the second coded data respectively to obtain a super priori feature for reconstructing a base layer and a super priori feature for reconstructing an enhancement layer.
[0193] S143. Process the base layer reconstruction super-prior features through a base layer super-prior decoding network to obtain base layer reconstruction potential features.
[0194] S144. Process the enhancement layer reconstruction super-prior features through an enhancement layer super-prior decoding network to obtain enhancement layer reconstruction potential features.
[0195] The network structures of the base layer super-a priori decoding network and the enhancement layer super-a priori decoding network in the above steps S413 and S414 can be the same as Figure 8 The network structure of the base layer super-prior decoding network shown is the same, and will not be described in detail here to avoid redundancy.
[0196] S145. Perform entropy decoding on the third coded data and the fourth coded data according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain the base layer reconstruction quantization potential feature and the enhancement layer reconstruction quantization potential feature.
[0197] S146. Obtain a reconstructed image of the target image according to the base layer reconstructed quantized latent features and the enhanced layer reconstructed quantized latent features.
[0198] The image decoding method provided in the embodiment of the present application can reconstruct the image encoded by the image encoding method provided in the above embodiment. Therefore, the image decoding method provided in the embodiment of the present application can improve the encoding performance and machine vision performance of the scalable image encoder based on machine learning while ensuring the normal decoding of the image.
[0199] As an extension and refinement of the above embodiment, the present application embodiment provides another image decoding method, referring to Fig.15 As shown, the image decoding method comprises the following steps:
[0200] S1501. Obtain the encoded data of the target image.
[0201] Among them, the encoding data of the target image includes: first encoding data, second encoding data, third encoding data and fourth encoding data, the first encoding data and the second encoding data are respectively encoding data obtained by entropy encoding the base layer super prior features and the enhancement layer super prior features of the target image, and the third encoding data and the fourth encoding data are respectively encoding data obtained by entropy encoding the base layer quantized latent features and the enhancement layer quantized latent features of the target image.
[0202] That is, the coded data of the target image includes the first coded data b1, the second coded data b2, the third coded data b3 and the fourth coded data b4 obtained by the image coding method provided by any of the above embodiments.
[0203] S1502: Perform entropy decoding on the first coded data to obtain super-prior features for base layer reconstruction.
[0204] S1503: Process the base layer reconstruction super-prior features through a base layer super-prior decoding network to obtain base layer reconstruction potential features.
[0205] S1504: Decode the second coded data to obtain a super-prior feature for reconstructing an enhancement layer.
[0206] S1505: Process the enhancement layer reconstruction super-prior features through an enhancement layer super-prior decoding network to obtain enhancement layer reconstruction potential features.
[0207] S1506. Perform entropy decoding on the third coded data and the fourth coded data according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain the base layer reconstruction quantization potential feature and the enhancement layer reconstruction quantization potential feature.
[0208] In some embodiments, the above step S1506 (entropy decoding the third coded data and the fourth coded data according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain the base layer reconstruction quantization potential feature and the enhancement layer reconstruction quantization potential feature) includes the following steps 1) to 4):
[0209] Step 1) obtaining enhancement layer context features according to the enhancement layer reconstructed potential features.
[0210] Step 2) entropy decoding the next element of the fourth encoded data according to the enhancement layer context feature.
[0211] Repeat the above steps 1) and 2) to complete the decoding of all elements of the fourth coded data to obtain the enhanced layer reconstruction quantization potential features.
[0212] Step 3) Obtaining base layer context features based on the base layer reconstructed potential features.
[0213] Step 4) entropy decoding the next element of the third encoded data according to the base layer context feature.
[0214] Repeat the above steps 3) and 4) to complete the decoding of all elements of the third coded data to obtain the base layer reconstruction quantization potential features.
[0215] In some embodiments, the above step S1506 (entropy decoding the third coded data and the fourth coded data according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain the base layer reconstruction quantization potential feature and the enhancement layer reconstruction quantization potential feature) includes the following steps 1 to 4:
[0216] Step 1: Obtain enhancement layer context features through the enhancement layer context network, the reconstructed enhancement layer quantized latent features (the decoding result of the decoded fourth coded data) and the enhancement layer reconstruction latent features.
[0217] Since the fourth coded data is coded data obtained by encoding the enhancement layer quantized latent features, the decoding result of the decoded fourth coded data is the reconstructed enhancement layer quantized latent features.
[0218] The network structure of the enhanced layer context network can be referred to Fig.10 As shown, the method for implementing the enhancement layer context network to obtain the enhancement layer context features based on the enhancement layer quantization potential features that have been reconstructed can refer to steps ①a to ①d in the above embodiment, the only difference is that steps ①a to ①d use the enhancement layer quantization potential features that have been encoded, while the present embodiment uses the enhancement layer quantization potential features that have been reconstructed.
[0219] Step 2: entropy decode the next element of the fourth encoded data according to the enhancement layer context feature.
[0220] Repeat the above steps 1 and 2 to complete the decoding of all elements of the fourth coded data to obtain the enhanced layer reconstruction quantization potential features.
[0221] Step three: obtain the base layer context features through the base layer context network, the enhanced layer reconstructed quantized latent features, the base layer reconstructed quantized latent features and the reconstructed base layer reconstructed quantized latent features (the decoding result of the decoded third encoded data).
[0222] Since the embodiment of the present application has acquired the enhanced layer reconstruction quantization potential features through the above steps 1 and 2, the embodiment of the present application can further acquire the base layer context features by using the enhanced layer reconstruction quantization potential features.
[0223] The network structure of the base layer context network can be referred to Fig.13As shown, the implementation method of the base layer context network obtaining the base layer context features according to the enhanced layer reconstructed quantization potential features, the base layer reconstructed quantization potential features and the reconstructed base layer reconstructed quantization potential features can refer to steps ③a to ③d in the above embodiment. The only difference is that the encoded base layer quantization potential features are used in steps ③a to ③d, while the reconstructed base layer quantization potential features are used in this embodiment.
[0224] Step 4: entropy decode the next element of the third encoded data according to the base layer context feature.
[0225] Since the third coded data is the coded data obtained by encoding the base layer quantization potential features, the decoding result of the decoded fourth coded data is the reconstructed base layer quantization potential features. Repeat the above steps 3 and 4 to complete the decoding of all elements of the third coded data to obtain the base layer reconstructed quantization potential features.
[0226] S1507 , concatenating the base layer reconstructed quantized latent features and the enhanced layer reconstructed quantized latent features to obtain reconstructed quantized latent features.
[0227] In some embodiments, concatenating the base layer reconstructed quantization potential feature and the enhancement layer reconstructed quantization potential feature comprises: adding feature values of corresponding positions of the base layer reconstructed quantization potential feature and the enhancement layer reconstructed quantization potential feature.
[0228] The base layer reconstructed quantized latent features are expressed as The enhanced layer reconstructs the quantized latent features and is expressed as The reconstructed quantized latent features are expressed as The concatenation operation is represented by +, then:
[0229]
[0230] S1508: Input the reconstructed quantized latent features into a synthetic transformation network, and obtain a reconstructed image of the target image output by the synthetic transformation network.
[0231] In some embodiments, reference Fig.16 As shown, the synthetic transformation network includes: a plurality of upsampling convolutional layers 161 and a plurality of residual activation units 162 connected in series between adjacent upsampling convolutional layers. Fig.16In the example, a synthetic transformation network includes four upsampling convolutional layers 161 and three residual activation units 162 connected in series between adjacent upsampling convolutional layers 161. The residual activation unit 162 includes: a first convolutional layer 1621, an activation function layer 1622, a second convolutional layer 1623 and a fusion layer 1624. The residual activation unit 162 is used to process the input feature I2 of the residual activation unit 162 through the first convolutional layer 1621, the activation function layer 1622 and the second convolutional layer 1623 in sequence, and add and fuse the output feature of the second convolutional layer 1623 and the input feature I2 of the residual activation unit through the fusion layer 1624 to obtain the output feature O2 of the residual activation unit 162.
[0232] In some embodiments, each upsampling convolution layer 161 in the synthetic transformation network is an upsampling convolution layer with a convolution kernel size of 3×3, the first convolution layer 1621 and the second convolution layer 1623 in each residual activation unit of the synthetic transformation network are convolution layers with a convolution kernel size of 3×3, and the activation area function of the activation function layer 1622 in each residual activation unit 162 of the synthetic transformation network is a LeakyReLU activation function.
[0233] The reconstructed quantized latent features are expressed as The reconstructed image is represented as The processing of features by the synthetic transformation network is represented as g s (), then:
[0234]
[0235] S1509: Process the reconstructed quantized latent features of the base layer based on a latent space transform network (Latent Space Transform, LST) to obtain machine vision features.
[0236] The base layer reconstructed quantized latent features are expressed as Machine vision features are expressed as The processing of features by the synthetic transformation network is expressed as T(), then:
[0237]
[0238] Reference Fig.17As shown, in some embodiments, the latent space transformation network includes: a first residual block 171, a first up-sampled residual block 172, a second residual block 173, a second up-sampled residual block 174, a third residual block 175, a third up-sampled residual block 176, a fourth residual block 177, an up-sampled convolution layer 178, and an activation function layer 179. The up-sampling multiples of the first up-sampled residual block 172, the second up-sampled residual block 174, the third up-sampled residual block 176, and the up-sampled convolution layer 178 are r1, r2, r3, and r4, respectively.
[0239] Reference Fig.17 As shown, in some embodiments, the residual blocks (first residual block 171, second residual block 173, third residual block 175, fourth residual block 177) in the latent space transformation network include: a first convolution layer 1701, a first LeakyLeLU activation function layer 1702, a second convolution layer 1703, a second LeakyLeLU activation function layer 1704, and a fusion module 1705. The feature processing step of the residual block includes: firstly, processing the input feature I3 of the residual block through the first convolution layer 1701, the first LeakyLeLU activation function layer 1702, the second convolution layer 1703, and the second LeakyLeLU activation function layer 1704 in sequence, and then adding and fusing the output feature of the second LeakyLeLU activation function layer 1704 and the input feature I3 of the residual block through the fusion module 1705 to obtain the output feature O3 of the residual block.
[0240] Reference Fig.17 As shown, in some embodiments, the upsampling residual blocks (first upsampling residual block 172, second upsampling residual block 174, third upsampling residual block 176) in the latent space transformation network include: an upsampling convolution layer 1706, a LeakyLeLU activation function layer 1707, a third convolution layer 1708, an inverse generalized divisive normalization (IGDN) layer 1709, a fourth convolution layer 1710 and a fusion module 1711. The feature processing steps of the upsampled residual block include: first, the upsampled convolution layer 1706, the LeakyLeLU activation function layer 1707, the third convolution layer 1708, and the IGDN layer 1709 process the input feature I4 of the upsampled residual block, and the fourth convolution layer 1710 processes the input feature I4 of the upsampled residual block, and then the output of the IGDN layer 1709 and the output of the fourth convolution layer 1710 are added and fused through the fusion module 1711 to obtain the output feature O4 of the upsampled residual block.
[0241] S1510. Input the machine vision feature into a machine vision task model to obtain a machine vision task processing result output by the machine vision task model.
[0242] The machine vision task model in the embodiments of the present application may be an image / video super-resolution network or a target (specified object such as a human face) detection network based on machine learning models such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a long short-term memory network (LSTM).
[0243] The processing of features by the machine vision task model is represented as V(), and the machine vision features are represented as The result of machine vision task processing is expressed as M, then:
[0244]
[0245] In some embodiments, the machine vision task model may be a model structure after the 13th layer of a YOLOv3 (You Only Look Once) model, and the machine vision task processing result is an average target detection accuracy. That is, the above step S112 includes: inputting the machine vision feature into the 13th layer of the YOLOv3 model to obtain the average target detection accuracy output by the YOLOv3 model.
[0246] Reference Fig.18 As shown, in Figure 2 Based on the image decoder shown in FIG. 1 , the image decoder in some embodiments of the present application further includes: a content adaptive filtering network 213. The content adaptive filtering network 213 is used to reconstruct the image Enhance to obtain enhanced image
[0247] Reference Fig.19As shown, in some embodiments, the content adaptive filtering network 213 includes: a first upsampling module 1901, a second upsampling module 1902, a first feature extraction module 1903, a second feature extraction module 1904, a third feature extraction module 1905, a splicing module 1906, a content adaptive inter-channel correlation information filter (Content adaptive Inter Channel Correlation Information filter, ICCI) 1907, a first processing module 1908, a second processing module 1909 and a third processing module 1910, a first fusion module 1911, a second fusion module 1912 and a third fusion module 1913, a first reconstruction module 1914, a second reconstruction module 1915 and a third reconstruction module 1916. The content adaptive filtering network 213 reconstructs the image The enhancement process includes: Step 1, respectively using the first upsampling module 1901 and the second upsampling module 1902 to represent the initial first chrominance component of the reconstructed image as and the initial second chrominance component is expressed as Upsample to obtain the same brightness component as the original The upsampled first chroma component with the same spatial size is expressed as And the upsampled second chroma component is expressed as Step 2: The initial brightness component is extracted by the first feature extraction module 1903, the second feature extraction module 1904, and the third feature extraction module 1905 respectively. The upsampled first chroma component is expressed as And the upsampled second chroma component is expressed as Perform feature extraction to obtain brightness features The first chromaticity characteristic The second chromaticity characteristic Step 3: Use the splicing module 1906 to combine the brightness features The first chromaticity characteristic The second chromaticity characteristic Splice to obtain the initial image feature F YUV Step 4: Use ICCI1907 to identify the initial image features F YUV Process and obtain enhanced features F enha Step 5: The first processing module 1908, the second processing module 1909 and the third processing module 1910 are used to enhance the feature F enha Processing to obtain brightness enhancement features First chroma enhancement feature and a second chroma enhancement feature Step 6: Fusion of brightness features through the first fusion module 1911 and brightness enhancement features Get enhanced brightness features Step 7: Fusing the first chromaticity feature through the second fusion module 1912 and the first chroma enhancement feature Get enhanced first chromaticity feature Step 8: Fusing the second chromaticity feature through the third fusion module 1913 and the second chroma enhancement feature Get enhanced second chromaticity feature Step 9: Reconstruct the brightness features of the image by using the first reconstruction module 1914, the second reconstruction module 1915 and the third reconstruction module 1916 respectively. Enhance the first brightness feature And enhance the second brightness feature Processing to obtain enhanced brightness component The enhanced first chrominance component is expressed as And the enhanced second chrominance component is expressed as Step 10: Combine and enhance the brightness component The enhanced first chrominance component is expressed as And the enhanced second chrominance component is expressed as Get enhanced image
[0248] Reference Fig. 20 As shown, based on any of the above-mentioned image decoding methods, the image decoding method provided in the embodiment of the present application further includes reconstructing the target image by using the content adaptive filtering network 213 Follow these steps:
[0249] S201. Acquire an initial luminance component, an initial first chrominance component, and an initial second chrominance component according to the reconstructed image.
[0250] That is, obtain the reconstructed image The Y component, U component and V component of the .
[0251] S202: Generate an upsampled first chroma component and an upsampled second chroma component based on the initial first chroma component and the initial second chroma component, respectively.
[0252] The spatial size of the upsampled first chroma component and the upsampled second chroma component are both the same as the spatial size of the initial luminance component.
[0253] That is, the initial first chroma component and the initial second chroma component are respectively up-sampled to chroma components having the same spatial size as the initial luminance component.
[0254] In some embodiments, bicubic interpolation may be performed on the initial first chroma component and the initial second chroma component, respectively, so as to obtain the upsampled first chroma component and the upsampled second chroma component, respectively.
[0255] The initial first chrominance component is expressed as The initial second chrominance component is expressed as The upsampled first chrominance component is expressed as The upsampled second chrominance component is expressed as The bicubic interpolation operation is represented by BI(), and the above step S202 can be represented by the following formula:
[0256]
[0257]
[0258] S203, performing feature extraction on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain a luminance feature, a first chrominance feature, and a second chrominance feature.
[0259] In some embodiments, the above step S203 (respectively extracting features from the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component to obtain luminance features, first chrominance features, and second chrominance features) includes:
[0260] Forward wavelet transform (WT) is performed on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component, respectively, to obtain the luminance feature, the first chrominance feature, and the second chrominance feature.
[0261] The initial brightness component is expressed as The upsampled first chrominance component is expressed as The upsampled second chrominance component is expressed as The forward wavelet transform is expressed as DWT(), and the brightness feature is expressed as The first chromaticity feature is expressed as The second chromaticity feature is expressed as Then the above S203 can be expressed as the following formula:
[0262]
[0263]
[0264]
[0265] S204: Acquire initial image features of the reconstructed image according to the brightness feature, the first chromaticity feature, and the second chromaticity feature.
[0266] In some embodiments, the initial image features of the reconstructed image are obtained based on the brightness feature, the first chromaticity feature, and the second chromaticity feature, including: splicing the brightness feature, the first chromaticity feature, and the second chromaticity feature from the channel dimension to obtain the initial image features of the reconstructed image.
[0267] The brightness feature is expressed as The first chromaticity feature is expressed as The second chromaticity feature is expressed as The concatenation operation in the channel dimension is represented as Concatenatio(), and the initial image features of the reconstructed image are represented as F YUV , then the above step S204 can be expressed as the following formula:
[0268]
[0269] S205 , extracting features from the initial image features through a content-adaptive inter-channel correlation information filter ICCI to obtain enhanced features.
[0270] Reference Fig.21 As shown, in some embodiments, the content adaptive channel-to-channel correlation information filter includes: a convolution layer 2101, a batch normalization layer 2102, an activation function layer 2103, and a residual block sequence 2104 composed of multiple residual blocks. The feature processing flow of the content adaptive channel-to-channel correlation information filter includes: processing the initial image features through the convolution layer 2101, the batch normalization layer 2102, the activation function layer 2103, and the residual block sequence 2104 in sequence, and scaling the output features of the residual block sequence 2104 by γ times.
[0271] In some embodiments, the convolution layer 2101 is a convolution layer with a convolution kernel size of 3×3.
[0272] In some embodiments, the activation function of the activation function layer 2103 is a ReLU activation function.
[0273] In some embodiments, the residual block sequence 2104 consists of 8 1D residual blocks.
[0274] Reference Fig. 22As shown, in some embodiments, the residual block in the residual block sequence 2104 includes: a first convolution layer 2201, a first batch of normalization layers 2202, a ReLU activation function layer 2203, a second convolution layer 2204, a second batch of normalization layers 2205, and a fusion unit 2206. The residual block is used to process the input feature I5 of the residual block through the first convolution layer 2201, the first batch of normalization layers 2202, the ReLU activation function layer 2203, the second convolution layer 2204, and the second batch of normalization layers 2205 in sequence, and perform γ on the output feature I5 of the second batch of normalization layers 2205. n The scaled feature is scaled by times, and the scaled feature and the input feature I5 of the residual block are added and fused by the fusion unit 2206 to obtain the output feature O5 of the residual block.
[0275] In some embodiments, the above step S205 (performing feature enhancement processing on the brightness feature, the first chromaticity feature and the second chromaticity feature respectively based on the enhanced feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature and an enhanced second chromaticity feature) includes:
[0276] Step 2051: Perform convolution processing on the initial image feature to obtain a first feature.
[0277] In some embodiments, the initial image feature may be convolved through a convolution layer with a convolution kernel size of 3×3 to obtain the first feature.
[0278] The initial image features of the reconstructed image are represented as F YUV , the convolution process is represented as Conv 3×3 (), the first feature is represented by F1, then the above step 2051 can be expressed as the following formula:
[0279] F1=Conv 3×3 (F YUV )
[0280] Step 2052: Perform batch normalization on the first feature to obtain a second feature.
[0281] In some embodiments, the first features may be batch normalized by a batch normalization layer to obtain the second features.
[0282] The first feature is represented as F1, the batch normalization process is represented as BN(), and the second feature is represented as F2. Then the above step 2052 can be represented as the following formula:
[0283] F2=BN(F1)
[0284] Step 2053: perform linear activation processing on the second feature to obtain the third feature.
[0285] In some embodiments, performing linear activation processing on the second feature to obtain the third feature includes: e The LU activation function layer performs linear activation processing on the second feature to obtain the third feature.
[0286] The second feature is represented as F2, the linear activation process is represented as ReLU(), and the third feature is represented as F3. Then the above step 2053 can be represented as the following formula:
[0287] F3 = ReLU(F2)
[0288] Step 2054: Process the third feature through a residual network to obtain the enhanced feature.
[0289] The residual network includes a plurality of residual blocks connected in series, and the residual block is used to sequentially perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.
[0290] In some embodiments, the residual network includes 8 serially connected residual blocks. The structure of the residual block can refer to Fig. 22 The structure of the residual block shown is not described in detail here to avoid redundancy.
[0291] The third feature is denoted as F3, the processing of the residual network is denoted as ReseN(), and the enhanced feature is denoted as F enha , then the above step 2054 can be expressed as the following formula:
[0292] F enha =ReseN(F3)
[0293] S206. Perform feature enhancement processing on the brightness feature, the first chromaticity feature, and the second chromaticity feature respectively based on the enhanced feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature.
[0294] Step 2061: Perform convolution processing on the enhancement feature to obtain a brightness enhancement feature.
[0295] In some embodiments, the enhancement feature may be convolved with a convolution kernel having a convolution kernel size of 3×3 to obtain the brightness enhancement feature.
[0296] Denote the enhanced feature as F enha, the convolution process is represented as Conv 3×3 (), the brightness enhancement feature is expressed as Then the above step 2061 can be expressed as the following formula:
[0297]
[0298] Step 2062: Add and fuse the brightness enhancement feature and the brightness feature to obtain the enhanced brightness feature.
[0299] The brightness enhancement feature is expressed as Additive fusion is represented by +, and enhanced brightness features are represented by Then the above step 2062 can be expressed as the following formula:
[0300]
[0301] Step 2063: Process the enhanced features through a first residual block sequence to obtain a first residual feature.
[0302] Among them, the first residual block sequence includes multiple serially connected residual blocks, and the residual block is used to perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block in sequence to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.
[0303] In some embodiments, the first residual block sequence includes 5 serially connected residual blocks. The structure of the residual block can refer to Fig. 22 The structure of the residual block shown is not described in detail here to avoid redundancy.
[0304] Denote the enhanced feature as F enha , the processing of the first residual block sequence is represented as RBS U (), the first residual feature is represented by Res U , then the above step 2063 can be expressed as the following formula:
[0305] Res U =RBS U (F en a )
[0306] Step 2064: Perform convolution processing on the first residual feature to obtain a first chroma enhancement feature.
[0307] The first residual feature is denoted as Res U , the convolution process is represented as Conv 3×3 (), the first chroma enhancement feature is expressed as Then the above step 2064 can be expressed as the following formula:
[0308]
[0309] Step 2065: Add and fuse the first chromaticity feature and the first chromaticity enhancement feature to obtain the enhanced first chromaticity feature.
[0310] The first chroma enhancement feature is expressed as Additive fusion is represented by +, and the enhanced first chromaticity feature is represented by Then the above step 2065 can be expressed as the following formula:
[0311]
[0312] Step 2066: Process the enhanced features through a second residual block sequence to obtain a second residual feature.
[0313] Among them, the second residual block sequence includes multiple serially connected residual blocks, and the residual block is used to perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block in sequence to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.
[0314] In some embodiments, the second residual block sequence includes 5 serially connected residual blocks. The structure of the residual block can refer to Fig. 22 The structure of the residual block shown is not described in detail here to avoid redundancy.
[0315] Denote the enhanced feature as F enha , the processing of the second residual block sequence is represented as RBS V (), the second residual feature is represented by Res V , then the above step 2066 can be expressed as the following formula:
[0316] Res V =RBS V (F en a )
[0317] Step 2067: Perform convolution processing on the second residual feature to obtain a second chroma enhancement feature.
[0318] The second residual feature is denoted as Res V , the convolution process is represented as Conv 3×3 (), the second chroma enhancement feature is expressed as Then the above step 2067 can be expressed as the following formula:
[0319]
[0320] Step 2068: Add and fuse the second chromaticity feature and the second chromaticity enhancement feature to obtain the enhanced second chromaticity feature.
[0321] The second chromaticity feature is expressed as The second chroma enhancement feature is expressed as Additive fusion is represented by +, and the enhanced second chromaticity feature is represented by Then the above step 2068 can be expressed as the following formula:
[0322]
[0323] S207. Acquire an enhanced image of the reconstructed image according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature.
[0324] In some embodiments, the image encoding method or image decoding method provided by the embodiments of the present application further includes: training each network model in the image encoder and the image decoder based on a sample data set.
[0325] Wherein, the sample data set includes: multiple sample data groups, and any sample data group includes: sample images and machine vision task processing results corresponding to the sample images. The process of training each network model in the image encoder and image decoder based on the sample data set may include: first inputting the sample image into the image encoder for encoding to obtain the encoded data of the image, and then decoding the encoded data through the image decoder to obtain the reconstructed image and the machine vision task processing results output by the machine vision task model, and then calculating the first loss value based on the reconstructed image and the sample image, calculating the second loss value based on the machine vision task processing results corresponding to the sample image and the machine vision task processing results output by the machine vision task model, and then adjusting the parameters of the network model in the image encoder and / or image decoder based on the first loss value and the second loss value until the image encoder and the image decoder converge.
[0326] In some embodiments, some embodiments of the present application provide an image encoding device, the image encoding device comprising:
[0327] a memory configured to store a computer program;
[0328] The processor is configured to, when calling a computer program, enable the image encoding device to implement the image encoding method described in any one of the above embodiments.
[0329] In some embodiments, some embodiments of the present application provide an image decoding device, the image decoding device comprising:
[0330] a memory configured to store a computer program;
[0331] The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method described in any one of the above embodiments.
[0332] In some embodiments, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image encoding method described in any of the above embodiments or the image decoding method described in any of the above embodiments.
[0333] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image encoding method described in any of the above embodiments or the image decoding method described in any of the above embodiments.
[0334] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0335] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. An image coding method, characterized in that: include: Obtaining latent features of the target image; Quantizing the latent feature based on a preset quantization parameter to obtain a quantized latent feature; Processing the quantized latent feature based on a channel attention mechanism to obtain a channel attention mask of the latent feature; dividing the latent features into base layer latent features and enhancement layer latent features according to a channel attention mask of the latent features; The encoded data of the target image is generated according to the base layer latent features and the enhancement layer latent features.
2. The method according to claim 1, characterized in that The acquiring of potential features of the target image comprises: The target image is processed based on an analysis transformation network to obtain potential features of the target image.
3. The method according to claim 1, characterized in that The processing of the quantized latent feature based on the channel attention mechanism to obtain a channel attention mask of the latent feature includes: Performing adaptive pooling processing on the quantized latent features to obtain a first feature; Performing convolution processing, normalization processing, and nonlinear activation processing on the first feature in sequence to obtain a second feature; Performing convolution processing on the second feature to obtain a third feature and a fourth feature, wherein the feature tensors of the third feature and the fourth feature are both C×1×1, where C is the number of channels of the potential feature; Gumbel sampling is performed on the third feature and the fourth feature to obtain a channel attention mask of the latent feature.
4. The method according to claim 1, characterized in that: The step of dividing the potential feature into a base layer potential feature and an enhancement layer potential feature according to the channel attention mask of the potential feature comprises: Performing a dot product operation on the latent feature and the channel attention mask to obtain the base layer latent feature; The difference between the potential feature and the base layer potential feature is calculated to obtain the enhancement layer potential feature.
5. The method according to claim 1, characterized in that Before generating the encoding data of the target image according to the base layer potential features and the enhancement layer potential features, the method further comprises: Adjusting the channels with zero feature values in the base layer potential feature to the first end of the base layer potential feature, and adjusting the channels with zero feature values in the enhancement layer potential feature to the second end of the enhancement layer potential feature; The first end is the head end and the second end is the tail end; or the first end is the tail end and the second end is the head end.
6. The method according to claim 1, characterized in that The step of generating the coded data of the target image according to the base layer potential features and the enhancement layer potential features comprises: Processing the base layer potential features through a base layer super priori coding network to obtain base layer super priori features; quantizing and entropy encoding the base layer super-prior features to obtain first encoded data; Processing the enhancement layer potential features through an enhancement layer super-prior coding network to obtain enhancement layer super-prior features; quantizing and entropy encoding the enhancement layer super-prior feature to obtain second encoded data; The encoded data of the target image is generated based on the first encoded data and the second encoded data.
7. The method according to claim 6, characterized in that The method further comprises: dividing the quantized latent features into base layer quantized latent features and enhancement layer quantized latent features according to a channel attention mask of the latent features; Decoding the first coded data to obtain a super-prior feature of base layer reconstruction; Processing the base layer reconstruction super-prior features according to the base layer super-prior decoding network to obtain base layer reconstruction potential features; Decoding the second coded data to obtain a super-prior feature for reconstruction of an enhancement layer; Processing the enhancement layer reconstruction super-prior features according to the enhancement layer super-prior decoding network to obtain enhancement layer reconstruction potential features; entropy encoding the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain third coded data and fourth coded data; The generating the encoding data of the target image according to the first encoding data and the second encoding data includes: generating the encoding data of the target image according to the first encoding data, the second encoding data, the third encoding data and the fourth encoding data.
8. The method according to claim 7, characterized in that The entropy encoding the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature comprises: Acquire enhancement layer context features according to the enhancement layer reconstruction potential features and the enhancement layer quantization potential features that have been encoded; entropy encoding a next element of the enhancement layer quantized latent feature according to the enhancement layer context feature; Acquire base layer context features according to the base layer reconstructed potential features and the encoded base layer quantized potential features; The next element of the base layer quantized latent feature is entropy encoded according to the base layer context feature.
9. The method according to claim 7, wherein entropy encoding the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature comprises: Acquire enhancement layer context features according to the enhancement layer reconstruction potential features and the enhancement layer quantization potential features that have been encoded; Encoding the following elements of the enhancement layer quantized latent features according to the enhancement layer context features; Acquire a base layer context feature according to the enhancement layer quantization potential feature, the base layer reconstruction potential feature and the encoded base layer quantization potential feature; The following elements of the base layer quantized latent features are encoded according to the base layer context features.
10. The method according to claim 9, wherein entropy encoding the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain third coded data and fourth coded data comprises: The channel context feature extraction is performed on the enhanced layer reconstruction potential features through the channel context module to obtain the enhanced layer channel context features; The spatial context feature extraction is performed on the quantized latent features of the enhanced layer that have been encoded by the spatial context module to obtain the spatial context features of the enhanced layer; splicing the enhancement layer reconstruction potential feature, the enhancement layer channel context feature, and the enhancement layer spatial context feature to obtain a first splicing feature; Processing the first splicing feature through an entropy parameter module to obtain an enhancement layer context feature; The next element of the enhancement layer quantized latent feature is entropy encoded according to the enhancement layer context feature.
11. The method according to claim 9, characterized in that The entropy encoding of the enhancement layer quantization potential feature and the base layer quantization potential feature according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain third coded data and fourth coded data also includes: splicing the enhancement layer quantized latent feature and the base layer reconstructed latent feature to obtain a second spliced feature; Performing channel context feature extraction on the second splicing feature through a channel context module to obtain a base layer channel context feature; The spatial context feature is extracted from the encoded base layer quantized latent features through the spatial context module to obtain the base layer spatial context feature; splicing the base layer reconstruction potential feature, the base layer channel context feature and the base layer spatial context feature to obtain a third splicing feature; Processing the third concatenated feature through an entropy parameter module to obtain a base layer context feature; The next element of the base layer quantized latent feature is entropy encoded according to the base layer context feature.
12. An image encoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to, when calling a computer program, enable the image encoding device to implement the image encoding method according to any one of claims 1 to 11.
Citation Information
Cited By
Extensible image coding method based on collaboration of human eyes and machine vision
CN119946274A
Privacy enhanced image compression method
CN121193942A
A privacy-enhanced image compression method
CN121193942B
