Image decoding method and device
By acquiring and utilizing the basic and enhancement layers in the image encoder to reconstruct potential features, the problem of failing to fully utilize the relationship between the two in the prior art is solved, and image encoding and machine vision performance is improved.
Patent Information
- Application Number
- CN202311575554.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-10-27
- Filing Date
- 2023-11-23
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-11-23
AI Technical Summary
Existing machine learning-based scalable image encoder fails to fully utilize the correlation between the potential features of the enhancement layer quantization and the potential features of the underlying layer quantization, resulting in the need to improve encoding performance and machine vision performance.
By acquiring the underlying features of the target image and reconstructing potential features, quantizing latent features with the enhancement layer and reconstructing quantizing latent features, predicting the context features of the base layer, thereby improving coding performance and machine vision performance.
By fully leveraging the correlation between the enhancement layer and the base layer, the encoding performance and machine vision performance of the scalable image encoder based on machine learning is improved.
Smart Images

Figure CN119094786A_ABST
Abstract
Description
[0001] This application claims the priority of the Chinese patent application filed with the Chinese Patent Office on June 6, 2023 with application number 2023106645409, and the priority of the Chinese patent application filed with the Chinese Patent Office on October 27, 2023 with application number 2023114114882. The contents of the prior applications are incorporated into this application by reference. Technical Field
[0002] Some embodiments of the present application relate to the field of image coding and decoding technology, and more specifically, to an image decoding method and device. Background Art
[0003] At present, image / image coding algorithms are generally optimized for the human visual system and suitable for human viewing applications, but they are not optimized for other application scenarios, such as the very common image processing and machine vision applications. For these applications, it is often necessary to fully decode the image to obtain the reconstructed image / image before the reconstructed image / image can be used as the input of these applications. This process is complex and time-consuming, and the lossy image after decoding often has a negative impact on the accuracy of machine vision tasks. Reducing the bit rate while maintaining the subjective quality of the image unchanged and providing a representation suitable for direct use as machine vision input data has become an urgent problem to be solved. Therefore, efficient machine image coding (Video Voding for Machines, VCM) has become an important topic in academia and industry. VCM technology is committed to connecting the two independent research directions of image / image compression and feature compression to a certain extent, while jointly optimizing high-accuracy machine vision tasks and high-fidelity human vision reconstruction as much as possible. Based on this, the Moving Picture Expert Group (MPEG) established a group called VCM to study the needs of VCM technology standardization. JPEG AI is a scalable image coding standard based on machine learning created by the Joint Photographic Experts Group (JPEG). Compared with traditional image coding, JPEG AI achieves a lower bit rate at the same subjective quality based on human vision. At the same time, the JPEG AI model also puts forward corresponding requirements for VCM.
[0004] In a scalable image encoder based on machine learning, it is necessary to predict the context features of the enhancement layer potential features and the context features of the base layer potential features respectively through a context network. At present, the relevant technologies generally predict the enhancement layer context features and the base layer context features based on the enhancement layer quantized potential features and the base layer quantized potential features respectively, and fail to fully utilize the correlation between the enhancement layer quantized potential features and the base layer quantized potential features. Therefore, the performance of the scalable image encoder based on machine learning needs to be further improved. Summary of the invention
[0005] The exemplary embodiments of the present application provide an image encoding method, an image decoding method and an apparatus for improving the encoding performance and machine vision performance of a scalable image encoder based on machine learning.
[0006] Some embodiments of the present application provide the following technical solutions:
[0007] In a first aspect, some embodiments of the present application provide an image decoding method, including:
[0008] Acquire a base layer reconstruction potential feature and an enhancement layer reconstruction potential feature of the target image according to the encoded data of the target image;
[0009] Acquire enhancement layer context features according to the enhancement layer reconstruction potential features and the enhancement layer reconstruction quantization potential features that have been reconstructed;
[0010] Decoding the following elements of the coded data of the enhancement layer quantized latent feature according to the enhancement layer context feature;
[0011] Acquire a base layer context feature according to the enhancement layer reconstruction quantization potential feature, the base layer reconstruction potential feature and the reconstructed base layer reconstruction quantization potential feature;
[0012] Decoding the following elements of the encoded data of the base layer quantized latent feature according to the base layer context feature;
[0013] A reconstructed image of the target image is obtained according to the enhanced layer reconstructed quantized latent features and the base layer reconstructed quantized latent features.
[0014] In a second aspect, some embodiments of the present application provide an image decoding device, including:
[0015] a memory configured to store a computer program;
[0016] The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method described in the second aspect.
[0017] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computing device, the computing device implements the image decoding method described in the first aspect.
[0018] In a fourth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image decoding method described in the first aspect.
[0019] It can be seen from the above technical scheme that the image decoding method provided in the embodiment of the present application obtains the base layer reconstruction potential features and the enhancement layer reconstruction potential features of the target image according to the encoding data of the target image, obtains the enhancement layer context features according to the enhancement layer reconstruction potential features and the enhanced layer quantization potential features that have been encoded, and decodes the lower elements of the encoding data of the enhancement layer quantization potential features according to the enhancement layer context features, obtains the base layer context features according to the enhancement layer quantization potential features, the base layer reconstruction potential features and the base layer quantization potential features that have been encoded, and decodes the lower elements of the encoding data of the base layer quantization potential features according to the base layer context features, and finally generates the encoding data of the target image according to the encoding data of the enhancement layer quantization potential features and the encoding data of the base layer quantization potential features. Since the embodiment of the present application not only uses the base layer reconstruction potential features and the reconstructed base layer reconstruction quantization potential features, but also uses the enhancement layer reconstruction quantization potential features when predicting the base layer context features, the embodiment of the present application can more fully utilize the correlation between the enhancement layer reconstruction quantization potential features and the base layer reconstruction quantization potential features, and thus more accurately predict the base layer context features. Therefore, the embodiment of the present application can improve the performance of the scalable image decoder based on machine learning and the machine vision performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.
[0021] Figure 1 A schematic diagram showing the structure of an image encoder in some embodiments of the present application is shown;
[0022] Figure 2 A schematic diagram showing the structure of an analysis transformation network in some embodiments of the present application is shown;
[0023] Figure 3A schematic diagram of the residual activation unit structure of the analysis transformation network in some embodiments of the present application is shown;
[0024] Figure 4 A schematic diagram of a super-a priori coding network structure in some embodiments of the present application is shown;
[0025] Figure 5 A schematic diagram of a super-a priori decoding network structure in some embodiments of the present application is shown;
[0026] Figure 6 A schematic diagram of a masked convolution kernel in some embodiments of the present application is shown;
[0027] Figure 7 A schematic diagram showing the structure of an image decoder in some embodiments of the present application is shown;
[0028] Figure 8 A schematic diagram showing the structure of a synthetic transformation network in some embodiments of the present application is shown;
[0029] Fig. 9 A schematic diagram showing the structure of a potential space transformation network in some embodiments of the present application is shown;
[0030] Fig.10 A schematic diagram showing the structure of an image encoder in some other embodiments of the present application is shown;
[0031] Fig.11 A schematic diagram showing the structure of an image decoder in some other embodiments of the present application is shown;
[0032] Fig.12 A schematic diagram showing the structure of a mask generation network in some embodiments of the present application is shown;
[0033] Fig.13 A schematic diagram showing the structure of an enhancement layer context network in some embodiments of the present application is shown;
[0034] Fig.14 A schematic diagram showing the structure of a channel context module in some embodiments of the present application is shown;
[0035] Fig.15 A schematic diagram showing the structure of an entropy parameter module in some embodiments of the present application is shown;
[0036] Fig.16 A schematic diagram showing the structure of a base layer context network in some embodiments of the present application is shown;
[0037] Fig.17 A flowchart showing the steps of the image encoding method in some embodiments of the present application is shown;
[0038] Fig.18A flowchart showing the steps of the image encoding method in some other embodiments of the present application is shown;
[0039] Fig.19 A flowchart showing the steps of the image decoding method in some embodiments of the present application is shown;
[0040] Fig. 20 A flowchart showing the steps of the image decoding method in some embodiments of the present application is shown;
[0041] Fig.21 A schematic diagram showing the structure of a content adaptive filtering network in some embodiments of the present application is shown;
[0042] Fig. 22 A flowchart showing the steps of the image decoding method in some other embodiments of the present application is shown;
[0043] Fig.23 A schematic diagram showing the structure of ICCI in other embodiments of the present application is shown;
[0044] Fig.24 A schematic diagram of the structure of a residual block in some embodiments of the present application is shown. DETAILED DESCRIPTION
[0045] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.
[0046] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.
[0047] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.
[0048] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).
[0049] Reference Figure 1 As shown, the image encoder provided by some embodiments of the present application includes: an analysis transformation network 101, a first quantization module 102, a first feature segmentation module 103, a super priori coding network 104, a second quantization module 105, a first encoding module 106, a decoding module 107, a super priori decoding network 108, a second feature segmentation module 109, an enhancement layer context network 110, a second encoding module 111, a base layer context network 112, and a third encoding module 113.
[0050] Figure 1 The process of image encoding by the image encoder shown includes the following steps 1 to 14:
[0051] Step 1: The analysis transformation network 101 extracts potential features of the target image X to obtain the potential features of the target image X.
[0052] Among them, the latent features Y are high-dimensional representations of the target image X. The latent features Y cannot be directly observed from the input data (target image), but they can be determined by the information learned from the input data. The above-mentioned latent features Y are obtained based on the information learned from the target image X.
[0053] In some embodiments, the number of channels of features of potential feature Y is C=192.
[0054] Reference Figure 2 As shown, in some embodiments, the analysis transformation network 101 includes: a plurality of downsampling convolutional layers 21 and a plurality of residual activation units 22 serially connected between adjacent downsampling convolutional layers 21, Figure 2 In the figure, an analysis transformation network including four downsampling convolutional layers 21 and three residual activation units 22 serially connected between adjacent downsampling convolutional layers 21 is taken as an example.
[0055] Reference Figure 3 As shown, the residual activation unit 22 includes: a convolution layer 51, an activation function layer 222, a second convolution layer 223 and a fusion layer 224. The convolution layer 51, the activation function layer 222 and the second convolution layer 223 process the input feature I1 of the residual activation unit 222 in sequence, and add and fuse the output feature of the second convolution layer and the input feature I1 of the residual activation unit through the fusion layer 224 to obtain the output feature O2 of the residual activation unit 222.
[0056] In some embodiments, each downsampling convolution layer 21 in the analysis transformation network is a downsampling convolution layer with a convolution kernel size of 3×3, the first convolution layer and the second convolution layer in each residual activation unit of the analysis transformation network are convolution layers with a convolution kernel size of 3×3, and the activation area function of the activation function layer in each residual activation unit of the analysis transformation network is the LeakyReLU activation function (a variant of the ReLU function, which introduces a small slope in the negative value interval, solving the problem that the ReLU function outputs zero in the negative value interval).
[0057] The target image is represented as X, the potential features of the target image are represented as Y, and the processing of the features by the analysis transformation network is represented as g a (), then:
[0058] Y=g a (X)
[0059] Step 2: The first quantization module 102 performs quantization processing on the latent feature Y to obtain a quantized latent feature y.
[0060] The quantization process refers to the process of approximating the continuous value of a signal (or a large number of possible discrete values) to a finite number of discrete values (or a small number of discrete values).
[0061] Step 3: The first feature segmentation module 103 segments the quantized latent feature y in the dimension of the channel to obtain the enhanced layer latent feature and the underlying latent features
[0062] In some embodiments, the enhancement layer quantizes the latent feature y enhance The number of channels C1=64, the base layer potential feature y base The number of channels c1=128.
[0063] In some embodiments, the first feature segmentation module 103 is specifically configured to segment the first 64 channels of the quantized latent feature y into enhanced layer quantized latent features y enhance , and split the last 128 channels of the quantized latent feature y into the base layer quantized latent feature y base That is, y enhance ={Y1, Y2, ..., Y 64},y base = {Y 65 , Y 66 , …, Y 192}.
[0064] Step 4: The super-prior encoding network 104 performs super-prior encoding on the latent feature Y to obtain the super-prior feature Z.
[0065] Reference Figure 4 As shown, in some embodiments, the super prior coding network 104 includes: a first convolution layer 41, a second convolution layer 42, a first downsampling layer 43, a third convolution layer 44, and a second downsampling layer 45 connected in series. The processing of the potential feature Y by the super prior coding network 104 includes: processing the potential feature Y through the first convolution layer 41, the second convolution layer 42, the first downsampling layer 43, the third convolution layer 44, and the second downsampling layer 45 in sequence to obtain the super prior feature Z.
[0066] In some embodiments, the first convolution layer 41, the second convolution layer 42, the first down-sampling layer 43, the third convolution layer 44 and the second down-sampling layer 45 are all convolution layers with a convolution kernel size of 3×3.
[0067] Step 5: The second quantization module 105 performs quantization processing on the super-prior feature Z to obtain a quantized super-prior feature z.
[0068] Step 6: The first encoding module 106 performs entropy encoding on the quantized super-prior feature z to obtain first encoded data b1.
[0069] In some embodiments, the first encoding module 106 may be a lossless entropy encoding module, and performing entropy encoding on the quantized super a priori feature z includes: performing lossless entropy encoding on the quantized super a priori feature z.
[0070] Step 7: The decoding module 107 performs entropy decoding on the first coded data b1 to obtain the reconstructed quantized hyper-prior feature
[0071] In some embodiments, the decoding module 107 may be a lossless entropy decoding module, and performing entropy decoding on the first encoded data b1 includes: performing lossless entropy decoding on the first encoded data b1.
[0072] It should be noted that when the first encoding module 106 is a lossless encoding module and the decoding module 107 is a lossless decoding module, since there is no information loss in the encoding and decoding process, the reconstructed quantized super-prior feature is exactly the same as the quantized hyper-prior feature z, so Figure 1 The image encoder shown in the figure can omit the decoding module 107 and the above step 7, and directly use the quantized super prior feature z to replace the reconstructed quantized super prior feature in the subsequent steps.
[0073] Step 8: The super-prior decoding network 108 reconstructs the quantized super-prior features Perform hyper-prior decoding to obtain reconstruction potential features
[0074] In some embodiments, the number of channels for reconstructing latent features is C3=384.
[0075] Reference Figure 5 As shown, in some embodiments, the super-prior decoding network 108 includes: a convolution layer 51, a first upsampling layer 52, a second upsampling layer 53, a third upsampling layer 54 and a fourth upsampling layer 55. The super-prior decoding network 108 reconstructs the quantized super-prior features. The processing includes: sequentially passing through a convolution layer 51, a first upsampling layer 52, a second upsampling layer 53, a third upsampling layer 54 and a fourth upsampling layer 55. Reconstructing the quantized super-prior features Processing to obtain reconstruction potential features
[0076] In some embodiments, the convolution layer 51, the first upsampling layer 52, the second upsampling layer 53, the third upsampling layer 54 and the fourth upsampling layer 55 are all convolution layers with a convolution kernel size of 3×3.
[0077] Step 9: The second feature segmentation module 109 reconstructs the potential features in the dimension of the channel Segmentation is performed to obtain the potential features for reconstruction of the enhanced layer and the base layer to reconstruct the latent features
[0078] In some embodiments, the second feature segmentation module 109 may segment the reconstructed latent feature y in the same proportion as the first feature segmentation module 103 segments the quantized latent feature y. That is, the enhancement layer reconstructs the potential features The number of channels C4 = 128, the base layer reconstructs the potential features The number of channels C5=256.
[0079] In some embodiments, the second feature segmentation module 109 is specifically used to reconstruct the potential features The first 128 channels of the segmentation are used to reconstruct the latent features of the enhancement layer and will reconstruct the latent features The last 256 channels are split into the base layer to reconstruct the latent features Right now,
[0080] Step 10: The enhancement layer context network 110 quantizes the potential feature y according to the already encoded enhancement layer ehhance and the enhancement layer to reconstruct the latent features Get the enhancement layer context feature Φ enhance .
[0081] Among them, the enhanced layer context feature Φ enhance That is, the enhanced layer quantized potential feature y ehhanceThe probability distribution of .
[0082] In some embodiments, a context network based on autoregressive convolution can be used to enhance the layer context feature Φ enhance Extraction.
[0083] The essence of the contextual network based on autoregressive convolution is a masked convolution layer. For example, refer to Figure 6 As shown, Figure 6 Schematic diagram of the mask convolution kernel used in the context network based on autoregressive convolution provided in the embodiment of the present application. Figure 6 As shown, the spatial context network based on autoregressive convolution uses the encoded elements before the current encoding position 601 in the latent space to predict the context features corresponding to the current encoding position, and uses the context features to subsequently predict the probability distribution of the current encoding position. Among them, the encoded elements are Figure 6 In some embodiments, the calculation formula of the spatial context network based on autoregressive convolution is:
[0084] g cm (x)=(M⊙W)x+b
[0085] Among them, x is the input feature of the spatial context network based on autoregressive convolution, M and W are the k×k mask and convolution weight respectively, ⊙ is the Hadamard operator, and b is the bias term.
[0086] Step 11: The second encoding module 111 is based on the enhancement layer context feature Φ enhance Quantize the latent features y for the enhancement layer ehhance The next element of is entropy encoded.
[0087] Repeat steps 10 and 11 above to complete the enhancement layer quantization potential feature y enhance The second encoded data b2 can be obtained by encoding all elements of .
[0088] Step 12: The base layer context network 112 quantizes the potential feature y according to the encoded base layer base and the base layer to reconstruct the latent features Get the base layer context feature Φ base .
[0089] Similarly, the context network based on autoregressive convolution can be used to perform the base layer context feature Φ base Extraction.
[0090] Step 13: The third encoding module 113 is based on the base layer context feature Φ base Quantize the latent features y of the base layer base The next element of is entropy encoded.
[0091] Repeat steps 12 and 13 above to complete the base layer quantization potential feature y base The third encoded data b3 can be obtained by encoding all the elements of .
[0092] Step 14: Generate coded data of the target image X according to the first coded data b1, the second coded data b2 and the third coded data b3.
[0093] That is, the coded data of the target image X includes first coded data b1, second coded data b2, and third coded data b3.
[0094] Reference Figure 7 As shown, in some embodiments, an image decoder for executing the image decoding method provided by an embodiment of the present application includes: a first decoding module 701, a super priori decoding network 702, a feature segmentation module 703, an enhancement layer context network 704, a second decoding module 705, a base layer context network 706, a third decoding module 707, a splicing module 708, a synthesis transformation network 709, a latent space transformation network 710 and a machine vision task model 711.
[0095] Figure 7 The process of image decoding by the image decoder shown includes the following steps a to l:
[0096] Step a: Obtain the encoded data of the target image.
[0097] The encoding data of the target image includes first encoding data b1, second encoding data b2 and third encoding data b3.
[0098] Step b: The first decoding module 701 performs entropy decoding on the first coded data b1 to obtain the reconstructed quantized super-prior feature
[0099] In some embodiments, the first decoding module 701 may be a lossless entropy decoding module, and performing entropy decoding on the first coded data b1 includes: performing lossless entropy decoding on the first coded data b1.
[0100] Step c: The super-prior decoding network 702 reconstructs the quantized super-prior features Perform super-prior synthesis processing to obtain reconstruction potential features
[0101] In some embodiments, the number of channels for reconstructing latent features is C3=384.
[0102] The network structure of the super-a priori decoding network 702 can be Figure 5The network structure of the super-a priori decoding network 108 shown is the same, and will not be described in detail here to avoid redundancy.
[0103] Step d: The feature segmentation module 703 reconstructs the potential features in the dimension of the channel. Segmentation is performed to obtain the potential features for reconstruction of the enhanced layer and the base layer to reconstruct the latent features
[0104] In some embodiments, the feature segmentation module 703 can reconstruct the potential features in a ratio of 1:2. That is, the enhancement layer reconstructs the potential features The number of channels C4 = 128, the base layer reconstructs the potential features The number of channels C5=256.
[0105] Step e: The enhancement layer context network 704 reconstructs the potential features according to the decoded second coded data b2 and the enhancement layer Get the enhancement layer context feature Φ enhance .
[0106] In some embodiments, a context network based on autoregressive convolution can be used to perform context feature Φ enhance Extraction.
[0107] Step f: The second decoding module 705 reconstructs the quantized latent features according to the enhancement layer The context feature Φ enhance The second code stream is entropy decoded.
[0108] Repeat the above steps e and f to complete the entropy decoding of all elements of the second coded data b2 to obtain the enhanced layer reconstruction quantization potential features.
[0109] when Figure 1 The first encoding module 106 and the second encoding module 111 in the image encoder are lossless entropy encoding modules. Figure 7 The first decoding module 701 and the second decoding module 705 in the image decoder are lossless decoding modules, and the enhancement layer quantization potential feature y enhance There is no information loss in the encoding and decoding process, and the enhancement layer quantizes the latent features y enhance Reconstruct quantized latent features with enhancement layer same.
[0110] Step g: The base layer context network 706 reconstructs the potential features according to the decoded third coded data b3 and the base layer Get the base layer context feature Φ base .
[0111] Similarly, we can use the context network based on autoregressive convolution to perform context feature Φ base Extraction.
[0112] Step h: The third decoding module 707 decodes the base layer context feature Φ base The next element of the third coded data b3 is entropy decoded.
[0113] Repeat the above steps g and h to complete the entropy decoding of all elements of the third coded data b3 to obtain the base layer reconstruction quantization potential features.
[0114] when Figure 1 The first encoding module 106 and the third encoding module 113 in the image encoder are lossless entropy encoding modules. Figure 7 The first decoding module 701 and the third decoding module 707 in the image decoder are lossless entropy decoding modules, and the base layer quantized potential feature y base There is no information loss in the encoding and decoding process, and the base layer quantizes the potential features y base Reconstructing quantized latent features with base layer same.
[0115] Step i: The splicing module 708 reconstructs the quantized latent features of the enhancement layer from the channel dimension and base layer reconstruction quantized latent features Splicing to obtain reconstruction quantification potential features
[0116] when Figure 1 The first encoding module 106, the second encoding module 111, and the third encoding module 113 in the image encoder are lossless entropy encoding modules. Figure 7 The first decoding module 701, the second decoding module 705, and the third decoding module 707 in the image decoder are lossless entropy decoding modules, and the enhancement layer quantization potential feature y enhance and the base layer quantized latent features y base There is no information loss during the encoding and decoding process, and the quantized latent features are reconstructed Same as quantizing the latent feature y.
[0117] Step j: Synthetic transformation network 708 reconstructs the quantized latent features Processing is performed to obtain a reconstructed image of the target image
[0118] In some embodiments, reference Figure 8 As shown, the synthetic transformation network 708 includes: a plurality of upsampling convolutional layers 81 and a plurality of residual activation units 82 connected in series between adjacent upsampling convolutional layers. Figure 8 In the figure, a synthetic transformation network including four upsampling convolutional layers 81 and three residual activation units 82 connected in series between adjacent upsampling convolutional layers 81 is taken as an example. The residual activation unit 82 includes: a first convolutional layer 821, an activation function layer 822, a second convolutional layer 823 and a fusion layer 824. The residual activation unit 82 is used to process the input feature I2 of the residual activation unit 82 through the first convolutional layer 821, the activation function layer 822 and the second convolutional layer 823 in sequence, and add and fuse the output feature of the second convolutional layer 823 and the input feature I2 of the residual activation unit through the fusion layer 824 to obtain the output feature O2 of the residual activation unit 82.
[0119] In some embodiments, each upsampling convolution layer 81 in the synthetic transformation network is an upsampling convolution layer with a convolution kernel size of 3×3, the first convolution layer 821 and the second convolution layer 823 in each residual activation unit of the synthetic transformation network are convolution layers with a convolution kernel size of 3×3, and the activation area function of the activation function layer 822 in each residual activation unit 82 of the synthetic transformation network is a LeakyReLU activation function.
[0120] Step k: Latent space transformation network 710 reconstructs the quantized latent features of the base layer Processing to obtain machine vision features
[0121] Reference Fig. 9 As shown, in some embodiments, the latent space transformation network 710 includes: a first residual block 91, a first up-sampled residual block 92, a second residual block 93, a second up-sampled residual block 94, a third residual block 95, a third up-sampled residual block 96, a fourth residual block 97, an up-sampled convolution layer 98, and an activation function layer 99. The up-sampling multiples of the first up-sampled residual block 92, the second up-sampled residual block 94, the third up-sampled residual block 96, and the up-sampled convolution layer 98 are r1, r2, r4, and r4, respectively.
[0122] Reference Fig. 9As shown, in some embodiments, the residual blocks (first residual block 91, second residual block 93, third residual block 95, fourth residual block 97) in the latent space transformation network 710 include: a first convolution layer 901, a first LeakyLeLU activation function layer 902, a second convolution layer 903, a second LeakyLeLU activation function layer 904, and a fusion module 905. The feature processing step of the residual block includes: firstly, the input feature I3 of the residual block is processed by the first convolution layer 901, the first LeakyLeLU activation function layer 902, the second convolution layer 903, and the second LeakyLeLU activation function layer 904 in sequence, and then the output feature of the second LeakyLeLU activation function layer 904 and the input feature I3 of the residual block are added and fused by the fusion module 905 to obtain the output feature O3 of the residual block.
[0123] Reference Fig. 9 As shown, in some embodiments, the upsampled residual blocks (first upsampled residual block 92, second upsampled residual block 94, third upsampled residual block 96) in the latent space transformation network 710 include: an upsampled convolution layer 906, a LeakyLeLU activation function layer 907, a third convolution layer 908, an Inverse Generalized Divisive Normalization (IGDN) layer 909, a fourth convolution layer 910, and a fusion module 911. The feature processing step of the upsampled residual block includes: firstly, the upsampled convolution layer 906, the LeakyLeLU activation function layer 907, the third convolution layer 908, and the IGDN layer 909 process the input feature I4 of the upsampled residual block, and the fourth convolution layer 910 processes the input feature I4 of the upsampled residual block, and then the fusion module 911 adds and fuses the output of the IGDN layer 909 and the output of the fourth convolution layer 910 to obtain the output feature O4 of the upsampled residual block.
[0124] Step 1: The machine vision task model 711 is based on the machine vision features Get the machine vision task processing result M.
[0125] Reference Fig.10As shown, the image encoder provided by other embodiments of the present application includes: an analysis transformation network 1001, a first quantization module 1002, a first feature segmentation module 1003, a base layer super-prior coding network 1004, a second quantization module 1005, a first encoding module 1006, an enhancement layer super-prior coding network 1007, a third quantization module 1008, a second encoding module 1009, a second feature segmentation module 1010, a first decoding module 1011, a base layer super-prior decoding network 1012, a base layer context network 1013, a third encoding module 1014, a second decoding module 1015, an enhancement layer super-prior decoding network 1016, an enhancement layer context network 1017, and a fourth encoding module 1018.
[0126] Fig.10 The image encoder shown in the figure performs image encoding in the following steps: Step 1: the analysis transformation network 1001 processes the target image (video frame) X to obtain the potential feature Y of the target image X. Step 2: the first quantization module 1002 quantizes the potential feature Y to obtain the quantized potential feature y. Step 3: the first feature segmentation module 1003 segments the potential feature Y into the base layer potential feature Y. base and the enhancement layer latent feature Y enhance Step 4: The base layer super prior coding network 1004 encodes the base layer potential feature Y base Processing is performed to obtain the base layer super prior feature Z base Step 5: The second quantization module 1005 quantizes the base layer super-prior feature Z base Perform quantization processing to obtain the basic layer quantization super prior feature z base Step 6: The first encoding module 1006 quantizes the super-prior feature z of the base layer base Perform entropy coding to obtain the first coded data b1. Step 7: The enhancement layer super-prior coding network 1007 encodes the enhancement layer potential feature Y enhance Processing is performed to obtain the enhanced layer super prior feature Z enhance Step 8: The third quantization module 1008 quantizes the enhancement layer super-prior feature Z enhance Perform quantization processing to obtain the enhanced layer quantization super prior feature z enhance Step 9: The second encoding module 1009 quantizes the super-prior feature z of the enhancement layer eenhancee Perform entropy coding to obtain the second coded data b2. Step 10: The second feature segmentation module 1010 segments the quantized latent feature y into the base layer quantized latent feature y bases and the enhancement layer latent feature y enhance Step 11: The first decoding module 1011 performs entropy decoding on the first coded data b1 to obtain the super prior features of the base layer reconstruction. Step 12: The base layer super-prior decoding network 1012 reconstructs super-prior features of the base layer Processing is performed to obtain the underlying features for reconstruction Step 13: Base layer context network 1013 reconstructs potential features based on the base layer and the encoded base layer quantized latent features y base Get the base layer context feature Φ base Step 14: The third encoding module 1014 generates a base layer context feature Φ base Quantize the latent features y of the base layer base The current element is entropy encoded, and finally the base layer quantization potential feature y is completed. base The entropy encoding of all elements is performed to obtain the third coded data b3. Step 15: The second decoding module 1015 performs entropy decoding on the second coded data b2 to obtain the enhanced layer reconstruction super prior feature Step 16: The enhancement layer super-prior decoding network 1016 reconstructs the super-prior features of the enhancement layer Processing is performed to obtain the potential features of the enhanced layer reconstruction Step 17: Enhanced layer context network 1017 reconstructs potential features based on the enhanced layer and the quantized latent feature y of the enhanced layer that has been coded enhance Get the enhancement layer context feature Φ enhance Step 18: The fourth encoding module 1018 generates the enhanced layer context feature Φ enhance Quantize the latent features y for the enhancement layer enhance The current element is entropy coded, and the enhancement layer quantization potential feature y is finally completed. enhance Step 19: The first coded data b1, the second coded data b2, the third coded data b3 and the fourth coded data b4 are used as coded data of the target image X and sent to the decoding end.
[0127] It should be noted that when the first encoding module 1006 is a lossless encoding module and the first decoding module 1011 is a lossless entropy decoding module, due to the base layer quantization super prior feature z base There is no information loss in the encoding and decoding process, and the base layer reconstructs the super prior features and the base layer quantized hyper-prior feature z base are exactly the same, so Fig.10 The image encoder shown in the figure can omit the decoding module 1011 and the above step 11, and directly use the base layer quantized super prior feature z base Replace the base layer in the subsequent steps to reconstruct the hyper-prior features Similarly, when the second encoding module 1009 is a lossless encoding module and the second decoding module 1015 is a lossless entropy decoding module, the decoding module 1014 and the above step 15 can be omitted, and the enhancement layer quantization super prior feature z is directly used. enhance Replace the base layer in the subsequent steps to reconstruct the hyper-prior features
[0128] Fig.10 The network structures of the analysis transformation network, super prior encoding network, super prior decoding network, etc. in the image encoder shown in FIG. Figure 1 The network structures of the analysis transformation network, super a priori encoding network, super a priori decoding network and other networks in the image encoder shown are the same. To avoid redundancy, they are not described in detail here.
[0129] Reference Fig.11 As shown, an image decoder for executing the image decoding method provided by some embodiments of the present application includes: a first decoding module 1101, a base layer super priori decoding network 1102, a base layer context network 1103, a second decoding module 1104, a third decoding module 1105, an enhancement layer super priori decoding network 1106, an enhancement layer context network 1107, a fourth decoding module 1108, a splicing module 1109, a synthesis transformation network 1110, a latent space transformation network 1111 and a machine vision task model 1112.
[0130] Fig.11 The process of image decoding by the image decoder shown in the figure includes the following steps: Step 1: The first decoding module 1101 performs entropy decoding on the first coded data b1 to obtain the super prior feature of the base layer reconstruction. Step 2: The base layer super-prior decoding network 1102 reconstructs the super-prior features of the base layer Processing is performed to obtain the underlying features for reconstruction Step 3: Base layer context network 1103 reconstructs potential features based on the base layer Reconstruct the quantified latent features from the completed base layer (Decoding result of the decoded third coded data b3) Obtaining the base layer context feature Φ base Step 4: The third decoding module 1104 decodes the base layer context feature Φ base The current element of the third coded data b3 is entropy decoded, and finally the entropy decoding of all elements of the third coded data b3 is completed to obtain the base layer reconstruction quantization potential feature Step 5: The third decoding module 1105 performs entropy decoding on the second coded data b2 to obtain the enhanced layer reconstruction super prior feature Step 6: The enhancement layer super-prior decoding network 1106 reconstructs the super-prior features of the enhancement layer Processing is performed to obtain the potential features of the enhanced layer reconstruction Step 7: Enhanced layer context network 1107 reconstructs potential features based on the enhanced layer and the enhanced layer reconstructed quantitative latent features (Decoding result of the fourth coded data b4 that has been decoded), obtain the enhancement layer context feature Φ enhance Step 8: The fourth decoding module 1108 decodes the enhanced layer context feature Φ enhance The current element of the fourth coded data b4 is entropy decoded, and finally the entropy decoding of all elements of the fourth coded data b4 is completed to obtain the enhanced layer reconstruction quantization potential feature Step 9: The splicing module 1109 splices the base layer to reconstruct the quantized potential features and enhancement layer reconstruction quantization latent features Obtaining reconstruction quantization potential features Step 10: Reconstruct the quantized latent features through the synthetic transformation network 1110 Processing is performed to obtain a reconstructed image of the target image Step 11: Reconstruct the quantized latent features of the base layer through the latent transformation network 1111 Processing to obtain machine vision features Step 12: Machine vision features The machine vision task model 1112 is input to obtain the machine vision task processing result M output by the machine vision task model 1112 .
[0131] Fig.11 The network structures of the super-prior decoding network, latent space variation network, synthetic transformation network, etc. in the image decoder shown in FIG. Figure 7 The network structures of the super-prior decoding network, latent space variation network, synthetic transformation network and other networks in the decoder shown are the same. To avoid redundancy, they are not described in detail here.
[0132] Figure 1 In the image encoder shown in FIG. 1 , the quantized potential feature y needs to be divided into the base layer quantized potential feature y by the first feature segmentation module 103. base and the enhancement layer quantized latent features y enhance , the second feature segmentation module 109 reconstructs the potential feature Segmentation is used to reconstruct latent features from the base layer and the enhancement layer to reconstruct the latent features Fig.10 In the image encoder shown in FIG. 1 , the potential feature Y needs to be divided into the base layer quantized potential feature Y by the first feature segmentation module 1003. base and the enhancement layer quantized latent feature Y enhanceThe second feature segmentation module 1010 segments the quantized potential feature y into the base layer quantized potential feature y base and the enhancement layer quantized latent features y enhance , and when segmenting features, they are segmented into base layer features and enhancement layer features according to fixed positions. However, if features are directly clipped into base layer features and enhancement layer features according to positions, it is likely that some channels of the base layer features will not be suitable for downstream tasks, which will affect the encoding performance and machine vision performance.
[0133] To address this problem, in some embodiments, Figure 1 The image encoder shown in FIG. 1 needs to pass through the first feature segmentation module 103 and the second feature segmentation module 109, Fig.10 The image encoder shown in the figure needs to pass through the first feature segmentation module 1003 and the second feature segmentation module 1010 and may also include a mask generation network, wherein the mask generation network is used to generate a channel attention mask Mask according to the quantized potential feature y, and to convert the potential feature Y or the quantized potential feature Y into base Or reconstruct the latent features Segmentation is into base layer features and enhancement layer features.
[0134] Reference Fig.12 As shown, in some embodiments, the mask generation network includes: an adaptive pooling layer 121, a first convolutional layer 122, a batch normalization layer 123, a nonlinear activation function layer 124, a second convolutional layer 125, and a sampling module 126. The process of the channel attention network 1203 processing the quantized potential feature y to obtain the channel attention mask Mask includes: first, the adaptive pooling layer 121 processes the quantized potential feature y to obtain the first feature F1; secondly, the first convolutional layer 122, the normalization layer 123, and the nonlinear activation function layer 124 process the first feature F1 in sequence to obtain the second feature F2; and again, the second convolutional layer 125 processes the second feature F2 to obtain the third feature F3 and the fourth feature F4. Finally, the sampling module 126 performs Gumbel sampling on the third feature F3 and the fourth feature F4 to obtain the channel attention mask Mask.
[0135] Figure 1 The image encoder shown, Fig.10 The image encoders shown in the figure all need to pass the enhancement layer context network to perform enhancement layer context feature Φ enhance The acquisition of the base layer context feature Φ through the base layer context network baseHowever, the prediction of the enhancement layer context features and the base layer context features based on the enhancement layer quantization potential features and the base layer quantization potential features respectively fails to fully utilize the correlation between the enhancement layer quantization potential features and the base layer quantization potential features.
[0136] In order to solve the problem of not fully utilizing the correlation between the quantized latent features of the enhancement layer and the quantized latent features of the base layer, Figure 1 The image encoder and Fig.10 The image encoder can be Fig.13 The enhancement layer context network, the encoded enhancement layer quantized latent features and the enhancement layer reconstructed latent features are shown to obtain the enhancement layer context features.
[0137] Reference Fig.13 As shown, in some embodiments, the enhancement layer context network includes: a channel context module 131, a spatial context module 132, a splicing module 133 and an entropy parameter module 134. The enhancement layer context network quantizes the potential feature y according to the enhancement layer enhance and the enhancement layer to reconstruct the latent features Get the enhancement layer context feature Φ enhance The process includes the following steps ①a to ①d:
[0138] Step ①a: Reconstruct the potential features of the enhancement layer through the channel context module 131 Perform channel context feature extraction to obtain enhanced layer channel context features
[0139] The channel context module includes multiple convolutional layers and activation function layers connected in series between adjacent convolutional layers.
[0140] Reference Fig.14 As shown, in some embodiments, the channel context module 131 includes: a first convolution layer 141, a first activation function layer 142, a second convolution layer 143, a second activation function layer 144 and a third convolution layer 145. The channel context module 11 is used to sequentially reconstruct the potential features of the enhancement layer through the first convolution layer 141, the first activation function layer 142, the second convolution layer 143, the second activation function layer 144 and the third convolution layer 145. Processing to obtain the enhanced layer channel context features
[0141] In some embodiments, the first convolution layer 141 and the second convolution layer 143 are convolution layers with a convolution kernel size of 3×3; the third convolution layer 145 is a convolution layer with a convolution kernel size of 5×5; the activation functions of the first activation function layer 142 and the second activation function layer 144 are Gaussian Error Linear Unit (GELU) activation functions.
[0142] Step ①b, the spatial context module 132 quantizes the potential feature y of the enhanced layer that has been encoded enhance Perform spatial context feature extraction to obtain enhanced layer spatial context features
[0143] In some embodiments, the spatial context module 132 may be an autoregressive convolution based spatial context network.
[0144] The schematic diagram of the mask convolution kernel used by the spatial context network based on autoregressive convolution can be shown as follows Figure 6 As shown, no detailed description is given here.
[0145] Step ①c: The splicing module 133 reconstructs the potential features of the enhancement layer Enhancement layer channel context features And the enhancement layer spatial context features Perform stitching to obtain the first stitching feature
[0146] In some embodiments, the stitching module 133 can reconstruct the latent features of the enhancement layer from the channel dimension. Channel context features And the enhancement layer spatial context features Perform stitching to obtain the first stitching feature
[0147] Step ①d, the entropy parameter module 134 performs the first splicing feature Processing is performed to obtain the enhanced layer context feature Φ enhance .
[0148] The entropy parameter module includes a plurality of convolutional layers connected in series.
[0149] Reference Fig.15 As shown, in some embodiments, the entropy parameter module 134 includes: a first convolution layer 151, a second convolution layer 152 and a third convolution layer 153. The entropy parameter module 134 is used to sequentially pass the first convolution layer 151, the second convolution layer 152 and the third convolution layer 153 to the first concatenation feature. Processing is performed to obtain the enhanced layer context feature Φ enhance .
[0150] In some embodiments, the first convolution layer 151, the second convolution layer 152 and the third convolution layer 153 are all convolution layers with a convolution kernel size of 1×1.
[0151] In order to solve the problem of not fully utilizing the correlation between the quantized latent features of the enhancement layer and the quantized latent features of the base layer, Figure 1 The image encoder and Fig.10 The image encoder can be Fig.16 The base layer context network, enhancement layer quantization potential features, base layer reconstruction potential features and encoded base layer quantization potential features are shown to obtain base layer context features.
[0152] Reference Fig.16 As shown, in some embodiments, the base layer context network includes: a first splicing module 161, a channel context module 162, a spatial context module 163, a second splicing module 164 and an entropy parameter module 165. The enhancement layer context network quantizes the potential feature y according to the enhancement layer enhance , base layer reconstruction potential features And the encoded base layer quantized potential features y base The process includes the following steps ②a to ②e:
[0153] Step ②a: The first splicing module 161 quantizes the potential feature y of the enhancement layer enhance and the base layer to reconstruct the latent features Perform stitching to obtain the second stitching feature
[0154] In some embodiments, the first splicing module 161 can quantize the potential features y of the already encoded enhancement layer from the channel dimension. enhance and the base layer to reconstruct the latent features Perform stitching to obtain the second stitching feature
[0155] Step ②b: The channel context module 162 processes the second concatenated feature Perform channel context feature extraction to obtain the base layer channel context feature
[0156] The structure of the channel context module 162 can be similar to Fig.14 The structure of the channel context module shown is the same as that of the channel context module 162, except that the number of channels in the channel context module 162 is different from that in the Fig.14 The number of channels of the channel context modules shown is different, and to avoid redundancy, they will not be described in detail here.
[0157] Step ②c, the spatial context module 163 quantizes the potential features t of the encoded base layer base Perform spatial context feature extraction to obtain the base layer spatial context features
[0158] In some embodiments, the spatial context module 163 may be an autoregressive convolution based spatial context network.
[0159] The schematic diagram of the mask convolution kernel used by the spatial context network based on autoregressive convolution can be shown as follows Figure 6 As shown, no detailed description is given here.
[0160] Step ②d: The second splicing module 164 reconstructs the potential features of the base layer Base layer channel context features And the base layer spatial context features Perform stitching to obtain the third stitching feature
[0161] In some embodiments, the second stitching module 164 can reconstruct the potential features of the base layer from the channel dimension. Base layer channel context features And the base layer spatial context features Perform stitching to obtain the third stitching feature
[0162] Step ②e, the entropy parameter module 165 performs the third splicing feature Processing is performed to obtain the base layer context feature Φ base .
[0163] The structure of the entropy parameter module 165 can be similar to Fig.15 The structure of the entropy parameter module is the same as that of the channel context module 165, except that the number of channels in the channel context module 165 is the same as that in the Fig.15 The number of channels of the entropy parameter modules shown is different, and to avoid redundancy, they will not be described in detail here.
[0164] Figure 7 The image decoder and Fig.11 The image decoder shown can also extract the base layer context features and the enhancement layer context features through the base layer context network and the enhancement layer context network, and Figure 1 The image encoder or Fig.10 The difference of the image encoder shown is that it is necessary to replace the already encoded enhancement layer quantization potential features with the already reconstructed enhancement layer reconstructed quantization potential features, and replace the already encoded base layer quantization potential features with the already reconstructed base layer reconstructed quantization potential features.
[0165] The present application also provides an image encoding method, referring to Fig.17 As shown, the image encoding method comprises the following steps:
[0166] S171. Potential features of the target image.
[0167] In some embodiments, Figure 2 or Figure 3 The analysis transformation network shown performs latent feature extraction on the target image to obtain the latent features of the target image.
[0168] In some embodiments, the feature channel data of the potential features of the target image is 192.
[0169] S172. Obtain base layer reconstruction potential features and enhancement layer reconstruction potential features according to the potential features.
[0170] In some embodiments, obtaining the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature according to the potential feature includes: firstly performing super-a priori coding, quantization, entropy coding, entropy decoding and super-a priori decoding on the potential feature in sequence to obtain the reconstruction potential feature, and then segmenting the reconstruction potential feature to obtain the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature ( Figure 1 That is, obtaining the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature according to the potential feature, including the following steps a to f:
[0171] Step a: perform super-a priori encoding on the latent features to obtain super-a priori features.
[0172] In some embodiments, Figure 4 The super-prior encoding network performs super-prior encoding on the latent features to obtain the super-prior features.
[0173] Step b: quantizing the super a priori features to obtain quantized super a priori features.
[0174] Step c: performing entropy coding on the quantized super-prior feature to obtain first coded data.
[0175] In some embodiments, entropy encoding the quantized super a priori features includes: performing lossless entropy encoding on the quantized super a priori features.
[0176] Step d: performing entropy decoding on the first encoded data to obtain reconstructed super-prior features.
[0177] In some embodiments, entropy encoding the quantized super-prior feature includes: lossless entropy decoding the first encoded data.
[0178] Step e: performing super-a priori decoding on the reconstructed super-a priori features to obtain reconstructed latent features.
[0179] In some embodiments, Figure 5 The super-prior decoding network performs super-prior decoding on the latent features to obtain the reconstructed latent features.
[0180] Step f: segmenting the reconstructed potential features to obtain the base layer reconstructed potential features and the enhanced layer reconstructed potential features.
[0181] In some embodiments, the reconstruction potential features may be segmented according to a fixed ratio of the number of channels to obtain the base layer reconstruction potential features and the enhancement layer reconstruction potential features. For example, the features of the first 1 / 3 channels of the reconstruction potential features are divided into the enhancement layer reconstruction potential features, and the features of the last 2 / 3 channels of the reconstruction potential features are divided into the base layer reconstruction potential features.
[0182] In some embodiments, Fig.12 The mask generation network shown obtains a channel attention mask, and divides the reconstructed potential features into the base layer reconstructed potential features and the enhanced layer reconstructed potential features through the channel attention mask.
[0183] In some embodiments, obtaining the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature according to the potential feature includes: firstly dividing the potential feature into the base layer potential feature and the enhancement layer potential feature, and then respectively performing super-a priori coding, quantization, entropy coding, entropy decoding and super-a priori decoding on the base layer potential feature and the enhancement layer potential feature to obtain the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature ( Fig.10 That is, obtaining the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature according to the potential feature includes the following steps 1 to 6:
[0184] Step 1: Segment the potential features to obtain base layer potential features and enhancement layer potential features.
[0185] In some embodiments, the potential features may be segmented according to a fixed channel number ratio to obtain the base layer potential features and the enhancement layer potential features.
[0186] In some embodiments, Fig.12 The mask generation network shown obtains a channel attention mask and divides the potential features into the base layer potential features and the enhancement layer potential features through the channel attention mask.
[0187] Step 2: Perform super-a priori coding on the base layer potential features and the enhancement layer potential features respectively to obtain base layer super-a priori features and enhancement layer super-a priori features.
[0188] In some embodiments, the two Figure 4 The super-prior coding networks shown are respectively set as a base layer super-prior coding network and an enhancement layer super-prior coding network, and then the base layer latent features are super-prior encoded by the base layer super-prior coding network to obtain the base layer super-prior features, and the enhancement layer latent features are super-prior encoded by the enhancement layer super-prior coding network to obtain the enhancement layer super-prior features.
[0189] Step 3: quantize the base layer super-prior feature and the enhancement layer super-prior feature respectively to obtain the base layer quantized super-prior feature and the enhancement layer quantized super-prior feature.
[0190] Step 4: Perform entropy coding on the base layer quantized super a priori features and the enhancement layer quantized super a priori features respectively to obtain first coded data and second coded data.
[0191] Step 5: Perform entropy decoding on the first coded data and the second coded data respectively to obtain the super prior features for reconstructing the base layer and the super prior features for reconstructing the enhancement layer.
[0192] Step 6: Perform super-prior decoding on the base layer reconstruction super-prior feature and the enhancement layer reconstruction super-prior feature respectively to obtain the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature.
[0193] In some embodiments, the two Figure 5 The super-prior decoding networks shown are respectively set as a base layer super-prior decoding network and an enhancement layer super-prior decoding network, and then the base layer super-prior decoding network is used to perform super-prior decoding on the base layer reconstructed super-prior features to obtain the base layer reconstructed potential features, and the enhancement layer super-prior decoding network is used to perform super-prior decoding on the enhancement layer reconstructed super-prior features to obtain the enhancement layer reconstructed potential features.
[0194] S173: quantize the potential features based on preset quantization parameters to obtain quantized potential features.
[0195] S174. Obtain a base layer quantized latent feature and an enhanced layer quantized latent feature according to the quantized latent feature.
[0196] In some embodiments, in some embodiments, the quantized latent features may be segmented according to a fixed channel number ratio to obtain the base layer quantized latent features and the enhanced layer quantized latent features.
[0197] In some embodiments, Fig.12 The mask generation network shown obtains a channel attention mask, and divides the quantized latent features into the base layer quantized latent features and the enhanced layer quantized latent features through the channel attention mask.
[0198] S175 . Obtain enhancement layer context features according to the enhancement layer reconstructed potential features and the coded enhancement layer quantized potential features.
[0199] In some embodiments, Fig.13 The enhancement layer context network, the enhancement layer reconstruction potential features and the encoded enhancement layer quantized potential features are used to obtain the enhancement layer context features.
[0200] S176: Encode the following elements of the enhancement layer quantized potential features according to the enhancement layer context features.
[0201] The above steps S175 and S176 are executed cyclically to complete the encoding of all elements of the enhancement layer quantization latent feature, thereby obtaining the encoding data of the enhancement layer quantization latent feature.
[0202] S177. Obtain base layer context features according to the enhancement layer quantization potential features, the base layer reconstruction potential features and the encoded base layer quantization potential features.
[0203] In some embodiments, Fig.16 The base layer context network, the enhancement layer quantization potential features, the base layer reconstruction potential features and the encoded base layer quantization potential features are used to obtain the base layer context features.
[0204] S178. Encode the following elements of the base layer quantized potential features according to the base layer context features.
[0205] The above steps S177 and S178 are executed in a loop, and the encoding data of the base layer quantized latent features can be obtained after the encoding of all elements of the base layer quantized latent features is completed.
[0206] S179. Generate the encoding data of the target image according to the encoding data of the enhanced layer quantized latent features and the encoding data of the base layer quantized latent features.
[0207] The image encoding method provided by the embodiment of the present application first obtains the potential features of the target image when encoding the target image, then obtains the base layer reconstruction potential features and the enhancement layer reconstruction potential features according to the potential features, quantizes the potential features based on preset quantization parameters to obtain the quantized potential features, obtains the base layer quantized potential features and the enhancement layer quantized potential features according to the quantized potential features, then obtains the enhancement layer context features according to the enhancement layer reconstruction potential features and the enhanced layer quantized potential features that have been encoded, and encodes the lower elements of the enhancement layer quantized potential features according to the enhancement layer context features, obtains the base layer context features according to the enhancement layer quantized potential features, the base layer reconstruction potential features and the base layer quantized potential features that have been encoded, encodes the lower elements of the base layer quantized potential features according to the base layer context features, and finally generates the encoding data of the target image according to the encoding data of the enhancement layer quantized potential features and the encoding data of the base layer quantized potential features. Since the embodiment of the present application not only uses the base layer reconstructed potential features and the encoded base layer quantized potential features when predicting the base layer context features, but also uses the enhanced layer quantized potential features, the embodiment of the present application can more fully utilize the correlation between the enhanced layer quantized potential features and the base layer quantized potential features, and thus more accurately predict the base layer context features. Therefore, the embodiment of the present application can improve the encoding performance and machine vision performance of the scalable image encoder based on machine learning.
[0208] As an extension and refinement of the above embodiment, the present application embodiment provides another image encoding method, referring to Fig.18 As shown, the image encoding method comprises the following steps:
[0209] S1801. Extract features of a target image based on an analysis transformation network to obtain potential features of the target image.
[0210] S1802: Obtain base layer reconstruction potential features and enhancement layer reconstruction potential features according to the potential features.
[0211] The implementation method of step S1802 can refer to the implementation method of the above-mentioned step 172. To avoid redundancy, it will not be described in detail here.
[0212] S1803: quantize the latent features based on preset quantization parameters to obtain quantized latent features.
[0213] S1804: Obtain a base layer quantized latent feature and an enhanced layer quantized latent feature according to the quantized latent feature.
[0214] S1805 . Perform channel context feature extraction on the enhancement layer reconstruction potential features based on the channel context module to obtain enhancement layer channel context features.
[0215] The structure of the channel context module can be as follows Fig.14 To avoid repetition, detailed description is not given here.
[0216] S1806 , performing spatial context feature extraction on the quantized latent features of the enhanced layer that have been encoded based on the spatial context module to obtain spatial context features of the enhanced layer.
[0217] S1807 , concatenating the enhancement layer reconstruction potential feature, the enhancement layer channel context feature, and the enhancement layer spatial context feature to obtain a first concatenated feature.
[0218] S1808. Process the first splicing feature based on an entropy parameter module to obtain the enhancement layer context feature.
[0219] The structure of the entropy parameter module can be as follows Fig.15 To avoid repetition, detailed description is not given here.
[0220] S1809: Encode the following elements of the enhancement layer quantization potential feature according to the enhancement layer context feature.
[0221] S1810: Splice the enhancement layer quantization potential feature and the base layer reconstruction potential feature to obtain a second splicing feature.
[0222] S1811. Perform channel context feature extraction on the second splicing feature based on a channel context module to obtain a base layer channel context feature.
[0223] The structure of the channel context module can be as follows Fig.14 To avoid repetition, detailed description is not given here.
[0224] S1812. Perform spatial context feature extraction on the encoded base layer quantized latent features based on the spatial context module to obtain base layer spatial context features.
[0225] S1813 . Concatenate the base layer reconstruction potential feature, the base layer channel context feature, and the base layer spatial context feature to obtain a third concatenated feature.
[0226] S1814. Process the third splicing feature based on an entropy parameter module to obtain the base layer context feature.
[0227] The structure of the entropy parameter module can be as follows Fig.15To avoid repetition, detailed description is not given here.
[0228] S1815. Encode the following elements of the base layer quantized potential features according to the base layer context features.
[0229] S1816. Generate the encoding data of the target image according to the encoding data of the enhanced layer quantized latent features and the encoding data of the base layer quantized latent features.
[0230] The present application also provides an image decoding method, referring to Fig.19 As shown, the image decoding method comprises the following steps:
[0231] S191. Obtain base layer reconstruction potential features and enhancement layer reconstruction potential features of the target image according to the encoded data of the target image.
[0232] In some embodiments, the encoded data of the target image includes first encoded data obtained by entropy encoding the quantized super-prior features, and the step of obtaining the base layer reconstruction potential features and the enhancement layer reconstruction potential features of the target image according to the encoded data of the target image includes the following steps a to c:
[0233] Step a: performing entropy decoding on the first encoded data to obtain reconstructed super-prior features.
[0234] Step b: performing super-prior decoding on the reconstructed super-prior features to obtain reconstructed potential features.
[0235] Step c: segmenting the reconstructed potential features to obtain the base layer reconstructed potential features and the enhanced layer reconstructed potential features.
[0236] In some embodiments, the reconstruction potential features may be segmented according to a fixed channel number ratio to obtain the base layer reconstruction potential features and the enhancement layer reconstruction potential features.
[0237] In some embodiments, a reconstruction quantization latent feature may be generated based on the reconstruction latent feature, and a reconstruction quantization latent feature ... may be generated based on the reconstruction quantization latent feature and the reconstruction quantization latent feature. Fig.12 The mask generation network shown obtains a channel attention mask, and divides the reconstructed potential features into the base layer reconstructed potential features and the enhanced layer reconstructed potential features through the channel attention mask.
[0238] In some embodiments, the coded data of the target image includes first coded data obtained by entropy encoding the base layer quantized super prior features and second coded data obtained by entropy encoding the enhancement layer quantized super prior features, and the step of obtaining the base layer reconstruction potential features and the enhancement layer reconstruction potential features of the target image according to the coded data of the target image includes the following steps 1 and 2:
[0239] Step 1: Entropy decode the first coded data and the second coded data respectively to obtain super-prior features for reconstructing a base layer and super-prior features for reconstructing an enhancement layer.
[0240] Step 2: Perform super-prior decoding on the base layer reconstruction super-prior feature and the enhancement layer reconstruction super-prior feature respectively to obtain the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature.
[0241] S192: Acquire enhancement layer context features according to the enhancement layer reconstruction potential features and the enhancement layer reconstruction quantization potential features that have been reconstructed.
[0242] In some embodiments, Fig.13 The enhancement layer context network, the enhancement layer reconstruction potential features and the reconstructed enhancement layer reconstruction quantization potential features are used to obtain the enhancement layer context features.
[0243] S193. Decode the following elements of the coded data of the enhancement layer quantized potential feature according to the enhancement layer context feature.
[0244] The above steps S192 and S193 are executed cyclically to complete the decoding of all elements of the coded data of the enhancement layer quantization potential feature, so as to obtain the enhancement layer reconstructed quantization potential feature.
[0245] S194, obtaining base layer context features according to the enhanced layer reconstructed quantized latent features, the base layer reconstructed quantized latent features and the reconstructed base layer reconstructed quantized latent features.
[0246] In some embodiments, Fig.16 The base layer context network, the enhanced layer quantized potential features, the base layer reconstruction potential features and the reconstructed base layer reconstruction quantized potential features are used to obtain the base layer context features.
[0247] Since the enhanced layer reconstruction quantization potential features can be obtained through the above steps S192 and S193, the enhanced layer reconstruction quantization potential features can be used to further obtain the base layer context features.
[0248] S195. Decode the following elements of the encoded data of the base layer quantized potential feature according to the base layer context feature.
[0249] The above steps S194 and S195 are executed in a loop, and the decoding of all elements of the encoded data of the base layer quantization potential features is completed to obtain the base layer reconstructed quantization potential features.
[0250] S196. Obtain a reconstructed image of the target image according to the enhanced layer reconstructed quantized latent features and the base layer reconstructed quantized latent features.
[0251] The image decoding method provided in the embodiment of the present application can reconstruct the image encoded by the image encoding method provided in the above embodiment. Therefore, the image decoding method provided in the embodiment of the present application can improve the encoding performance and machine vision performance of the scalable image encoder based on machine learning while ensuring the normal decoding of the image.
[0252] As an extension and refinement of the above embodiment, the present application embodiment also provides an image decoding method, referring to Fig. 20 As shown, the image decoding method comprises the following steps:
[0253] S2001. Obtain base layer reconstruction potential features and enhancement layer reconstruction potential features of the target image according to the encoded data of the target image.
[0254] S2002: extract channel context features from the enhancement layer reconstruction potential features based on a channel context module to obtain enhancement layer channel context features.
[0255] The structure of the channel context module can be as follows Fig.14 To avoid repetition, detailed description is not given here.
[0256] S2003: Extract spatial context features from the reconstructed quantized latent features of the enhancement layer based on the spatial context module to obtain spatial context features of the enhancement layer.
[0257] S2004: Splice the enhancement layer reconstruction potential feature, the enhancement layer channel context feature, and the enhancement layer spatial context feature to obtain a first splicing feature.
[0258] S2005. Process the first splicing feature based on an entropy parameter module to obtain the enhancement layer context feature.
[0259] The structure of the entropy parameter module can be as follows Fig.15 To avoid repetition, detailed description is not given here.
[0260] S2006: Decode the following elements of the coded data of the enhancement layer quantized latent feature according to the enhancement layer context feature.
[0261] The above steps S2003 to S2006 are executed cyclically to complete the decoding of all elements of the coded data of the enhancement layer quantization latent feature, so as to obtain the enhancement layer reconstructed quantization latent feature.
[0262] S2007 , splicing the enhancement layer quantized latent feature and the base layer reconstructed latent feature to obtain a second splicing feature.
[0263] S2008. Perform channel context feature extraction on the second splicing feature based on the channel context module to obtain a base layer channel context feature.
[0264] The structure of the channel context module can be as follows Fig.14 To avoid repetition, detailed description is not given here.
[0265] S2009. Extract spatial context features from the reconstructed base layer quantization latent features based on the spatial context module to obtain base layer spatial context features.
[0266] S2010 , stitching the base layer reconstruction potential feature, the base layer channel context feature, and the base layer spatial context feature to obtain a third stitching feature.
[0267] S2011. Process the third splicing feature based on an entropy parameter module to obtain the base layer context feature.
[0268] The structure of the entropy parameter module can be as follows Fig.15 To avoid repetition, detailed description is not given here.
[0269] S2012. Decode the following elements of the encoded data of the base layer quantized potential features according to the base layer context features.
[0270] The above steps S2009 to S2012 are executed in a loop, and the decoding of all elements of the coded data of the base layer quantized latent features is completed to obtain the base layer reconstructed quantized latent features.
[0271] S2013: Concatenate the base layer reconstructed quantized latent features and the enhanced layer reconstructed quantized latent features to obtain reconstructed quantized latent features.
[0272] S2014. Input the reconstructed quantized latent features into a synthetic transformation network, and obtain a reconstructed image of the target image output by the synthetic transformation network.
[0273] The structure of the synthetic transformation network can be as follows Figure 8 To avoid repetition, detailed description is not given here.
[0274] S2015. Process the reconstructed quantitative latent features of the base layer based on a latent space transform network (Latent Space Transform, LST) to obtain machine vision features.
[0275] The structure of the latent space transformer network can be as follows Fig. 9 To avoid repetition, detailed description is not given here.
[0276] S2016. Input the machine vision feature into a machine vision task model to obtain a machine vision task processing result output by the machine vision task model.
[0277] The machine vision task model in the embodiments of the present application can be an image / video super-resolution network or a target (specified object such as a human face) detection network based on machine learning models such as a convolutional neural network (CNN), a convolutional neural network (RNN), or a long short-term memory network (LSTM).
[0278] In order to reduce the bit rate of image coding data, the image encoder will process the chrominance components (U component and V component) in the image data through additional downsampling and a smaller number of channels, which will affect the quality of the reconstructed image. In order to improve the quality of the reconstructed image, Figure 7 or Fig.11 Based on the image decoder shown in FIG. 1 , the image decoder in some embodiments of the present application further includes: a content adaptive filtering network. The content adaptive filtering network is used to reconstruct the image Enhance to obtain enhanced image
[0279] Reference Fig.21 As shown, in some embodiments, the content adaptive filtering network includes: a first upsampling module 2101, a second upsampling module 2102, a first feature extraction module 2103, a second feature extraction module 2104, a third feature extraction module 2105, a splicing module 2106, a content adaptive inter-channel correlation information filter (Content adaptive InterChannel Correlation Information filter, ICCI) 2107, a first processing module 2108, a second processing module 2109 and a third processing module 2110, a first fusion module 2111, a second fusion module 2112 and a third fusion module 2113, a first reconstruction module 2114, a second reconstruction module 2115 and a third reconstruction module 2116. The content adaptive filtering network reconstructs the image The enhancement process includes: Step 1, respectively using the first upsampling module 2101 and the second upsampling module 2102 to represent the initial first chrominance component of the reconstructed image as and the initial second chrominance component is expressed as Upsample to obtain the same brightness component as the original The upsampled first chroma component with the same spatial size is expressed as And the upsampled second chroma component is expressed as Step 2: The initial brightness component is extracted by the first feature extraction module 2103, the second feature extraction module 2104, and the third feature extraction module 2105 respectively. The upsampled first chroma component is expressed as And the upsampled second chroma component is expressed as Perform feature extraction to obtain brightness features The first chromaticity characteristic The second chromaticity characteristic Step 3: Use the splicing module 2106 to combine the brightness features The first chromaticity characteristic The second chromaticity characteristic Splice to obtain the initial image feature F YUV Step 4: Use ICCI 2107 to identify the initial image features F YUV Process and obtain enhanced features F enha Step 5: The first processing module 2108, the second processing module 2109 and the third processing module 2110 are used to enhance the feature F enha Processing to obtain brightness enhancement features First chroma enhancement feature and a second chroma enhancement feature Step 6: Fusion of brightness features through the first fusion module 2111 and brightness enhancement features Get enhanced brightness features Step 7: Fusion of the first chromaticity feature by the second fusion module 2112 and the first chroma enhancement feature Get enhanced first chromaticity feature Step 8: Fusion of the second chromaticity feature by the third fusion module 2113 and the second chroma enhancement feature Get enhanced second chromaticity feature Step 9: Reconstruct the brightness features of the image by using the first reconstruction module 2114, the second reconstruction module 2115 and the third reconstruction module 2116 respectively. Enhance the first brightness feature And enhance the second brightness feature Processing to obtain enhanced brightness component The enhanced first chrominance component is expressed as And the enhanced second chrominance component is expressed as Step 10: Combine and enhance the brightness component The enhanced first chrominance component is expressed as And the enhanced second chrominance component is expressed as Get enhanced image
[0280] Reference Fig. 22 As shown, based on any of the above-mentioned image decoding methods, the image decoding method provided in the embodiment of the present application also includes reconstructing the target image through a content adaptive filtering network Follow these steps:
[0281] S221. Acquire an initial luminance component, an initial first chrominance component, and an initial second chrominance component according to the reconstructed image.
[0282] That is, obtain the reconstructed image The Y component, U component and V component of the .
[0283] S222: Generate an upsampled first chroma component and an upsampled second chroma component based on the initial first chroma component and the initial second chroma component, respectively.
[0284] The spatial size of the upsampled first chroma component and the upsampled second chroma component are both the same as the spatial size of the initial luminance component.
[0285] That is, the initial first chroma component and the initial second chroma component are respectively up-sampled to chroma components having the same spatial size as the initial luminance component.
[0286] In some embodiments, bicubic interpolation may be performed on the initial first chroma component and the initial second chroma component, respectively, so as to obtain the upsampled first chroma component and the upsampled second chroma component, respectively.
[0287] The initial first chrominance component is expressed as The initial second chrominance component is expressed as The upsampled first chrominance component is expressed as The upsampled second chrominance component is expressed as The bicubic interpolation operation is represented by BI(), and the above step S222 can be represented by the following formula:
[0288]
[0289]
[0290] S223, performing feature extraction on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain a luminance feature, a first chrominance feature, and a second chrominance feature.
[0291] In some embodiments, the above step S223 (respectively extracting features from the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component to obtain luminance features, first chrominance features, and second chrominance features) includes:
[0292] Forward wavelet transform (WT) is performed on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component, respectively, to obtain the luminance feature, the first chrominance feature, and the second chrominance feature.
[0293] The initial brightness component is expressed as The upsampled first chrominance component is expressed as The upsampled second chrominance component is expressed as The forward wavelet transform is expressed as DWT(), and the brightness feature is expressed as The first chromaticity feature is expressed as The second chromaticity feature is expressed as Then the above S223 can be expressed as the following formula:
[0294]
[0295]
[0296]
[0297] S224. Acquire initial image features of the reconstructed image according to the brightness feature, the first chromaticity feature, and the second chromaticity feature.
[0298] In some embodiments, the initial image features of the reconstructed image are obtained based on the brightness feature, the first chromaticity feature, and the second chromaticity feature, including: splicing the brightness feature, the first chromaticity feature, and the second chromaticity feature from the channel dimension to obtain the initial image features of the reconstructed image.
[0299] The brightness feature is expressed as The first chromaticity feature is expressed as The second chromaticity feature is expressed as The concatenation operation in the channel dimension is represented as Concatenatio(), and the initial image features of the reconstructed image are represented as F YUV , then the above step S224 can be expressed as the following formula:
[0300]
[0301] S225 , extracting features of the initial image through a content-adaptive inter-channel correlation information filter ICCI to obtain enhanced features.
[0302] Reference Fig.23 As shown, in some embodiments, the content-adaptive channel-to-channel correlation information filter includes: a convolution layer 2301, a batch normalization layer 2302, an activation function layer 2303, and a residual block sequence 2304 composed of multiple residual blocks. The feature processing flow of the content-adaptive channel-to-channel correlation information filter includes: processing the initial image features through the convolution layer 2301, the batch normalization layer 2302, the activation function layer 2303, and the residual block sequence 2304 in sequence, and scaling the output features of the residual block sequence 2304 by γ times.
[0303] In some embodiments, the convolution layer 2301 is a convolution layer with a convolution kernel size of 3×3.
[0304] In some embodiments, the activation function of the activation function layer 2303 is a ReLU activation function.
[0305] In some embodiments, the residual block sequence 2304 consists of 8 1D residual blocks.
[0306] Reference Fig.24 As shown, in some embodiments, the residual block in the residual block sequence 2404 includes: a first convolution layer 2401, a first batch of normalization layers 2402, a ReLU activation function layer 2403, a second convolution layer 2404, a second batch of normalization layers 2405, and a fusion unit 2406. The residual block is used to process the input feature I5 of the residual block through the first convolution layer 2401, the first batch of normalization layers 2402, the ReLU activation function layer 2403, the second convolution layer 2404, and the second batch of normalization layers 2405 in sequence, and perform γ on the output feature I5 of the second batch of normalization layers 2405. n The scaled feature is scaled by times, and the scaled feature and the input feature I5 of the residual block are added and fused by the fusion unit 2406 to obtain the output feature O5 of the residual block.
[0307] In some embodiments, the above step S225 (performing feature enhancement processing on the brightness feature, the first chromaticity feature and the second chromaticity feature respectively based on the enhanced feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature and an enhanced second chromaticity feature) includes:
[0308] Step 2251: Perform convolution processing on the initial image feature to obtain a first feature.
[0309] In some embodiments, the initial image feature may be convolved through a convolution layer with a convolution kernel size of 3×3 to obtain the first feature.
[0310] The initial image features of the reconstructed image are represented as F YUV , the convolution process is represented as Conv 3×3 (), the first feature is represented by F1, then the above step 2251 can be expressed as the following formula:
[0311] F1=Conv 3×3 (F YUV )
[0312] Step 2252: Perform batch normalization on the first feature to obtain a second feature.
[0313] In some embodiments, the first features may be batch normalized by a batch normalization layer to obtain the second features.
[0314] The first feature is represented as F1, the batch normalization process is represented as BN(), and the second feature is represented as F2. Then the above step 2252 can be represented as the following formula:
[0315] F2=BN(F1)
[0316] Step 2253: perform linear activation processing on the second feature to obtain the third feature.
[0317] In some embodiments, performing linear activation processing on the second feature to obtain the third feature includes: performing linear activation processing on the second feature through a ReLU activation function layer to obtain the third feature.
[0318] The second feature is represented as F2, the linear activation process is represented as ReLU(), and the third feature is represented as F3. Then the above step 2253 can be expressed as the following formula:
[0319] F3 = ReLU(F2)
[0320] Step 2254: Process the third feature through a residual network to obtain the enhanced feature.
[0321] The residual network includes a plurality of residual blocks connected in series, and the residual block is used to sequentially perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.
[0322] In some embodiments, the residual network includes 8 serially connected residual blocks. The structure of the residual block can refer to Fig.24 The structure of the residual block shown is not described in detail here to avoid redundancy.
[0323] The third feature is represented as F3, the processing of the residual network is represented as ReseN(), and the enhanced feature is represented as F enha , then the above step 2254 can be expressed as the following formula:
[0324] F enha =ReseN(F3)
[0325] S226. Perform feature enhancement processing on the brightness feature, the first chromaticity feature, and the second chromaticity feature respectively based on the enhanced feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature.
[0326] Step 2261: Perform convolution processing on the enhancement feature to obtain a brightness enhancement feature.
[0327] In some embodiments, the enhancement feature may be convolved with a convolution kernel having a convolution kernel size of 3×3 to obtain the brightness enhancement feature.
[0328] Denote the enhanced feature as F enha , the convolution process is represented as Conv 3×3 (), the brightness enhancement feature is expressed as Then the above step 2261 can be expressed as the following formula:
[0329]
[0330] Step 2262: Add and fuse the brightness enhancement feature and the brightness feature to obtain the enhanced brightness feature.
[0331] The brightness enhancement feature is expressed as Additive fusion is represented by +, and enhanced brightness features are represented by Then the above step 2262 can be expressed as the following formula:
[0332]
[0333] Step 2263: Process the enhanced features through a first residual block sequence to obtain a first residual feature.
[0334] Among them, the first residual block sequence includes multiple serially connected residual blocks, and the residual block is used to perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block in sequence to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.
[0335] In some embodiments, the first residual block sequence includes 5 serially connected residual blocks. The structure of the residual block can refer to Fig.24 The structure of the residual block shown is not described in detail here to avoid redundancy.
[0336] Denote the enhanced feature as F enha , the processing of the first residual block sequence is represented as RBS U (), the first residual feature is represented by Res U , then the above step 2263 can be expressed as the following formula:
[0337] Res U =RBS U (F en a )
[0338] Step 2264: perform convolution processing on the first residual feature to obtain a first chroma enhancement feature.
[0339] The first residual feature is denoted as Res U , the convolution process is represented as Conv 3×3 (), the first chroma enhancement feature is expressed as Then the above step 2264 can be expressed as the following formula:
[0340]
[0341] Step 2265: Add and fuse the first chromaticity feature and the first chromaticity enhancement feature to obtain the enhanced first chromaticity feature.
[0342] The first chroma enhancement feature is expressed as Additive fusion is represented by +, and the enhanced first chromaticity feature is represented by Then the above step 2265 can be expressed as the following formula:
[0343]
[0344] Step 2266: Process the enhanced features through a second residual block sequence to obtain a second residual feature.
[0345] Among them, the second residual block sequence includes multiple serially connected residual blocks, and the residual block is used to perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block in sequence to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.
[0346] In some embodiments, the second residual block sequence includes 5 serially connected residual blocks. The structure of the residual block can refer to Fig.24 The structure of the residual block shown is not described in detail here to avoid redundancy.
[0347] Denote the enhanced feature as F enha , the processing of the second residual block sequence is represented as RBS V (), the second residual feature is represented by Res V , then the above step 2266 can be expressed as the following formula:
[0348] Res V =RBS V (F ena )
[0349] Step 2267: Perform convolution processing on the second residual feature to obtain a second chroma enhancement feature.
[0350] The second residual feature is denoted as Res V , the convolution process is represented as Conv 3×3 (), the second chroma enhancement feature is expressed as Then the above step 2267 can be expressed as the following formula:
[0351]
[0352] Step 2268: Add and fuse the second chromaticity feature and the second chromaticity enhancement feature to obtain the enhanced second chromaticity feature.
[0353] The second chromaticity feature is expressed as The second chroma enhancement feature is expressed as Additive fusion is represented by +, and the enhanced second chromaticity feature is represented by Then the above step 2268 can be expressed as the following formula:
[0354]
[0355] S227. Obtain an enhanced image of the reconstructed image according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature.
[0356] In some embodiments, the image encoding method or image decoding method provided by the embodiments of the present application further includes: training each network model in the image encoder and the image decoder based on a sample data set.
[0357] Wherein, the sample data set includes: multiple sample data groups, and any sample data group includes: sample images and machine vision task processing results corresponding to the sample images. The process of training each network model in the image encoder and image decoder based on the sample data set may include: first inputting the sample image into the image encoder for encoding to obtain the encoded data of the image, and then decoding the encoded data through the image decoder to obtain the reconstructed image and the machine vision task processing results output by the machine vision task model, and then calculating the first loss value based on the reconstructed image and the sample image, calculating the second loss value based on the machine vision task processing results corresponding to the sample image and the machine vision task processing results output by the machine vision task model, and then adjusting the parameters of the network model in the image encoder and / or image decoder based on the first loss value and the second loss value until the image encoder and the image decoder converge.
[0358] In some embodiments, some embodiments of the present application provide an image encoding device, the image encoding device comprising:
[0359] a memory configured to store a computer program;
[0360] The processor is configured to, when calling a computer program, enable the image encoding device to implement the image encoding method described in any one of the above embodiments.
[0361] In some embodiments, some embodiments of the present application provide an image decoding device, the image decoding device comprising:
[0362] a memory configured to store a computer program;
[0363] The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method described in any one of the above embodiments.
[0364] In some embodiments, some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image encoding method described in any of the above embodiments or the image decoding method described in any of the above embodiments.
[0365] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image encoding method described in any of the above embodiments or the image decoding method described in any of the above embodiments.
[0366] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0367] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.
Claims
1. An image decoding method, characterized in that: include: Acquire a base layer reconstruction potential feature and an enhancement layer reconstruction potential feature of the target image according to the encoded data of the target image; Acquire enhancement layer context features according to the enhancement layer reconstruction potential features and the enhancement layer reconstruction quantization potential features that have been reconstructed; Decoding the following elements of the coded data of the enhancement layer quantized latent feature according to the enhancement layer context feature; Acquire a base layer context feature according to the enhancement layer reconstruction quantization potential feature, the base layer reconstruction potential feature, and the reconstructed base layer reconstruction quantization potential feature; Decoding the following elements of the encoded data of the base layer quantized latent feature according to the base layer context feature; A reconstructed image of the target image is obtained according to the enhanced layer reconstructed quantized latent features and the base layer reconstructed quantized latent features.
2. The method according to claim 1, characterized in that The step of obtaining enhancement layer context features according to the enhancement layer reconstruction potential features and the reconstructed enhancement layer reconstruction quantization potential features includes: Performing channel context feature extraction on the enhancement layer reconstruction potential features based on the channel context module to obtain enhancement layer channel context features; Based on the spatial context module, spatial context feature extraction is performed on the quantized latent features of the enhanced layer that has been reconstructed to obtain the spatial context features of the enhanced layer; splicing the enhancement layer reconstruction potential feature, the enhancement layer channel context feature, and the enhancement layer spatial context feature to obtain a first splicing feature; The first splicing feature is processed based on an entropy parameter module to obtain the enhancement layer context feature.
3. The method according to claim 1, characterized in that The acquiring of base layer context features according to the enhancement layer reconstruction quantization potential features, the base layer reconstruction potential features, and the reconstructed base layer reconstruction quantization potential features includes: splicing the enhancement layer quantized latent feature and the base layer reconstructed latent feature to obtain a second spliced feature; Performing channel context feature extraction on the second splicing feature based on the channel context module to obtain a base layer channel context feature; Based on the spatial context module, spatial context features are extracted from the reconstructed base layer quantization potential features to obtain the base layer spatial context features; splicing the base layer reconstruction potential feature, the base layer channel context feature, and the base layer spatial context feature to obtain a third splicing feature; The third splicing feature is processed based on an entropy parameter module to obtain the base layer context feature.
4. The method according to claim 2 or 3, characterized in that The channel context module includes: a first convolution layer, a first activation function layer, a second convolution layer, a second activation function layer and a third convolution layer connected in series.
5. The method according to claim 2 or 3, characterized in that The entropy parameter module is composed of multiple convolutional layers connected in series.
6. The method according to claim 1, characterized in that The step of obtaining a reconstructed image of the target image according to the enhanced layer reconstructed quantized latent features and the base layer reconstructed quantized latent features includes: splicing the base layer reconstructed quantized latent features and the enhancement layer reconstructed quantized latent features to obtain a reconstructed quantized latent feature; The reconstructed quantized latent features are input into a synthetic transformation network, and a reconstructed image of the target image output by the synthetic transformation network is obtained.
7. The method according to claim 6, characterized in that After obtaining the reconstructed image of the target image output by the synthetic transformation network, the method further includes: Acquire an initial luminance component, an initial first chrominance component, and an initial second chrominance component according to the reconstructed image; generating an upsampled first chroma component and an upsampled second chroma component based on the initial first chroma component and the initial second chroma component, respectively; wherein the spatial size of the upsampled first chroma component and the upsampled second chroma component are the same as the spatial size of the initial luma component; Performing feature extraction on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component, respectively, to obtain a luminance feature, a first chrominance feature, and a second chrominance feature; Acquire an initial image feature according to the brightness feature, the first chromaticity feature, and the second chromaticity feature; Extracting features from the initial image using a content-adaptive inter-channel correlation information filter to obtain enhanced features; performing feature enhancement processing on the luminance feature, the first chrominance feature, and the second chrominance feature based on the enhancement feature to obtain an enhanced luminance feature, an enhanced first chrominance feature, and an enhanced second chrominance feature; An enhanced image of the reconstructed image is acquired according to the enhanced luminance feature, the enhanced first chrominance feature, and the enhanced second chrominance feature.
8. The method according to claim 1, characterized in that The method further comprises: After decoding all elements of the encoded data of the base layer quantized latent features, processing the base layer reconstructed quantized latent features based on a latent space transform network to obtain machine vision features; The machine vision feature is input into a machine vision task model to obtain a machine vision task processing result output by the machine vision task model.
9. The method according to claim 1, characterized in that The coded data of the target image includes: first coded data obtained by entropy coding the quantized super-prior features of the target image, and obtaining the base layer reconstruction potential features and the enhancement layer reconstruction potential features of the target image according to the coded data of the target image includes: Performing entropy decoding on the first coded data to obtain a reconstructed super-prior feature; Performing super-prior decoding on the reconstructed super-prior feature to obtain a reconstructed latent feature; The reconstruction potential features are segmented to obtain the base layer reconstruction potential features and the enhancement layer reconstruction potential features.
10. The method according to claim 9, characterized in that The segmenting of the reconstruction potential features to obtain the base layer reconstruction potential features and the enhancement layer reconstruction potential features includes: Quantifying the reconstructed latent features to obtain reconstructed quantized latent features; Processing the reconstructed quantized latent features based on a channel attention mechanism to obtain a channel attention mask; The reconstructed latent features are divided into the base layer reconstructed latent features and the enhanced layer reconstructed latent features according to the channel attention mask.
11. The method according to claim 10, characterized in that The processing of the reconstructed quantized latent features based on the channel attention mechanism to obtain a channel attention mask includes: Performing adaptive pooling processing on the reconstructed quantized latent features to obtain a first feature; performing convolution processing, normalization processing, and nonlinear activation processing on the first features in sequence to obtain a second feature; Performing convolution processing on the second feature to obtain a third feature and a fourth feature, wherein the feature tensors of the third feature and the fourth feature are both C×1×1, where C is the number of channels of the potential feature; Gumbel sampling is performed on the third feature and the fourth feature to obtain the channel attention mask.
12. The method according to claim 1, characterized in that The coded data of the target image includes: first coded data obtained by entropy coding the base layer quantized super-prior features of the target image, and second coded data obtained by entropy coding the enhancement layer quantized super-prior features of the target image, and obtaining the base layer reconstruction potential features and the enhancement layer reconstruction potential features of the target image according to the coded data of the target image includes: Performing entropy decoding on the first coded data and the second coded data respectively to obtain a base layer reconstruction super priori feature and an enhancement layer reconstruction super priori feature; The base layer reconstruction super a priori feature and the enhancement layer reconstruction super a priori feature are respectively super a priori decoded to obtain the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature.
13. An image decoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Enhancement layer coding and decoding method and device
CN115134597A
Multi-code-rate depth image compression system and method applied to mixed context
CN115866265A
Scalable coding of video and associated features
US20230065862A1
