Image decoding method and device

By performing feature extraction and enhancement processing on JPEG AI coded data, the problem of image quality degradation when JPEG AI reduces the code rate is solved, higher quality image reconstruction is achieved, and the structure of the image decoder is simplified.

CN119094773APending Publication Date: 2024-12-06HISENSE VISUAL TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202311575516.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-10-27
Filing Date
2023-11-23
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Existing learning-based image coding standards such as JPEG AI, when reducing the code rate of image coding data, lead to a degradation of reconstruction image quality through downsampling and reducing chroma component channel processing.

Method used

By extracting and splicing the initial brightness component and upsampling chrominance component, feature enhancement is used to use content adaptive channel-related information filters to generate enhanced brightness and chrominance features, thereby improving the quality of the reconstruction image.

Benefits of technology

It effectively improves the quality of the reconstructed image, reduces the complexity of the image decoder, reduces the amount of parameters, and improves the image decoding efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119094773A_ABST
    Figure CN119094773A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image decoding method and device, and relates to the technical field of image coding and decoding. Comprises: obtaining a reconstructed image of a target image; generating up-sampled first and second chroma components having the same spatial size as the initial luma component from the initial first and second chroma components; respectively carrying out feature extraction on the initial brightness component and the up-sampling first and second chrominance components to obtain a brightness feature, a first chrominance feature and a second chrominance feature; splicing the brightness feature, the first chrominance feature and the second chrominance feature to obtain an initial image feature; performing feature extraction on the initial image features through a content adaptive inter-channel related information filter to obtain enhanced features; performing feature enhancement processing on the brightness feature, the first chrominance feature and the second chrominance feature based on the enhancement feature to obtain an enhanced brightness feature, an enhanced first chrominance feature and an enhanced second chrominance feature; and enhancing the first and second chromaticity features according to the enhanced brightness feature, and obtaining an enhanced image of the reconstructed image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims the priority of the Chinese patent application filed with the Chinese Patent Office on June 6, 2023 with application number 2023106645409, and the priority of the Chinese patent application filed with the Chinese Patent Office on October 27, 2023 with application number 2023114114882. The contents of the prior applications are incorporated into this application by reference. Technical Field

[0002] Some embodiments of the present application relate to the technical field of video coding and decoding, and more specifically, to an image decoding method and device. Background Art

[0003] Image compression is one of the basic tasks of computer vision and also one of the important tasks of image processing. Image compression is to convert a two-dimensional image into binary data with smaller bits, and then restore the binary data into a high-quality image. Image compression is widely used in image transmission, image storage and image processing. Image compression is divided into lossy image compression and lossless image compression. Lossless image compression will lose some data that has little impact on image quality during the compression process, which can greatly increase its compression rate, but the reconstructed image will often be reduced.

[0004] In recent years, deep learning has developed rapidly, and learning-based image compression has received widespread attention, which includes processes such as transformation, quantization, and entropy coding. Some learning-based image compression models have surpassed traditional image compression on the rate-distortion curve, and have great prospects in the future. JPEG AI is a learning-based image coding standard proposed by the Joint Photographic Experts Group (JPEG). Although the JPEG AI image encoder has better subjective quality than the commonly used image coding standards at the same bit rate, in order to reduce the bit rate of the image coding data, the JPEG AI image encoder processes the chrominance components (U component and V component) in the image data through additional downsampling and a smaller number of channels, which affects the quality of the reconstructed image. Summary of the invention

[0005] The exemplary embodiments of the present application provide an image decoding method and apparatus for improving the quality of a reconstructed image.

[0006] Some embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, some embodiments of the present application provide an image decoding method, including:

[0008] Decoding the coded data of the target image to obtain a reconstructed image of the target image, wherein the image data of the reconstructed image includes: an initial luminance component, an initial first chrominance component, and an initial second chrominance component;

[0009] Generate an upsampled first chroma component and an upsampled second chroma component according to the initial first chroma component and the initial second chroma component respectively; the spatial sizes of the upsampled first chroma component and the upsampled second chroma component are the same as the spatial size of the initial luminance component;

[0010] Performing feature extraction on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain a luminance feature, a first chrominance feature, and a second chrominance feature;

[0011] splicing the brightness feature, the first chromaticity feature, and the second chromaticity feature to obtain an initial image feature;

[0012] Extracting features of the initial image using a content-adaptive inter-channel related information filter to obtain enhanced features;

[0013] Based on the enhancement feature, feature enhancement processing is performed on the brightness feature, the first chromaticity feature, and the second chromaticity feature respectively to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature;

[0014] An enhanced image of the reconstructed image is acquired according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature.

[0015] In a second aspect, some embodiments of the present application provide an image decoding device, including:

[0016] a memory configured to store a computer program;

[0017] The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method described in the first aspect.

[0018] In a third aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a computing device, the computing device implements the image decoding method described in the first aspect or the image decoding method described in the second aspect.

[0019] In a fourth aspect, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image decoding method described in the first aspect or the image decoding method described in the second aspect.

[0020] The image decoding method provided in the embodiment of the present application decodes the encoded data of the target image to obtain a reconstructed image of the target image, firstly generates an upsampled first chroma component and an upsampled second chroma component whose spatial sizes are the same as the spatial size of the initial luminance component according to the initial first chroma component and the initial second chroma component, and then performs feature extraction on the initial luminance component, the upsampled first chroma component and the upsampled second chroma component respectively to obtain luminance features, first chroma features and second chroma features, and concatenates the luminance features, the first chroma features and the second chroma features to obtain initial image features; then performs feature extraction on the initial image features through a content adaptive inter-channel related information filter to obtain enhanced features, and performs feature enhancement processing on the luminance features, the first chroma features and the second chroma features respectively based on the enhanced features to obtain enhanced luminance features, enhanced first chroma features and enhanced second chroma features, and finally obtains an enhanced image of the reconstructed image according to the enhanced luminance features, the enhanced first chroma features and the enhanced second chroma features. Since the embodiment of the present application can extract the features of the initial image through a content-adaptive channel-related information filter to obtain enhanced features, and then perform feature enhancement processing on the brightness features, the first chromaticity features, and the second chromaticity features based on the enhanced features to obtain enhanced brightness features, enhanced first chromaticity features, and enhanced second chromaticity features, and obtain an enhanced image of the reconstructed image based on the enhanced brightness features, the enhanced first chromaticity features, and the enhanced second chromaticity features, the embodiment of the present application can improve the quality of the reconstructed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0022] Figure 1 A schematic diagram showing the structure of an image encoder in some embodiments of the present application is shown;

[0023] Figure 2 A schematic diagram of an analysis transformation network provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a residual activation unit provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a residual non-local attention block provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a residual block provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of a super-a priori coding network provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of a super-a priori scale coding network provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a masked convolution kernel used in a context network of an autoregressive convolution provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of a masked convolution kernel used in a context network provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of a wavefront parallel processing mechanism of a latent space provided in an embodiment of the present application;

[0032] Fig.11 A schematic diagram of a structural diagram of an image decoder provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of a synthetic transformation network provided in an embodiment of the present application;

[0034] Fig.13 A schematic diagram of a structural diagram of another image decoder provided in an embodiment of the present application;

[0035] Fig.14 A schematic diagram of a structural diagram of an ICCI sub-network provided in an embodiment of the present application;

[0036] Fig.15 A schematic diagram of a structural schematic diagram of an ICCI processing module provided in an embodiment of the present application;

[0037] Fig.16 A schematic diagram of a structural diagram of another ICCI sub-network provided in an embodiment of the present application;

[0038] Fig.17 A schematic diagram of a structural schematic diagram of another ICCI processing module provided in an embodiment of the present application;

[0039] Fig.18 A schematic diagram of a residual block in an ICCI subnetwork provided in an embodiment of the present application;

[0040] Fig.19A schematic diagram showing the structure of an image encoder in some embodiments of the present application is shown;

[0041] Fig. 20 A schematic diagram showing the structure of an analysis transformation network in some embodiments of the present application is shown;

[0042] Fig.21 A schematic diagram showing the structure of a super-a priori coding network in some embodiments of the present application is shown;

[0043] Fig. 22 A schematic diagram showing the structure of a super-a priori decoding network in some embodiments of the present application is shown;

[0044] Fig.23 A schematic diagram showing the structure of an image decoder in some embodiments of the present application is shown;

[0045] Fig.24 A schematic diagram showing the structure of a synthetic transformation network in some embodiments of the present application is shown;

[0046] Fig.25 A schematic diagram showing the structure of a potential space transformation network in some embodiments of the present application is shown;

[0047] Fig.26 A schematic diagram showing the structure of an image encoder in some embodiments of the present application is shown;

[0048] Fig. 27 A schematic diagram showing the structure of a mask generation network in some embodiments of the present application is shown;

[0049] Fig.28 A schematic diagram showing the structure of an image decoder in some embodiments of the present application is shown;

[0050] Fig.29 A flowchart of the steps of the image decoding method in some embodiments provided in the embodiments of the present application;

[0051] Fig.30 A flowchart of the steps of the image decoding method in some other embodiments provided in the embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0053] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0054] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0055] References to "some implementations", "some embodiments", etc. in the specification indicate that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may include the specific features, structures, or characteristics. In addition, such phrases do not necessarily refer to the same implementation. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, it is considered to be within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in connection with other implementations (whether or not explicitly described herein).

[0056] Image compression is one of the basic tasks in computer vision and image processing. The development of lossy image compression technology has brought benefits to various fields. Image compression technology can greatly reduce the amount of data for image storage and image transmission, but the image quality obtained after reconstructing the image may be slightly reduced. In the field of lossy image compression technology, traditional codecs including the Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG) and Versatile Video Coding (VVC) use similar transforms, quantization and entropy coding to compress images.

[0057] In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression technology. JPEG AI is a learning-based image coding standard created by the Joint Photographic Experts Group (JPEG). JPEG AI targets human vision and achieves a lower bit rate than traditional image coding at the same subjective quality.

[0058] Figure 1 This is a schematic diagram of the structure of the encoder under the JPEG AI standard. Figure 1As shown, the encoder under the JPEG AI standard includes: an analysis transform network (Analysis Transform Net) 101, a residual unit 102, a gain unit 103, a first quantization unit 104, a super priori coding network 105, a second quantization unit 106, a decomposition entropy model 107, a first encoding module 108, a decoding module 109, a super priori scale decoding network 110, a second encoding module 111, a super priori decoding network 112, an inverse gain unit 113, a fusion unit 114, a context network 115 and a prediction fusion network 116. Figure 1 The process of image encoding by the encoder shown includes the following steps 1 to 16:

[0059] Step 1: Process the target image X through the analysis transformation network 101 to obtain the potential feature Y of the target image X.

[0060] Among them, the latent features Y are high-dimensional representations of the target image X. The latent features Y cannot be directly observed from the target image, but they can be obtained through the information learned from the target image. The above latent features Y are obtained based on the information learned from the target image X.

[0061] Reference Figure 2 As shown, in some embodiments, the analysis transformation network 101 includes: a first downsampling layer 21, a first residual activation unit 22, a residual non-local attention block (Residual non-local attention networks for image restoration) 23, a second downsampling layer 24, a second residual activation unit 25, a third downsampling layer 26, a third residual activation unit 27, a fourth downsampling layer 28 and a convolution layer 29, which are sequentially connected. The feature processing process of the analysis transformation network 101 includes: processing the target image X through the first downsampling layer 21, the first residual activation unit 22, the residual non-local attention block 23, the second downsampling layer 24, the second residual activation unit 25, the third downsampling layer 26, the third residual activation unit 27, the fourth downsampling layer 28 and the convolution layer 29 in sequence to obtain the potential feature Y.

[0062] In some embodiments, the first downsampling layer 21, the second downsampling layer 24, the third downsampling layer 26 and the fourth downsampling layer 28 are all downsampling convolution layers with a convolution kernel size of 3×3.

[0063] In some embodiments, the convolution layer 29 is a convolution layer with a convolution kernel size of 1×1.

[0064] The residual activation units (the first residual activation unit 22, the second residual activation unit 25, and the third residual activation unit 27) in the analysis transformation network 101 are residual activation units with an attention mechanism. Figure 3 As shown, in some embodiments, the residual activation unit in the analysis transformation network 101 includes: a LeakyReLU activation function layer 31, a convolution layer 32, a tanh activation function layer 33, a nonlinear transformation unit 34 and a fusion unit 35. The feature processing process of the residual activation unit includes: first, the LeakyReLU activation function layer 31, the convolution layer 32, and the tanh activation function layer 33 process the input features I1 of the residual activation unit in sequence, then, the nonlinear transformation unit 34 performs a nonlinear transformation on the input features I1 of the residual activation unit through the output features of the tanh activation function layer 33, and finally, the fusion unit 35 adds and fuses the output features of the nonlinear transformation unit 34 and the input features I1 of the residual activation unit to obtain the output features O1 of the residual activation unit.

[0065] Reference Figure 4 As shown, in some embodiments, the residual non-local attention block 23 in the analysis transformation network 101 includes: a first residual block 401, a second residual block 402, a third residual block 403, a fourth residual block 404, a fifth residual block 405, a first downsampling layer 406, a sixth residual block 407, a seventh residual block 408, a second downsampling layer 409, an eighth residual block 410, a ninth residual block 411, a convolution layer 412, a Sigmoid activation function layer 413, a nonlinear transformation unit 414 and a fusion unit 415. The process of processing the feature by the residual non-local attention block 23 in the analysis transformation network 101 includes: on the one hand, the input feature (the output feature of the first residual activation unit 22 in the analysis transformation network 101) I2 of the residual non-local attention block 23 is processed in sequence by the first residual block 401, the second residual block 402, and the third residual block 403 to obtain the first residual feature On the other hand, the input feature I2 of the residual non-local attention block 23 is processed in sequence by the fourth residual block 404, the fifth residual block 405, the first downsampling layer 406, the sixth residual block 407, the seventh residual block 408, the second downsampling layer 409, the eighth residual block 410, the ninth residual block 411, the convolution layer 412 and the Sigmoid activation function layer 413 to obtain the second residual feature The first residual feature is then transformed by the nonlinear transformation unit 414 and the second residual feature Perform multiplication to obtain the third residual feature Finally, the third residual feature is fused by the fusion unit 415 The input feature I2 of the residual non-local attention block 23 is added and fused to obtain the output feature O2 of the residual non-local attention block 23.

[0066] Reference Figure 5 As shown, in some embodiments, any residual block in the residual non-local attention block 23 (the first residual block 401, the second residual block 402, the third residual block 403, the fourth residual block 404, the fifth residual block 405, the sixth residual block 407, the seventh residual block 408, the eighth residual block 410, and the ninth residual block 411) includes: a first convolution layer 51, a LeakyReLU activation function layer 52, a second convolution layer 53, and a fusion unit 54. The process of processing the feature by the residual block in the residual non-local attention block 23 includes: processing the input feature I3 of the residual block through the first convolution layer 51, the LeakyReLU activation function layer 52, and the second convolution layer 53 in sequence, and adding and fusing the input feature I3 of the residual block with the output feature of the second convolution layer 53 through the fusion unit 54 to obtain the output feature O3 of the residual block.

[0067] Step 2: The residual calculation unit 102 calculates the difference between the potential feature Y and the predicted feature μ to obtain the residual feature R.

[0068] Among them, the predicted feature μ is the reconstructed potential feature output by the prediction fusion network 116 based on the context feature Φ output by the context network 115 and the super prior decoding network 112 Predict the potential feature Y and obtain the prediction information of the potential feature Y.

[0069] Step 3: The gain unit 103 performs gain processing on the residual feature R to obtain the residual feature Rg after gain processing.

[0070] In some embodiments, the gain unit 103 is used to multiply each element of the residual feature R by a forward gain m t The gain factor specified in is used to obtain the residual feature Rg after gain processing. The residual feature R is the difference between the potential feature Y and the predicted feature μ, where the predicted feature μ is the predicted feature based on the elements in the potential feature Y. The gain unit 103 is used to implement the variable rate capability of the encoder. The gain unit 103 uses a parameter β to control the compression ratio. The parameter β specifies the weight between the rate and the distortion in the loss function during training. During the training of the image encoder, a pair of forward gain vectors and reverse gain vectors are obtained for multiple βs, which can be expressed as Among them, m t is the forward gain vector, is the reverse gain vector. During encoding and decoding, the forward gain vector and the reverse gain vector are known to the encoder and decoder (they are part of the encoder), and the gain vector mt and The dimension is equal to the number of channels of the residual feature R.

[0071] Step 4: The first quantization unit 104 quantizes the residual feature Rg after the gain processing to obtain a quantized residual feature r.

[0072] The quantization process refers to the process of approximating the continuous value of a signal (or a large number of possible discrete values) to a finite number of discrete values ​​(or a small number of discrete values).

[0073] Step 5: The super-a priori coding network 105 performs super-a priori coding on the latent feature Y to obtain the super-a priori feature Z, and extracts additional auxiliary information during the super-a priori coding process so that an accurate probability model of entropy coding can be obtained based on the coding result.

[0074] Reference Figure 6 As shown, in some embodiments, the super prior coding network 105 includes: an absolute operation layer 61, a first convolution layer 62, a first LeakyReLU activation function layer 63, a second convolution layer 64, a first downsampling layer 65, a second LeakyReLU activation function layer 66, a third convolution layer 67, and a second downsampling layer 68 connected in series. The super prior coding network 105 is used to process the potential feature Y through the absolute operation layer 61, the first convolution layer 62, the first LeakyReLU activation function layer 63, the second convolution layer 64, the first downsampling layer 65, the second LeakyReLU activation function layer 66, the third convolution layer 67, and the second downsampling layer 68 in sequence to obtain the super prior feature Z.

[0075] In some embodiments, the first convolution layer 92, the second convolution layer 94, and the third convolution layer 97 are convolution layers with a convolution kernel size of 3×3.

[0076] In some embodiments, the first downsampling layer 95 and the second downsampling layer 98 are downsampling convolution layers with a convolution kernel size of 3×3.

[0077] Step 6: The second quantization unit 106 performs quantization processing on the super a priori feature Z to obtain a quantized super a priori feature z.

[0078] Step 7: Decompose the entropy model 107 to obtain the cumulative distribution function cdf (Cumulative Distribution Function, CDF).

[0079] In some embodiments, the decomposition entropy model 107 can be integerized to solve the problem of device interoperability, so that the decomposition entropy model 107 can be used on any device. The cumulative distribution function is also called a probability table, which is pre-calculated and quantized, rather than trained based on trainable parameters (matrix, bias and factor). The table is stored in integer form, so there is no need to store the parameters of the quantized hyper-prior feature z distribution.

[0080] Step 8: The first encoding module 108 performs entropy encoding on the quantized super-prior feature z according to the cumulative distribution function cdf to obtain the first encoded data b1.

[0081] In some embodiments, the first encoding module 108 may be a lossless entropy encoder, and the process of entropy encoding the quantized super prior feature z according to the cumulative distribution function cdf includes: integrating the cumulative distribution function cdf and the elements to be encoded in the quantized super prior feature z.

[0082] Step 9: The decoding module 109 performs entropy decoding on the first coded data b1 according to the cumulative distribution function cdf to obtain the reconstructed super-prior feature

[0083] In some embodiments, the decoding module 109 may be a lossless entropy decoding module, and the process of entropy decoding the first coded data b1 according to the cumulative distribution function cdf includes: integrating the cumulative distribution function cdf and the elements to be decoded in the first coded data b1.

[0084] Step 10: The super-prior scale coding network 110 reconstructs the super-prior features Processing is performed to obtain the variance N(0,σ) of a Gaussian distribution with a probability distribution mean of 0.

[0085] In some embodiments, the super-prior scale decoding network 110 is used to reconstruct the super-prior features. The inverse operation of the super-prior coding network 105 is performed, and the variance σ of the Gaussian distribution N(0,σ) with a probability distribution of 0 mean is output. The super-prior scale decoding network 113 may include two 5×5 inverse convolution sequences with an upsampling stride of 2, two 3×3 convolution layers, and a LeakyReLU activation function layer. The super-prior scale decoding network 110 is symmetrical to the super-prior coding network 105, and each upsampling convolution layer is followed by a cropping operation.

[0086] Reference Figure 7As shown, in some embodiments, the super prior scale coding network 110 includes: a first upsampling layer 701, a first LeakyReLU activation function layer 702, a second upsampling layer 703, a second LeakyReLU activation function layer 704, a first convolution layer 705, a third LeakyReLU activation function layer 706, a second convolution layer 707, a fourth LeakyReLU activation function layer 708, a third convolution layer 709, a fifth LeakyReLU activation function layer 710 and a fourth convolution layer 711, which are connected in series. The super prior scale coding network 110 reconstructs the super prior features. The processing includes: reconstructing the super prior features through the first upsampling layer 701, the first LeakyReLU activation function layer 702, the second upsampling layer 703, the second LeakyReLU activation function layer 704, the first convolution layer 705, the third LeakyReLU activation function layer 706, the second convolution layer 707, the fourth LeakyReLU activation function layer 708, the third convolution layer 709, the fifth LeakyReLU activation function layer 710 and the fourth convolution layer 711 in sequence Processing is performed to obtain the variance σ.

[0087] In some embodiments, the first upsampling layer 701 and the second upsampling layer 702 are convolution layers with convolution kernel sizes of C×3×3 and 3 / 2C×3×3, respectively, where C is the reconstructed hyper-prior feature. The number of channels.

[0088] In some embodiments, the first convolution layer 705, the second convolution layer 707, the third convolution layer 709 and the fourth convolution layer 711 are convolution layers with convolution kernel sizes of 2C×3×3, 5 / 3C×3×3, 4 / 3C×3×3, and C×3×3, respectively.

[0089] Step 11: The second encoding module 111 performs lossless encoding on the quantized residual feature r based on the variance N(0,σ) to obtain second encoded data b2.

[0090] Furthermore, in order to obtain the predicted feature μ of the next element of the potential feature Y, the encoding process of the encoder also includes the following steps:

[0091] Step 12: The super-prior decoding network 112 reconstructs the super-prior features Perform super-prior decoding processing to obtain reconstruction potential features

[0092] In some embodiments, the super-prior decoding network 112 includes a series of inverse convolution layers with an upsampling step size of 2, three convolutions without tensor size changes, and ReLU activation function layers. The super-prior decoding network 112 can receive features of size C×h×w as input and generate output features with dual channels 4C×h×w. The super-prior decoding network 112 is symmetrical to the super-prior encoding network 107, and each upsampling convolution layer is followed by a cropping operation.

[0093] Step 13: The inverse gain unit 113 performs inverse gain processing on the quantized residual feature r to obtain an inverse gain residual feature rig.

[0094] In some embodiments, the inverse gain unit 113 is used to multiply each element of the quantized residual feature r by the inverse gain vector The inverse gain factor specified in is used to obtain the residual tensor rig after the inverse gain processing. As described above, the inverse gain unit 113 can realize the variable rate capability of the encoder. The inverse gain unit 113 uses the parameter β to control the compression ratio. The parameter β specifies the weight between the rate and the distortion in the loss function during training.

[0095] Step 14: The fusion unit 114 adds and fuses the inverse gain residual feature rig and the predicted feature μ to obtain the fused potential feature y.

[0096] Step 15: The context network 115 performs context feature extraction on the fused latent feature y to obtain the context feature Φ.

[0097] In some embodiments, a context network based on autoregressive convolution may be used to extract context features Φ.

[0098] The essence of the contextual network based on autoregressive convolution is a masked convolution layer. For example, refer to Figure 8 As shown, Figure 8 Schematic diagram of the mask convolution kernel used in the context network based on autoregressive convolution provided in the embodiment of the present application. Figure 8 As shown, the spatial context network based on autoregressive convolution uses the completed coding elements before the current coding position 801 in the latent space to predict the context features corresponding to the current coding position, and uses the context features to subsequently predict the probability distribution of the current coding position. Among them, the completed coding elements are Figure 8 In some embodiments, the calculation formula of the spatial context network based on autoregressive convolution is:

[0099] g cm (x)=(M⊙W)x+b

[0100] Among them, x is the input feature of the spatial context network based on autoregressive convolution, M and W are the k×k mask and convolution weight respectively, ⊙ is the Hadamard operator, and b is the bias term.

[0101] The context network of autoregressive convolution can significantly improve the rate-distortion performance of the super-prior model. However, due to its autoregressive structure, the context network of the autoregressive convolution can only be decoded in a fixed raster scanning order. The latent space decoded later always depends on the latent space decoded earlier, resulting in a very slow decoding speed for this model.

[0102] In some embodiments, in order to achieve parallel processing in the context network 115 and solve the problem of slow decoding speed, the wavefront parallel processing (WPP) mechanism of the latent space can be further used in the context network 115. In this mechanism, the decoded elements are used to predict the elements of the fused latent feature y. The prediction process is an autoregressive process, and the elements of different rows in the fused latent feature y can be processed independently in a wavefront parallel manner, where the delay in processing consecutive rows depends on the shape of the masked convolution kernel.

[0103] Fig. 9 A schematic diagram of a masked convolution kernel used by a context network 115 provided in an embodiment of the present application. Fig. 9 In the figure, the row and column coordinates are established with the current coding position 91 as the coordinate origin. The current coding position is in the 0th row and the 0th column. n=0 indicates the row where the current coding position is located, n=-1 indicates the row before the row where the current coding position is located, and n=-2 indicates the row before the row where the current coding position is located. Fig. 9 The "-2", "-1", "0", "1" and "2" in the figure indicate different columns, where "0" indicates the column at the current encoding position, "-1" indicates 1 column before the column at the current encoding position 91, "-2" indicates 2 columns before the column at the current encoding position, "1" indicates 1 column after the column at the current encoding position, and "2" indicates 2 columns after the column at the current encoding position.

[0104] Fig.10 A schematic diagram of a wavefront parallel processing mechanism of a latent space in the related technology provided by the embodiment of the present application. In order to improve the utilization rate of computing resources, the wavefront parallel processing mechanism is introduced. Fig.10 "Columu" means column, "Row" means row, Fig.10 The latent space with 5 rows and 10 columns is used as an example. Fig.10 The elements of all channels of the tensor of the current pixel are predicted independently (simultaneously) and can be predicted based on the pixels that have been processed in the preset process. Fig.10The masked convolution kernel shown is designed in such a way that Fig.10 Each row of samples (i.e., the prediction of each row of pixels) can be processed in parallel, with the next row being processed delayed by one element compared to the previous row, i.e., delayed by the time of one wave movement (T).

[0105] Step 16: The prediction fusion network 116 reconstructs the super prior features based on the context features Φ and The predicted feature μ of the next element of the potential feature is obtained to encode the next element based on the predicted feature μ.

[0106] Finally, the first coded data b1 and the second coded data b2 are used as coded data of the target image X and sent to the decoding end.

[0107] Fig.11 This is a schematic diagram of the structure of the image decoder under the JPEG AI standard. Figure 1 As shown, the image decoder under the JPEG AI standard includes: a decomposition entropy model 1101, a first decoding module 1102, a super priori scale decoding network 1103, a second decoding module 1104, an inverse gain unit 1105, a fusion module 1106, a synthetic transformation network 1107, a context network 1108, a super priori decoding network 1109, and a prediction fusion network 1110.

[0108] Fig.11 The process of image decoding by the image decoder shown includes the following steps a to k:

[0109] Step a: Decompose the entropy model 1101 to obtain the cumulative distribution function cdf.

[0110] Step b: The first decoding module 1102 performs entropy decoding on the first coded data b1 according to the cumulative distribution function cdf to obtain the reconstructed super-prior feature

[0111] Step c: The super-prior scale decoding network 1103 reconstructs the super-prior features Perform super-prior scale decoding to obtain the variance N(0,σ) of a Gaussian distribution with a probability distribution of zero mean.

[0112] The network structure of the super prior scale decoding network 1103 in step c above can be as follows: Figure 7 To avoid repetition, detailed description is not given here.

[0113] Step d: The second decoding module 1104 performs entropy decoding on the second coded data b2 according to the variance N(0,σ) to obtain the reconstructed quantized residual feature.

[0114] Step e: the inverse gain unit 1105 reconstructs the quantized residual feature Perform inverse gain processing to obtain the reconstructed inverse gain residual feature

[0115] The inverse gain unit 1105 in step e above reconstructs the quantized residual feature The inverse gain processing of is similar to the inverse gain processing of the quantized residual feature r by the inverse gain unit 113, and will not be described in detail here to avoid redundancy.

[0116] Step f: The super-prior decoding network 1106 reconstructs the super-prior features Perform hyper-prior decoding to obtain reconstruction potential features

[0117] The structure of the super priori decoding network 1106 in step f above can be similar to Figure 1 The structure of the super-a priori decoding network 112 in the encoder is the same, and will not be described in detail here to avoid redundancy.

[0118] Step i: Prediction fusion network 1107 reconstructs potential features The context feature Φ output by the context network 1110 is used to obtain the reconstructed prediction feature

[0119] It should be noted that the context network 1110 performs prediction based on the completed reconstructed fused latent features to obtain the context feature Φ.

[0120] Step j: The fusion unit 1108 performs inverse gain residual feature and reconstructed predictive features Perform additive fusion to obtain reconstruction fusion potential features

[0121] Step g: Synthetic transformation network 1109 reconstructs the fused latent features Processing to obtain the reconstructed image

[0122] Reference Fig.12As shown, in some embodiments, the synthesis transformation network 1107 includes: a first residual block 121, a second residual block 122, a first upsampling layer 123, a first residual activation unit 124, a second upsampling layer 125, a second residual activation unit 126, a third upsampling layer 127, a residual non-local attention block 128, a third residual activation unit 129, and a fourth upsampling layer 1210, which are connected in series. The synthesis transformation network 1107 is used to reconstruct the fusion potential features through the first residual block 121, the second residual block 122, the first upsampling layer 123, the first residual activation unit 124, the second upsampling layer 125, the second residual activation unit 126, the third upsampling layer 127, the residual non-local attention block 128, the third residual activation unit 129, and the fourth upsampling layer 1210 in sequence. Processing to obtain the reconstructed image

[0123] Among them, the first residual block 801 and the second residual block 802 are placed at the starting position of the synthetic transformation network 1107, which enhance the reconstruction of the fusion potential features. Reduced quantization error.

[0124] It should be noted that each residual activation unit in the synthetic coding network 1107 is Figure 4 The structure and function of the residual activation unit shown in FIG. 1 are the same. The residual non-local attention block 128 in the synthetic coding network 1107 is the same as the above Figure 5 The structure and function of the residual non-local attention block 23 shown in FIG. 1 are the same. Each residual block in the synthetic coding network 1107 is the same as Figure 5 The structure and function of the residual blocks shown are the same.

[0125] Step k, context network 1110 reconstructs fusion potential features Context feature extraction is performed to obtain context feature Φ so as to reconstruct an element of the latent feature through the context feature Φ.

[0126] Fig.13 This is a schematic diagram of the structure of another image decoder provided in an embodiment of the present application. Fig.13 As shown in the above Fig.11 The content adaptive filtering network 130 is added to the image decoder shown in FIG. The content adaptive filtering network 130 is used to reconstruct the image The Y, U, and V components are enhanced to reconstruct the image output by the synthesis transformation network 1109. Enhance to obtain enhanced reconstructed image The content adaptive filtering network 130 includes a content adaptive Inter Channel Correlation Information filter sub network (ICCI) sub network for reconstructing and enhancing the three components Y, U, and V of the image, and the Y, U, and V components are enhanced respectively through the ICCI sub network.

[0127] Fig.14 A schematic diagram of a content adaptive filtering network provided in an embodiment of the present application. Fig.14 As shown, the content adaptive filtering network 130 includes three ICCI sub-networks, and the three ICCI sub-networks have different ICCI processing modules. The inputs of the ICCI sub-networks are the brightness component Y, the first chrominance component U, and the second chrominance component V of the image, respectively. The brightness component Y is represented by The first chrominance component U is expressed as The second chrominance component V is expressed as The first chroma component U is obtained after bicubic interpolation to obtain the U component of the same size as the luminance component Y The first chroma component V is obtained after bicubic interpolation to obtain the V component with the same spatial size as the luminance component Y Then, the Y component U Component and V component Perform forward wavelet transform and express the three components obtained by the transform as Fig.14 Components 1401, 1402 and 1403 in the channel dimension are spliced, and the spliced ​​component 1404 and the respective components are sent to the first ICCI processing module 1405, the second ICCI processing module 1406 and the third ICCI processing module 1407 respectively. The first ICCI processing module 1405, the second ICCI processing module 1406 and the third ICCI processing module 1407 process the Y, U and V components respectively to obtain processed components 1408, 1409 and 1410, and the processed components 1408, 1409 and 1410 are respectively subjected to inverse wavelet transform to obtain the reconstructed Y, U and V components. The reconstructed Y component is The U component after reconstruction is The reconstructed V component is Among them, the above-mentioned bicubic interpolation is a method of enlarging the image size and increasing the pixel density. Bicubic interpolation can be used to double the number of image pixels, and these expanded pixels are predicted based on existing pixels.

[0128] Fig.15 for Fig.14 Schematic diagram of the ICCI processing modules (first ICCI processing module 1405, second ICCI processing module 1406, third ICCI processing module 1407). Fig.15 As shown, the ICCI processing module includes: a 3×3 convolution layer 151, a batch normalization layer 152, a ReLU activation function layer 153, a residual block sequence 154 consisting of Nb 1D residual blocks, and a fusion module 155. The feature processing flow of the ICCI processing module includes: sequentially processing the spliced ​​component 1404 through the 3×3 convolution layer 151, the batch normalization layer 152, the ReLU activation function layer 153, and the residual block sequence 154, and scaling the output features of the residual block sequence 154 by γ times, and then adding and fusing the scaled features and the components after wavelet transformation (component 1401 or component 1402 or component 1403) through the fusion module 155 to obtain processed components (processed component 1408, processed component 1409, and processed component 1410). Among them, the number of residual blocks in the ICCI processing module 1405 for processing the Y component is Nb=8, the number of residual blocks in the ICCI processing module 1006 for processing the U component is Nb=13, and the number of residual blocks in the ICCI processing module 1407 for processing the V component is Nb=13.

[0129] against Fig.13 The image decoder shown in the figure can enhance the image through post-processing because the processing of the secondary components U and V is done through additional downsampling and a smaller number of channels. Specifically, the ICCI network ( Fig.14 As shown in the figure, the Y, U, and V components of the image are reconstructed and enhanced. The input of the ICCI network is the Y, U, and V components obtained by decoding the image encoding data through the image decoder under JPEGAI, and the output is the enhanced Y, U, and V components. First, the U and V components are respectively subjected to bicubic interpolation operations, and the U and V components are upsampled to the same spatial size as the Y component. Then Y, U, and V are spliced ​​after forward wavelet transform, and then input into the three ICCI processing modules with Y, U, and V respectively, and finally the reconstructed Y, U, and V components are generated through inverse wavelet transform. ICCI enhances the components by processing the Y, U, and V components. However, the shortcoming of this technology is that the three ICCI processing modules ( Fig.14 The first ICCI processing module 1405, the second ICCI processing module 1406 and the third ICCI processing module 1407 in the content adaptive filtering network have similar structures and are independent of each other. There is a certain redundancy, which increases the complexity of the content adaptive filtering network.

[0130] To solve the above problems, the embodiment of the present application proposes a simplified content adaptive filter, which concatenates the three components Y, U, and V and inputs them into an ICCI processing module for processing to learn the correlation characteristics of the three components. The concatenated components are then processed by different residual block sequences and convolutions, and finally the Y, U, and V components are added to generate enhanced Y, U, and V components. Fig.14 The content adaptive filter shown requires three ICCI processing modules. Since only one ICCI processing module is required in the simplified content adaptive filter, and a residual block sequence is added at the same time, the commonalities of the three components Y, U, and V are considered and the characteristics of the Y, U, and V components are enhanced. Therefore, the embodiment of the present application can reduce the number of parameters of the ICCI processing module and reduce the complexity of the decoder.

[0131] Fig.16 A schematic diagram of a content adaptive filtering network provided in an embodiment of the present application. Fig.16 As shown, in some embodiments, the content adaptive filtering network 130 includes: a first upsampling module 1601, a second upsampling module 1602, a first feature extraction module 1603, a second feature extraction module 1604, a third feature extraction module 1605, a splicing module 1606, a content adaptive inter-channel correlation information filter (Content adaptive Inter Channel Correlation Information filter, ICCI) 1607, a first processing module 1608, a second processing module 1609 and a third processing module 1610, a first fusion module 1611, a second fusion module 1612 and a third fusion module 1613, a first reconstruction module 1614, a second reconstruction module 1615 and a third reconstruction module 1616. The content adaptive filtering network 213 reconstructs the image The enhancement process includes: Step 1, respectively using the first upsampling module 1601 and the second upsampling module 1602 to represent the initial first chrominance component of the target image as and the initial second chrominance component is expressed as Upsample to obtain the same brightness component as the original The upsampled first chroma component with the same spatial size is expressed as And the upsampled second chroma component is expressed as Step 2: The initial brightness component is extracted by the first feature extraction module 1603, the second feature extraction module 1604, and the third feature extraction module 1605 respectively. The upsampled first chroma component is expressed as And the upsampled second chroma component is expressed as Perform feature extraction to obtain brightness features The first chromaticity characteristic The second chromaticity characteristic Step 3: Use the splicing module 1606 to combine the brightness features The first chromaticity characteristic The second chromaticity characteristic Splice to obtain the initial image feature F YUV Step 4: Use ICCI1607 to identify the initial image features F YUV Process and obtain enhanced features F enha Step 5: The first processing module 1608, the second processing module 1609 and the third processing module 1610 are used to enhance the feature F enha Processing to obtain brightness enhancement features First chroma enhancement feature and a second chroma enhancement feature Step 6: Fusion of brightness features through the first fusion module 1611 and brightness enhancement features Get enhanced brightness features Step 7: Fusion of the first chromaticity feature by the second fusion module 1612 and the first chroma enhancement feature Get enhanced first chromaticity feature Step 8: Fusion of the second chromaticity feature by the third fusion module 1613 and the second chroma enhancement feature Get enhanced second chromaticity feature Step 9: Reconstruct the brightness features of the image by using the first reconstruction module 1614, the second reconstruction module 1615 and the third reconstruction module 1616 respectively. Enhance the first brightness feature And enhance the second brightness feature Processing to obtain enhanced brightness component The enhanced first chrominance component is expressed as And the enhanced second chrominance component is expressed as Step 10: Combine and enhance the brightness component The enhanced first chrominance component is expressed as And the enhanced second chrominance component is expressed as Get enhanced image

[0132] Fig.17 for Fig.16 The schematic diagram of the structure of ICCI 1607 is shown in FIG. Fig.17As shown, in some embodiments, the ICCI includes: a convolution layer 171, a batch normalization layer 172, an activation function layer 173, and a residual block sequence 174 composed of multiple residual blocks. The feature processing flow of the content adaptive inter-channel correlation information filter includes: processing the initial image features through the convolution layer 171, the batch normalization layer 172, the activation function layer 173, and the residual block sequence 174 in sequence, and scaling the output features of the residual block sequence 2004 by γ times.

[0133] In some embodiments, the convolution layer 171 is a convolution layer with a convolution kernel size of 3×3.

[0134] In some embodiments, the activation function of the activation function layer 173 is a ReLU activation function.

[0135] In some embodiments, the residual block sequence 174 consists of 8 1D residual blocks.

[0136] Fig.18 Schematic diagram of a content adaptive filtering network and a residual block sequence 174 and a residual block in ICCI 1607. Fig.18 As shown, in some embodiments, the residual block includes: a first convolution layer 181, a first batch normalization layer 182, a ReLU activation function layer 183, a second convolution layer 184, a second batch normalization layer 185, and a fusion unit 186. The residual block is used to process the input feature I4 of the residual block through the first convolution layer 181, the first batch normalization layer 182, the ReLU activation function layer 183, the second convolution layer 184, and the second batch normalization layer 185 in sequence, and perform *γ on the output feature of the second batch normalization layer 185 n The scaled feature is scaled by times, and the scaled feature and the input feature I4 of the residual block are added and fused by the fusion unit 186 to obtain the output feature O4 of the residual block.

[0137] Reference Fig.19 As shown, the encoder provided by other embodiments of the present application includes: an analysis transformation network 1901, a first quantization module 1902, a first feature segmentation module 1903, a super priori coding network 1904, a second quantization module 1905, a first encoding module 1906, a decoding module 1907, a super priori decoding network 1908, a second feature segmentation module 1909, an enhancement layer context network 1910, a second encoding module 1911, a base layer context network 1912, and a third encoding module 1913.

[0138] Fig.19 The process of image encoding by the encoder shown includes the following steps 1 to 14:

[0139] Step 1: The analysis transformation network 1901 extracts potential features of the target image X to obtain the potential features of the target image X.

[0140] Among them, the latent features Y are high-dimensional representations of the target image X. The latent features Y cannot be directly observed from the input data (target image), but they can be determined by the information learned from the input data. The above-mentioned latent features Y are obtained based on the information learned from the target image X.

[0141] In some embodiments, the number of channels of features of potential feature Y is C=192.

[0142] Reference Fig. 20 As shown, in some embodiments, the analysis transformation network 1901 includes: a plurality of downsampling convolutional layers 21 and a plurality of residual activation units 22 serially connected between adjacent downsampling convolutional layers 21, Fig. 20 In the example, the analysis transformation network includes four downsampling convolutional layers 21 and three residual activation units 22 connected in series between adjacent downsampling convolutional layers 21. The residual activation unit 22 includes: a convolutional layer 221, an activation function layer 222, a second convolutional layer 223 and a fusion layer 224. The convolutional layer 221, the activation function layer 222 and the second convolutional layer 223 process the input feature I5 ​​of the residual activation unit 222 in sequence, and the output feature of the second convolutional layer and the input feature I5 ​​of the residual activation unit are added and fused through the fusion layer 224 to obtain the output feature O5 of the residual activation unit 222.

[0143] In some embodiments, each downsampling convolution layer 21 in the analysis transformation network is a downsampling convolution layer with a convolution kernel size of 3×3, the first convolution layer and the second convolution layer in each residual activation unit of the analysis transformation network are convolution layers with a convolution kernel size of 3×3, and the activation area function of the activation function layer in each residual activation unit of the analysis transformation network is the LeakyReLU activation function (a variant of the ReLU function, which introduces a small slope in the negative value interval, solving the problem that the ReLU function outputs zero in the negative value interval).

[0144] The target image is represented as X, the potential features of the target image are represented as Y, and the processing of the features by the analysis transformation network is represented as g a (), then:

[0145] Y=g a (X)

[0146] Step 2: The first quantization module 1902 performs quantization processing on the latent feature Y to obtain a quantized latent feature y.

[0147] The quantization process refers to the process of approximating the continuous value of a signal (or a large number of possible discrete values) to a finite number of discrete values ​​(or a small number of discrete values).

[0148] Step 3: The first feature segmentation module 103 segments the quantized latent feature y in the dimension of the channel to obtain the enhanced layer latent feature and the underlying latent features

[0149] In some embodiments, the enhancement layer quantizes the latent feature y enhance The number of channels C1=64, the base layer potential feature y base The number of channels C1=128.

[0150] In some embodiments, the first feature segmentation module 1903 is specifically configured to segment the first 64 channels of the quantized latent feature y into enhanced layer quantized latent features y enhance , and split the last 128 channels of the quantized latent feature y into the base layer quantized latent feature y base That is, y enhabce ={Y1, Y2, ..., Y 64},y base = {Y 65 , Y 66 , …, Y 192}.

[0151] Step 4: The super-prior coding network 1904 performs super-prior coding on the latent feature Y to obtain the super-prior feature Z.

[0152] Reference Fig.21 As shown, in some embodiments, the super a priori coding network 1904 includes: a first convolution layer 211, a second convolution layer 212, a first downsampling layer 213, a third convolution layer 214, and a second downsampling layer 215 connected in series. The processing of the potential feature Y by the super a priori coding network includes: processing the potential feature Y through the first convolution layer 211, the second convolution layer 212, the first downsampling layer 213, the third convolution layer 214, and the second downsampling layer 215 in sequence to obtain the super a priori feature Z.

[0153] In some embodiments, the first convolution layer 211, the second convolution layer 212, the first down-sampling layer 213, the third convolution layer 214 and the second down-sampling layer 215 are all convolution layers with a convolution kernel size of 3×3.

[0154] Step 5: The second quantization module 1905 performs quantization processing on the super-prior feature Z to obtain a quantized super-prior feature z.

[0155] Step 6: The first encoding module 1906 performs entropy encoding on the quantized super-prior feature z to obtain first encoded data b1.

[0156] In some embodiments, the first encoding module 1906 may be a lossless entropy encoding module, and performing entropy encoding on the quantized super a priori feature z includes: performing lossless entropy encoding on the quantized super a priori feature z.

[0157] Step 7: The decoding module 1907 performs entropy decoding on the first coded data b1 to obtain the reconstructed quantized super-prior feature

[0158] In some embodiments, the decoding module 1907 may be a lossless entropy decoding module, and performing entropy decoding on the first encoded data b1 includes: performing lossless entropy decoding on the first encoded data b1.

[0159] It should be noted that when the first encoding module 106 is a lossless encoding module and the decoding module 107 is a lossless decoding module, since there is no information loss in the encoding and decoding process, the reconstructed quantized super-prior feature It is exactly the same as the quantized super-prior feature z, so the encoder can omit the decoding module 1907 and the above step 7, and directly use the quantized super-prior feature z to replace the reconstructed quantized super-prior feature in the subsequent steps

[0160] Step 8: The super-prior decoding network 108 reconstructs the quantized super-prior features Perform hyper-prior decoding to obtain reconstruction potential features

[0161] In some embodiments, the number of channels for reconstructing latent features is C3=384.

[0162] Reference Fig. 22 As shown, in some embodiments, the super-prior decoding network 108 includes: a convolution layer 221, a first upsampling layer 222, a second upsampling layer 223, a third upsampling layer 224 and a fourth upsampling layer 225. The super-prior decoding network 108 reconstructs the quantized super-prior features. The processing includes: sequentially reconstructing the quantized super-prior features through the convolution layer 221, the first upsampling layer 222, the second upsampling layer 223, the third upsampling layer 224 and the fourth upsampling layer 225 Processing to obtain reconstruction potential features

[0163] In some embodiments, the convolution layer 221 , the first upsampling layer 222 , the second upsampling layer 223 , the third upsampling layer 224 , and the fourth upsampling layer 225 are all convolution layers with a convolution kernel size of 3×3.

[0164] Step 9: The second feature segmentation module 1909 reconstructs the potential features in the dimension of the channel Segmentation is performed to obtain the potential features for reconstruction of the enhanced layer and the base layer to reconstruct the latent features

[0165] In some embodiments, the second feature segmentation module 1909 may segment the reconstructed latent feature y in the same proportion as the first feature segmentation module 1903 segments the quantized latent feature y. That is, the enhancement layer reconstructs the potential features The number of channels C4 = 128, the base layer reconstructs the potential features The number of channels C5=256.

[0166] In some embodiments, the second feature segmentation module 1909 is specifically used to reconstruct the potential features The first 128 channels of the segmentation are used to reconstruct the latent features of the enhancement layer and will reconstruct the latent features The last 256 channels are segmented into the base layer to reconstruct the latent features Right now,

[0167] Step 10: The enhancement layer context network 1910 quantizes the potential feature y according to the already encoded enhancement layer ehhance and the enhancement layer to reconstruct the latent features Get the enhancement layer context feature Φ enhance .

[0168] Among them, the enhanced layer context feature Φ enhance That is, the enhanced layer quantized potential feature y ehhance The probability distribution of .

[0169] In some embodiments, a context network based on autoregressive convolution can be used to enhance the layer context feature Φ enhance Extraction. The schematic diagram of the mask convolution kernel used by the spatial context network based on autoregressive convolution can be shown as follows Figure 8 As shown, no detailed description is given here.

[0170] Step 11: The second encoding module 1911 is based on the enhancement layer context feature Φ enhance Quantize the latent features y for the enhancement layer ehhance The next element of is entropy encoded.

[0171] Repeat steps 10 and 11 above to complete the enhancement layer quantization potential feature y enhance The second encoded data b2 can be obtained by encoding all elements of .

[0172] Step 12: The base layer context network 1912 quantizes the potential feature y according to the encoded base layer base and the base layer to reconstruct the latent features Get the base layer context feature Φ base .

[0173] Similarly, the context network based on autoregressive convolution can be used to perform the base layer context feature Φ base Extraction.

[0174] Step 13: The third encoding module 1913 is based on the base layer context feature Φ base Quantize the latent features y of the base layer base The next element of is entropy encoded.

[0175] Repeat steps 12 and 13 above to complete the base layer quantization potential feature y base The third encoded data b3 can be obtained by encoding all the elements of .

[0176] Step 14: Generate coded data of the target image X according to the first coded data b1, the second coded data b2 and the third coded data b3.

[0177] That is, the coded data of the target image X includes first coded data b1, second coded data b2, and third coded data b3.

[0178] Reference Fig.23 As shown, in some embodiments, an image decoder for executing the image decoding method provided by an embodiment of the present application includes: a first decoding module 2301, a super priori decoding network 2302, a feature segmentation module 2303, an enhancement layer context network 2304, a second decoding module 2305, a base layer context network 2306, a third decoding module 2307, a splicing module 2308, a synthesis transformation network 2309, a content adaptive filtering network 2310, a latent space transformation network 2311 and a machine vision task model 2312.

[0179] Fig.23 The process of image decoding by the image decoder shown includes the following steps a to j:

[0180] Step a: Obtain the encoded data of the target image.

[0181] The encoding data of the target image includes first encoding data b1, second encoding data b2 and third encoding data b3.

[0182] Step b: The first decoding module 2301 performs entropy decoding on the first coded data b1 to obtain the reconstructed quantized super-prior feature

[0183] In some embodiments, the first decoding module 2301 may be a lossless entropy decoding module, and performing entropy decoding on the first coded data b1 includes: performing lossless entropy decoding on the first coded data b1.

[0184] Step c: The super-prior decoding network 2302 reconstructs the quantized super-prior features Perform super-prior decoding processing to obtain reconstruction potential features

[0185] In some embodiments, the number of channels for reconstructing latent features is C3=384.

[0186] The network structure of the super priori decoding network 2302 can be Fig. 22 The network structure of the super-prior decoding network 1908 shown is the same, and will not be described in detail here to avoid redundancy.

[0187] Step d: The feature segmentation module 2303 reconstructs the potential features in the dimension of the channel. Segmentation is performed to obtain the potential features for reconstruction of the enhanced layer and the base layer to reconstruct the latent features

[0188] In some embodiments, the feature segmentation module 2303 can reconstruct the potential features in a ratio of 1:2. That is, the enhancement layer reconstructs the potential features The number of channels C4 = 128, the base layer reconstructs the potential features The number of channels C5=256.

[0189] Step e: The enhancement layer context network 2304 reconstructs the quantized potential features based on the completed enhancement layer reconstruction. (the decoding result of the second coded data b2 after decoding) and the enhancement layer reconstruction potential feature Get the enhancement layer context feature Φ enhance .

[0190] In some embodiments, a context network based on autoregressive convolution can be used to perform context feature Φ enhance Extraction.

[0191] Step f: The second decoding module 2305 reconstructs the quantized latent features according to the enhanced layer The context feature Φ enhance The second encoded data is entropy decoded.

[0192] Repeat the above steps e and f to complete the entropy decoding of all elements of the second coded data b2 to obtain the enhanced layer reconstruction quantization potential features.

[0193] when Fig.19 The first encoding module 1906 and the second encoding module 1911 in the encoder are lossless encoders. Fig.23 The first decoding module 2301 and the second decoding module 2305 in the image decoder are lossless decoding modules, and the enhancement layer quantization potential feature y enhance There is no information loss in the encoding and decoding process, and the enhancement layer quantizes the latent features y enhance Reconstruct quantized latent features with enhancement layer same.

[0194] Step g: The base layer context network 2306 reconstructs the quantized potential features based on the reconstructed base layer. (Decoding result of the decoded third coded data b3) and base layer reconstruction potential features Get the base layer context feature Φ base .

[0195] Similarly, we can use the context network based on autoregressive convolution to perform context feature Φ base Extraction.

[0196] Step h: The third decoding module 2307 decodes the base layer context feature Φ base The next element of the third coded data b3 is entropy decoded.

[0197] Repeat the above steps g and h to complete the entropy decoding of all elements of the third coded data b3 to obtain the base layer reconstruction quantization potential features.

[0198] when Fig.19 The first encoding module 1906 and the third encoding module 1913 in the encoder are lossless encoding modules. Fig.23 The first decoding module 2301 and the third decoding module 2307 in the image decoder are lossless entropy decoding modules, and the base layer quantized potential feature y base There is no information loss in the encoding and decoding process, and the base layer quantizes the potential features y base Reconstructing quantized latent features with base layer same.

[0199] Step j: The splicing module 2308 reconstructs the quantized latent features of the enhancement layer from the channel dimension. and base layer reconstruction quantized latent features Splicing to obtain reconstruction quantification potential features

[0200] when Fig.19The first encoding module 1906, the second encoding module 1911, and the third encoding module 1913 in the encoder are lossless encoding modules. Fig.23 The first decoding module 2301, the second decoding module 2305, and the third decoding module 2307 in the image decoder are lossless entropy decoding modules, and the enhancement layer quantization potential feature y enhance and the base layer quantized latent features y base There is no information loss during the encoding and decoding process, and the quantized latent features are reconstructed Same as quantizing the latent feature y.

[0201] Step k, synthesis transformation network 2309 reconstructs quantized latent features Processing is performed to obtain a reconstructed image of the target image

[0202] In some embodiments, reference Fig.24 As shown, the synthetic transformation network 2309 includes: a plurality of upsampling convolutional layers 241 and a plurality of residual activation units 242 serially connected between adjacent upsampling convolutional layers, Fig.24 In the example, a synthetic transformation network includes four upsampling convolutional layers 241 and three residual activation units 242 connected in series between adjacent upsampling convolutional layers 241. The residual activation unit 242 includes: a first convolutional layer 2421, an activation function layer 2422, a second convolutional layer 2423 and a fusion layer 2424. The residual activation unit 242 is used to process the input feature I6 of the residual activation unit 242 through the first convolutional layer 2421, the activation function layer 2422 and the second convolutional layer 2423 in sequence, and add and fuse the output feature of the second convolutional layer 2423 and the input feature I6 of the residual activation unit through the fusion layer 2424 to obtain the output feature O6 of the residual activation unit 242.

[0203] In some embodiments, each upsampling convolution layer 241 in the synthetic transformation network is an upsampling convolution layer with a convolution kernel size of 3×3, the first convolution layer 2421 and the second convolution layer 2423 in each residual activation unit of the synthetic transformation network are convolution layers with a convolution kernel size of 3×3, and the activation area function of the activation function layer 2422 in each residual activation unit 242 of the synthetic transformation network is a LeakyReLU activation function.

[0204] Step 1: Content adaptive filtering network 2310 reconstructs the image Perform enhancement processing to obtain an enhanced image

[0205] The network structure of the content adaptive filtering network 2310 can refer to Fig.16 To avoid repetition, detailed description is not given here.

[0206] Step i: Latent space transformation network 2311 reconstructs the quantized latent features of the base layer Processing to obtain machine vision features

[0207] Reference Fig.25 As shown, in some embodiments, the latent space transformation network 2311 includes: a first residual block 251, a first up-sampled residual block 252, a second residual block 253, a second up-sampled residual block 254, a third residual block 255, a third up-sampled residual block 256, a fourth residual block 257, an up-sampled convolution layer 258, and an activation function layer 259. The up-sampling multiples of the first up-sampled residual block 252, the second up-sampled residual block 254, the third up-sampled residual block 256, and the up-sampled convolution layer 258 are r1, r2, r4, and r4, respectively.

[0208] Reference Fig.25 As shown, in some embodiments, the residual blocks (first residual block 251, second residual block 253, third residual block 255, fourth residual block 257) in the latent space transformation network 2311 include: a first convolution layer 2501, a first LeakyLeLU activation function layer 2502, a second convolution layer 2503, a second LeakyLeLU activation function layer 2504, and a fusion module 2505. The feature processing step of the residual block includes: firstly, the input feature I7 of the residual block is processed by the first convolution layer 2501, the first LeakyLeLU activation function layer 2502, the second convolution layer 2503, and the second LeakyLeLU activation function layer 2504 in sequence, and then the output feature of the second LeakyLeLU activation function layer 2504 and the input feature I7 of the residual block are added and fused by the fusion module 2505 to obtain the output feature O7 of the residual block.

[0209] Reference Fig.25As shown, in some embodiments, the upsampling residual blocks (first upsampling residual block 252, second upsampling residual block 254, third upsampling residual block 256) in the latent space transformation network 2311 include: an upsampling convolution layer 2506, a LeakyLeLU activation function layer 2507, a third convolution layer 2508, an inverse generalized divisive normalization (IGDN) layer 2509, a fourth convolution layer 2510 and a fusion module 2511. The feature processing steps of the upsampled residual block include: first, the upsampled convolution layer 2506, the LeakyLeLU activation function layer 2507, the third convolution layer 2508, and the IGDN layer 2509 process the input feature I8 of the upsampled residual block, and the fourth convolution layer 2510 processes the input feature I8 of the upsampled residual block, and then the output of the IGDN layer 2509 and the output of the fourth convolution layer 2510 are added and fused through the fusion module 2511 to obtain the output feature O8 of the upsampled residual block.

[0210] Step j: Machine vision task model 2312 is based on machine vision features Get the machine vision task processing result M.

[0211] The embodiments of the present application relate to the field of image coding and decoding technology. The following first describes an image coding and decoding framework for executing the image coding method and image decoding method provided by the embodiments of the present application.

[0212] Reference Fig.26 As shown, the image encoder for executing the image encoding method provided by some embodiments of the present application includes: an analysis transformation network 2601, a first quantization module 2602, a mask generation network 2603, a first channel selection module 2604, a base layer super-prior encoding network 2605, a second quantization module 2606, a first encoding module 2607, an enhancement layer super-prior encoding network 2608, a third quantization module 2609, a second encoding module 2610, a second channel selection module 2611, a first decoding module 2612, a base layer super-prior decoding network 2613, a base layer context network 2614, a third encoding module 2615, a second decoding module 2616, an enhancement layer super-prior decoding network 2617, an enhancement layer context network 2618, and a fourth encoding module 2619.

[0213] Fig.26The process of image encoding performed by the image encoder shown includes the following steps: Step 1, the analysis transformation network 2601 processes the target image (video frame) X to obtain the potential feature Y of the target image X. Step 2, the first quantization module 2602 quantizes the potential feature Y to obtain the quantized potential feature y. Step 3, the quantized potential feature y is input into the mask generation network 2603, and the channel attention mask Mask of the potential feature Y output by the mask generation network 2603 is obtained. Step 4, the first channel selection module 2604 performs channel selection on the potential feature U based on the channel attention mask Mask to divide the potential feature Y into the base layer potential feature Y base and the enhancement layer latent feature Y enhance Step 5: The base layer super prior coding network 2605 encodes the base layer potential feature Y base Processing is performed to obtain the base layer super prior feature Z base Step 6: The second quantization module 2606 quantizes the base layer super-prior feature Z base Perform quantization processing to obtain the basic layer quantization super prior feature z base Step 7: The first encoding module 2607 quantizes the super-prior feature z of the base layer base Perform entropy coding to obtain the first coded data b1. Step 8: The enhancement layer super-prior coding network 2608 encodes the enhancement layer potential feature Y enhance Processing is performed to obtain the enhanced layer super prior feature Z enhance Step 9: The third quantization module 2609 quantizes the enhancement layer super-prior feature Z enhance Perform quantization processing to obtain the enhanced layer quantization super prior feature z enhance Step 10: The second encoding module 2610 quantizes the super-prior feature z of the enhancement layer enhance Step 11: The second channel selection module 2611 performs channel selection on the quantized potential feature y based on the channel attention mask Mask to divide the quantized potential feature y into the base layer quantized potential feature y base and the enhancement layer quantized latent features y enhance Step 12: The first decoding module 2612 performs entropy decoding on the first coded data b1 to obtain the super prior features of the base layer reconstruction. Step 13: The base layer super-prior decoding network 2613 reconstructs super-prior features of the base layer Processing is performed to obtain the underlying features for reconstruction Step 14: Base layer context network 2614 reconstructs potential features based on the base layer and the encoded base layer quantized latent features y base Get the base layer context feature Φ baseStep 15: The third encoding module 2615 generates a base layer context feature Φ base Quantize the latent features y of the base layer base The current element is entropy encoded, and finally the base layer quantization potential feature y is completed. base The entropy encoding of all elements is performed to obtain the third coded data b3. Step 16: The second decoding module 2616 performs entropy decoding on the second coded data b2 to obtain the enhanced layer reconstruction super prior feature Step 17: The enhancement layer super-prior decoding network 2617 reconstructs the super-prior features of the enhancement layer Processing is performed to obtain the potential features of the enhanced layer reconstruction Step 18: Enhanced layer context network 2618 reconstructs potential features based on the enhanced layer and the quantized latent feature y of the enhanced layer that has been coded enhance Get the enhancement layer context feature Φ enhance Step 19: The fourth encoding module 2619 generates the enhanced layer context feature Φ enhance Quantize the latent features y for the enhancement layer enhance The current element is entropy coded, and the enhancement layer quantization potential feature y is finally completed. enhance The entropy coding of all elements of is performed to obtain the fourth coded data b4. Step 20: The first coded data b1, the second coded data b2, the third coded data b3 and the fourth coded data b4 are used as the coded data of the target image X and sent to the decoding end.

[0214] It should be noted that when the first encoding module 2607 is a lossless encoding module and the first decoding module 2612 is a lossless entropy decoding module, due to the base layer quantization super prior feature z base There is no information loss in the encoding and decoding process, and the base layer reconstructs the super prior features and the base layer quantized hyper-prior feature z base are exactly the same, so Fig.26 The image encoder shown in the figure can omit the decoding module 2612 and the above step 12, and directly use the base layer quantized super prior feature z base Replace the base layer in the subsequent steps to reconstruct the hyper-prior features Similarly, when the second encoding module 2610 is a lossless encoding module and the second decoding module 2616 is a lossless entropy decoding module, the decoding module 2615 and the above step 16 can be omitted, and the enhancement layer quantization super prior feature z is directly used. enhance Replace the base layer in the subsequent steps to reconstruct the hyper-prior features

[0215] Reference Fig. 27As shown, in some embodiments, the mask generation network 2603 includes: an adaptive pooling layer 271, a first convolution layer 272, a batch normalization layer 273, a nonlinear activation function layer 274, a second convolution layer 275, and a sampling module 276. The process of the channel attention network 2603 processing the quantized potential feature y to obtain the channel attention mask Mask includes: first, the adaptive pooling layer 271 processes the quantized potential feature y to obtain the first feature F1; secondly, the first convolution layer 272, the normalization layer 273, and the nonlinear activation function layer 274 process the first feature F1 in sequence to obtain the second feature F2; and again, the second convolution layer 275 processes the second feature F2 to obtain the third feature F3 and the fourth feature F4. Finally, the sampling module 276 performs Gumbel sampling on the third feature F3 and the fourth feature F4 to obtain the channel attention mask Mask of the potential feature.

[0216] Fig.26 The network structures of the analysis transformation network, super prior encoding network, super prior decoding network, etc. in the image encoder shown in FIG. Fig.19 The network structures of the analysis transformation network, super a priori encoding network, super a priori decoding network and other networks in the image encoder shown are the same. To avoid redundancy, they are not described in detail here.

[0217] Reference Fig.28 As shown, an image decoder for executing the image decoding method provided by some embodiments of the present application includes: a first decoding module 2801, a base layer super priori decoding network 2802, a base layer context network 2803, a second decoding module 2804, a third decoding module 2805, an enhancement layer super priori decoding network 2806, an enhancement layer context network 2807, a fourth decoding module 2808, a splicing module 2809, a synthesis transformation network 2810, a content adaptive filtering network 2811, a latent space transformation network 2812 and a machine vision task model 2813.

[0218] Fig.28 The process of image decoding by the image decoder shown in the figure includes the following steps: Step 1, the first decoding module 2801 performs entropy decoding on the first coded data b1 to obtain the super prior feature of the base layer reconstruction Step 2: The base layer super-prior decoding network 2802 reconstructs the super-prior features of the base layer Processing is performed to obtain the underlying features for reconstruction Step 3: Base layer context network 2803 reconstructs potential features based on the base layer Reconstruct the quantified latent features from the completed base layer (Decoding result of the decoded third coded data b3), obtain the base layer context feature Φ baseStep 4: The third decoding module 2804 decodes the base layer context feature Φ base The current element of the third coded data b3 is entropy decoded, and finally the entropy decoding of all elements of the third coded data b3 is completed to obtain the base layer reconstruction quantization potential feature Step 5: The third decoding module 2805 performs entropy decoding on the second coded data b2 to obtain the enhanced layer reconstruction super prior feature Step 6: The enhancement layer super-prior decoding network 2806 reconstructs the super-prior features of the enhancement layer Processing is performed to obtain the potential features of the enhanced layer reconstruction Step 7: Enhanced layer context network 2807 reconstructs potential features based on the enhanced layer and the enhanced layer reconstructed quantitative latent features (Decoding result of the fourth coded data b4 that has been decoded), obtain the enhancement layer context feature Φ enhaace Step 8: The fourth decoding module 2808 decodes the enhanced layer context feature Φ eaaaaae The current element of the fourth coded data b4 is entropy decoded, and finally the entropy decoding of all elements of the fourth coded data b4 is completed to obtain the enhanced layer reconstruction quantization potential feature Step 9: The splicing module 2809 splices the base layer to reconstruct the quantized potential features and enhancement layer reconstruction quantization latent features Obtaining reconstruction quantization potential features Step 10: Reconstruct quantized latent features through synthetic transformation network 2810 Processing is performed to obtain a reconstructed image of the target image Step 11: Content adaptive filtering network 2811 reconstructs the image Processing to obtain enhanced image Step 12: Reconstruct the quantized latent features of the base layer through the latent transformation network 2812 Processing to obtain machine vision features Step 13: Machine vision features Input the machine vision task model 2813 to obtain the machine vision task processing result M output by the machine vision task model 2813.

[0219] Fig.28 The network structures of the super-prior decoding network, latent space variation network, synthetic transformation network, etc. in the decoder shown can be compared with Fig.23 The network structures of the super-prior decoding network, latent space variation network, synthetic transformation network, etc. in the decoder shown are the same. To avoid redundancy, they will not be described in detail here. Fig.28The network structure of the content adaptive filtering network 2310 in the decoder shown can be referred to Fig.16 To avoid repetition, detailed description is not given here.

[0220] Fig.29 A flowchart of a method for decoding an image provided by an embodiment of the present application. Fig.29 As shown, the image decoding method comprises the following steps:

[0221] S291. Decode the encoded data of the target image to obtain a reconstructed image of the target image.

[0222] The image data of the reconstructed image includes: an initial brightness component (Y component), an initial first chrominance component (U component) and an initial second chrominance component (V component).

[0223] In some embodiments, the Fig.13 or Fig.23 Or the image encoder shown in 28 decodes the encoded data of the target image to obtain a reconstructed image of the target image.

[0224] It should be noted that the initial reconstructed data of the target image may also be the reconstructed data obtained by decoding the encoded data of the target image through a decoder in other codec architectures. The embodiment of the present application is not limited to this, and the reconstructed image of the target image can be obtained by decoding the encoded data of the target image.

[0225] S292: Generate an upsampled first chroma component and an upsampled second chroma component according to the initial first chroma component and the initial second chroma component, respectively.

[0226] The spatial size of the upsampled first chroma component and the upsampled second chroma component are both the same as the spatial size of the initial luminance component.

[0227] That is, the initial first chroma component and the initial second chroma component are respectively up-sampled to chroma components having the same spatial size as the initial luminance component.

[0228] In some embodiments, bicubic interpolation may be performed on the initial first chroma component and the initial second chroma component, respectively, to obtain the upsampled first chroma component and the upsampled second chroma component, respectively.

[0229] S293, performing feature extraction on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain luminance features, first chrominance features, and second chrominance features.

[0230] In some embodiments, the above step S293 (respectively extracting features from the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component to obtain luminance features, first chrominance features, and second chrominance features) includes:

[0231] Forward wavelet transform (WT) is performed on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component, respectively, to obtain the luminance feature, the first chrominance feature, and the second chrominance feature.

[0232] S294, concatenating the brightness feature, the first chromaticity feature, and the second chromaticity feature to obtain an initial image feature.

[0233] In some embodiments, the above step S294 (joining the brightness feature, the first chromaticity feature and the second chromaticity feature to obtain an initial image feature) includes:

[0234] The brightness feature, the first chromaticity feature, and the second chromaticity feature are concatenated from a channel dimension to obtain an initial image feature of the target image.

[0235] S295 , extracting features from the initial image features through a content adaptive InterChannel Correlation Information filter (ICCI) to obtain enhanced features.

[0236] In some embodiments, the content adaptive channel-to-channel correlation information filter may be used with Fig.17 The structure of ICCI1607 shown is the same and will not be described in detail here to avoid redundancy.

[0237] S296. Perform feature enhancement processing on the brightness feature, the first chromaticity feature, and the second chromaticity feature respectively based on the enhanced feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature.

[0238] In some embodiments, the above step S296 (based on the enhancement feature, the brightness feature, the first chromaticity feature and the second chromaticity feature are respectively subjected to feature enhancement processing to obtain an enhanced brightness feature, an enhanced first chromaticity feature and an enhanced second chromaticity feature) includes: based on Fig.16The first processing module 1606, the second processing module 1607 and the third processing module 1608 shown process the enhanced features, and respectively add and fuse the output features of the first processing module 1606, the second processing module 1607 and the third processing module 1608 with the brightness feature, the first chromaticity feature and the second chromaticity feature to obtain enhanced brightness features, enhanced first chromaticity features and enhanced second chromaticity features.

[0239] S297. Obtain an enhanced image of the reconstructed image according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature.

[0240] That is, the enhanced brightness feature, the enhanced first chromaticity feature and the enhanced second chromaticity feature are arranged according to the data format of the YUV color space to obtain enhanced reconstructed data of the target image.

[0241] Compared to Fig.13 The content adaptive filter shown requires three ICCI processing modules. Since the above embodiment only requires one content adaptive inter-channel correlation information filter to enhance the target image, the above embodiment can reduce the complexity of the image decoder.

[0242] As an extension and refinement of the above embodiment, the present application embodiment provides another image decoding method, referring to Fig.30 As shown, the image decoding method comprises the following steps:

[0243] S3001. Obtain the encoded data of the target image.

[0244] In some embodiments, the encoded data of the target image is based on Figure 1 , Fig.19 , Fig.23 The image encoder encodes the target image to obtain the encoded data of the target image. The encoded data of the target image includes: inputting the original image data of the target image into Figure 1 When the image encoder is shown, Figure 1 The first coded data b1 and the second coded data b2 output by the image encoder are shown, or the original image data of the target image is input Fig.19 When the image encoder is shown, Fig.19 The first coded data b1, the second coded data b2 and the third coded data b3 output by the image encoder, or the original image data input of the target image Fig.23 When the image encoder is shown, Fig.23 The image encoder outputs first coded data b1, second coded data b2, third coded data b3 and third coded data b4.

[0245] S3002. Decode the encoded data of the target image to obtain a reconstructed image of the target image.

[0246] In some embodiments, the encoded data of the target image includes Figure 1 The first coded data b1 and the second coded data b2 output by the image encoder are as follows: Fig.13 The image decoder decodes the first coded data b1 and the second coded data b2 to obtain a reconstructed image.

[0247] In some embodiments, the encoded data of the target image includes Figure 1 The first coded data b1, the second coded data b2 and the third coded data b3 output by the image encoder are as follows: Fig.23 The image decoder decodes the first coded data b1, the second coded data b2, and the third coded data b3 to obtain a reconstructed image.

[0248] In some embodiments, the encoded data of the target image includes Figure 1 The first coded data b1, the second coded data b2, the third coded data b3 and the fourth coded data b4 output by the image encoder are as follows: Fig.28 The image decoder decodes the first coded data b1, the second coded data b2, the third coded data b3 and the fourth coded data b4 to obtain a reconstructed image.

[0249] S3003 . Perform bicubic interpolation on the initial first chroma component and the initial second chroma component respectively to obtain the upsampled first chroma component and the upsampled second chroma component respectively.

[0250] The spatial size of the upsampled first chroma component and the upsampled second chroma component are both the same as the spatial size of the initial luminance component.

[0251] The initial first chrominance component is expressed as The initial second chrominance component is expressed as The upsampled first chrominance component is expressed as The upsampled second chrominance component is expressed as The bicubic interpolation operation is represented by BI(), and the above step S3003 can be represented by the following formula:

[0252]

[0253]

[0254] S3004, performing forward wavelet transform on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain the luminance feature, the first chrominance feature, and the second chrominance feature.

[0255] The initial brightness component is expressed as The upsampled first chrominance component is expressed as The upsampled second chrominance component is expressed as The forward wavelet transform is expressed as DWT(), and the brightness feature is expressed as The first chromaticity feature is expressed as The second chromaticity feature is expressed as Then the above S3003 can be expressed as the following formula:

[0256]

[0257]

[0258]

[0259] S3005. Concatenate the brightness feature, the first chromaticity feature, and the second chromaticity feature from a channel dimension to obtain an initial image feature of the target image.

[0260] The brightness feature is expressed as The first chromaticity feature is expressed as The second chromaticity feature is expressed as The concatenation operation in the channel dimension is represented as Concatenatio(), and the initial image features of the target image are represented as F YUV , then the above step S3005 can be expressed as the following formula:

[0261]

[0262] S3006: Perform convolution processing on the initial image feature to obtain a first feature.

[0263] In some embodiments, the initial image feature may be convolved through a convolution layer with a convolution kernel size of 3×3 to obtain the first feature.

[0264] The initial image features of the target image are represented as F YUV , the convolution process is represented as Conv a×3 (), the first feature is represented by F a , then the above step S3006 can be expressed as the following formula:

[0265] F1=Conva×3 (F YUV )

[0266] S3007. Perform batch normalization on the first feature to obtain a second feature.

[0267] In some embodiments, the first features may be batch normalized by a batch normalization layer to obtain the second features.

[0268] The first feature is represented as F1, the batch normalization process is represented as BN(), and the second feature is represented as F2. Then the above step 2052 can be represented as the following formula:

[0269] F2=BN(F1)

[0270] S3008. Perform linear activation processing on the second feature to obtain a third feature.

[0271] In some embodiments, performing linear activation processing on the second feature to obtain the third feature includes: performing linear activation processing on the second feature through a ReLU activation function layer to obtain the third feature.

[0272] The second feature is represented as F2, the linear activation process is represented as ReLU(), and the third feature is represented as F3, then the above step S3008 can be expressed as the following formula:

[0273] F3 = ReLU(F2)

[0274] S3009. Process the third feature through a residual network to obtain the enhanced feature.

[0275] The residual network includes a plurality of residual blocks connected in series, and the residual block is used to sequentially perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.

[0276] In some embodiments, the residual network includes 8 serially connected residual blocks. The structure of the residual block can refer to Fig.18 The structure of the residual block shown is not described in detail here to avoid redundancy.

[0277] The third feature is represented as F3, the processing of the residual network is represented as ReseN(), and the enhanced feature is represented as F enha , then the above step S3009 can be expressed as the following formula:

[0278] Fenha =ReseN(F3)

[0279] S3010: Perform convolution processing on the enhancement feature to obtain a brightness enhancement feature.

[0280] In some embodiments, the enhancement feature may be convolved with a convolution kernel having a convolution kernel size of 3×3 to obtain the brightness enhancement feature.

[0281] Denote the enhanced feature as F enha , the convolution process is represented as Conv 3×3 (), the brightness enhancement feature is expressed as Then the above step S3010 can be expressed as the following formula:

[0282]

[0283] S3011. Add and fuse the brightness enhancement feature and the brightness feature to obtain the enhanced brightness feature.

[0284] The brightness enhancement feature is expressed as Additive fusion is represented by +, and enhanced brightness features are represented by Then the above step 2062 can be expressed as the following formula:

[0285]

[0286] S3012. Process the enhanced feature through a first residual block sequence to obtain a first residual feature.

[0287] Among them, the first residual block sequence includes multiple serially connected residual blocks, and the residual block is used to perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block in sequence to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.

[0288] In some embodiments, the first residual block sequence includes 5 serially connected residual blocks. The structure of the residual block can refer to Fig.18 The structure of the residual block shown is not described in detail here to avoid redundancy.

[0289] Denote the enhanced feature as F enha , the processing of the first residual block sequence is represented as RBS U (), the first residual feature is represented by Res U , then the above step S3012 can be expressed as the following formula:

[0290] ResU =RBS U (F ena )

[0291] S3013. Perform convolution processing on the first residual feature to obtain a first chroma enhancement feature.

[0292] The first residual feature is denoted as Res U , the convolution process is represented as Conv 3×3 (), the first chroma enhancement feature is expressed as Then the above S3013 can be expressed as the following formula:

[0293]

[0294] S3014. Add and fuse the first chromaticity feature and the first chromaticity enhancement feature to obtain the enhanced first chromaticity feature.

[0295] The first chroma enhancement feature is expressed as Additive fusion is represented by +, and the enhanced first chromaticity feature is represented by Then the above S3014 can be expressed as the following formula:

[0296]

[0297] S3015. Process the enhanced features through a second residual block sequence to obtain a second residual feature.

[0298] Among them, the second residual block sequence includes multiple serially connected residual blocks, and the residual block is used to perform a first convolution operation, a normalization operation, a linear activation operation, a second convolution operation and a scaling operation on the input features of the residual block in sequence to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.

[0299] In some embodiments, the second residual block sequence includes 5 serially connected residual blocks. The structure of the residual block can refer to Fig.18 The structure of the residual block shown is not described in detail here to avoid redundancy.

[0300] Denote the enhanced feature as F enha , the processing of the second residual block sequence is represented as RBS V (), the second residual feature is represented by Res V , then the above step 2066 can be expressed as the following formula:

[0301] Res V =RBS V (F en a)

[0302] S3016. Perform convolution processing on the second residual feature to obtain a second chroma enhancement feature.

[0303] The second residual feature is denoted as Res V , the convolution process is represented as Conv 3×3 (), the second chroma enhancement feature is expressed as Then the above S3016 can be expressed as the following formula:

[0304]

[0305] S3017. Add and fuse the second chromaticity feature and the second chromaticity enhancement feature to obtain the enhanced second chromaticity feature.

[0306] The second chromaticity feature is expressed as The second chroma enhancement feature is expressed as Additive fusion is represented by +, and the enhanced second chromaticity feature is represented by Then the above step 20612 can be expressed as the following formula:

[0307]

[0308] S3018. Obtain an enhanced image of the target image according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature.

[0309] Some embodiments of the present application provide an image decoding device, the image decoding device comprising:

[0310] a memory configured to store a computer program;

[0311] The processor is configured to, when calling a computer program, enable the image decoding device to implement the image decoding method described in any one of the above embodiments.

[0312] In some embodiments, the image decoding method provided by the embodiments of the present application further includes: training each network model in the image encoder and the image decoder based on a sample data set.

[0313] Wherein, the sample data set includes: multiple sample data groups, and any sample data group includes: a sample image and an enhanced image corresponding to the sample image. The process of training each network model in the image encoder and the image decoder based on the sample data set may include: first inputting the sample image into the image encoder for encoding to obtain the encoded data of the image, then decoding the encoded data through the image decoder to obtain the enhanced image, then calculating the loss value based on the enhanced image and the sample image, and then adjusting the parameters of the network model in the image encoder and / or the image decoder based on the loss value until the image encoder and the image decoder converge.

[0314] Some embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image decoding method described in any of the above embodiments.

[0315] In some embodiments, some embodiments of the present application provide a computer program product, which, when executed on a computer, enables the computer to implement the image decoding method described in any of the above embodiments.

[0316] In the embodiment of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0317] In the embodiments of the present application, the memory may include a non-permanent memory in a computer-readable medium, a random access memory (RAM) and / or a non-volatile memory, such as a read-only memory (ROM) or a flash RAM.

[0318] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0319] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. An image decoding method, characterized in that: include: Decoding the coded data of the target image to obtain a reconstructed image of the target image, wherein the image data of the reconstructed image includes: an initial luminance component, an initial first chrominance component, and an initial second chrominance component; Generate an upsampled first chroma component and an upsampled second chroma component according to the initial first chroma component and the initial second chroma component respectively; the spatial sizes of the upsampled first chroma component and the upsampled second chroma component are the same as the spatial size of the initial luminance component; Performing feature extraction on the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain a luminance feature, a first chrominance feature, and a second chrominance feature; splicing the brightness feature, the first chromaticity feature, and the second chromaticity feature to obtain an initial image feature; Extracting features of the initial image using a content-adaptive inter-channel related information filter to obtain enhanced features; Based on the enhancement feature, feature enhancement processing is performed on the brightness feature, the first chromaticity feature, and the second chromaticity feature respectively to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature; An enhanced image of the reconstructed image is acquired according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature.

2. The method according to claim 1, characterized in that The extracting the features of the initial image by using a content-adaptive inter-channel related information filter to obtain enhanced features includes: Performing convolution processing on the initial image feature to obtain a first feature; Performing batch normalization on the first feature to obtain a second feature; Performing linear activation processing on the second feature to obtain a third feature; The third feature is processed through a residual network to obtain the enhanced feature; the residual network includes a plurality of serially connected residual blocks, and the residual block is used to sequentially perform a first convolution process, a normalization process, a linear activation process, a second convolution process and a scaling process on the input features of the residual block to obtain the residual features of the residual block, and to add and fuse the input features of the residual block and the residual features of the residual block to obtain the output features of the residual block.

3. The method according to claim 2, characterized in that The residual network includes 8 serially connected residual blocks.

4. The method according to claim 1, characterized in that: The performing feature enhancement processing on the brightness feature, the first chromaticity feature, and the second chromaticity feature based on the enhancement feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature, comprises: Performing convolution processing on the enhancement feature to obtain a brightness enhancement feature; The brightness enhancement feature and the brightness feature are added and fused to obtain the enhanced brightness feature.

5. The method according to claim 1, characterized in that The performing feature enhancement processing on the brightness feature, the first chromaticity feature, and the second chromaticity feature based on the enhancement feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature, comprises: The enhanced features are processed by a first residual block sequence to obtain a first residual feature; the first residual block sequence includes a plurality of serially connected residual blocks, the residual blocks are used to sequentially perform a first convolution process, a normalization process, a linear activation process, a second convolution process and a scaling process on the input features of the residual blocks to obtain the residual features of the residual blocks, and the input features of the residual blocks and the residual features of the residual blocks are added and fused to obtain the output features of the residual blocks; Performing convolution processing on the first residual feature to obtain a first chroma enhancement feature; The first chromaticity feature and the first chromaticity enhancement feature are added and fused to obtain the enhanced first chromaticity feature.

6. The method according to claim 5, characterized in that The first residual block sequence includes 5 serially connected residual blocks.

7. The method according to claim 1, characterized in that The performing feature enhancement processing on the brightness feature, the first chromaticity feature, and the second chromaticity feature based on the enhancement feature to obtain an enhanced brightness feature, an enhanced first chromaticity feature, and an enhanced second chromaticity feature, comprises: The enhanced features are processed by a second residual block sequence to obtain a second residual feature; the second residual block sequence includes a plurality of serially connected residual blocks, the residual blocks are used to sequentially perform a first convolution process, a normalization process, a linear activation process, a second convolution process and a scaling process on the input features of the residual blocks to obtain the residual features of the residual blocks, and the input features of the residual blocks and the residual features of the residual blocks are added and fused to obtain the output features of the residual blocks; Performing convolution processing on the second residual feature to obtain a second chroma enhancement feature; The second chromaticity feature and the second chromaticity enhancement feature are added and fused to obtain the enhanced second chromaticity feature.

8. The method according to claim 7, characterized in that The second residual block sequence includes 5 serially connected residual blocks.

9. The method according to claim 1, characterized in that: The generating of the upsampled first chroma component and the upsampled second chroma component according to the initial first chroma component and the initial second chroma component respectively comprises: Bicubic interpolation is performed on the initial first chroma component and the initial second chroma component respectively to obtain the upsampled first chroma component and the upsampled second chroma component respectively.

10. The method according to claim 1, characterized in that The extracting features of the initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component respectively to obtain luminance features, first chrominance features, and second chrominance features includes: The initial luminance component, the upsampled first chrominance component, and the upsampled second chrominance component are respectively subjected to forward wavelet transform to obtain the luminance feature, the first chrominance feature, and the second chrominance feature.

11. The method according to claim 10, characterized in that The step of acquiring enhanced reconstruction data of the target image according to the enhanced brightness feature, the enhanced first chromaticity feature, and the enhanced second chromaticity feature comprises: Performing inverse wavelet transform on the enhanced brightness feature, the enhanced first chrominance feature and the enhanced second chrominance feature respectively to obtain an enhanced brightness component, an enhanced first chrominance component and an enhanced second chrominance component; The enhanced luminance component, the enhanced first chrominance component, and the enhanced second chrominance component are combined to obtain enhanced reconstructed data of the target image.

12. The method according to any one of claims 1 to 11, characterized in that: The step of decoding the coded data of the target image to obtain a reconstructed image of the target image comprises: The encoded data of the target image is decoded based on an image decoder in the JPEG AI image coding standard to obtain a reconstructed image of the target image.

13. The method according to any one of claims 1 to 11, characterized in that: The coded data of the target image includes: first coded data, second coded data, third coded data and fourth coded data, wherein the first coded data and the second coded data are coded data obtained by entropy coding the base layer super prior features and the enhancement layer super prior features of the target image, respectively, and the third coded data and the fourth coded data are coded data obtained by entropy coding the base layer quantized potential features and the enhancement layer quantized potential features of the target image, respectively; the entropy decoding of the coded data of the target image to obtain a reconstructed image of the target image includes: Performing entropy decoding on the first coded data and the second coded data respectively to obtain a base layer reconstruction super priori feature and an enhancement layer reconstruction super priori feature; Processing the base layer reconstruction super-prior features through a base layer super-prior decoding network to obtain base layer reconstruction potential features; Processing the enhancement layer reconstruction super-prior features through an enhancement layer super-prior decoding network to obtain enhancement layer reconstruction potential features; Performing entropy decoding on the third coded data and the fourth coded data according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain a base layer reconstruction quantization potential feature and an enhancement layer reconstruction quantization potential feature; The target image is reconstructed according to the base layer reconstructed quantized latent features and the enhanced layer reconstructed quantized latent features to obtain a reconstructed image of the target image.

14. The method according to any one of claims 1 to 11, characterized in that: The coded data of the target image includes: first coded data, second coded data and third coded data, the first coded data is coded data obtained by entropy coding the super-prior features of the target image, the second coded data and the third coded data are coded data obtained by entropy coding the enhancement layer quantized potential features and the base layer quantized potential features of the target image respectively; the decoding of the coded data of the target image to obtain the reconstructed image of the target image includes: Performing entropy decoding on the first coded data to obtain a reconstructed super-prior feature; Performing super-prior decoding processing on the reconstructed super-prior feature to obtain a reconstructed potential feature; Segmenting the reconstruction potential features to obtain base layer reconstruction potential features and enhancement layer reconstruction potential features; Performing entropy decoding on the second coded data and the third coded data according to the base layer reconstruction potential feature and the enhancement layer reconstruction potential feature to obtain the enhancement layer reconstruction quantization potential feature and the base layer reconstruction quantization potential feature; The target image is reconstructed according to the enhancement layer reconstructed quantized latent features and the base layer reconstructed quantized latent features to obtain a reconstructed image of the target image.

15. A video decoding device, characterized in that: include: a memory configured to store a computer program; The processor is configured to enable the video decoding device to implement the image decoding method according to any one of claims 1 to 14 when calling a computer program.

Citation Information

Patent Citations

  • Modified entropy encoding for images and videos

    CA2751803A1

  • Filtering, encoding and decoding method and device, computer readable medium and electronic equipment

    CN115883842A

  • Scalable coding of video and associated features

    US20230065862A1