Image reconstruction enhancement method, decoding device and storage medium

The adapter-based ICCI sub-network in JPEG AI verification models simplifies the content adaptive filter by unifying the processing of Y, U, and V components, reducing complexity and enhancing image quality.

CN119090726AActive Publication Date: 2024-12-06HISENSE VISUAL TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310870502.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-06
Filing Date
2023-07-14
Publication Date
2024-12-06
Estimated Expiration
2043-07-14

AI Technical Summary

Technical Problem

The complexity of content adaptive filters in JPEG AI verification models, which require separate ICCI processing modules for Y, U, and V components, increases computational complexity and reduces image quality.

Method used

An adapter-based ICCI sub-network is introduced to process Y, U, and V components adaptively, using a unified ICCI module to enhance image features, reducing the need for separate processing and simplifying the content adaptive filter architecture.

Benefits of technology

This approach reduces computational complexity while maintaining or improving image quality by adaptively processing each component, enhancing the final image with more features.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090726A_ABST
    Figure CN119090726A_ABST
Patent Text Reader

Abstract

The invention relates to an image reconstruction enhancement method, decoding equipment and a storage medium, and particularly relates to the technical field of image coding and decoding. Comprising the following steps: respectively carrying out forward wavelet transformation on a first Y component, a first U component and a first V component of an input image to obtain a Y component transformation result, a U component transformation result and a V component transformation result, and splicing to obtain a spliced component; inputting the splicing component into an adapter, and obtaining an adaptive enhancement component output by the adapter; inputting the adaptive enhancement component, the Y component transformation result, the U component transformation result and the V component transformation result into an ICCI processing module, and obtaining a Y enhancement component, a U enhancement component and a V enhancement component output by the ICCI processing module; respectively carrying out inverse wavelet transform based on the Y enhancement component, the U enhancement component and the V enhancement component to obtain an output Y component, an output U component and an output V component; and obtaining an output image based on the output Y component, the output U component and the output V component.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 6, 2023, with application number 2023106645409. The entire contents of this application are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the technical field of image coding and decoding, and in particular to an image reconstruction and enhancement method, a decoding device and a storage medium. Background Art

[0003] Image compression is one of the basic tasks in computer vision and image processing. The development of lossy image compression technology has brought benefits to various fields. Image compression technology can greatly reduce the amount of data for image storage and image transmission, but the image quality obtained after reconstructing the image may be slightly reduced. In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression technology. JPEG AI is a learning-based image coding standard created by the Joint Photographic Experts Group (JPEG). The JPEG AI verification model achieves a lower bit rate at the same subjective quality compared to traditional image coding for human vision.

[0004] The JPEG AI verification model uses a content adaptive filter to enhance image features. The characteristics of different Y, U, and V components are taken into consideration in the content adaptive filter, and three different ICCI processing modules are set up to process the characteristics of the three channels of the image, respectively, and enhance the Y, U, and V components. However, this requires the setting of three different ICCI processing modules, which increases the complexity of the content adaptive filter. Summary of the invention

[0005] In order to solve the above technical problem or at least partially solve the above technical problem, the present disclosure provides an image reconstruction enhancement method, a decoding device and a storage medium.

[0006] In order to achieve the above objectives, some embodiments of the present application provide the following technical solutions:

[0007] In a first aspect, an image reconstruction enhancement method is provided, comprising:

[0008] Performing forward wavelet transform on the first Y component, the first U component and the first V component of the input image respectively to obtain a Y component transform result, a U component transform result and a V component transform result;

[0009] splicing the Y component transformation result, the U component transformation result and the V component transformation result to obtain a spliced ​​component;

[0010] Input the spliced ​​component into an adapter, and obtain an adaptive enhancement component output by the adapter, wherein the adapter is trained based on a convolutional layer framework, and has adaptive processing capabilities for the three components Y, U, and V by learning the features of the three components Y, U, and V during the training process;

[0011] Inputting the adaptive enhancement component, the Y component transformation result, the U component transformation result and the V component transformation result into an ICCI processing module, and obtaining the Y enhancement component, the U enhancement component and the V enhancement component output by the ICCI processing module;

[0012] Perform inverse wavelet transform on the Y enhancement component, the U enhancement component and the V enhancement component to obtain an output Y component, an output U component and an output V component;

[0013] An output image is obtained based on the output Y component, the output U component and the output V component.

[0014] In a second aspect, a decoding device is provided, including:

[0015] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image reconstruction enhancement method as described in the first aspect or any one of its optional embodiments.

[0016] In a third aspect, a computer-readable storage medium is provided, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the image reconstruction enhancement method as described in the first aspect or any one of its optional embodiments is implemented.

[0017] According to a fourth aspect, a computer program product is provided, wherein a computer program is stored in the computer program product, and when the computer program is executed by a processor, the image reconstruction enhancement method as described in the first aspect or any optional embodiment thereof is implemented.

[0018] The image reconstruction enhancement method provided by the embodiment of the present application uses an adapter-based ICCI sub-network in the content adaptive filter in the JPEG AI verification model, and introduces an adapter in the adapter-based ICCI sub-network to adaptively process the Y, U, and V components to enhance the input image according to the characteristics of each Y, U, and V component, so that there is no need to process the three components separately through different ICCI modules, but instead a unified ICCI module is used for processing, so that the content adaptive filter architecture is simplified from three ICCI modules to one ICCI module, effectively reducing the computational complexity of the JPEG AI verification model. When the method uses a unified ICCI module for processing, each Y, U, and V component and the result after the adapter-based adaptive processing are used as the input of the ICCI module processing, so that the ICCI module can obtain the Y enhanced component, U enhanced component, and V enhanced component corresponding to Y, U, and V, respectively, so that the characteristics of different components can be further taken into account, so that the final enhanced image includes more features of each component. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0020] In order to more clearly illustrate the technical solutions in some embodiments of the present application, the drawings required for describing the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A schematic diagram of the architecture of a JPEG AI verification model in the related technology provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of an analysis and transformation network 101 in the related technology provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a residual activation unit in the related technology provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a residual non-local attention block 203 in the related art provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a residual block in the related technology provided in an embodiment of the present application;

[0026] Figure 6A schematic diagram of a masked convolution kernel used in a context model of an autoregressive convolution in the related technology provided in an embodiment of the present application;

[0027] Fig. 7A A schematic diagram of a mask convolution kernel used by a context model network 105 in a JPEG AI verification model in the related technology provided in an embodiment of the present application;

[0028] Figure 7B A schematic diagram of a wavefront parallel processing mechanism of a latent space in a related technology provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of a synthesis transformation network 117 in a related technology provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of another JPEG AI verification model in the related art provided in an embodiment of the present application;

[0031] Fig.10 A schematic diagram of an ICCI sub-network provided in an embodiment of the present application;

[0032] Fig.11 A schematic flow chart of a method for obtaining a predicted value corresponding to an input potential tensor provided in an embodiment of the present application;

[0033] Fig.12 A schematic diagram of a potential space divided into three parts provided in an embodiment of the present application;

[0034] Fig.13 A schematic diagram of a first potential tensor provided in an embodiment of the present application;

[0035] Fig.14 A schematic diagram of obtaining a first context feature provided in an embodiment of the present application;

[0036] Fig.15 A schematic diagram of a second potential tensor provided in an embodiment of the present application;

[0037] Fig.16 A schematic diagram of obtaining a second context feature provided in an embodiment of the present application;

[0038] Fig.17 A schematic diagram of obtaining a predicted value corresponding to a predicted input potential tensor provided in an embodiment of the present application;

[0039] Fig.18 A schematic flow chart of a method for obtaining an output potential tensor provided in an embodiment of the present application;

[0040] Fig.19AA schematic diagram of obtaining a second context feature provided in an embodiment of the present application;

[0041] Fig.19B A schematic diagram of obtaining a first context feature provided in an embodiment of the present application;

[0042] Fig.19C A schematic diagram of obtaining an output potential tensor provided in an embodiment of the present application;

[0043] Fig. 20A A schematic diagram of an adapter-based ICCI sub-network provided in an embodiment of the present application;

[0044] Fig. 20B Provided in the embodiments of this application Fig. 20A A schematic diagram of the adapter in FIG.

[0045] Fig.21 A flowchart of an image reconstruction enhancement method provided in an embodiment of the present application;

[0046] Fig. 22 A method for Fig.10 Schematic diagram of ICCI processing module;

[0047] Fig.23 A method provided in the embodiment of the present application Fig. 20A Schematic diagram of ICCI processing module;

[0048] Fig.24 A schematic diagram of a 1D residual block provided in an embodiment of the present application;

[0049] Fig.25 A schematic diagram of a flow chart of another image reconstruction enhancement method provided in an embodiment of the present application;

[0050] Fig.26 A schematic diagram of another adapter-based ICCI sub-network provided in an embodiment of the present application;

[0051] Fig. 27 A schematic diagram of the hardware structure of a decoding device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0052] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0053] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0054] The terms "first" and "second" in the specification and claims of this application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first context feature and a second context feature are used to distinguish different context features rather than to describe a specific order of context features.

[0055] Image compression is one of the basic tasks in computer vision and image processing. The development of lossy image compression technology has brought benefits to various fields. Image compression technology can greatly reduce the amount of data for image storage and image transmission, but the image quality obtained after reconstructing the image may be slightly reduced. In the field of lossy image compression technology, traditional codecs including the Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG) and Versatile Video Coding (VVC) use similar transforms, quantization and entropy coding to compress images.

[0056] In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression technology. JPEG AI is a learning-based image coding standard created by the Joint Photographic Experts Group (JPEG). JPEG AI targets human vision and achieves a lower bit rate than traditional image coding at the same subjective quality.

[0057] Figure 1A schematic diagram of the architecture of a JPEG AI verification model (Verification Model 1.0) in the related technology provided in an embodiment of the present application. The JPEG AI verification model includes: an analysis transform network (Analysis Transform Net) 101, a gain unit 102, a first quantization unit 103, a first inverse gain unit 104, a context model network 105, a prediction fusion network 106, a super priori encoding network 107, a super priori decoding network 108, a second quantization unit 109, a first lossless encoder 110, a first lossless decoder 111, a decomposition entropy model 112, a super priori scale decoding network 113, a second lossless encoder 114, a second lossless decoder 115, a second inverse gain unit 116, and a synthesis transform network 117. The JPEG AI verification model can compress the input original image x into a code stream, and reconstruct the compressed code stream into a reconstructed image . In the verification model of JPEGAI, the network modules such as the Analysis Transform Net 101, the gain unit 102, the first quantization unit 103, the first inverse gain unit 104, the super-prior decoding network 108, the second quantization unit 109, the first lossless encoder 110, and the second lossless encoder 114 only exist in the encoder; the network modules such as the second lossless decoder 115, the second inverse gain unit 116, and the synthesis transform network 117 only exist in the decoder. The network modules such as the context model network 105, the prediction fusion network 106, the super-prior coding network 107, the first lossless decoder 111, the decomposition entropy model 112, and the super-prior scale decoding network 113 exist in both the encoder and the decoder.

[0058] The above-mentioned analysis transformation network 101 is used to transform the original image x input into the verification model of the JPEG AI into a latent tensor y. The analysis transformation network 101 only works in the encoder. The latent tensor is a high-dimensional representation of the data. The latent tensor cannot be directly observed from the input data, but they can be determined by the information learned from the input data. The above-mentioned latent tensor y is determined based on the information learned from the original image x.

[0059] Figure 2 A schematic diagram of an analysis transformation network 101 in the related technology provided in the embodiment of the present application is shown as follows: Figure 2As shown, the analysis transformation network 101 includes, from the input end to the output end, a 3×3 downsampling convolution 201, a residual activation unit 202, a residual non-local attention block (Residual non-local attention networks for image restoration) 203, a 3×3 downsampling convolution 204, a residual activation unit 205, a 3×3 downsampling convolution 206, a residual activation unit 207, a 3×3 downsampling convolution 208 and a 1×1 convolution 209.

[0060] Among them, the analysis transformation network 101 uses a residual activation unit with an attention mechanism, for example, Figure 2 The residual activation unit 202, the residual activation unit 205 and the residual activation unit 207 in are all residual activation units with attention mechanism. Figure 2 The 3×3 downsampling convolution 201, the 3×3 downsampling convolution 204, the 3×3 downsampling convolution 206 and the 3×3 downsampling convolution 208 are all 3×3 downsampling convolutions with the same structure.

[0061] Figure 3 Schematic diagram of a residual activation unit 202 in the related technology provided in the embodiment of the present application. Figure 3 As shown, the residual activation unit 202 may include: a LeakyReLU activation function 301, a convolution layer 302, a tanh activation function 303 and a residual connection unit 304. The residual activation unit is used to add nonlinear transformation.

[0062] In the JPEG AI verification model, the attention mechanism can be used Figure 2 The residual non-local attention block 203 shown is implemented. The residual non-local attention block 203 can extract deep features. In the process of extracting deep features, by capturing the correlation between pixels, the pixel redundancy of the feature map can be reduced. Figure 4 Schematic diagram of a residual non-local attention block 203 in the related art provided in the embodiment of the present application. Figure 4 As shown, the residual non-local attention block 203 includes two branches: a trunk branch 41 and a mask branch 42. The trunk branch 41 contains three residual blocks for extracting deep features. The mask branch 42 contains a 3×3 downsampling convolution 401, a 3×3 upsampling convolution 402, a 1×1 convolution 403, a sigmoid activation function 404, and a series of residual blocks. Figure 4The three residual blocks of the trunk branch 41 are respectively represented as residual block 1, residual block 2, and residual block 3; the series of residual blocks included in the mask branch 42 are respectively represented as residual block 4, residual block 5, residual block 6, residual block 7, residual block 8, and residual block 9. It should be noted that the structure and function of each residual block in the above-mentioned residual block 1 to residual block 9 are the same.

[0063] Figure 5 for Figure 4 A schematic diagram of a residual block in the embodiment shown, each residual block includes a LeakyReLU activation function 501 and two 3×3 convolutions, wherein the two 3×3 convolutions are Figure 5 They are represented as 3×3 convolution 502 and 3×3 convolution 503 respectively.

[0064] like Figure 1 As shown, in the verification model (Verification Model 1.0) of JPEG AI, the interpolation gain unit is used to realize the variable bit rate. The parameter of the interpolation gain unit is controlled by the parameter β to scale the channel dimension of the potential residual variable. The interpolation gain unit includes the above-mentioned gain unit 102 and the first inverse gain unit 104.

[0065] The gain unit 102 is used to perform gain processing on the residual tensor r, and multiply each element of the residual tensor r by the forward gain vector m t The gain factor specified in is used to obtain the residual tensor r′ after gain processing, where the residual tensor r is the difference between the potential tensor y and the predicted value μ, where the predicted value μ is the predicted value based on the elements in the potential tensor y. The gain unit 102 is used to implement the variable rate capability of the verification model of JPEG AI. The gain unit 102 uses a parameter β to control the compression ratio. The parameter β specifies the weight between the rate and the distortion in the loss function during training. During model training, for multiple β∈{β t}Get the paired forward gain vector and reverse gain vector, which can be expressed as Among them, m t is the forward gain vector, is the reverse gain vector. During encoding and decoding, the forward gain vector and the reverse gain vector are known to the encoder and decoder (they are part of the model), and the gain vector m t and The dimension of is equal to the number of channels of the residual tensor r. The gain unit 102 works only in the encoder.

[0066] The first quantization unit 103 is used to quantize the residual tensor after gain processing to obtain a quantized residual tensor The quantization process refers to the process of approximating the continuous value (or a large number of possible discrete values) of the signal to a finite number of discrete values ​​(or a small number of discrete values). The first quantization unit 103 only works in the encoder.

[0067] The first inverse gain unit 104 is used to quantize the residual tensor After inverse gain processing, it is the quantized residual tensor Each element of is multiplied by the inverse gain vector The gain factor specified in , to obtain the residual tensor after inverse gain processing The first inverse gain unit 104 can realize the variable rate capability of the verification model of JPEG AI. The first inverse gain unit 104 uses a parameter β to control the compression ratio, and the parameter β specifies the weight between rate and distortion in the loss function during training. The first inverse gain unit 104 only works in the encoder.

[0068] The context model network 105 is used to extract context features. The context model network 105 is present in both the encoder and the decoder. In the encoder, the context model network 105 is used to extract the context features. Perform context feature extraction to obtain the latent tensor The context features of . Among them, the latent tensor in the encoder The residual tensor after processing according to the inverse gain In the decoder, the context model network 105 is used to calculate the potential tensor Perform context feature extraction to obtain the latent tensor Context features. In the decoder, the bitstream ( Figure 1 The bitstream shown in 2) is the residual tensor after lossless decoding and inverse gain processing Adding it to the predicted value μ, we can get the potential tensor

[0069] In some learning-based image coding methods, an autoregressive convolutional context model is used to extract contextual features. Figure 6 A schematic diagram of a mask convolution kernel used in a context model of an autoregressive convolution in the related technology provided in the embodiment of the present application. Figure 6 As shown, the context model of the autoregressive convolution uses the encoded elements before the current encoding position 601 in the latent space to predict the context features corresponding to the current encoding position, and uses the context features to subsequently predict the probability distribution of the current encoding position. Among them, the encoded elements are Figure 6The elements at the encoding position shown in the dotted box in . The context model of the autoregressive convolution can significantly improve the rate-distortion performance of the super-prior model. However, due to its autoregressive structure, the context model of the autoregressive convolution can only be decoded in a fixed raster scanning order, and the latent space decoded later always depends on the latent space decoded earlier, resulting in a very slow decoding speed for this model.

[0070] To solve the above problems, in some embodiments, the context model network 105 in the verification model of JPEG AI can be further used in the context model network 105 to achieve parallel processing. The Wavefront Parallel Processing (WPP) mechanism of the latent space is used in the context model network 105, in which the decoded elements are used for latent tensors. The prediction process is an autoregressive process and is for the potential tensor Elements in different rows of can be processed independently in a wavefront-parallel manner, where the delay in processing consecutive rows depends on the shape of the masked convolution kernel.

[0071] Fig. 7A A schematic diagram of a masked convolution kernel used by a context model network 105 in a JPEG AI verification model in the related technology provided in an embodiment of the present application. Fig. 7A In the column coordinates, the current coding position is the origin of the coordinates. The current coding position is in row 0 and column 0. n=0 indicates the row where the current coding position is located, n=-1 indicates the row before the row where the current coding position is located, and n=-2 indicates the row before the row where the current coding position is located. Fig. 7A The "-2", "-1", "0", "1" and "2" in the table indicate different columns, where "0" indicates the column at the current encoding position, "-1" indicates 1 column before the column at the current encoding position, "-2" indicates 2 columns before the column at the current encoding position, "1" indicates 1 column after the column at the current encoding position, and "2" indicates 2 columns after the column at the current encoding position.

[0072] Figure 7B A schematic diagram of a wavefront parallel processing mechanism of a latent space in the related technology provided by the embodiment of the present application. In order to improve the utilization rate of computing resources, the wavefront parallel processing mechanism is introduced. Figure 7B In the figure, “columu” means column and “row” means row. The figure shows a potential space with 5 rows and 10 columns as an example. Figure 7B The elements of all channels of the tensor of the current pixel are predicted independently (simultaneously) and can be predicted based on the pixels that have been processed in the preset process. Fig. 7A The masked convolution kernel shown is designed in such a way that Figure 7BEach row of samples (i.e., the prediction of each row of pixels) can be processed in parallel, with the next row being processed delayed by one element compared to the previous row, i.e., delayed by the time of one wave movement (T).

[0073] The super-a priori coding network 107 is used to perform super-a priori coding on the potential tensor y to obtain the potential tensor z, and can extract additional auxiliary information during the super-a priori coding process so that an accurate probability model of entropy coding can be obtained based on the coding result. The super-a priori coding network 107 only works in the encoder, and the super-a priori coding network 107 may include two downsampling convolutions, three convolutions and a ReLU activation function. The above-mentioned ReLU activation function is a neural network activation function. When the input value is greater than 0, it remains unchanged, and when the input value is less than 0, it will output 0.

[0074] The second quantization unit 109 is used to quantize the potential tensor z to obtain a quantized coding potential tensor The second quantization unit 109 only works in the encoder.

[0075] The first lossless encoder 110 is used to quantize the potential tensor Entropy encoding is performed to obtain a bit stream 1. Entropy encoding is a lossless operation used to encode a bit stream. The entropy encoding and entropy decoding process takes the cumulative distribution function (CDF) output by the decomposition entropy model 112 as input. The cumulative distribution function (CDF) is essentially the integration of the probability density function (PDF) and the symbol to be encoded at the encoding end, or the integration of the probability density function (PDF) and the bit stream to be decoded at the decoding end. The first lossless encoder 110 only works at the encoder.

[0076] The first lossless decoder 111 is used to entropy decode the bit stream 1 obtained after entropy coding to obtain a potential tensor after entropy decoding. Among them, entropy decoding is a lossless operation for decoding a bit stream. The cumulative distribution function (CDF) can be received as input during the entropy decoding process. Entropy decoding is essentially the integral of the probability density function (PDF) and the bit stream to be decoded (at the decoder end). The first lossless decoder 111 only works at the decoder.

[0077] The decomposition entropy model 112 is used to input a cumulative distribution function (CDF) to the first lossless encoder 110 or the first lossless decoder 111, and the cumulative distribution function (CDF) is used for entropy encoding or entropy decoding. The decomposition entropy model 112 works in the decoder and also in the encoder. The decomposition entropy model 112 is integerized to solve the problem of device interoperability so that the decomposition entropy model 112 can be used on any device. Among them, the cumulative distribution function (CDF) is also called a probability table, which is pre-calculated and quantized, rather than trained based on trainable parameters (matrix, bias and factor). The table is stored in integer form, so there is no need to store the above Parameters of the distribution.

[0078] The super-a priori decoding network 108 is used to perform super-a priori decoding. The super-a priori decoding network 108 is used to decode the potential tensor after entropy decoding. Decode to get the potential tensor The super-prior decoding network 108 includes a series of two inverse convolutions with upsampling stride 2, three convolutions without tensor size change, and a ReLU activation function. The super-prior decoding network 108 can receive a tensor of size C×h×w as input and generate an output tensor with dual channels 4C×h×w. The super-prior decoding network 108 is symmetrical to the super-prior encoding network 107, and each upsampling convolution layer is followed by a clipping operation. The super-prior decoding network 108 works in the decoder and also in the encoder.

[0079] The prediction fusion network 106 can receive two inputs, the first input is the potential tensor obtained by the super prior decoding network 108 after super prior decoding The second input is the context feature output by the context model network 105. The prediction fusion network 106 generates a prediction value μ (size C×h×w) of size C×h×w based on the two inputs of size 4C×h×w. In the encoder, the prediction value μ can be added to the residual tensor after the inverse gain processing. To obtain the latent tensor As the input of the context model network 105. In the decoder, the bitstream ( Figure 1 The bitstream shown in 2) is the residual tensor after lossless decoding and inverse gain processing Adding it to the predicted value μ, we can get the potential tensor As input to the context model network 105, the prediction fusion network 106 works in the decoder and also in the encoder.

[0080] Super prior scale decoding network 113, used to decode the potential tensor after entropy The inverse operation of the super-prior coding network is performed, and the variance of the Gaussian distribution N(0,σ) with a probability distribution of 0 mean is output. The super-prior scale decoding network 113 includes two 5×5 inverse convolution sequences with an upsampling stride of 2, two 3×3 convolutions and a LeakyReLU activation function. The super-prior scale decoding network 113 is symmetrical to the super-prior coding network 107, and each upsampling convolution layer is followed by a cropping operation. The prediction fusion network 106 works in the decoder and also in the encoder.

[0081] The second lossless encoder 114 is used to quantize the residual tensor based on N(0,σ) Entropy encoding is performed to obtain a code stream 2. It should be noted that the second lossless encoder 114 has a similar structure and function to the first lossless encoder 110. The second lossless encoder 114 only works in the encoder.

[0082] The second lossless decoder 115 is used to perform entropy decoding on the bitstream 2 based on N(0,σ) to obtain a residual tensor after entropy decoding. It should be noted that the structure and function of the second lossless decoder 115 are similar to those of the first lossless decoder 111. The second lossless decoder 115 only works in the decoder.

[0083] The second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding, and multiply each element of the residual tensor after entropy decoding by the inverse gain vector The gain factor specified in , to obtain the residual tensor after inverse gain processing The second inverse gain unit 116 operates only in the decoder.

[0084] Synthetic Transformation Network 117, used to transform the latent tensor Representation is converted into the reconstructed image Among them, the potential tensor is the residual tensor after inverse gain processing The result is added to the prediction value μ. The synthesis transformation network 117 only works in the decoder.

[0085] Figure 8 FIG. 1 is a schematic diagram of a synthesis transformation network 117 in the related art provided in an embodiment of the present application. Figure 8 As shown, the synthesis transformation network 117 includes, from the input end to the output end, a residual block 801, a residual block 802, a 3×3 upsampling convolution 803, a residual activation unit 804, a 3×3 upsampling convolution 805, a residual activation unit 806, a 3×3 upsampling convolution 807, a residual non-local attention block 808, a residual activation unit 809, and a 3×3 upsampling convolution 810. Among them, two residual blocks are placed at the beginning of the synthesis transformation, which enhance the reconstructed potential tensor Reduced quantization error.

[0086] It should be noted that each residual activation unit in the synthetic coding network 117 is similar to Figure 3 The structure and function of the residual activation unit shown in FIG. 8 are the same as those of the residual non-local attention block 808 in the synthetic coding network 117. Figure 4 The structure and function of the residual non-local attention block 203 shown in FIG. 1 are the same. Each residual block in the synthetic coding network 117 is the same as Figure 4 The structure and function of the residual blocks shown are the same.

[0087] Fig. 9 A schematic diagram of another JPEG AI verification model in the related technology provided in an embodiment of the present application.

[0088] like Fig. 9 As shown in the above Figure 1 The content adaptive filter 118 is added to the verification model of JPEG AI shown in the figure. The content adaptive filter 118 is used to enhance the Y, U, and V components and synthesize the reconstructed image output by the transformation network 117. Enhance the Y, U, and V components to obtain the enhanced reconstructed image The adaptive filter 118 includes a content adaptive Inter Channel Correlation Information filter sub network (ICCI) sub network for reconstructing and enhancing the three components Y, U, and V of the image. The purpose of enhancing the Y, U, and V components is achieved through this ICCI sub network.

[0089] Fig.10 A schematic diagram of an ICCI sub-network provided in an embodiment of the present application. Fig.10 As shown in FIG. 1 , the ICCI subnetwork includes three different ICCI processing modules. The input of the ICCI subnetwork is the Y, U, and V components of the image. Fig.10 The Y component of the image is expressed as The U component of the image is expressed as The V component of the image is expressed as The U component of the image is bicubicly interpolated to obtain a U component of the same shape as Y. The V component of the image is bicubicly interpolated to obtain a V component of the same shape as Y Then the Y component U Component and V component All of them are forward wavelet transformed, and the three components obtained by the transformation are expressed as Fig.10 Components 1001, 1002 and 1003 in the image are spliced ​​in the channel dimension, and the spliced ​​component 1004 and their respective components are respectively sent to three ICCI processing modules as inputs. The three ICCI processing modules process the Y, U and V components respectively to obtain three processed components, so as to obtain processed components 1008, 1009 and 1010. The processed components 1008, 1009 and 1010 are all subjected to inverse wavelet transform to obtain the reconstructed Y, U and V components. The reconstructed Y component is The U component after reconstruction is The reconstructed V component is The number of residual blocks in the three ICCI processing modules is Nb, the number of residual blocks in the ICCI processing module 1005 for the Y component is Nb=8, the number of residual blocks in the ICCI processing module 1006 for the U component is Nb=13, and the number of residual blocks in the ICCI processing module 1007 for the V component is Nb=13. The bicubic interpolation is a method for enlarging the image size and increasing the pixel density, and the bicubic difference can be used to double the number of image pixels, wherein these expanded pixels are predicted based on the existing pixels.

[0090] Above Fig.10 In the ICCI subnetwork shown, in the process of image feature enhancement, in order to consider the characteristics of different Y, U, and V components, three different ICCI processing modules are set to enhance different components to achieve the purpose of adaptive processing.

[0091] Figure 1 In the context model network 105, in order to achieve parallel processing, the convolution kernel is used to calculate the context features of the elements of the current pixel. When calculating the context features of the elements of the current pixel, only the elements to the upper left of the current pixel are concerned, and other elements around the current pixel are not concerned. Compared with the context model of the autoregressive convolution, the receptive field is reduced, and other elements to the lower right of the current pixel are not utilized. Therefore, the accuracy of the obtained context features is low. Among them, the above-mentioned receptive field refers to the size of the area in which a single element in the feature map output by each network layer in the convolutional neural network is mapped back to the original input feature, that is, an element point on the feature map output by the network layer in the convolutional neural network, the size of its receptive field is the size of the area mapped from the original input, and the deeper the network layer, the larger the receptive field corresponding to the element of its output feature.

[0092] In order to improve the accuracy of context features, the present application embodiment proposes a step-by-step context model, by embedding the step-by-step context model into Figure 1 In the context model network 105 of the verification model of the JPEG AI shown, the calculation speed of the entropy coding part can be improved. The step-by-step context model can divide the latent space into 3 or more parts, calculate the context features for a part of the space in the latent space, and when calculating the context features, not only the elements to the upper left of the current pixel can be used, but also the elements to the lower right of the current pixel can be used, thereby improving the accuracy of the calculated context features, and the subsequent calculation of the prediction value corresponding to this part of the space based on the calculated context features can also improve the accuracy. It is also possible to not calculate the context features for another part of the space in the latent space, and then directly calculate the prediction value of the other part of the space in the latent space based on the output of the super-prior decoding network 108, thereby improving the efficiency of calculating the prediction value of the latent space. The step-by-step context model can be applied in an encoder and / or a decoder.

[0093] Fig.11 A flowchart of a method for obtaining a predicted value corresponding to an input potential tensor provided in an embodiment of the present application. In this method, a step-by-step context model is applied in a context model network in an encoder to extract context features, and the method includes but is not limited to the following steps:

[0094] 2001. The latent space corresponding to the input latent tensor is divided into three parts.

[0095] Among them, the entire latent space of the input potential tensor can be regarded as being composed of multiple 2×2 blocks, and the first pixel position of the upper left corner of each 2×2 block in the multiple 2×2 blocks can be determined as the first part; the second pixel position of the lower right corner of each 2×2 block in the multiple 2×2 blocks is determined as the second part; and the third pixel position of the upper right corner and the lower left corner of each 2×2 block in the 2×2 blocks is determined as the third part.

[0096] Among them, in the latent space, the upper left corner of each 2×2 block corresponds to the first pixel position, the lower right corner of each 2×2 block corresponds to the second pixel position, and the upper right corner and lower left corner of each 2×2 block correspond to the third pixel position.

[0097] Fig.12 A schematic diagram of a potential space divided into three parts is provided in an embodiment of the present application. For example, Fig.12 As shown, in the input potential tensor, the first part corresponding to the first pixel position can be marked as 1, the second part corresponding to the second pixel position can be marked as 2, and the third part corresponding to each third pixel position can be marked as 3.

[0098] The input potential tensor is the potential tensor input to the context model network 105. When the context model network 105 works in the encoder, the input potential tensor is Figure 1 The latent tensor shown in The latent tensor The residual tensor after processing according to the inverse gain It is obtained by adding the predicted value μ.

[0099] 2002. Perform a zeroing operation on the third pixel position in the latent space corresponding to the input latent tensor to obtain a first latent tensor.

[0100] The above-mentioned zeroing operation is performed on the third pixel position in the latent space corresponding to the input latent tensor, that is, the data of the first pixel position and the second pixel position in the latent space corresponding to the input latent tensor are retained, and the data of the third pixel position is set to zero. It should be noted that in the embodiment of the present application, setting the data of the third pixel position to zero means setting the data of the third pixel position in the latent space corresponding to the input latent tensor to zero, rather than setting Fig.12 The "3" used to mark the third part corresponding to the third pixel position in the latent space corresponding to the input potential tensor is set to zero. After the data of the third pixel position in the latent space corresponding to the input potential tensor is set to zero, the third part corresponding to the third pixel position can still be marked as 3. In order to show which pixel positions are set to zero in the above step 2002, Fig.13 The third part corresponding to the third pixel position is marked as empty, but this does not mean that the "3" used to mark the third part corresponding to the third pixel position is set to empty.

[0101] For example, Fig.13 The potential space partitioning method shown in Fig.13 A schematic diagram of a first potential tensor provided in an embodiment of the present application. Fig.13 In the first latent tensor shown, only the data of the first pixel position and the second pixel position are retained.

[0102] 2003. Perform a convolution operation on the first latent tensor through a first mask convolution kernel to obtain a first context feature corresponding to a third pixel position.

[0103] The size of the first mask convolution kernel is 5×5.

[0104] Fig.14 A schematic diagram of obtaining a first context feature provided in an embodiment of the present application. Fig.14 As shown, Fig.14 The first potential tensor 2301 shown in FIG. 2 is convolved with a 5×5 mask convolution 2302 to obtain Fig.14The first context feature 2303 is shown in .

[0105] 2004. Perform a zeroing operation on the second pixel position and the third pixel position in the latent space corresponding to the input latent tensor to obtain a second latent tensor.

[0106] Among them, the above-mentioned zeroing operation on the second pixel position and the third pixel position in the latent space corresponding to the input latent tensor is to retain the data of the first pixel position and the second pixel position in the latent space corresponding to the input latent tensor, and set the data of the third pixel position to zero.

[0107] For example, Fig.13 The potential space partitioning method shown in Fig.15 A schematic diagram of a second potential tensor provided in an embodiment of the present application. Fig.15 In the second latent tensor shown, only the data at the first pixel position is retained.

[0108] 2005. Perform a convolution operation on the second latent tensor through a second mask convolution kernel to obtain a second context feature corresponding to a second pixel position.

[0109] The size of the second mask convolution kernel is 3×3.

[0110] Fig.16 A schematic diagram of obtaining a second context feature provided in an embodiment of the present application. Fig.16 As shown, Fig.16 The first potential tensor 2501 shown in FIG. 1 is convolved with the 3×3 mask convolution 2502 to obtain Fig.16 The second context feature 2503 shown in .

[0111] After obtaining the first context feature and the second context feature, a predicted value μ corresponding to the input potential tensor may be predicted based on the first context feature and the second context feature.

[0112] The above steps 2001 to 2005 can be performed by Figure 1 The context model network 105 of the verification model of JPEG AI shown in FIG. 1 is implemented, and a step-by-step context model is applied in the context model network 105 .

[0113] 2006. Perform entropy probability derivation on the first context feature, the second context feature and the target latent tensor to obtain prediction values ​​corresponding to all pixel positions in the input latent tensor.

[0114] The entropy probability derivation of the first context feature, the second context feature and the target latent tensor to obtain the prediction values ​​corresponding to all pixel positions in the input latent tensor may include but is not limited to: adding the first context feature and the second context feature to obtain the target context feature, and splicing the target context feature and the target latent tensor to obtain a splicing result, and performing convolution processing on the splicing result to obtain the prediction values ​​corresponding to all pixel positions in the input latent tensor. The convolution processing may be a 1×1 convolution operation.

[0115] The above step 2006 can be performed by Figure 1 The preset fusion network 106 of the verification model of JPEG AI shown in FIG.

[0116] Fig.17 A schematic diagram of obtaining a predicted value corresponding to a predicted input potential tensor provided in an embodiment of the present application. In an embodiment of the present application, a step-by-step context model is embedded into Figure 1 The context model network 105 of the JPEG AI verification model shown in FIG. 1 is implemented based on the JPEG AI verification model. Fig.16 The process of obtaining the predicted value μ is shown.

[0117] like Fig.17 As shown, the input potential tensor is input to the context model network 105, which is based on Fig.11 After the method flow shown in the figure is processed, the first context feature and the second context feature are output, and the first context feature and the second context feature are used as Figure 1 The preset fusion network 106 of the verification model of JPEG AI shown in the figure can also receive the potential tensor after entropy decoding by the super priori decoding network 108 in the verification model of JPEG AI. The potential tensor obtained by decoding The preset fusion network 106 combines the first context feature, the second context feature and the potential tensor By deducing the entropy probability, we can get the predicted value μ corresponding to the input potential tensor.

[0118] In the above embodiment of the present application, when calculating the predicted value corresponding to the input potential tensor of the context model network 105 in the encoder, the context feature is not calculated for the first pixel position in the input potential tensor, and the predicted value of the latent space is calculated based on the context features of the second pixel position and the third pixel position in the input potential tensor, and the target latent tensor. In the process of calculating the predicted value corresponding to the input potential tensor, since there is no need to calculate the context feature corresponding to the first pixel position, the efficiency of calculating the predicted value of the latent space is improved.

[0119] The above method of calculating the predicted value corresponding to the input potential tensor can utilize not only the upper left element of the current pixel, but also the lower right element of the current pixel when calculating the context features of the second pixel position and the third pixel position, thereby improving the accuracy of the calculated context features.

[0120] Fig.18 A flow chart of a method for obtaining an output potential tensor provided in an embodiment of the present application. In this method, a step-by-step context model is applied in a context model network in a decoder to extract context features, and the method includes but is not limited to the following steps:

[0121] 2701. Perform convolution processing on the target latent tensor to calculate the prediction value corresponding to the first pixel position in the latent space of the target latent tensor.

[0122] Among them, the target potential tensor is the code stream (i.e. Figure 1 The potential tensor obtained by decoding the potential tensor after entropy decoding of the bit stream 1) in the above convolution operation is a 1×1 convolution operation.

[0123] In the above latent space, the upper left corner of each 2×2 block corresponds to the first pixel position, the lower right corner of each 2×2 block corresponds to the second pixel position, and the upper right corner and lower left corner of each 2×2 block correspond to the third pixel position.

[0124] 2702. Add the first residual tensor and the predicted value corresponding to the first pixel position to obtain a second latent tensor.

[0125] Among them, there are data at the first pixel position and the second pixel position in the latent space of the second latent tensor, and the data at the third pixel position is zero.

[0126] The first residual tensor is the residual tensor corresponding to the first pixel position in the target residual tensor, and the target residual tensor is the residual tensor for the second bitstream (i.e. Figure 1 The potential tensor obtained after entropy decoding and inverse gain processing of the bitstream 2) in.

[0127] 2703. Perform a convolution operation on the second potential tensor using a second mask convolution kernel to obtain a second context feature corresponding to a second pixel position.

[0128] Among them, the size of the second mask convolution kernel is 3×3.

[0129] For the description of the above step 2703, you can refer to the above related description of step 2705, which will not be repeated here.

[0130] Fig.19AA schematic diagram of obtaining a second context feature provided in an embodiment of the present application. In an embodiment of the present application, the context model performed step by step is embedded into Figure 1 The context model network 105 of the JPEG AI verification model shown in FIG. 1 is implemented based on the JPEG AI verification model. Fig.19A The process of obtaining the second context feature is shown.

[0131] like Fig.19A As shown, the potential tensor after entropy decoding by the super prior scale decoding network 113 The inverse operation of the super-a priori coding network 107 is performed to obtain the probability distribution N(0,σ), and then the second lossless decoder 115 performs entropy decoding on the bitstream 2 based on the N(0,σ) to obtain the residual tensor after entropy decoding, and the second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding to obtain the residual tensor after inverse gain processing. The target potential tensor in the above step 2701 is the potential tensor after entropy decoding by the super-a priori decoding network 108. Decode the potential tensor

[0132] like Fig.19A The preset fusion network 106 shown can perform a 1×1 convolution on the residual tensor after inverse gain processing (i.e., the above-mentioned target potential tensor) to calculate the prediction value μ1 corresponding to the first pixel position, and then add μ1 to the first residual tensor corresponding to the first pixel position in the residual tensor after inverse gain processing to obtain a second potential tensor 2801. Among them, the first residual tensor in step 2701 is the residual tensor corresponding to the first pixel position in the residual tensor after inverse gain processing.

[0133] like Fig.19A As shown, after the second latent tensor 2801 is input into the context model network 105, the context model network 105 can execute the above step 2703 to obtain the second context feature 2802.

[0134] 2704. The second context feature and the target potential tensor are subjected to entropy probability derivation to obtain prediction values ​​corresponding to the two pixel positions.

[0135] The two pixel positions include a first pixel position and a second pixel position.

[0136] 2705. Add the second residual tensor to the predicted values ​​at the two pixel positions to obtain a first latent tensor.

[0137] Among them, the above-mentioned second residual tensor is the residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor.

[0138] 2706. Perform a convolution operation on the first latent tensor using a first mask convolution kernel to obtain a first context feature corresponding to a third pixel position.

[0139] Among them, the size of the first mask convolution kernel is 5×5.

[0140] Fig.19B A schematic diagram of obtaining a first context feature provided in an embodiment of the present application. In an embodiment of the present application, a context model is embedded in a step-by-step manner. Figure 1 The context model network 105 of the JPEG AI verification model shown in FIG. 1 is implemented based on the JPEG AI verification model. Fig.19B The process of obtaining the first context feature is shown.

[0141] like Fig.19B As shown, the potential tensor after entropy decoding by the super prior scale decoding network 113 The inverse operation of the super-a priori coding network 107 is performed to obtain the probability distribution N(0,σ), and then the second lossless decoder 115 performs entropy decoding on the bitstream 2 based on the N(0,σ) to obtain the residual tensor after entropy decoding, and the second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding to obtain the residual tensor after inverse gain processing. The target potential tensor in the above step 2704 is the potential tensor after entropy decoding by the super-a priori decoding network 108. Decode the potential tensor

[0142] Fig.19B The preset fusion network 106 can use the second context feature 2802 and the potential tensor After entropy probability derivation, the predicted values ​​μ of the two pixel positions are obtained 12 , and then the second residual tensor is combined with the predicted values ​​μ at the two pixel locations 12 The addition can obtain the first potential tensor 2803. The second residual tensor in step 2705 is the residual tensor corresponding to the first pixel position and the second pixel position in the residual tensor after the inverse gain processing.

[0143] Fig.19B After the first latent tensor 2803 is input into the context model network 105 , the context model network 105 executes the above step 2706 to obtain the first context feature 2804 .

[0144] 2707. The first context feature, the second context feature and the target potential tensor are subjected to entropy probability derivation to obtain prediction values ​​corresponding to all pixel positions.

[0145] In some embodiments of the present application, the first context feature, the second context feature and the target latent tensor are subjected to entropy probability derivation to obtain prediction values ​​corresponding to all pixel positions, which may include but are not limited to: adding the first context feature and the second context feature to obtain the target context feature; splicing the target context feature and the target latent tensor to obtain a splicing result; and performing convolution processing on the splicing result to obtain prediction values ​​corresponding to all pixel positions.

[0146] 2708. Add the third residual tensor to the predicted values ​​at all pixel locations to obtain an output latent tensor.

[0147] Among them, the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor.

[0148] Fig.19C A schematic diagram of obtaining an output potential tensor provided in an embodiment of the present application. In an embodiment of the present application, the context model is embedded in a step-by-step manner. Figure 1 The context model network 105 of the JPEG AI verification model shown in FIG. 1 is implemented based on the JPEG AI verification model. Fig.19B The process of obtaining the output potential tensor is shown.

[0149] like Fig.19B As shown, the potential tensor after entropy decoding by the super prior scale decoding network 113 The inverse operation of the super-a priori coding network 107 is performed to obtain the probability distribution N(0,σ), and then the second lossless decoder 115 performs entropy decoding on the bitstream 2 based on the N(0,σ) to obtain the residual tensor after entropy decoding, and the second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding to obtain the residual tensor after inverse gain processing. The target potential tensor in the above step 2707 is the potential tensor after entropy decoding by the super-a priori decoding network 108. Decode the potential tensor

[0150] Fig.19B The preset fusion network 106 can use the second context feature 2802 and the first context feature 2804 with the potential tensor after entropy decoding After entropy probability derivation, the predicted value μ of all pixel positions is obtained 123 , and then the third residual tensor is combined with the predicted values ​​μ at all pixel locations 123 The addition can obtain the output potential tensor 2805. The third residual tensor in step 2708 is the residual tensor corresponding to all pixel positions in the residual tensor after the inverse gain processing.

[0151] In the above embodiment of the present application, when obtaining the output potential tensor in the decoder, since there is no need to calculate the context feature corresponding to the first pixel position, the efficiency of calculating the predicted value of the latent space is improved.

[0152] In the above process of obtaining the output potential tensor, when calculating the context features of the second pixel position and the third pixel position, not only the elements to the upper left of the current pixel can be used, but also the elements to the lower right of the current pixel can be used, thereby improving the accuracy of the calculated context features.

[0153] In the embodiment of the present application, Fig. 9 The content adaptive filter 118 in the decoder is used in the JPEG AI verification model shown. Fig.10 When image feature enhancement is performed in the ICCI subnetwork shown, three different ICCI processing modules need to be set, thereby increasing the complexity of the content adaptive filter.

[0154] In order to reduce the complexity of the content adaptive filter, the embodiment of the present application Fig. 9 The content adaptive filter 118 in the decoder shown in the figure is improved, and an adapter-based ICCI sub-network is proposed. The adapter-based ICCI sub-network uses the introduced adapter to adaptively process the Y, U, and V components, and then sends them to a unified ICCI module, so that the content adaptive filter 118 architecture is simplified from three ICCI modules to one ICCI module, which effectively reduces the computational complexity of the JPEG AI verification model and also reduces the complexity of the content adaptive filter 118.

[0155] Fig. 20A A schematic diagram of an ICCI sub-network based on an adapter provided in an embodiment of the present application. Fig. 20A As shown, the adapter-based ICCI sub-network includes: the ICCI sub-network includes a unified ICCI processing module, the input of the ICCI sub-network is the three components Y, U, and V of the image, Fig. 20A The Y component of the image is expressed as The U component of the image is expressed as The V component of the image is expressed as The U component of the image is bicubicly interpolated to obtain a U component of the same shape as Y. The V component of the image is bicubicly interpolated to obtain a V component of the same shape as Y Then the Y component U Component and V component All of them are forward wavelet transformed, and the three components obtained by the transformation are expressed as Fig. 20AComponent 2901, component 2902 and component 2903 in the image are spliced ​​in the channel dimension, and the spliced ​​component 2904 after splicing is input into the adapter 2905. The adapter 2905 performs adaptive processing on the spliced ​​component 2904 to obtain an adaptive enhancement component, and the adaptively enhanced component is input into the ICCI processing module 2906 for enhancement processing to obtain a target enhanced component 2907 after enhancement processing, and then the target enhanced component 2907 is split in the channel dimension to obtain a Y component 2908, a U component 2909 and a V component 2910 after splitting, and then the three split components are inversely wavelet transformed to obtain an output Y component, an output U component and an output V component, and an output image is obtained based on the output Y component, the output U component and the output V component. Fig. 20A The output Y component, output U component and output V component are respectively expressed as

[0156] The adapter 2905 in the adapter-based ICCI subnetwork is used to extract different features of the components based on the attention mechanism, and introduce channel attention and spatial attention at the same time to achieve a combination of channel attention and spatial attention. The adapter 2905 takes the total component (spinning component 2904) after the three components are spliced ​​as input, first passes through the normalization layer, and then performs channel attention and spatial attention on the input components respectively. The results processed by channel attention and spatial attention are added and residually connected, and then the processed results are normalized and fully connected, and then the residual connection is performed again as the output of the adapter module. In the ICCI subnetwork of the JPEG AI image coding framework, only convolution operations are used to extract local features of the components, and no attention is paid to the global features of the components. Therefore, the global features of the components are fully extracted through the structure that combines channel attention and spatial attention in the adapter module.

[0157] Fig. 20B for Fig. 20A FIG. 2 is a schematic diagram of an adapter 2905 in an adapter-based ICCI subnetwork. Fig. 20BAs shown, the adapter 2905 in the adapter-based ICCI subnetwork includes: 1×1 convolution 29051, normalization layer (LayerNormalization) 29052, channel attention block (Channel Attention) 29053, spatial attention block (SpatialAttention) 29054, additive fusion module 29055, normalization layer 29056, multilayer perceptron (MultilayerPerceptron, MLP) 29057, additive fusion module 29058 and 1×1 convolution 29059. The adapter 2905 processes the spliced ​​components by first processing the spliced ​​components 2904 through a 1×1 convolution 29051 and a normalization layer 29052, and then inputting the output features of the normalization layer 29052 into a channel attention block 29053 and a network mutual inductor 29054 based on a sliding window, and then merging the output of the 1×1 convolution 29051, the output of the channel attention block 29053, and the network mutual inductor 29054 based on a sliding window through an addition fusion module 29055. 4 are added and fused, and the addition fusion result of the addition fusion module 29055 is processed in turn by the normalization layer 29056 and the multilayer perceptron 29057, and finally the addition fusion result of the addition fusion module 29055 and the output of the multilayer perceptron 29057 are added and fused by the addition fusion module 29058, and the output of the addition fusion module 29058 is convolved by a 1×1 convolution 29059 to obtain the output (adaptive enhancement component) of the adapter 2905.

[0158] In some embodiments, the spatial attention block may be a sliding window based network transformer (SwinTransformer).

[0159] contrast Fig.10 and Fig. 20A It can be seen that Fig.10 Three ICCI processing modules are used for different components. Fig. 20A In the above, a unified ICCI processing module is used for the three components, so Fig. 20A The adapter-based ICCI sub-network shown reduces the number of ICCI processing modules and reduces the complexity of the ICCI sub-network.

[0160] This application is an application of an adapter-based content adaptive filter in the JPEG AI image compression standard, and proposes an image reconstruction enhancement method. The method acts on Figure 1The content adaptive filter 118 in the verification model of the JPEG AI shown. In an embodiment of the present application, an image reconstruction enhancement method may include: determining a splicing component based on a first Y component, a first U component, and a first V component of an input image; inputting the splicing component into an adapter, and obtaining an adaptive enhancement component output by the adapter, wherein the adapter is obtained by training based on a convolutional layer framework, and during the training process, the features of the three components Y, U, and V are learned to have adaptive processing capabilities for the three components Y, U, and V; inputting the adaptive enhancement component into an ICCI processing module, and obtaining a target enhancement component output by the ICCI processing module; splitting and inversely transforming the target enhancement component in terms of channel dimensions to obtain an output Y component, an output U component, and an output V component; and obtaining an output image based on the output Y component, the output U component, and the output V component.

[0161] The input image refers to the image input to Figure 1 The content adaptive filter 118 is shown as an image. The first Y component, the first U component and the first V component respectively represent the Y, U and V components of the input image.

[0162] Fig.21 The following is a flow chart of an image reconstruction enhancement method provided in an embodiment of the present application. Fig.21 As shown, the method comprises the following steps:

[0163] 3001. Use bicubic interpolation on a first U component and a first V component of an input image to obtain a second U component and a second V component having the same shape as the first Y component of the input image.

[0164] The size of the first Y component is S times the size of the first U component and the first V component. In the JPEG AI verification model, S=2.

[0165] The above-mentioned determination of the splicing component based on the first Y component, the first U component and the first V component of the input image can be specifically implemented through steps 3001 and 3002.

[0166] The first Y component of the above input image has a different shape from the first U component and the first V component. Since the U component, V component and Y component need to be spliced ​​in the channel dimension later, the input images of the first U component and the first V component need to be adjusted by bicubic interpolation to obtain the second U component and the second V component that are consistent with the shape of the first Y component of the input image.

[0167] 3002. Perform forward wavelet transform on the first Y component, the second U component, and the second V component respectively, and perform splicing operation on the transform results to obtain splicing components.

[0168] The above processes 3001 and 3002 can be expressed as:

[0169]

[0170]

[0171]

[0172]

[0173] Among them, in the above formulas (1) to (4), and are the three components of the input image, namely, the first Y component, the first U component and the first V component, BiCubic() represents bicubic interpolation, DWT() represents forward wavelet transform, and Concatenation() represents concatenation function; Indicates the transformation result of the forward wavelet transform of the first Y component; Indicates that the first U component is subjected to a forward wavelet transform to obtain a second U component, and the result of the forward wavelet transform performed on the second U component; Indicates the result of performing bicubic interpolation on the first V component to obtain the second V component, and performing forward wavelet transform on the second V component; x YUV Represents the above concatenated components.

[0174] 3003. Input the spliced ​​component into an adapter, and obtain the adaptive enhancement component output by the adapter.

[0175] In the embodiment of the present application, the main function of the adapter is to learn the characteristics of the three components Y, U, and V to achieve adaptive processing for the three components Y, U, and V.

[0176] The structure of the above adapter can be expressed as:

[0177] r(x YUV ; θ)=AB Τ x YUV (5)

[0178] In the above formula (5), A and B represent matrices, which are learnable parameters of the adapter. A, B ∈ R C×M A and B are of length C (and x YUV The number of channels is the same as that of the matrix), and the width is M; x YUV is the input of the adapter, x YUV ∈R C×H×W , x YUV ∈R C×H×W Represents x YUVThe feature has a channel number of C, a height of H, and a width of W. It can be seen that the lengths of A and B are related to x YUV The number of channels is the same. θ=[A,B] represents the adapter parameters, r(x YUv ; θ) represents the adapter.

[0179] The adapter is trained based on a convolutional layer framework, and during the training process, the features of the three components Y, U, and V are learned to have adaptive processing capabilities for the three components Y, U, and V.

[0180] In some embodiments of the present application, the training set of the framework can be a JPEG AI data set and images compressed at different bit rates corresponding to the JPEG AI data set. During the training process, the input of the adapter module is the total component after the Y, U, and V components of the image are spliced. The total component is sent to the adapter module to extract the features of each component, and then enters the ICCI processing module for feature extraction. The output of the ICCI processing module is the total component after processing, that is, the target enhancement component, and then the target enhancement component is split and inverse wavelet transformed to obtain the processed Y, U, and V components.

[0181] The JPEG AI dataset contains 5,284 images, including people, animals, landscapes, buildings and other types of images. The JPEG AI verification model and ICCI processing module are trained using the JPEG AI dataset. We use the JPEG AI dataset to train the ICCI module. Each image in the dataset can extract three components, Y, U, and V. The training process also uses the three components of Y, U, and V as the input of the ICCI module at the same time, that is, the Y, U, and V components of each image in the JPEG AI dataset are used for training.

[0182] After the training is completed and the adapter is obtained, the adapter can extract different features of the components according to the different inputs (three different components of Y, U, and V). After unified processing by the ICCI processing module, it can realize the input of three different components of Y, U, and V, realize different feature extraction, and achieve adaptation for the three components, thereby achieving the purpose of simplifying the three ICCI modules and reducing the complexity of the model.

[0183] 3004. Input the adaptive enhancement component to the ICCI processing module, and obtain the target enhancement component output by the ICCI processing module.

[0184] The ICCI processing module may be: Fig. 20A The ICCI processing module 2905 shown in FIG. Fig.10The ICCI processing modules 1008, 1009 and 1010 shown in FIG. 1 are different, as follows:

[0185] Fig. 22 A method for Fig.10 Schematic diagram of the ICCI processing module.

[0186] like Fig. 22 As shown in FIG. 1 , the ICCI processing module includes: 3×3 convolution 3101, batch normalization layer 3102, ReLU activation function 3103, Nb 1D residual blocks (one-dimensional residual blocks) 3104, and 3×3 convolution 3105. Among them, the Batch Normalization layer is used to standardize the data. The normalized data can make the network converge faster. The ICCI processing module includes two inputs, one of which is as follows: Fig.10 The component 1004 in the ICCI processing module, and the other component 100X corresponding to the ICCI processing module, wherein the ICCI processing module is Fig.10 When the ICCI processing module 1008 is in the Fig.10 Component 1001 shown in , and Nb = 8; in the ICCI processing module is Fig.10 When the ICCI processing module 1009 in the Fig.10 Component 1002 shown in FIG. 1 , Nb=13; in the ICCI processing module, Fig.10 When the ICCI processing module 1010 in FIG. 1 is used, the component 100X is Fig.10 Component 1003 shown in , Nb = 13. The result of multiplying component 1001 by the scaling factor γ after being processed by 3×3 convolution 3101, batch normalization layer 3102, ReLU activation function 3103, Nb 1D residual blocks 3104 and 3×3 convolution 3105 is summed with component 100X to obtain the output of the ICCI processing module. Among them, the scaling factor γ is a parameter in the ICCI processing module, which can be obtained through the training process of the ICCI processing module.

[0187] Fig.23 A method provided in the embodiment of the present application Fig. 20ASchematic diagram of the ICCI processing module in FIG. The ICCI processing module includes: 3×3 convolution 3201, batch normalization layer 3202, ReLU activation function 3203, Nb 1D residual blocks 3204 and 3×3 convolution 3215. The ICCI processing module 1010 includes an input path, which uses the adaptive enhancement component processed by the adapter as the input of the ICCI module, and passes through a sequence of 3×3 convolution 3201, batch normalization layer 3202, Nb 1D residual blocks and 3×3 convolution 3215. After being processed by the sequence, it is multiplied by the scaling factor γ as the output of the ICCI module. Fig.23 Nb in can be a fixed value. Wherein, the scaling factor γ is a parameter in the ICCI processing module, and the parameter can be obtained through the training process of the ICCI processing module.

[0188] In some embodiments of the present application, Fig.23 The Nb in can be an integer greater than 8 and less than 13.

[0189] Compare the above Fig. 22 and Fig.23 It can be seen that Fig. 22 The ICCI processing module shown in FIG. 1 has two inputs, and the number of 1D residual blocks in different ICCI processing modules is different. Fig.23 The ICCI processing module shown has only one input, and the number of 1D residual blocks in the ICCI processing module is a fixed value. Fig.23 The ICCI processing module architecture shown is similar to Fig. 22 The architecture of the ICCI processing module shown in is different.

[0190] Above Fig.23 The number of 1D residual blocks in can be based on Fig. 22 The number of 1D residual blocks in is determined by Fig. 22 The number of 1D residual blocks for different components is different. For the Y component, the number of 1D residual blocks is 8; for the U component, the number of 1D residual blocks is 13; for the V component, the number of 1D residual blocks is 13. Then in some embodiments of the present application, Fig.23 The number of 1D residual blocks can be calculated according to Fig. 22 The number of 1D residual blocks in is determined by 8, 13 and 13, Fig.23 The number of 1D residual blocks in can be any integer between 8 and 13. For example, the number of 1D residual blocks can be 10.

[0191] In some embodiments of the present application, the above Fig.23 The number of 1D residual blocks in can be based on Fig. 22The number of 1D residual blocks in 8, 13 and 13 is determined by the average value, for example, it can be rounded up based on the average value to determine it as 12, or it can be rounded down based on the average value to determine it as 11.

[0192] Above Fig. 22 and Fig.23 The structures of the ID residual blocks in are the same. Fig.24 A schematic diagram of a 1D residual block provided in an embodiment of the present application. Fig.24 As shown, the 1D residual block includes convolution 3301, batch normalization layer (BatchNormalization, BN) 3302, ReLU activation function 3303, convolution 3304 and batch normalization layer 3305. Among them, after the input of the ID residual block is processed by convolution 3301, batch normalization layer 3302, ReLU activation function 3303, convolution 3304 and batch normalization layer 3305 and the scaling factor γ n Multiply them, and add the multiplication result to the input of the ID residual block to get the output of the ID residual block. n Represents the scaling factor of the nth ID residual block. There is a scaling factor for each ID residual block. Therefore, these scaling factors can be obtained through the training process of the ICCI processing module. The 1D residual block can improve the ability to express image features, extract effective image features, and solve the problems of gradient disappearance and network degradation in the image encoding and decoding process.

[0193] 3005. Split the target enhancement component in channel dimension to obtain an enhanced Y component, an enhanced U component and an enhanced V component.

[0194] 3006. Perform inverse wavelet transform on the enhanced Y component, the enhanced U component and the enhanced V component respectively to obtain an output Y component, an output U component and an output V component, and obtain an output image based on the output Y component, the output U component and the output V component.

[0195] The above-mentioned splitting and inverse transformation of the target enhancement component in the channel dimension to obtain the output Y component, the output U component and the output V component can be implemented through steps 3005 and 3006.

[0196] Among them, after being processed by the ICCI module, the obtained target enhancement component is split into channel dimensions to obtain enhanced Y component, enhanced U component and enhanced V component, and then the three components are inversely transformed to obtain the reconstructed output Y component component, output U component and output V component. The above operation can be expressed as:

[0197]

[0198]

[0199]

[0200]

[0201] Among them, in the above formulas (6) to (9), represents the target enhancement component, Split represents the channel dimension splitting, Y, U, V represents the enhanced Y component, enhanced U component and enhanced V component obtained by splitting, IDWT represents the inverse wavelet transform processing, They represent the output Y component, output U component and output V component respectively.

[0202] The image reconstruction enhancement method provided in the embodiment of the present application uses an adapter-based ICCI sub-network in the content adaptive filter in the JPEG AI verification model, and introduces an adapter in the adapter-based ICCI sub-network to adaptively process the Y, U, and V components, so as to enhance the input image according to the characteristics of each Y, U, and V component, thereby eliminating the need to process the three components separately through different ICCI modules, and instead using a unified ICCI module for processing, thereby simplifying the content adaptive filter architecture from three ICCI modules to one ICCI module, effectively reducing the computational complexity of the JPEG AI verification model.

[0203] Based on the application of the content adaptive filter of the adapter in the JPEG AI image compression standard, the embodiment of the present application proposes another image reconstruction enhancement method, which acts on Figure 1 The content adaptive filter 118 in the verification model of the JPEG AI shown. In an embodiment of the present application, the other image reconstruction enhancement method may include: performing forward wavelet transform on the first Y component, the first U component and the first V component of the input image respectively to obtain the Y component transform result, the U component transform result and the V component transform result; splicing the Y component transform result, the U component transform result and the V component transform result to obtain the spliced ​​component; inputting the spliced ​​component into the adapter, and obtaining the adaptive enhancement component output by the adapter; inputting the adaptive enhancement component, the Y component transform result, the U component transform result and the V component transform result into the ICCI processing module, and obtaining the Y enhancement component, the U enhancement component and the V enhancement component output by the ICCI processing module; performing inverse wavelet transform based on the Y enhancement component, the U enhancement component and the V enhancement component respectively to obtain the output Y component, the output U component and the output V component; obtaining the output image based on the output Y component, the output U component and the output V component.

[0204] The input image refers to the image input to Figure 1The content adaptive filter 118 is shown as an image. The first Y component, the first U component and the first V component respectively represent the Y, U and V components of the input image.

[0205] Fig.25 A flowchart of another image reconstruction enhancement method provided in an embodiment of the present application is shown below. Fig.25 As shown, the method comprises the following steps:

[0206] 3401. Use bicubic interpolation on a first U component and a first V component of an input image to obtain a second U component and a second V component having the same shape as the first Y component of the input image.

[0207] The first Y component of the above input image has a different shape from the first U component and the first V component. Since channel dimension stitching is required later, the input images of the first U component and the first V component need to be adjusted by bicubic interpolation to obtain the second U component and the second V component that are consistent with the shape of the first Y component of the input image.

[0208] 3402. Perform forward wavelet transform on the first Y component, the second U component, and the second V component respectively to obtain a Y component transformation result, a U component transformation result, and a V component transformation result.

[0209] The forward wavelet transform of the first Y component, the first U component and the first V component of the input image to obtain the Y component transformation result, the U component transformation result and the V component transformation result can be achieved through the above steps 3401 and 3402.

[0210] The above-mentioned Y component transformation result refers to the transformation result obtained after performing forward wavelet transformation on the first Y component, the U component transformation result refers to the transformation result obtained after performing forward wavelet transformation on the second U component, and the V component transformation result refers to the transformation result obtained after performing forward wavelet transformation on the second V component.

[0211] 3403. Concatenate the Y component transformation result, the U component transformation result, and the V component transformation result to obtain a concatenated component.

[0212] 3404. Input the spliced ​​component into an adapter, and obtain the adaptive enhancement component output by the adapter.

[0213] For the description of the above steps 3401, 3402, 3403 and 3404, you can refer to the relevant description of the above steps 3001, 3002 and 3003. The processes implemented by these two parts of steps are similar and will not be repeated here.

[0214] The adapter in step 3404 is the same as the adapter described in the above step 3003 and will not be described again here.

[0215] 3405. Input the adaptive enhancement component, the Y component transformation result, the U component transformation result and the V component transformation result to the ICCI processing module, and obtain the Y enhancement component, the U enhancement component and the V enhancement component output by the ICCI processing module.

[0216] Among them, inputting the adaptive enhancement component, the Y component transformation result, the U component transformation result and the V component transformation result into the ICCI processing module, and obtaining the Y enhancement component, the U enhancement component and the V enhancement component output by the ICCI processing module can include: inputting the adaptive enhancement component and the Y component transformation result into the ICCI processing module, and obtaining the Y enhancement component output by the ICCI processing module; inputting the adaptive enhancement component and the U component transformation result into the ICCI processing module, and obtaining the U enhancement component output by the ICCI processing module; inputting the adaptive enhancement component and the V component transformation result into the ICCI processing module, and obtaining the V enhancement component output by the ICCI processing module.

[0217] The above-mentioned ICCI processing module can perform enhancement processing on the Y component based on the Y component transformation result and the adaptive enhancement component to obtain the Y enhanced component. The ICCI processing module can also perform enhancement processing on the U component based on the U component transformation result and the adaptive enhancement component to obtain the U enhanced component. It can also perform enhancement processing on the V component based on the V component transformation result and the adaptive enhancement component to obtain the V enhanced component.

[0218] Fig.25 In the image reconstruction enhancement method shown in FIG. 1 , there is only one ICCI processing module in the adapter-based ICCI subnetwork, but the ICCI subnetwork corresponding to the method is different from the above-mentioned ICCI subnetwork. Fig. 20A The existence of the ICCI subnetwork shown is different, and the difference lies in: the input of the ICCI processing module is different. At this time, the input not only includes the adaptive enhancement component after adapter processing, but also includes the Y component transformation result, the U component transformation result and the V component transformation result, and the output of the ICCI processing module is also changed from the original one to three, namely the Y enhancement component, the U enhancement component and the V enhancement component.

[0219] Fig.26 A schematic diagram of another adapter-based ICCI sub-network provided in an embodiment of the present application. Fig.26 The ICCI subnetwork shown is Fig. 20A Compared with the schematic diagram of the ICCI sub-network shown in FIG. 1 , the similarities are: the ICCI sub-network includes a unified ICCI processing module, the input of the ICCI sub-network is the Y, U, and V components of the image, Fig.26 The Y component of the image is expressed as The U component of the image is expressed as The V component of the image is expressed as The U component of the image is bicubicly interpolated to obtain a U component of the same shape as Y. The V component of the image is bicubicly interpolated to obtain a V component of the same shape as Y Then the Y component U Component and V component All components are forward wavelet transformed to obtain component 2901, component 2902 and component 2903. Component 2901, component 2902 and component 2903 are spliced ​​in the channel dimension, and the spliced ​​component 2904 is input into the adapter 2905. The adapter 2905 performs adaptive processing on the spliced ​​component 2904 to obtain an adaptive enhancement component, and the adaptive enhancement component is input into the ICCI processing module 2906 for enhancement processing.

[0220] Fig.26 The ICCI subnetwork shown is Fig. 20A The difference between the schematic diagram of the ICCI sub-network shown is that: Fig.26 Component 2901, component 2902 and component 2903 are also input into the ICCI processing module 2906, wherein the ICCI processing module 2906 can obtain the Y enhanced component 3501 based on component 2901 and the adaptive enhanced component, the ICCI processing module 2906 can obtain the U enhanced component 3502 based on component 2902 and the adaptive enhanced component, and the ICCI processing module 2906 can also obtain the V enhanced component 3503 based on component 2903 and the adaptive enhanced component.

[0221] Then, the Y enhanced component 3501, the U enhanced component 3502, and the V enhanced component 3503 are respectively subjected to inverse wavelet transformation to obtain an output Y component, an output U component, and an output V component, and an output image is obtained based on the output Y component, the output U component, and the output V component. Fig.26 The output Y component, output U component and output V component are respectively expressed as

[0222] exist Fig.26 In the ICCI subnetwork shown in FIG. 1 , the structure of the ICCI processing module is similar to that of Fig. 22 The structure shown is similar, but Fig.26 In the ICCI subnetwork shown, since there is only one ICCI processing module, Fig.26 Adopted in Fig. 22 When the ICCI processing module is shown, the number Nb of 1D residual blocks is a constant value.

[0223] Fig.26 In the ICCI subnetwork shown, the number Nb of 1D residual blocks in the ICCI processing module can be an integer greater than 8 and less than 13.

[0224] in, Fig.26 The number of 1D residual blocks in can be based on Fig. 22 The number of 1D residual blocks in 8, 13 and 13 is determined by the average value. For example, the average value may be rounded up to determine 12, or the average value may be rounded down to determine 11.

[0225] 3406. Perform inverse wavelet transform on the Y enhancement component, the U enhancement component, and the V enhancement component to obtain an output Y component, an output U component, and an output V component.

[0226] Among them, the Y enhancement component is subjected to inverse wavelet transform to obtain the output Y component; the U enhancement component is subjected to inverse wavelet transform to obtain the output U component; the V enhancement component is subjected to inverse wavelet transform to obtain the output V component.

[0227] 3407. Obtain an output image based on the output Y component, the output U component, and the output V component.

[0228] The image reconstruction enhancement method provided in the embodiment of the present application uses an adapter-based ICCI subnetwork in the content adaptive filter in the JPEG AI verification model, and introduces an adapter in the adapter-based ICCI subnetwork to adaptively process the Y, U, and V components to enhance the input image according to the characteristics of each Y, U, and V component, so that it is no longer necessary to process the three components separately through different ICCI module processing, but to use a unified ICCI module processing for processing, so that the content adaptive filter architecture is simplified from three ICCI modules to one ICCI module, effectively reducing the computational complexity of the JPEG AI verification model. In some embodiments of the method, when using a unified ICCI module for processing, each Y, U, and V component and the result after the adapter-based adaptive processing are used as inputs for ICCI module processing, so that the ICCI module can obtain the Y enhanced component, U enhanced component, and V enhanced component corresponding to Y, U, and V, respectively, so that the characteristics of different components can be further considered, so that the final enhanced image includes more features of each component.

[0229] In some embodiments of the present application, before determining the splicing component based on the first Y component, the first U component, and the first V component of the input image, the following scheme may also be performed: performing a 1×1 convolution on the target potential tensor to calculate a predicted value corresponding to the first pixel position in the latent space of the target potential tensor, where the target potential tensor is a potential tensor obtained after entropy decoding and super-prior decoding of the first bitstream; adding the first residual tensor to the predicted value corresponding to the first pixel position to obtain a second potential tensor, where the first residual tensor is the predicted value corresponding to the first pixel position in the target residual tensor. The target residual tensor is a residual tensor obtained after entropy decoding and inverse gain processing of the second bitstream; the second potential tensor is convolved by a 3×3 mask convolution to obtain a second context feature corresponding to the second pixel position; the second context feature and the target potential tensor are subjected to entropy probability derivation to obtain a prediction value corresponding to the two pixel positions; the second residual tensor is added to the prediction value of the two pixel positions to obtain the first potential tensor, and the second residual tensor is a residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor;

[0230] The first latent tensor is convolved with a 5×5 mask convolution to obtain a first context feature corresponding to the third pixel position. The first context feature, the second context feature, and the target latent tensor are subjected to entropy probability derivation to obtain prediction values ​​corresponding to all pixel positions. Based on the prediction values ​​of all pixel positions, the input image is determined.

[0231] Wherein, in the latent space, the upper left corner of each 2×2 block corresponds to the first pixel position, the lower right corner of each 2×2 block corresponds to the second pixel position, and the upper right corner and lower left corner of each 2×2 block correspond to the third pixel position;

[0232] In some embodiments of the present application, determining the input image based on the predicted values ​​of all pixel positions may include but is not limited to: adding the third residual tensor to the predicted values ​​of all pixel positions to obtain an output potential tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor; performing a synthetic transformation on the output potential tensor to obtain the input image.

[0233] That is, before determining the splicing component based on the first Y component, the first U component, and the first V component of the input image, the embodiment of the present application further provides the following Fig.18 A method for determining the predicted value is shown in Fig.18 The method shown calculates the prediction values ​​corresponding to all pixel positions and obtains the output potential tensor based on the prediction values ​​of all pixel positions, and then the input image can be further determined.

[0234] Among them, based on the predicted values ​​of all pixel positions, determining the input image may include but is not limited to, adding the above-mentioned third residual tensor to the predicted values ​​of all pixel positions to obtain an output potential tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor, and performing a synthetic transformation on the output potential tensor to obtain the above-mentioned input image.

[0235] The above synthetic transformation process can be Fig. 9 The synthesis transformation network 117 in is implemented.

[0236] In the present application embodiment, by Fig.18 According to the method for determining the prediction value shown, after calculating the prediction values ​​corresponding to all pixel positions, the input image can be determined based on the prediction values ​​of all pixel positions, and the reconstruction enhancement processing is performed based on the input image based on the above-mentioned image reconstruction enhancement method to obtain the final output image. Through such a scheme, on the one hand, since when calculating the context features of the second pixel position and the third pixel position, not only the elements at the upper left of the current pixel but also the elements at the lower right of the current pixel can be used, the accuracy of the calculated context features is improved, so a more accurate input image can be obtained. On the other hand, when reconstructing and enhancing the input image, an adapter-based ICCI subnetwork is used in the content adaptive filter, and an adapter is introduced in the adapter-based ICCI subnetwork to adaptively process the Y, U, and V components to enhance the input image according to the characteristics of each Y, U, and V component, so that it is no longer necessary to process the three components separately through different ICCI module processing, but a unified ICCI module processing is used for processing, thereby effectively reducing the computational complexity of the JPEG AI verification model.

[0237] Some embodiments of the present application provide a decoding device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-mentioned prediction value determination method when executed by the processor.

[0238] Fig. 27 A schematic diagram of the hardware structure of a decoding device provided for some embodiments of the present application.

[0239] like Fig. 27 As shown, the decoding device provided by the embodiment of the present disclosure includes: a processor 271 and a memory 272;

[0240] In some embodiments, the processor 271 is configured to: perform forward wavelet transform on the first Y component, the first U component and the first V component of the input image, respectively, to obtain a Y component transform result, a U component transform result and a V component transform result; splice the Y component transform result, the U component transform result and the V component transform result to obtain a spliced ​​component; input the spliced ​​component into an adapter, and obtain an adaptive enhancement component output by the adapter, wherein the adapter is obtained by training based on a convolutional layer framework, and has adaptive processing capabilities for the three components Y, U and V by learning the features of the three components Y, U and V during the training process; input the adaptive enhancement component, the Y component transform result, the U component transform result and the V component transform result into an ICCI processing module, and obtain the Y enhancement component, U enhancement component and V enhancement component output by the ICCI processing module; perform inverse wavelet transform based on the Y enhancement component, the U enhancement component and the V enhancement component, respectively, to obtain an output Y component, an output U component and an output V component; and obtain an output image based on the output Y component, the output U component and the output V component.

[0241] In some embodiments of the present application, the processor 271 is specifically configured to: perform forward wavelet transform on the first Y component, the first U component and the first V component of the input image, respectively, to obtain a Y component transformation result, a U component transformation result and a V component transformation result, including: using bicubic interpolation on the first U component and the first V component of the input image to obtain a second U component and a second V component that are consistent with the shape of the first Y component of the input image; perform forward wavelet transform on the first Y component, the second U component and the second V component, respectively, to obtain the Y component transformation result, the U component transformation result and the V component transformation result.

[0242] In some embodiments of the present application, the processor 271 is specifically configured to: input the adaptive enhancement component, the Y component transformation result, the U component transformation result and the V component transformation result into the ICCI processing module, and obtain the Y enhancement component, U enhancement component and V enhancement component output by the ICCI processing module, including: inputting the adaptive enhancement component and the Y component transformation result into the ICCI processing module, and obtaining the Y enhancement component output by the ICCI processing module; inputting the adaptive enhancement component and the U component transformation result into the ICCI processing module, and obtaining the U enhancement component output by the ICCI processing module; inputting the adaptive enhancement component and the V component transformation result into the ICCI processing module, and obtaining the V enhancement component output by the ICCI processing module.

[0243] In some embodiments of the present application, the training set of the framework for training the convolutional layer includes a JPEG AI data set, and images compressed at different bit rates corresponding to the JPEG AI data.

[0244] In some embodiments of the present application, the ICCI processing module includes: 3×3 convolution, batch normalization layer, ReLU activation function and Nb 1D residual blocks.

[0245] In some embodiments of the present application, Nb is greater than 8 and less than 13.

[0246] In some embodiments of the present application, the processor 271 is configured to: before determining the splicing component based on the first Y component, the first U component and the first V component of the input image, perform a 1×1 convolution on the target potential tensor to calculate the prediction value corresponding to the first pixel position in the latent space of the target potential tensor, where the target potential tensor is the potential tensor obtained after entropy decoding and super-prior decoding of the first bitstream; add the first residual tensor to the prediction value corresponding to the first pixel position to obtain a second potential tensor, where the first residual tensor is the residual tensor corresponding to the first pixel position in the target residual tensor, and the target residual tensor is the residual tensor obtained after entropy decoding and inverse gain processing of the second bitstream; perform a convolution operation on the second potential tensor through a 3×3 mask convolution to obtain a second context feature corresponding to the second pixel position; add the second context feature to the target potential tensor The first latent tensor is convolved with a 5×5 mask convolution to obtain a first context feature corresponding to a third pixel position; the first context feature, the second context feature and the target latent tensor are convolved with an entropy probability to obtain prediction values ​​corresponding to all pixel positions; the input image is determined based on the prediction values ​​of all pixel positions; in the latent space, the upper left corner of each 2×2 block corresponds to the first pixel position, the lower right corner of each 2×2 block corresponds to the second pixel position, and the upper right corner and lower left corner of each 2×2 block correspond to the third pixel position.

[0247] In some embodiments of the present application, the processor 271 is specifically configured to: determine the input image based on the predicted values ​​of all pixel positions, including: adding the third residual tensor to the predicted values ​​of all pixel positions to obtain an output potential tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor; and perform a synthetic transformation on the output potential tensor to obtain the input image.

[0248] Some embodiments of the present application provide a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, one or more of the image feature processing method, the prediction value determination method and the image reconstruction enhancement method in the above method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. In the present application, computer-readable media include permanent and non-permanent, removable and non-removable storage media. The storage medium can implement information storage by any method or technology, and the information can be a computer-readable instruction, a data structure, a module of a program or other data. Among them, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0249] Some embodiments of the present application provide a computer program product, which stores a computer program. When the computer program is executed by a processor, it implements one or more of the image feature processing method, prediction value determination method and image reconstruction enhancement method in the above method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0250] In the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0251] In the present application, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM.

[0252] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image reconstruction enhancement method, It is characterized in that include: Performing forward wavelet transform on the first Y component, the first U component and the first V component of the input image respectively to obtain a Y component transform result, a U component transform result and a V component transform result; splicing the Y component transformation result, the U component transformation result and the V component transformation result to obtain a spliced ​​component; Input the spliced ​​component into an adapter, and obtain an adaptive enhancement component output by the adapter, wherein the adapter is trained based on a convolutional layer framework, and has adaptive processing capabilities for the three components Y, U, and V by learning the features of the three components Y, U, and V during the training process; Inputting the adaptive enhancement component, the Y component transformation result, the U component transformation result and the V component transformation result into an ICCI processing module, and obtaining the Y enhancement component, the U enhancement component and the V enhancement component output by the ICCI processing module; Perform inverse wavelet transform on the Y enhancement component, the U enhancement component and the V enhancement component to obtain an output Y component, an output U component and an output V component; An output image is obtained based on the output Y component, the output U component and the output V component.

2. The method according to claim 1, It is characterized in that The performing forward wavelet transform on the first Y component, the first U component and the first V component of the input image respectively to obtain a Y component transform result, a U component transform result and a V component transform result includes: Using bicubic interpolation on a first U component and a first V component of the input image to obtain a second U component and a second V component that are consistent in shape with the first Y component of the input image; The first Y component, the second U component and the second V component are respectively subjected to forward wavelet transformation to obtain the Y component transformation result, the U component transformation result and the V component transformation result.

3. The method according to claim 1, It is characterized in that The step of inputting the adaptive enhancement component, the Y component transformation result, the U component transformation result, and the V component transformation result into an ICCI processing module, and obtaining the Y enhancement component, the U enhancement component, and the V enhancement component output by the ICCI processing module comprises: Inputting the adaptive enhancement component and the Y component transformation result into an ICCI processing module to obtain the Y enhancement component output by the ICCI processing module; Inputting the adaptive enhancement component and the U component transformation result into an ICCI processing module to obtain the U enhancement component output by the ICCI processing module; The adaptive enhancement component and the V component transformation result are input into the ICCI processing module to obtain the V enhancement component output by the ICCI processing module.

4. The method according to claim 1, It is characterized in that The training set of the framework for training the convolutional layer includes a JPEG AI data set, and images compressed at different bit rates corresponding to the JPEG AI data.

5. The method according to claim 1, It is characterized in that The ICCI processing module includes: 3×3 convolution, batch normalization layer, ReLU activation function and Nb 1D residual blocks.

6. The method according to claim 5, It is characterized in that The Nb is greater than 8 and less than 13.

7. The method according to claim 1, It is characterized in that Before determining the splicing component based on the first Y component, the first U component and the first V component of the input image, the method further includes: Performing a 1×1 convolution on a target latent tensor to calculate a prediction value corresponding to a first pixel position in a latent space of the target latent tensor, wherein the target latent tensor is a latent tensor obtained after entropy decoding and super-a priori decoding processing are performed on the first bitstream; Adding a first residual tensor and a predicted value corresponding to the first pixel position to obtain a second potential tensor, wherein the first residual tensor is a residual tensor corresponding to the first pixel position in a target residual tensor, and the target residual tensor is a residual tensor obtained after entropy decoding and inverse gain processing of the second bitstream; Performing a convolution operation on the second latent tensor through a 3×3 masked convolution to obtain a second context feature corresponding to a second pixel position; The second context feature and the target potential tensor are subjected to entropy probability derivation to obtain prediction values ​​corresponding to two pixel positions; Adding a second residual tensor to the predicted values ​​of two pixel positions to obtain a first potential tensor, wherein the second residual tensor is a residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor; Performing a convolution operation on the first latent tensor through a 5×5 masked convolution to obtain a first context feature corresponding to a third pixel position; The first context feature, the second context feature and the target potential tensor are subjected to entropy probability derivation to obtain prediction values ​​corresponding to all pixel positions; Wherein, in the latent space, the upper left corner of each 2×2 block corresponds to the first pixel position, the lower right corner of each 2×2 block corresponds to the second pixel position, and the upper right corner and lower left corner of each 2×2 block correspond to the third pixel position; The input image is determined based on the predicted values ​​of all pixel positions.

8. The method according to claim 7, It is characterized in that The step of determining the input image based on the predicted values ​​of all pixel positions comprises: Adding the third residual tensor to the predicted values ​​of all pixel positions to obtain an output potential tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor; The output potential tensor is subjected to a synthetic transformation process to obtain the input image.

9. A decoding device, It is characterized in that include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image reconstruction enhancement method according to any one of claims 1 to 8.

10. A computer-readable storage medium, It is characterized in that include: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image reconstruction enhancement method according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Controlling spatial audio coding parameters as a function of auditory events

    CN101410889A

  • Image enhancement method and device fusing wavelet transform and attention mechanism

    CN115170410A

  • Efficient End-to-End Single Layer Inverse Display Management Coding

    US20210150680A1