Image reconstruction enhancement method, decoding device and storage medium

Abstract

The present disclosure relates to an image reconstruction enhancement method, a decoding device and a storage medium, and particularly relates to the field of image coding technology. The method comprises: performing forward wavelet transform on a first Y component, a first U component and a first V component of an input image respectively to obtain Y component transformation results, U component transformation results and V component transformation results, and splicing the Y component transformation results, the U component transformation results and the V component transformation results to obtain spliced components; inputting the spliced components into an adapter and obtaining adaptive enhancement components output by the adapter; inputting the adaptive enhancement components, the Y component transformation results, the U component transformation results and the V component transformation results into an ICCI processing module, and obtaining Y enhancement components, U enhancement components and V enhancement components output by the ICCI processing module; performing inverse wavelet transform on the Y enhancement components, the U enhancement components and the V enhancement components respectively to obtain output Y components, output U components and output V components; and obtaining an output image based on the output Y components, the output U components and the output V components.

Description

[0001] This application claims priority to Chinese Patent Application No. 2023106645409, filed with the Chinese Patent Office on June 6, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of image encoding and decoding technology, and in particular to an image reconstruction and enhancement method, decoding device and storage medium. Background Technology

[0003] Image compression is one of the fundamental tasks in computer vision and image processing. The development of lossy image compression techniques has brought benefits to various fields, significantly reducing the amount of data required for image storage and transmission; however, the image quality after reconstruction may slightly decrease. In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression techniques. JPEG AI is a learning-based image coding standard created by the Joint Picture Experts Group (JPEG). JPEG AI's validated model, designed for human vision, achieves a lower bitrate compared to traditional image coding while maintaining the same subjective quality.

[0004] The JPEG AI verification model uses a content-adaptive filter for image feature enhancement. This content-adaptive filter considers the characteristics of different Y, U, and V components and sets up three different ICCI processing modules to process the features of the three channels Y, U, and V in the image respectively, and enhance the Y, U, and V components. However, this requires setting up three different ICCI processing modules, thus increasing the complexity of the content-adaptive filter. Summary of the Invention

[0005] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides an image reconstruction and enhancement method, a decoding device, and a storage medium.

[0006] To achieve the above objectives, some embodiments of this application provide the following technical solutions: Firstly, an image reconstruction and enhancement method is provided, including: Forward wavelet transforms are performed on the first Y component, the first U component, and the first V component of the input image to obtain the Y component transform result, the U component transform result, and the V component transform result, respectively. The transformation results of the Y component, the U component, and the V component are concatenated to obtain the concatenated component. The spliced ​​components are input into the adapter, and the adaptive enhanced components output by the adapter are obtained. The adapter is trained based on a convolutional layer framework. During the training process, it learns the features of the three components Y, U, and V to have adaptive processing capabilities for the three components Y, U, and V. The adaptive enhancement component, the Y component transformation result, the U component transformation result, and the V component transformation result are input to the ICCI processing module, and the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module are obtained. Inverse wavelet transforms are performed on the Y-enhanced component, the U-enhanced component, and the V-enhanced component respectively to obtain the output Y-component, the output U-component, and the output V-component; The output image is obtained based on the output Y component, the output U component, and the output V component.

[0007] Secondly, a decoding device is provided, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image reconstruction enhancement method as described in the first aspect or any alternative embodiment thereof.

[0008] Thirdly, a computer-readable storage medium is provided, comprising: storing a computer program on the computer-readable storage medium, wherein the computer program, when executed by a processor, implements the image reconstruction enhancement method as described in the first aspect or any alternative embodiment thereof.

[0009] Fourthly, a computer program product is provided, wherein a computer program is stored in the computer program product, and when the computer program is executed by a processor, it implements the image reconstruction and enhancement method as described in the first aspect or any of its optional embodiments.

[0010] The image reconstruction and enhancement method provided in this application uses an adapter-based ICCI sub-network in the content adaptive filter of the JPEG AI verification model. This adapter-based ICCI sub-network introduces an adapter to adaptively process the Y, U, and V components, enhancing the input image based on the features of each Y, U, and V component. This eliminates the need to process the three components separately through different ICCI modules; instead, a unified ICCI module is used. This simplifies the content adaptive filter architecture from three ICCI modules to a single ICCI module, effectively reducing the computational complexity of the JPEG AI verification model. Furthermore, when using a unified ICCI module, this method also uses each Y, U, and V component and the result of the adapter-based adaptive processing as input to the ICCI module. This allows the ICCI module to obtain the Y-enhanced component, U-enhanced component, and V-enhanced component corresponding to Y, U, and V respectively, further considering the characteristics of different components and resulting in an enhanced image that includes more features of each component. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] To more clearly illustrate the technical solutions in some embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of the architecture of a JPEG AI verification model provided in the embodiments of this application; Figure 2 A schematic diagram of an analytical transformation network 101 provided in the embodiments of this application; Figure 3 A schematic diagram of a residual activation unit in the related technology provided in the embodiments of this application; Figure 4 A schematic diagram of a residual nonlocal attention block 203 provided in the related technology of this application embodiment; Figure 5 A schematic diagram of a residual block in the related art provided in the embodiments of this application; Figure 6 A schematic diagram of a masked convolution kernel used in a context model of autoregressive convolution provided in the embodiments of this application; Figure 7AA schematic diagram of the masked convolutional kernel used in the context model network 105 of a JPEG AI verification model provided in the embodiments of this application; Figure 7B A schematic diagram of a wavefront parallel processing mechanism for latent space in a related art provided for an embodiment of this application; Figure 8 A schematic diagram of a synthetic transformation network 117 in a related art provided in an embodiment of this application; Figure 9 A schematic diagram of another JPEG AI verification model provided in the related technology for embodiments of this application; Figure 10 A schematic diagram of an ICCI subnetwork provided for an embodiment of this application; Figure 11 This application provides a flowchart illustrating a method for obtaining predicted values ​​corresponding to input latent tensors in an embodiment of the present application. Figure 12 A schematic diagram illustrating a potential space divided into three parts, provided for an embodiment of this application; Figure 13 A schematic diagram of a first potential tensor provided in an embodiment of this application; Figure 14 A schematic diagram illustrating how to obtain a first context feature, provided as an embodiment of this application; Figure 15 A schematic diagram of a second potential tensor provided in an embodiment of this application; Figure 16 A schematic diagram illustrating the acquisition of a second context feature, provided as an embodiment of this application; Figure 17 This application provides a schematic diagram of obtaining the predicted value corresponding to the predicted input latent tensor. Figure 18 This application provides a schematic flowchart of a method for obtaining output potential tensors in an embodiment of the present application. Figure 19A A schematic diagram illustrating the acquisition of a second context feature, provided as an embodiment of this application; Figure 19B A schematic diagram illustrating the acquisition of a first context feature, provided as an embodiment of this application; Figure 19C A schematic diagram illustrating the acquisition of an output potential tensor provided in an embodiment of this application; Figure 20A A schematic diagram of an adapter-based ICCI subnetwork provided for an embodiment of this application; Figure 20B Provided for the embodiments of this application Figure 20A A schematic diagram of the adapter in the diagram; Figure 21 A schematic flowchart of an image reconstruction and enhancement method provided in an embodiment of this application; Figure 22 A method for providing embodiments of this application Figure 10 A schematic diagram of the ICCI processing module; Figure 23 An embodiment provided in this application Figure 20A A schematic diagram of the ICCI processing module; Figure 24 A schematic diagram of a 1D residual block provided for an embodiment of this application; Figure 25 A schematic flowchart of another image reconstruction and enhancement method provided in an embodiment of this application; Figure 26 A schematic diagram of another adapter-based ICCI subnetwork provided for an embodiment of this application; Figure 27 This is a schematic diagram of the hardware structure of a decoding device provided in some embodiments of this application. Detailed Implementation

[0014] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0015] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0016] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first contextual feature" and "second contextual feature," etc., are used to distinguish different contextual features, not to describe a specific order of contextual features.

[0017] Image compression is one of the fundamental tasks in computer vision and image processing. The development of lossy image compression techniques has brought benefits to various fields. Image compression can significantly reduce the amount of data required for image storage and transmission, but the quality of the reconstructed image may slightly decrease. In the field of lossy image compression, traditional codecs, including the Joint Photographic Experts Group (JPEG), Better Portable Graphics (BPG), and Versatile Video Coding (VVC), use similar transform, quantization, and entropy coding techniques for image compression.

[0018] In recent years, deep neural networks have achieved great success in various tasks, and image compression based on deep neural networks has begun to surpass traditional image compression techniques. JPEG AI is a learning-based image coding standard created by the Joint Picture Experts Group (JPEG). Compared to traditional image coding, JPEG AI, designed for human vision, achieves a lower bitrate while maintaining the same subjective quality.

[0019] Figure 1 This is a schematic diagram of the architecture of a JPEG AI verification model (Verification Model 1.0) provided in the embodiments of this application. The JPEG AI verification model includes: an AnalysisTransform Net 101, a gain unit 102, a first quantization unit 103, a first inverse gain unit 104, a context model network 105, a prediction fusion network 106, a super-prior coding network 107, a super-prior decoding network 108, a second quantization unit 109, a first lossless encoder 110, a first lossless decoder 111, a decomposition entropy model 112, a super-prior scale decoding network 113, a second lossless encoder 114, a second lossless decoder 115, a second inverse gain unit 116, and a synthesis transform network 117. This JPEG AI verification model can compress the input original image x into a bitstream and reconstruct the reconstructed image from the compressed bitstream. In the JPEGAI validation model, the following network modules exist only in the encoder: Analysis Transform Net 101, Gain Unit 102, First Quantization Unit 103, First Inverse Gain Unit 104, Super Prior Decoding Network 108, Second Quantization Unit 109, First Lossless Encoder 110, and Second Lossless Encoder 114. The following network modules exist only in the decoder: Second Lossless Decoder 115, Second Inverse Gain Unit 116, and Synthesis Transform Network 117. The following network modules exist in both the encoder and decoder: Context Model Network 105, Prediction Fusion Network 106, Super Prior Encoding Network 107, First Lossless Decoder 111, Decomposition Entropy Model 112, and Super Prior Scale Decoding Network 113.

[0020] The aforementioned analysis transform network 101 is used to transform the original image x input to the validation model of the JPEG AI into a latent tensor. The analysis transform network 101 operates only in the encoder. Latent tensors are high-dimensional representations of data; they cannot be directly observed from the input data, but they can be determined through information learned from the input data. The aforementioned latent tensors... That is, it is determined based on the information learned from the original image x.

[0021] Figure 2 A schematic diagram of an analytical transformation network 101 provided in the embodiments of this application is shown below. Figure 2 As shown, the analysis transformation network 101 includes, from input to output, the following components in sequence: a 3×3 downsampled convolution 201, a residual activation unit 202, a residual non-local attention block 203, a 3×3 downsampled convolution 204, a residual activation unit 205, a 3×3 downsampled convolution 206, a residual activation unit 207, a 3×3 downsampled convolution 208, and a 1×1 convolution 209.

[0022] In this analysis, the transformation network 101 uses residual activation units with attention mechanisms, for example, Figure 2 The residual activation units 202, 205, and 207 mentioned above are all residual activation units with attention mechanisms. Figure 2 The 3×3 downsampling convolutions 201, 204, 206, and 208 are all 3×3 downsampling convolutions with the same structure.

[0023] Figure 3This is a schematic diagram of a residual activation unit 202 provided in the related technology of this application. For example... Figure 3 As shown, the residual activation unit 202 may include: a LeakyReLU activation function 301, a convolutional layer 302, a tanh activation function 303, and a residual connection unit 304. This residual activation unit is used to add nonlinear transformations.

[0024] In the JPEG AI validation model, attention mechanisms can be used. Figure 2 The residual nonlocal attention block 203 shown is implemented. The residual nonlocal attention block 203 can extract deep features. In the process of extracting deep features, by capturing the correlation between pixels, pixel redundancy in the feature map can be reduced. Figure 4 This is a schematic diagram of a residual nonlocal attention block 203 provided in the related technology of this application embodiment. For example... Figure 4 As shown, the residual nonlocal attention block 203 includes two branches: a main branch 41 and a mask branch 42. The main branch 41 contains three residual blocks used to extract deep features. The mask branch 42 contains a 3×3 downsampling convolution 401, a 3×3 upsampling convolution 402, a 1×1 convolution 403, a sigmoid activation function 404, and a series of residual blocks. Figure 4 The three residual blocks in the main branch 41 are represented as: residual block 1, residual block 2, and residual block 3; the series of residual blocks included in the mask branch 42 are represented as: residual block 4, residual block 5, residual block 6, residual block 7, residual block 8, and residual block 9. It should be noted that the structure and function of each residual block in the above residual blocks 1 to 9 are the same.

[0025] Figure 5 for Figure 4 The illustrated embodiment shows a schematic diagram of a residual block, each residual block including a LeakyReLU activation function 501 and two 3×3 convolutions, wherein the two 3×3 convolutions are in Figure 5 These are represented as 3×3 convolution 502 and 3×3 convolution 503, respectively.

[0026] like Figure 1 As shown, in the JPEG AI Verification Model 1.0, the extrapolation gain unit is used to implement the variable bit rate, through parameters... The parameters of the extrapolation gain unit are controlled to scale the potential residual variable by the channel dimension. The extrapolation gain unit includes the aforementioned gain unit 102 and the first inverse gain unit 104.

[0027] Gain unit 102 is used for the residual tensor Perform gain processing to convert it into a residual tensor. Multiply each element by The specified gain factor is used to obtain the residual tensor after gain processing. Among them, residual tensor For potential tensors Compared with the predicted value The difference, where the predicted value For based on latent tensors The predicted values ​​of the elements. Gain unit 102 is used to implement the variable rate capability of the JPEG AI verification model. Gain unit 102 uses parameters to control the compression ratio. This parameter Specifies the weights between rate and distortion in the loss function during training. Obtaining the paired positive and negative gain vectors can be represented as follows: ,in, It is the positive gain vector. This is the inverse gain vector. During encoding and decoding, the forward and inverse gain vectors are known to the encoder and decoder (they are part of the model), and the gain vector... and The dimension is equal to the residual tensor. The number of channels. This gain unit 102 only operates in the encoder.

[0028] The first quantization unit 103 is used to quantize the residual tensor after gain processing to obtain the quantized residual tensor. Quantization refers to the process of approximating a signal's continuous values ​​(or a large number of possible discrete values) to a finite number of discrete values ​​(or a small number of discrete values). This first quantization unit 103 operates only in the encoder.

[0029] The first inverse gain unit 104 is used to process the quantized residual tensor. Inverse gain processing was performed to obtain the quantized residual tensor. Each element is multiplied by the inverse gain vector The specified gain factor is used to obtain the residual tensor after inverse gain processing. The first inverse gain unit 104 enables the variable rate capability of the JPEG AI verification model. The first inverse gain unit 104 uses parameters to control the compression ratio. This parameter Specifies the weights between rate and distortion in the loss function during training. This first inverse gain unit 104 operates only in the encoder.

[0030] The aforementioned context model network 105 is used to extract contextual features, and such a network module exists in both the encoder and decoder. In the encoder, the context model network 105 is used to process the latent tensor. Contextual feature extraction is performed to obtain the latent tensor. Contextual features. Among them, the latent tensor in the encoder. Based on the residual tensor after inverse gain processing and predicted value The sum is obtained by addition. In the decoder, the context model network 105 is used to process the latent tensors. Contextual feature extraction is performed to obtain the latent tensor. Contextual features. Specifically, in the decoder, the bitstream ( Figure 1 The residual tensor of the bitstream 2) shown in the figure after lossless decoding and inverse gain processing Compared with the predicted value By adding them together, we can obtain the potential tensor. .

[0031] In some learning-based image coding methods, autoregressive convolutional context models are used to extract contextual features. Figure 6 This is a schematic diagram of a masked convolution kernel used in a context model of autoregressive convolution provided in the embodiments of this application. For example... Figure 6 As shown, this autoregressive convolutional context model uses the encoded elements 601 before the current encoding position in the latent space to predict the context features corresponding to the current encoding position, and uses these context features to predict the probability distribution of the current encoding position in subsequent calculations. Here, the encoded elements are... Figure 6 The elements at the encoded positions are shown in the dashed box. Autoregressive convolutional context models can significantly improve the rate-distortion performance of super-prior models. However, due to their autoregressive structure, these context models can only decode according to a fixed raster scan order. The latent space of later decoded models always depends on the latent space of earlier decoded models, resulting in very slow decoding speeds.

[0032] To address the aforementioned issues, in some embodiments, the context model network 105 in the JPEG AI verification model further employs a wavefront parallel processing (WPP) mechanism in the latent space to enable parallel processing. This mechanism uses already decoded elements to generate latent tensors. The element prediction is an autoregressive process, and it is performed on the latent tensor. Elements in different rows can be processed independently in a wavefront parallel manner, where the delay in processing consecutive rows depends on the shape of the masked convolution kernel.

[0033] Figure 7A This is a schematic diagram of the masked convolutional kernel used in the context model network 105 of a JPEG AI verification model provided in the embodiments of this application. Figure 7A The coordinates are established with the current encoding position as the origin. The current encoding position is in row 0 and column 0. n=0 indicates the row where the current encoding position is located, n=-1 indicates the row before the current encoding position, and n=-2 indicates the row before the current encoding position. Figure 7A The symbols “-2”, “-1”, “0”, “1” and “2” indicate different columns. “0” indicates the column where the current encoding position is located, “-1” indicates the column before the current encoding position, “-2” indicates the column before the current encoding position, “1” indicates the column after the current encoding position, and “2” indicates the column after the current encoding position.

[0034] Figure 7B This diagram illustrates a wavefront parallel processing mechanism for a potential space, as provided in the embodiments of this application. The wavefront parallel processing mechanism is introduced to improve the utilization of computing resources. Figure 7B In this context, "column" represents a column and "row" represents a row. The diagram illustrates a potential space of 5 rows and 10 columns. Figure 7B The tensor of the current pixel shows that all channels are predicted independently (simultaneously), and prediction can be based on previously processed pixels during the preset process. For example... Figure 7A The design of the masked convolution kernel shown makes Figure 7B Each row of samples (i.e., the prediction of each row of pixels) can be processed in parallel, only with a delay of one element compared to the previous processing time when processing the next row, which is a delay of one wave shift (T).

[0035] Super-prior coding network 107, used for latent tensors Perform prior encoding to obtain the latent tensor. Furthermore, additional auxiliary information can be extracted during the super-prior encoding process, enabling the subsequent acquisition of an accurate probabilistic model based on the entropy encoding result. The super-prior encoding network 107 operates only in the encoder and can include two downsampling convolutions, three convolutions, and a ReLU activation function. The ReLU activation function is a neural network activation function. It remains unchanged when the input value is greater than 0, and outputs 0 when the input value is less than 0.

[0036] The second quantization unit 109 is used for quantization of potential tensors. Quantization is performed to obtain the quantized encoded latent tensor. The second quantization unit 109 only operates in the encoder.

[0037] The first lossless encoder 110 is used to process the quantized latent tensor. Entropy encoding is performed to obtain bitstream 1. Entropy encoding is a lossless operation used to encode the bitstream. The entropy encoding and decoding processes take the cumulative distribution function (CDF) output by the decomposed entropy model 112 as input. The cumulative distribution function (CDF) is essentially the integration of the probability density function (PDF) with the symbol to be encoded at the encoding end, or the integration of the probability density function (PDF) with the bitstream to be decoded at the decoding end. This first lossless encoder 110 operates only as an encoder.

[0038] The first lossless decoder 111 is used to perform entropy decoding on the bitstream 1 obtained after entropy encoding to obtain the latent tensor after entropy decoding. Entropy decoding is a lossless operation used to decode bitstreams. It can receive a cumulative distribution function (CDF) as input and is essentially the integral of the probability density function (PDF) and the bitstream to be decoded (at the decoder). This first lossless decoder 111 operates only at the decoder.

[0039] The decomposition entropy model 112 is used to input the cumulative distribution function (CDF) into the first lossless encoder 110 or the first lossless decoder 111. The CDF is used for entropy encoding or decoding. The decomposition entropy model 112 operates in both the decoder and the encoder. The decomposition entropy model 112 is integerized to address device interoperability issues, ensuring its usability on any device. The cumulative distribution function (CDF), also known as a probability table, is pre-computed and quantized, rather than trained based on trainable parameters (matrices, biases, and factors). This table is stored in integer form, thus eliminating the need to store the aforementioned parameters. Parameters of the distribution.

[0040] The super-prior decoding network 108 is used for super-prior decoding, specifically for processing the latent tensor after entropy decoding. Decoding is performed to obtain the latent tensor. The hyperprior decoding network 108 comprises a series of two deconvolutions with an upsampling stride of 2, three convolutions without tensor size changes, and a ReLU activation function. The hyperprior decoding network 108 can accept a tensor of size C×h×w as input and generate an output tensor of 4C×h×w with two channels. This hyperprior decoding network 108 is symmetric to the hyperprior coding network 107, with a pruning operation following each upsampling convolutional layer. The hyperprior decoding network 108 operates in both the decoder and encoder.

[0041] The predictive fusion network 106 can receive two inputs. The first input is the latent tensor obtained after super-prior decoding by the super-prior decoding network 108. The second input is the contextual features output by the context model network 105. The prediction fusion network 106 generates a predicted value of size C×h×w based on the two inputs of size 4C×h×w. (Size is C×h×w). In the encoder, this predicted value... It can be added to the residual tensor after inverse gain processing. To obtain the potential tensor This is used as input to the context model network 105. In the decoder, the bitstream ( Figure 1 The residual tensor of the bitstream 2) shown in the figure after lossless decoding and inverse gain processing Compared with the predicted value Adding them together yields the potential tensor. As input to the context model network 105, the prediction fusion network 106 operates in both the decoder and the encoder.

[0042] Super-prior scale decoding network 113 is used for the latent tensor after entropy decoding. Perform the inverse operation of the hyperprior coding network and output a Gaussian distribution with zero mean. The variance of the super-prior scale decoding network 113 is calculated. The super-prior scale decoding network 113 consists of two 5×5 deconvolutional sequences with an upsampling stride of 2, two 3×3 convolutions, and a LeakyReLU activation function. The super-prior scale decoding network 113 is symmetric to the super-prior encoder network 107, with a pruning operation following each upsampling convolutional layer. The prediction fusion network 106 operates in both the decoder and encoder.

[0043] The second lossless encoder 114 is used for... For the quantized residual tensor Entropy encoding is performed to obtain bitstream 2. It should be noted that the second lossless encoder 114 has a similar structure and function to the first lossless encoder 110. The second lossless encoder 114 only operates within the encoder.

[0044] The second lossless decoder 115 is used for... Entropy decoding is performed on bitstream 2 to obtain the residual tensor after entropy decoding. It should be noted that the second lossless decoder 115 has a similar structure and function to the first lossless decoder 111. The second lossless decoder 115 only operates within the decoder.

[0045] The second inverse gain unit 116 is used to perform inverse gain processing on the residual tensor after entropy decoding, multiplying each element of the residual tensor after entropy decoding by an inverse gain vector. The specified gain factor is used to obtain the residual tensor after inverse gain processing. The second inverse gain unit 116 only operates in the decoder.

[0046] Synthetic transformation network 117 is used to transform latent tensors This indicates conversion to a reconstructed image. Among them, the latent tensor The residual tensor after inverse gain processing Compared with the predicted value The result obtained by addition. The synthetic transform network 117 only operates in the decoder.

[0047] Figure 8 This is a schematic diagram of a synthetic transformation network 117 provided in the embodiments of this application. For example... Figure 8 As shown, the synthesis transform network 117 includes, from input to output, the following components in sequence: residual block 801, residual block 802, 3×3 upsampling convolution 803, residual activation unit 804, 3×3 upsampling convolution 805, residual activation unit 806, 3×3 upsampling convolution 807, residual nonlocal attention block 808, residual activation unit 809, and 3×3 upsampling convolution 810. Two residual blocks are placed at the beginning of the synthesis transform, enhancing the reconstructed latent tensor. This reduces quantization error.

[0048] It should be noted that each residual activation unit in the synthetic coding network 117 is related to, for example, Figure 3 The residual activation unit shown has the same structure and function. The residual nonlocal attention block 808 in the synthetic coding network 117 is the same as described above. Figure 4 The residual nonlocal attention block 203 shown has the same structure and function. Each residual block in the synthetic coding network 117 is similar to... Figure 4 The residual blocks shown have the same structure and function.

[0049] Figure 9 This is a schematic diagram of another JPEG AI verification model provided in the related technology of this application embodiment.

[0050] like Figure 9 As shown above, in the above Figure 1 The JPEG AI validation model shown is supplemented with a content-adaptive filter 118. This content-adaptive filter 118 is used to enhance the Y, U, and V components, and to reconstruct the image output by the synthetic transform network 117. Enhancement is performed on the Y, U, and V components to obtain the enhanced reconstructed image. The adaptive filter 118 includes a Content Adaptive Inter Channel Correlation Information filter subnetwork (ICCI) for reconstructing and enhancing the Y, U, and V components of the image. This ICCI subnetwork is used to enhance the Y, U, and V components.

[0051] Figure 10 This is a schematic diagram of an ICCI subnetwork provided in an embodiment of this application. (See diagram below.) Figure 10 As shown, this ICCI sub-network includes three different ICCI processing modules. The input to this ICCI sub-network is the Y, U, and V components of the image. Figure 10 The Y component of the image is represented as The U component of the image is represented as The V component of the image is represented as The U component of the image is obtained by bicubic interpolation to obtain a U component with the same shape as the Y component. The V component of the image is obtained by bicubic interpolation to obtain a V component with the same shape as the Y component. Then the Y component U component and V component All are subjected to forward wavelet transform, and the three components obtained by the transform are represented as follows: Figure 10 Components 1001, 1002, and 1003 are concatenated along the channel dimension. The concatenated component 1004, along with the individual components, is then fed into three ICCI processing modules as input. These modules process the Y, U, and V components respectively, resulting in processed components 1008, 1009, and 1010. These components are then subjected to inverse wavelet transform to obtain the reconstructed Y, U, and V components. The reconstructed Y component is... The reconstructed U component is The reconstructed V component is In the three ICCI processing modules mentioned above, the number of residual blocks is Nb. Specifically, the number of residual blocks in ICCI processing module 1005 (for the Y component) is Nb=8, in ICCI processing module 1006 (for the U component) is Nb=13, and in ICCI processing module 1007 (for the V component) is Nb=13. The aforementioned bicubic interpolation is a method for enlarging image size and increasing pixel density. It can double the number of image pixels using bicubic interpolation, where these expanded pixels are predicted based on existing pixels.

[0052] The above Figure 10 In the ICCI subnetwork shown, during the image feature enhancement process, in order to take into account the characteristics of different Y, U, and V components, three different ICCI processing modules are set up to enhance different components, thereby achieving the purpose of adaptive processing.

[0053] Figure 1 In the context model network 105, to achieve parallel processing, convolutional kernels are used to calculate the context features of the elements of the current pixel. However, when calculating the context features of the current pixel, only the elements to the upper left of the current pixel are considered, neglecting other elements around the current pixel. Compared to the autoregressive convolutional context model, this reduces the receptive field and does not utilize the elements to the lower right of the current pixel, resulting in lower accuracy of the obtained context features. The receptive field refers to the size of the region in the original input features that a single element in the feature map output by each network layer in the convolutional neural network maps back to. In other words, the receptive field of an element point in the feature map output by a network layer in the convolutional neural network is determined by the size of the region in the original input it maps to; the deeper the network layer, the larger the receptive field corresponding to the elements of its output features.

[0054] To improve the accuracy of context features, embodiments of this application propose a step-by-step context model, which is embedded into... Figure 1In the context model network 105 of the JPEG AI verification model shown, the computation speed of the entropy coding part can be improved. This step-by-step context model can divide the latent space into three or more parts, calculate context features for one part of the latent space, and utilize not only the elements to the upper left and lower right of the current pixel when calculating context features, thereby improving the accuracy of the calculated context features. This also improves the accuracy of subsequent predictions based on the calculated context features for this part of the space. Alternatively, context features can be omitted for another part of the latent space, and the predictions for this other part of the latent space can be directly calculated based on the output of the super-prior decoding network 108, improving the efficiency of calculating predictions for the latent space. This step-by-step context model can be applied in encoders and / or decoders.

[0055] Figure 11 This is a flowchart illustrating a method for obtaining predicted values ​​corresponding to input latent tensors, provided in an embodiment of this application. In this method, a step-by-step context model is applied in the context model network of the encoder to extract context features. The method includes, but is not limited to, the following steps: 2001. Divide the latent space corresponding to the input latent tensor into 3 parts.

[0056] Specifically, the entire latent space of the input latent tensor can be viewed as being composed of multiple 2×2 blocks. The first pixel position of the upper left corner of each 2×2 block can be determined as part 1; the second pixel position of the lower right corner of each 2×2 block can be determined as part 2; and the third pixel positions of the upper right and lower left corners of each 2×2 block can be determined as part 3.

[0057] In the potential space, the top left corner of each 2×2 block corresponds to the first pixel position, the bottom right corner of each 2×2 block corresponds to the second pixel position, and the top right and bottom left corners of each 2×2 block correspond to the third pixel position.

[0058] Figure 12 This is a schematic diagram illustrating a potential space divided into three parts, as provided in an embodiment of this application. For example, as shown... Figure 12 As shown, in the input latent tensor, the first part corresponding to the first pixel position can be marked as 1, the second part corresponding to the second pixel position can be marked as 2, and the third part corresponding to each third pixel position can be marked as 3.

[0059] Here, the input latent tensor is the latent tensor input to the context model network 105. When the context model network 105 operates in the encoder, this input latent tensor is... Figure 1 The potential tensor shown The potential tensor Based on the residual tensor after inverse gain processing and predicted value Add them together to get the result.

[0060] 2002. Zero out the third pixel position in the latent space corresponding to the input latent tensor to obtain the first latent tensor.

[0061] The above-described operation of zeroing out the third pixel position in the latent space corresponding to the input latent tensor means that the data at the first and second pixel positions in the latent space corresponding to the input latent tensor are retained, while the data at the third pixel position is set to zero. It should be noted that, in this embodiment, zeroing out the data at the third pixel position refers to zeroing out the data at the third pixel position in the latent space corresponding to the input latent tensor, and not zeroing out the data at the third pixel position. Figure 12 The "3" used to mark the third part corresponding to the third pixel position is set to zero. After setting the data of the third pixel position in the latent space corresponding to the input latent tensor to zero, the third part corresponding to the third pixel position can still be marked as 3. To demonstrate which pixel positions' data were set to zero in step 2002 above. Figure 13 The third part corresponding to the third pixel position is marked as empty, but this does not mean that the "3" used to mark the third part corresponding to the third pixel position is set to empty.

[0062] For example, with Figure 13 The potential space partitioning method shown is as follows: Figure 13 This is a schematic diagram of a first potential tensor provided for an embodiment of this application. For example... Figure 13 In the first potential tensor shown, only the data of the first pixel position and the second pixel position are retained.

[0063] 2003. Perform convolution operation on the first potential tensor using the first mask convolution kernel to obtain the first context feature corresponding to the third pixel position.

[0064] The size of the first mask convolution kernel is 5×5.

[0065] Figure 14 This is a schematic diagram illustrating how a first context feature is obtained, as provided in an embodiment of this application. Figure 14 As shown, Figure 14 The first potential tensor 2301 shown is convolved with a 5×5 mask convolution 2302 to obtain the following: Figure 14 The first context feature 2303 is shown in the figure.

[0066] 2004. Zero out the second and third pixel positions in the latent space corresponding to the input latent tensor to obtain the second latent tensor.

[0067] Specifically, the above-mentioned operation of setting the second and third pixel positions in the latent space corresponding to the input latent tensor to zero means that the data of the first and second pixel positions in the latent space corresponding to the input latent tensor are retained, while the data of the third pixel position is set to zero.

[0068] For example, with Figure 13 The potential space partitioning method shown is as follows: Figure 15 This is a schematic diagram of a second potential tensor provided in an embodiment of this application. Figure 15 In the second potential tensor shown, only the data of the first pixel position is retained.

[0069] 2005. Perform convolution operation on the second potential tensor using the second mask convolution kernel to obtain the second context feature corresponding to the second pixel position.

[0070] The second mask convolution kernel has a size of 3×3.

[0071] Figure 16 This is a schematic diagram illustrating how a second context feature is obtained, as provided in an embodiment of this application. Figure 16 As shown, Figure 16 The first potential tensor 2501 shown is convolved with a 3×3 mask convolution 2502 to obtain the following: Figure 16 The second context feature 2503 is shown.

[0072] After obtaining the first and second context features mentioned above, the predicted value corresponding to the input latent tensor can be predicted based on the first and second context features. .

[0073] Steps 2001 to 2005 described above can be derived from the above. Figure 1 The context model network 105 of the JPEG AI verification model shown is implemented, and a step-by-step context model is applied in the context model network 105.

[0074] 2006. The first context features, the second context features, and the target latent tensor are subjected to entropy probability derivation to obtain the predicted values ​​corresponding to all pixel positions in the input latent tensor.

[0075] The process of deriving entropy probabilities from the first context features, the second context features, and the target latent tensor to obtain predicted values ​​for all pixel positions in the input latent tensor may include, but is not limited to: adding the first and second context features to obtain the target context features; concatenating the target context features and the target latent tensor to obtain the concatenated result; and performing convolution processing on the concatenated result to obtain predicted values ​​for all pixel positions in the input latent tensor. This convolution processing can be a 1×1 convolution operation.

[0076] The above step 2006 can be performed by Figure 1 The preset fusion network 106 of the JPEG AI verification model shown is executed.

[0077] Figure 17 This is a schematic diagram illustrating the acquisition of predicted values ​​corresponding to predicted input latent tensors, provided as an embodiment of this application. In this embodiment, a step-by-step context model is embedded into... Figure 1 The context model network 105 of the JPEG AI verification model is shown, and the verification model of the JPEG AI is implemented based on this model. Figure 16 The example shown is for obtaining the predicted value. The process.

[0078] like Figure 17 As shown, the input latent tensor is fed into the context model network 105, which follows... Figure 11 After processing by the method shown, the first context feature and the second context feature are output, and the first context feature and the second context feature are used as... Figure 1 The input to the preset fusion network 106 of the JPEG AI verification model shown is the input to the preset fusion network 106. The preset fusion network 106 can also receive the latent tensor after entropy decoding by the super-prior decoding network 108 in the JPEG AI verification model. The latent tensor obtained by decoding The pre-defined fusion network 106 integrates the first context features, the second context features, and the latent tensor. By performing entropy probability derivation, we can obtain the predicted value corresponding to the input latent tensor. .

[0079] In the above embodiments of this application, when calculating the predicted value corresponding to the input latent tensor of the context model network 105 in the encoder, the context features for the first pixel position in the input latent tensor are not calculated. Instead, the context features for the second and third pixel positions in the input latent tensor are used, and based on the context features of the second and third pixel positions and the target latent tensor, the predicted value of the latent space is calculated. Since it is not necessary to calculate the context features corresponding to the first pixel position during the calculation of the predicted value of the input latent tensor, the efficiency of calculating the predicted value of the latent space is improved.

[0080] The above method for calculating the predicted value corresponding to the input latent tensor can utilize not only the elements above and to the left of the current pixel, but also the elements below and to the right of the current pixel when calculating the context features at the second and third pixel positions, thereby improving the accuracy of the calculated context features.

[0081] Figure 18 This is a flowchart illustrating a method for obtaining output latent tensors provided in an embodiment of this application. In this method, a step-by-step context model is applied in the context model network of the decoder to extract context features. The method includes, but is not limited to, the following steps: 2701. Perform convolution processing on the target latent tensor to calculate the predicted value corresponding to the first pixel position in the latent space of the target latent tensor.

[0082] Wherein, the target latent tensor is a pair of bitstreams (i.e. Figure 1 The latent tensor obtained by decoding the latent tensor after entropy decoding of the bitstream 1) is described above. The convolution process described above is a 1×1 convolution operation.

[0083] In the aforementioned potential space, the top left corner of each 2×2 block corresponds to the first pixel position, the bottom right corner of each 2×2 block corresponds to the second pixel position, and the top right and bottom left corners of each 2×2 block correspond to the third pixel position.

[0084] 2702. Add the first residual tensor to the predicted value corresponding to the first pixel position to obtain the second potential tensor.

[0085] In the latent space of the second latent tensor, there are data for the first and second pixel positions, and the data for the third pixel position is zero.

[0086] The aforementioned first residual tensor is the residual tensor corresponding to the first pixel position in the target residual tensor, and the target residual tensor is the residual tensor for the second bitstream (i.e. Figure 1 The latent tensor obtained after entropy decoding and inverse gain processing of the bitstream 2) in the code stream.

[0087] 2703. Perform convolution operation on the second potential tensor using the second mask convolution kernel to obtain the second context feature corresponding to the second pixel position.

[0088] The size of the second mask convolution kernel is 3×3.

[0089] The description of step 2703 above can be referred to the relevant description of step 2005 above, and will not be repeated here.

[0090] Figure 19A This is a schematic diagram illustrating the acquisition of a second context feature according to an embodiment of this application. In this embodiment, a step-by-step context model is embedded into... Figure 1 The context model network 105 of the JPEG AI verification model is shown, and the verification model of the JPEG AI is implemented based on this model. Figure 19A The process of obtaining the second contextual features is shown.

[0091] like Figure 19A As shown, the advanced prior scale decoding network 113 decodes the latent tensor after entropy decoding. The inverse operation of the priori coding network 107 is performed to obtain the probability distribution N(0,σ). Then, the second lossless decoder 115 performs entropy decoding on the bitstream 2 based on N(0,σ) to obtain the entropy-decoded residual tensor. The second inverse gain unit 116 performs inverse gain processing on the entropy-decoded residual tensor to obtain the inverse-gain processed residual tensor. The target latent tensor in step 2701 is the latent tensor after entropy decoding by the priori decoding network 108. Decoding is performed to obtain the latent tensor. .

[0092] like Figure 19A The preset fusion network 106 shown can perform a 1×1 convolution on the residual tensor after inverse gain processing (i.e., the aforementioned target latent tensor) to calculate the predicted value corresponding to the first pixel position. 1. The next one Adding the first residual tensor corresponding to the first pixel position in the residual tensor after inverse gain processing 1 yields the second potential tensor 2801. Here, the first residual tensor in step 2701 is the residual tensor corresponding to the first pixel position in the residual tensor after inverse gain processing.

[0093] like Figure 19A As shown, after inputting the second potential tensor 2801 into the context model network 105, the context model network 105 can perform the above step 2703 to obtain the second context feature 2802.

[0094] 2704. By deriving the second context features and the target latent tensor through entropy probability, the predicted values ​​corresponding to the two pixel positions can be obtained.

[0095] The two pixel positions include the first pixel position and the second pixel position.

[0096] 2705. Add the second residual tensor to the predicted values ​​at the two pixel locations to obtain the first latent tensor.

[0097] The second residual tensor mentioned above is the residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor.

[0098] 2706. Perform convolution operation on the first potential tensor using the first mask convolution kernel to obtain the first context feature corresponding to the third pixel position.

[0099] The size of the first mask convolution kernel is 5×5.

[0100] Figure 19B This is a schematic diagram illustrating the acquisition of a first context feature according to an embodiment of this application. In this embodiment, a step-by-step context model is embedded into... Figure 1 The context model network 105 of the JPEG AI verification model is shown, and the verification model of the JPEG AI is implemented based on this model. Figure 19B The process of obtaining the first contextual features is shown.

[0101] like Figure 19B As shown, the advanced prior scale decoding network 113 decodes the latent tensor after entropy decoding. The inverse operation of the priori coding network 107 is performed to obtain the probability distribution N(0,σ). Then, the second lossless decoder 115 performs entropy decoding on the bitstream 2 based on N(0,σ) to obtain the entropy-decoded residual tensor. The second inverse gain unit 116 performs inverse gain processing on the entropy-decoded residual tensor to obtain the inverse-gain processed residual tensor. The target latent tensor in step 2704 is the latent tensor after entropy decoding by the priori decoding network 108. Decoding is performed to obtain the latent tensor. .

[0102] Figure 19B The pre-defined fusion network 106 can use the second context feature 2802 and the latent tensor Through entropy probability derivation, the predicted values ​​for the two pixel positions are obtained. Then the second residual tensor is combined with the predicted values ​​at the two pixel locations. The summation yields the first potential tensor 2803. The second residual tensor in step 2705 is the residual tensor corresponding to the first and second pixel positions within the residual tensor after inverse gain processing.

[0103] Figure 19B After the first potential tensor 2803 is input into the context model network 105, the context model network 105 performs the above step 2706 to obtain the first context feature 2804.

[0104] 2707. By deriving the first context features, the second context features, and the target latent tensor through entropy probability, the predicted values ​​corresponding to all pixel positions can be obtained.

[0105] In some embodiments of this application, the prediction values ​​corresponding to all pixel positions can be obtained by entropy probability derivation of the first context feature, the second context feature, and the target latent tensor. This may include, but is not limited to: adding the first context feature and the second context feature to obtain the target context feature; concatenating the target context feature and the target latent tensor to obtain the concatenation result; and performing convolution processing on the concatenation result to obtain the prediction values ​​corresponding to all pixel positions.

[0106] 2708. Add the third residual tensor to the predicted values ​​at all pixel locations to obtain the output latent tensor.

[0107] Among them, the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor.

[0108] Figure 19C This application provides a schematic diagram of obtaining the output latent tensor as an embodiment of the present application. In this embodiment, a step-by-step context model is embedded into... Figure 1 The context model network 105 of the JPEG AI verification model is shown, and the verification model of the JPEG AI is implemented based on this model. Figure 19B The process of obtaining the output potential tensor is shown.

[0109] like Figure 19B As shown, the advanced prior scale decoding network 113 decodes the latent tensor after entropy decoding. The inverse operation of the priori coding network 107 is performed to obtain the probability distribution N(0,σ). Then, the second lossless decoder 115 performs entropy decoding on the bitstream 2 based on N(0,σ) to obtain the entropy-decoded residual tensor. The second inverse gain unit 116 performs inverse gain processing on the entropy-decoded residual tensor to obtain the inverse-gain processed residual tensor. The target latent tensor in step 2707 is the latent tensor after entropy decoding by the priori decoding network 108. Decoding is performed to obtain the latent tensor. .

[0110] Figure 19B The pre-defined fusion network 106 can use the second context feature 2802 and the first context feature 2804 with the entropy-decoded latent tensor. Through entropy probability derivation, the predicted values ​​for all pixel positions are obtained. Then the third residual tensor is combined with the predicted values ​​for all pixel locations. Adding them together yields the output latent tensor 2805. The third residual tensor in step 2708 is the residual tensor corresponding to all pixel positions in the residual tensor after inverse gain processing.

[0111] In the above embodiments of this application, when obtaining the output latent tensor in the decoder, since it is not necessary to calculate the context features corresponding to the first pixel position, the efficiency of calculating the predicted value of the latent space is improved.

[0112] In the process of obtaining the output potential tensor described above, when calculating the context features at the second and third pixel positions, not only the elements above and to the left of the current pixel can be used, but also the elements below and to the right of the current pixel, thereby improving the accuracy of the calculated context features.

[0113] In the embodiments of this application, Figure 9 The content adaptive filter 118 in the decoder of the JPEG AI verification model shown uses... Figure 10 When performing image feature enhancement in the ICCI subnetwork shown, the complexity of the content adaptive filter is increased because three different ICCI processing modules need to be set up.

[0114] To reduce the complexity of content-adaptive filters, embodiments of this application... Figure 9 The content adaptive filter 118 in the decoder shown is improved by proposing an adapter-based ICCI sub-network. This adapter-based ICCI sub-network uses the introduced adapter to adaptively process the Y, U, and V components before feeding them into a unified ICCI module. This simplifies the architecture of the content adaptive filter 118 from three ICCI modules to one ICCI module, effectively reducing the computational complexity of the JPEG AI verification model and also reducing the complexity of the content adaptive filter 118.

[0115] Figure 20A This is a schematic diagram of an adapter-based ICCI subnetwork provided as an embodiment of this application. Figure 20A As shown, the adapter-based ICCI subnetwork includes: a unified ICCI processing module; the input to the ICCI subnetwork is the Y, U, and V components of the image; Figure 20AThe Y component of the image is represented as The U component of the image is represented as The V component of the image is represented as The U component of the image is obtained by bicubic interpolation to obtain a U component with the same shape as the Y component. The V component of the image is obtained by bicubic interpolation to obtain a V component with the same shape as the Y component. Then the Y component U component and V component All are subjected to forward wavelet transform, and the three components obtained by the transform are represented as follows: Figure 20A The components 2901, 2902, and 2903 are concatenated along the channel dimension. The concatenated component 2904 is then input into adapter 2905. Adapter 2905 performs adaptive processing on the concatenated component 2904 to obtain an adaptively enhanced component. This adaptively enhanced component is then input into ICCI processing module 2906 for enhancement processing, resulting in the enhanced target component 2907. The target enhanced component 2907 is then split along the channel dimension to obtain the split Y component 2908, U component 2909, and V component 2910. Inverse wavelet transform is then performed on the three split components to obtain the output Y component, output U component, and output V component. The output image is then obtained based on the output Y component, output U component, and output V component. Figure 20A The output Y component, output U component, and output V component are respectively represented as follows: , , .

[0116] In the adapter-based ICCI subnetwork, adapter 2905 is used to extract different features of components based on an attention mechanism, simultaneously introducing channel attention and spatial attention to achieve a combination of these two approaches. Adapter 2905 takes the concatenated three-component component (concatenated component 2904) as input. It first passes through a normalization layer, then applies channel attention and spatial attention to each input component separately. The results of the channel and spatial attention processing are summed and residual connections are formed. The processed result is then normalized and fully connected, followed by another residual connection, which serves as the output of the adapter module. In the JPEG AI image coding framework ICCI subnetwork, only convolutional operations are used to extract local features of the components, neglecting global features. Therefore, the structure combining channel and spatial attention in the adapter module effectively extracts global features from the components.

[0117] Figure 20B for Figure 20AThe diagram shows an adapter 2905 in an adapter-based ICCI subnetwork. Figure 20B As shown, the adapter 2905 in the adapter-based ICCI sub-network includes: a 1×1 convolution 29051, a layer normalization layer 29052, a channel attention block 29053, a spatial attention block 29054, an additive fusion module 29055, a normalization layer 29056, a multilayer perceptron (MLP) 29057, an additive fusion module 29058, and a 1×1 convolution 29059. The processing of the spliced ​​components by the adapter 2905 includes: first, processing the spliced ​​component 2904 sequentially through a 1×1 convolution 29051 and a normalization layer 29052; then, inputting the output features of the normalization layer 29052 into the channel attention block 29053 and the spatial attention block 29054 respectively; and finally, processing the outputs of the 1×1 convolution 29051, the channel attention block 29053, and the spatial attention block 29054 through the addition and fusion module 29055. The additive fusion is performed, and the additive fusion result of the additive fusion module 29055 is processed sequentially through the normalization layer 29056 and the multilayer perceptron 29057. Finally, the additive fusion module 29058 adds and fuses the additive fusion result of the additive fusion module 29055 and the output of the multilayer perceptron 29057. The output of the additive fusion module 29058 is then convolved by a 1×1 convolution 29059 to obtain the output of the adapter 2905 (adaptive augmentation component).

[0118] In some embodiments, the spatial attention block can be a network transformer based on a sliding window (SwinTransformer).

[0119] contrast Figure 10 and Figure 20A It can be seen that, Figure 10 Three ICCI processing modules were used for different components, and Figure 20A The middle part uses a unified ICCI processing module for the three components, therefore Figure 20A The adapter-based ICCI subnetwork shown reduces the number of ICCI processing modules and lowers the complexity of the ICCI subnetwork.

[0120] This application describes the application of an adapter-based content adaptive filter in the JPEG AI image compression standard, proposing an image reconstruction and enhancement method that operates on... Figure 1The JPEG AI verification model shown includes a content adaptive filter 118. In this embodiment, an image reconstruction enhancement method may include: determining stitching components based on the first Y component, first U component, and first V component of the input image; inputting the stitching components into an adapter and obtaining the adaptive enhancement components output by the adapter, wherein the adapter is trained using a convolutional layer-based framework, and learns the features of the Y, U, and V components during training to have adaptive processing capabilities for the Y, U, and V components; inputting the adaptive enhancement components into an ICCI processing module and obtaining the target enhancement components output by the ICCI processing module; performing channel-dimensional splitting and inverse transformation on the target enhancement components to obtain output Y components, output U components, and output V components; and obtaining the output image based on the output Y components, output U components, and output V components.

[0121] Here, the input image refers to the image input to... Figure 1 The image shown is from the content adaptive filter 118. The first Y component, first U component, and first V component mentioned above represent the Y, U, and V components of the input image, respectively.

[0122] Figure 21 This is a schematic flowchart illustrating an image reconstruction and enhancement method provided in an embodiment of this application. Figure 21 As shown, the method includes the following steps: 3001. Use bicubic interpolation on the first U component and the first V component of the input image to obtain a second U component and a second V component with the same shape as the first Y component of the input image.

[0123] The size of the first Y component is S times the size of the first U component and the first V component. In the JPEG AI verification model, S=2.

[0124] The above determination of the stitching components based on the first Y component, first U component, and first V component of the input image can be specifically achieved through steps 3001 and 3002.

[0125] The first Y component of the input image has a different shape than the first U component and the first V component. Since the U component, V component and Y component need to be stitched together in the channel dimension, the input images of the first U component and the first V component need to be adjusted by bicubic interpolation to obtain the second U component and the second V component with the same shape as the first Y component of the input image.

[0126] 3002. Perform forward wavelet transform on the first Y component, the second U component, and the second V component respectively, and then concatenate the transform results to obtain the concatenated component.

[0127] The processes 3001 and 3002 described above can be represented as follows: (1) (2) (3) (4) Among them, in the above formulas (1) to (4), , and These are the three components of the input image: the first Y component, the first U component, and the first V component. ( ) indicates bicubic interpolation. ( ) represents the forward wavelet transform. () represents a concatenation function; This represents the result of the positive wavelet transform of the first Y component; This represents the result of performing a forward wavelet transform on the first U component to obtain the second U component, and then performing a forward wavelet transform on the second U component. This represents the result of bicubic interpolation of the first V component to obtain the second V component, followed by a forward wavelet transform of the second V component. This indicates the above spliced ​​components.

[0128] 3003. Input the spliced ​​components into the adapter and obtain the adaptive augmented components output by the adapter.

[0129] In this embodiment, the main function of the adapter is to learn the features of the three components Y, U, and V in order to achieve adaptive processing for the three components Y, U, and V.

[0130] The structure of the above adapter can be represented as follows: (5) In formula (5) above, The representative matrix contains learnable parameters of the adapter. express B is a length of C (and (with the same number of channels), a matrix with a width of M; , express Given a feature with C channels, H height, and W width, it can be seen that the lengths of A and B are related to... The number of channels is the same. Indicates adapter parameters, This refers to an adapter.

[0131] The adapter is trained using a convolutional layer-based framework. During training, it learns the features of the three components Y, U, and V to have adaptive processing capabilities for these three components.

[0132] In some embodiments of this application, the training set of the framework can be a JPEG AI dataset and images compressed at different bitrates corresponding to the JPEG AI dataset. During training, the input to the adapter module is the total component obtained by concatenating the Y, U, and V components of the image. The total component is sent to the adapter module to extract the features of each component, and then enters the ICCI processing module for feature extraction. The output of the ICCI processing module is the processed total component, i.e., the target enhancement component. Then, the target enhancement component is split and subjected to inverse wavelet transform to obtain the processed Y, U, and V components.

[0133] The JPEG AI dataset contains 5,284 images, including various image types such as people, animals, landscapes, and architecture. The JPEG AI validation model and the ICCI processing module are trained using this dataset. We use the JPEG AI dataset to train the ICCI module. Each image in the dataset can be used to extract Y, U, and V components. The training process also uses all three components simultaneously as input to the ICCI module; that is, it uses the Y, U, and V components of each image in the JPEG AI dataset for training.

[0134] After training is completed and the adapter is obtained, the adapter can extract different features of the components according to different inputs (three different components Y, U, and V). After unified processing by the ICCI processing module, different features can be extracted for the inputs of the three different components Y, U, and V, achieving adaptation for the three components and simplifying the three ICCI modules and reducing the complexity of the model.

[0135] 3004. Input the adaptive enhancement component into the ICCI processing module and obtain the target enhancement component output by the ICCI processing module.

[0136] The ICCI processing module can be, for example, Figure 20A The ICCI processing module 2905 shown is described. This ICCI processing module is related to... Figure 10 The ICCI processing modules 1008, 1009, and 1010 shown are different, as detailed below: Figure 22 A method for providing embodiments of this application Figure 10 A schematic diagram of the ICCI processing module.

[0137] like Figure 22As shown, the ICCI processing module includes: a 3×3 convolution 3101, a batch normalization layer 3102, a ReLU activation function 3103, Nb 1D residual blocks (one-dimensional residual blocks) 3104, and a 3×3 convolution 3105. The batch normalization layer is used to standardize the data; normalized data can accelerate network convergence. The ICCI processing module has two inputs, one of which is as follows: Figure 10 Component 1004 is one of the components, and component 100X is the component corresponding to the ICCI processing module. In this ICCI processing module... Figure 10 When the ICCI processing module 1008 is in the middle, the component 100X is Figure 10 The component shown is 1001, and Nb=8; in this ICCI processing module, Figure 10 When the ICCI processing module 1009 is in the middle, the component 100X is Figure 10 The component shown is 1002, Nb=13; in this ICCI processing module, it is... Figure 10 When the ICCI processing module 1010 is in the middle, the component 100X is Figure 10 The component 1003 shown has Nb=13. Component 1001, after being processed by a 3×3 convolution 3101, a batch normalization layer 3102, a ReLU activation function 3103, Nb 1D residual blocks 3104, and a 3×3 convolution 3105, is then scaled by a scaling factor. The result of the multiplication is summed with the component 100X to obtain the output of the ICCI processing module. Here, the scaling factor is... These are parameters in the ICCI processing module, which can be obtained through the training process of the ICCI processing module.

[0138] Figure 23 An embodiment provided in this application Figure 20A A schematic diagram of the ICCI processing module is shown. This ICCI processing module includes: a 3×3 convolution 3201, a batch normalization layer 3202, a ReLU activation function 3203, Nb 1D residual blocks 3204, and a 3×3 convolution 3215. The ICCI processing module 1010 includes one input channel, taking the adaptive enhancement component processed by the adapter as the input to the ICCI module. This input is then processed through a sequence of 3×3 convolutions 3201, batch normalization layers 3202, Nb 1D residual blocks, and 3×3 convolutions 3215. After processing this sequence, the result is combined with a scaling factor. The product is then multiplied and used as the output of the ICCI module. Figure 23 Nb can be a fixed value. Here, the scaling factor is... These are parameters in the ICCI processing module, which can be obtained through the training process of the ICCI processing module.

[0139] In some embodiments of this application, Figure 23 Nb in the equation can be an integer greater than 8 and less than 13.

[0140] In comparison with the above Figure 22 and Figure 23 It can be seen that, Figure 22 The ICCI processing module shown has two inputs, and the number of 1D residual blocks varies in different ICCI processing modules. Figure 23 The ICCI processing module shown has only one input, and the number of 1D residual blocks in this ICCI processing module is a fixed value. Figure 23 The architecture of the ICCI processing module shown is similar to... Figure 22 The architecture of the ICCI processing module shown is different.

[0141] The above Figure 23 The number of 1D residual blocks can be based on Figure 22 The number of 1D residual blocks is determined. Figure 22 The number of 1D residual blocks varies depending on the component. For the Y component, there are 8 1D residual blocks; for the U component, there are 13 1D residual blocks; and for the V component, there are 13 1D residual blocks. Therefore, in some embodiments of this application... Figure 23 The number of 1D residual blocks can be calculated according to... Figure 22 The number of residual blocks in 1D is determined as 8, 13, and 13. Figure 23 The number of 1D residual blocks can be any integer between 8 and 13. For example, the number of 1D residual blocks can be 10.

[0142] In some embodiments of this application, the above Figure 23 The number of 1D residual blocks can be determined according to Figure 22 The number of 1D residual blocks, 8, 13, and the average value of 13 are determined. For example, it can be determined by rounding up based on the average value to 12, or by rounding down based on the average value to 11.

[0143] The above Figure 22 and Figure 23 The structures of the ID residual blocks in the data are all the same. Figure 24 This is a schematic diagram of a 1D residual block provided in an embodiment of this application. Figure 24As shown, the 1D residual block includes convolution 3301, batch normalization (BN) layer 3302, ReLU activation function 3303, convolution 3304, and batch normalization layer 3305. Specifically, the input to the ID residual block, after being processed by convolution 3301, batch normalization layer 3302, ReLU activation function 3303, convolution 3304, and batch normalization layer 3305, is then compared with a scaling factor. Multiply the results and add the result to the input of the ID residual block to obtain the output of the ID residual block. This represents the scaling factor for the nth ID residual block. Since there exists a scaling factor for each ID residual block, these scaling factors can be obtained through the training process of the ICCI processing module. 1D residual blocks can improve the expressive power of image features, extract effective image features, and solve the problems of gradient vanishing and network degradation during image encoding and decoding.

[0144] 3005. The target enhancement component is split into channel dimensions to obtain the enhanced Y component, enhanced U component, and enhanced V component.

[0145] 3006. Perform inverse wavelet transform on the enhanced Y component, enhanced U component, and enhanced V component respectively to obtain the output Y component, output U component, and output V component, and obtain the output image based on the output Y component, output U component, and output V component.

[0146] The above-mentioned channel-dimensional splitting and inverse transformation of the target enhancement component to obtain the output Y component, output U component and output V component can be achieved through steps 3005 and 3006.

[0147] After processing by the ICCI module, the target enhancement components are decomposed into enhanced Y, enhanced U, and enhanced V components through channel dimension decomposition. Then, inverse wavelet transforms are performed on these three components to obtain the reconstructed output Y, output U, and output V components. This process can be represented as follows: (6) (7) (8) (9) Among them, in formulas (6) to (9) above, Indicates the target enhancement component, This indicates that channel dimension splitting is being performed. This represents the enhanced Y component, enhanced U component, and enhanced V component obtained from the splitting. This indicates inverse wavelet transform processing. , , These represent the output Y component, output U component, and output V component, respectively.

[0148] The image reconstruction enhancement method provided in this application uses an adapter-based ICCI subnetwork in the content adaptive filter of the JPEG AI verification model. In this adapter-based ICCI subnetwork, an adapter is introduced to adaptively process the Y, U, and V components to enhance the input image according to the features of each Y, U, and V component. This eliminates the need to process the three components separately through different ICCI modules. Instead, a unified ICCI module is used for processing, thus simplifying the content adaptive filter architecture from three ICCI modules to one ICCI module, effectively reducing the computational complexity of the JPEG AI verification model.

[0149] In addition to the application of adapter-based content adaptive filters in the JPEG AI image compression standard, this application proposes another image reconstruction and enhancement method, which operates on... Figure 1 The content adaptive filter 118 in the JPEG AI verification model is shown. In this embodiment, the other image reconstruction enhancement method may include: performing forward wavelet transforms on the first Y component, first U component, and first V component of the input image respectively to obtain Y component transform results, U component transform results, and V component transform results; concatenating the Y component transform results, U component transform results, and V component transform results to obtain a concatenated component; inputting the concatenated component into an adapter and obtaining the adaptive enhancement component output by the adapter; inputting the adaptive enhancement component, Y component transform results, U component transform results, and V component transform results into an ICCI processing module and obtaining the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module; performing inverse wavelet transforms on the Y enhancement component, U enhancement component, and V enhancement component respectively to obtain the output Y component, output U component, and output V component; and obtaining the output image based on the output Y component, output U component, and output V component.

[0150] Here, the input image refers to the image input to... Figure 1 The image shown is from the content adaptive filter 118. The first Y component, first U component, and first V component mentioned above represent the Y, U, and V components of the input image, respectively.

[0151] Figure 25 This is a schematic flowchart illustrating another image reconstruction and enhancement method provided in an embodiment of this application. Figure 25 As shown, the method includes the following steps: 3401. Use bicubic interpolation on the first U component and the first V component of the input image to obtain a second U component and a second V component with the same shape as the first Y component of the input image.

[0152] The first Y component of the input image has a different shape from the first U component and the first V component. Since channel-dimensional stitching is required later, the input images of the first U component and the first V component need to be adjusted by bicubic interpolation to obtain the second U component and the second V component with the same shape as the first Y component of the input image.

[0153] 3402. Perform forward wavelet transform on the first Y component, the second U component, and the second V component respectively to obtain the transform results of the Y component, the U component, and the V component.

[0154] Specifically, the above-mentioned forward wavelet transform of the first Y component, the first U component, and the first V component of the input image to obtain the Y component transformation result, the U component transformation result, and the V component transformation result can be achieved through the above steps 3401 and 3402.

[0155] The above-mentioned Y component transformation result refers to the transformation result obtained by performing a forward wavelet transform on the first Y component, the U component transformation result refers to the transformation result obtained by performing a forward wavelet transform on the second U component, and the V component transformation result refers to the transformation result obtained by performing a forward wavelet transform on the second V component.

[0156] 3403. The transformation results of the Y component, U component, and V component are concatenated to obtain the concatenated component.

[0157] 3404. Input the spliced ​​components into the adapter and obtain the adaptive augmented components output by the adapter.

[0158] The descriptions of steps 3401, 3402, 3403, and 3404 above can be referenced to the descriptions of steps 3001, 3002, and 3003 above. The processes implemented in these two parts are similar, and will not be repeated here.

[0159] The adapter in step 3404 is the same as the adapter described in step 3003 above, and will not be repeated here.

[0160] 3405. Input the adaptive enhancement component, Y component transformation result, U component transformation result and V component transformation result into the ICCI processing module, and obtain the Y enhancement component, U enhancement component and V enhancement component output by the ICCI processing module.

[0161] The process of inputting the adaptive enhancement component, Y component transformation results, U component transformation results, and V component transformation results into the ICCI processing module and obtaining the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module can include: inputting the adaptive enhancement component and Y component transformation results into the ICCI processing module to obtain the Y enhancement component output by the ICCI processing module; inputting the adaptive enhancement component and U component transformation results into the ICCI processing module to obtain the U enhancement component output by the ICCI processing module; and inputting the adaptive enhancement component and V component transformation results into the ICCI processing module to obtain the V enhancement component output by the ICCI processing module.

[0162] The aforementioned ICCI processing module can enhance the Y component based on the Y component transformation result and the adaptive enhancement component to obtain the Y-enhanced component. The ICCI processing module can also enhance the U component based on the U component transformation result and the adaptive enhancement component to obtain the U-enhanced component. Furthermore, it can enhance the V component based on the V component transformation result and the adaptive enhancement component to obtain the V-enhanced component.

[0163] Figure 25 In the image reconstruction and enhancement method shown, the adapter-based ICCI subnetwork also has only one ICCI processing module, but the ICCI subnetwork corresponding to this method is different from that described above. Figure 20A The ICCI subnetwork shown is different in that the input of the ICCI processing module is different. At this time, the input not only includes the adaptive enhancement component after the adapter processing, but also the Y component transformation result, U component transformation result and V component transformation result. In addition, the output of the ICCI processing module is also changed from one channel to three channels, namely the Y enhancement component, U enhancement component and V enhancement component.

[0164] Figure 26 This is a schematic diagram of another adapter-based ICCI subnetwork provided in an embodiment of this application. Figure 26 The ICCI subnetwork shown is Figure 20A Compared to the schematic diagram of the ICCI subnetwork shown, the similarity lies in that: this ICCI subnetwork includes a unified ICCI processing module, and the input of this ICCI subnetwork is the Y, U, and V components of the image. Figure 26 The Y component of the image is represented as The U component of the image is represented as The V component of the image is represented as The U component of the image is obtained by bicubic interpolation to obtain a U component with the same shape as the Y component. The V component of the image is obtained by bicubic interpolation to obtain a V component with the same shape as the Y component. Then the Y component U component and V component All components undergo a forward wavelet transform to obtain components 2901, 2902, and 2903. Components 2901, 2902, and 2903 are then concatenated along the channel dimension. The concatenated component 2904 is then input into adapter 2905. After adaptive processing of the concatenated component 2904 by adapter 2905, an adaptively enhanced component is obtained. This adaptively enhanced component is then input into ICCI processing module 2906 for enhancement processing.

[0165] Figure 26 The ICCI subnetwork shown is Figure 20A The differences between the schematic diagram of the ICCI subnetwork shown are as follows: Figure 26 The components 2901, 2902, and 2903 are also input to the ICCI processing module 2906. The ICCI processing module 2906 can obtain the Y-enhanced component 3501 based on component 2901 and the adaptive enhancement component, obtain the U-enhanced component 3502 based on component 2902 and the adaptive enhancement component, and obtain the V-enhanced component 3503 based on component 2903 and the adaptive enhancement component.

[0166] Then, the Y-enhancement component 3501, U-enhancement component 3502, and V-enhancement component 3503 are subjected to inverse wavelet transform to obtain the output Y component, output U component, and output V component, and the output image is obtained based on the output Y component, output U component, and output V component. Figure 26 The output Y component, output U component, and output V component are respectively represented as follows: , , .

[0167] exist Figure 26 In the ICCI sub-network shown, the structure of the ICCI processing module is similar to... Figure 22 The structures shown are similar, but... Figure 26 In the ICCI subnetwork shown, since there is only one ICCI processing module, therefore... Figure 26 China adopts Figure 22 When the ICCI processing module is shown, the number of 1D residual blocks Nb is a constant.

[0168] Figure 26 In the ICCI sub-network shown, the number Nb of 1D residual blocks in the ICCI processing module can be an integer greater than 8 and less than 13.

[0169] in, Figure 26 The number of 1D residual blocks can be determined according to Figure 22The number of 1D residual blocks is determined by 8, 13, and the average of 13. For example, it can be determined by rounding up from the average value to 12, or by rounding down from the average value to 11.

[0170] 3406. Perform inverse wavelet transform on the Y-enhanced component, U-enhanced component and V-enhanced component respectively to obtain the output Y component, output U component and output V component.

[0171] Specifically, the Y-enhanced component is subjected to inverse wavelet transform to obtain the output Y component; the U-enhanced component is subjected to inverse wavelet transform to obtain the output U component; and the V-enhanced component is subjected to inverse wavelet transform to obtain the output V component.

[0172] 3407. Based on the output Y component, output U component and output V component, the output image is obtained.

[0173] The image reconstruction and enhancement method provided in this application uses an adapter-based ICCI sub-network in the content adaptive filter of the JPEG AI verification model. This adapter-based ICCI sub-network introduces an adapter to adaptively process the Y, U, and V components, enhancing the input image based on the characteristics of each Y, U, and V component. This eliminates the need for separate processing of the three components through different ICCI modules; instead, a unified ICCI module is used. This simplifies the content adaptive filter architecture from three ICCI modules to a single ICCI module, effectively reducing the computational complexity of the JPEG AI verification model. In some embodiments, when using the unified ICCI module, each Y, U, and V component, along with the result of the adapter-based adaptive processing, is used as input to the ICCI module. This allows the ICCI module to obtain the Y-enhanced component, U-enhanced component, and V-enhanced component corresponding to Y, U, and V respectively, further considering the characteristics of different components and resulting in an enhanced image that includes more features of each component.

[0174] In some embodiments of this application, before determining the stitching components based on the first Y component, first U component, and first V component of the input image, the following scheme may also be performed: A 1×1 convolution is performed on the target latent tensor to calculate the predicted value corresponding to the first pixel position in the latent space of the target latent tensor, where the target latent tensor is the latent tensor obtained after entropy decoding and super-prior decoding of the first bitstream; the first residual tensor is added to the predicted value corresponding to the first pixel position to obtain the second latent tensor, where the first residual tensor is the value corresponding to the first pixel position in the target residual tensor. The residual tensor is obtained by performing entropy decoding and inverse gain processing on the second bitstream. The second potential tensor is convolved with a 3×3 mask to obtain the second context feature corresponding to the second pixel position. The second context feature and the target potential tensor are deduced by entropy probability to obtain the predicted value corresponding to the two pixel positions. The second residual tensor is added to the predicted value of the two pixel positions to obtain the first potential tensor. The second residual tensor is the residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor. The first latent tensor is convolved using a 5×5 mask to obtain the first context feature corresponding to the third pixel position. The first context feature, the second context feature, and the target latent tensor are then used to derive the prediction value corresponding to all pixel positions through entropy probability derivation. Based on the prediction value of all pixel positions, the input image is determined.

[0175] In the latent space, the top left corner of each 2×2 block corresponds to the first pixel position, the bottom right corner of each 2×2 block corresponds to the second pixel position, and the top right and bottom left corners of each 2×2 block correspond to the third pixel position. In some embodiments of this application, determining the input image based on the predicted values ​​of all pixel positions may include, but is not limited to: adding the third residual tensor to the predicted values ​​of all pixel positions to obtain the output latent tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor; and performing a synthesis transformation on the output latent tensor to obtain the input image.

[0176] In other words, before determining the stitching components based on the first Y component, first U component, and first V component of the input image, embodiments of this application also provide, as follows: Figure 18 The method shown is for determining the predicted value by... Figure 18 The method shown calculates the predicted values ​​corresponding to all pixel locations, and after obtaining the output latent tensor based on the predicted values ​​of all pixel locations, the input image can be further determined.

[0177] The determination of the input image based on the predicted values ​​of all pixel positions may include, but is not limited to, adding the third residual tensor to the predicted values ​​of all pixel positions to obtain the output latent tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor, and performing a synthesis transformation on the output latent tensor to obtain the input image.

[0178] The above-mentioned synthetic transformation process can be performed by Figure 9 The synthesis transformation network 117 is implemented in the model.

[0179] In this embodiment of the application, by means of Figure 18 The method for determining the predicted values, as shown, calculates the predicted values ​​corresponding to all pixel positions. Based on these predicted values, the input image can be determined, and then reconstructed and enhanced using the image reconstruction and enhancement method described above, yielding the final output image. This approach achieves two advantages: First, when calculating the contextual features of the second and third pixel positions, not only the elements to the upper left and lower right of the current pixel can be utilized, improving the accuracy of the calculated contextual features and resulting in a more accurate input image. Second, during the reconstruction and enhancement of the input image, an adapter-based ICCI subnetwork is used in the content adaptive filter. This adapter-based ICCI subnetwork introduces an adapter to adaptively process the Y, U, and V components, enhancing the input image based on the features of each Y, U, and V component. This eliminates the need to process the three components separately using different ICCI modules; instead, a unified ICCI module is used, effectively reducing the computational complexity of the JPEG AI verification model.

[0180] Some embodiments of this application provide a decoding device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the above-described prediction value determination method.

[0181] Figure 27 This is a schematic diagram of the hardware structure of a decoding device provided in some embodiments of this application.

[0182] like Figure 27 As shown, the decoding device provided in this embodiment includes: a processor 271 and a memory 272; In some embodiments, the processor 271 is configured to: perform forward wavelet transforms on the first Y component, the first U component, and the first V component of the input image respectively to obtain Y component transform results, U component transform results, and V component transform results; concatenate the Y component transform results, the U component transform results, and the V component transform results to obtain a concatenated component; input the concatenated component into an adapter and obtain the adaptive enhancement component output by the adapter, wherein the adapter is trained based on a convolutional layer framework and learns the features of the three Y, U, and V components during training to have adaptive processing capabilities for the three Y, U, and V components; input the adaptive enhancement component, the Y component transform results, the U component transform results, and the V component transform results into an ICCI processing module and obtain the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module; perform inverse wavelet transforms on the Y enhancement component, the U enhancement component, and the V enhancement component respectively to obtain an output Y component, an output U component, and an output V component; and obtain an output image based on the output Y component, the output U component, and the output V component.

[0183] In some embodiments of this application, the processor 271 is specifically configured to: perform forward wavelet transform on the first Y component, first U component, and first V component of the input image respectively to obtain the Y component transform result, the U component transform result, and the V component transform result, including: using bicubic interpolation on the first U component and the first V component of the input image to obtain a second U component and a second V component with the same shape as the first Y component of the input image; and performing forward wavelet transform on the first Y component, the second U component, and the second V component respectively to obtain the Y component transform result, the U component transform result, and the V component transform result.

[0184] In some embodiments of this application, the processor 271 is specifically configured to: input the adaptive enhancement component, the Y component transformation result, the U component transformation result, and the V component transformation result to the ICCI processing module, and obtain the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module, including: inputting the adaptive enhancement component and the Y component transformation result to the ICCI processing module to obtain the Y enhancement component output by the ICCI processing module; inputting the adaptive enhancement component and the U component transformation result to the ICCI processing module to obtain the U enhancement component output by the ICCI processing module; and inputting the adaptive enhancement component and the V component transformation result to the ICCI processing module to obtain the V enhancement component output by the ICCI processing module.

[0185] In some embodiments of this application, the training set for training the framework of the convolutional layer includes a JPEG AI dataset and images compressed at different bitrates corresponding to the JPEG AI data.

[0186] In some embodiments of this application, the ICCI processing module includes: a 3×3 convolution, a batch normalization layer, a ReLU activation function, and Nb 1D residual blocks.

[0187] In some embodiments of this application, Nb is greater than 8 and less than 13.

[0188] In some embodiments of this application, the processor 271 is configured to: before determining the splicing components based on the first Y component, first U component, and first V component of the input image, perform a 1×1 convolution on the target latent tensor to calculate the predicted value corresponding to the first pixel position in the latent space of the target latent tensor, wherein the target latent tensor is a latent tensor obtained after entropy decoding and super-prior decoding of the first bitstream; add the first residual tensor to the predicted value corresponding to the first pixel position to obtain a second latent tensor, wherein the first residual tensor is the residual value corresponding to the first pixel position in the target residual tensor. The target residual tensor is the residual tensor obtained after entropy decoding and inverse gain processing of the second bitstream; the second potential tensor is convolved by a 3×3 mask convolution to obtain the second context feature corresponding to the second pixel position; the second context feature and the target potential tensor are deduced by entropy probability to obtain the predicted value corresponding to the two pixel positions; the second residual tensor is added to the predicted value of the two pixel positions to obtain the first potential tensor, and the second residual tensor is the residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor; A first latent tensor is convolved using a 5×5 mask convolution to obtain a first context feature corresponding to the third pixel position. The first context feature, the second context feature, and the target latent tensor are then subjected to entropy probability derivation to obtain predicted values ​​corresponding to all pixel positions. Based on the predicted values ​​for all pixel positions, the input image is determined. In the latent space, the top-left corner of each 2×2 block corresponds to the first pixel position, the bottom-right corner of each 2×2 block corresponds to the second pixel position, and the top-right and bottom-left corners of each 2×2 block correspond to the third pixel position.

[0189] In some embodiments of this application, the processor 271 is specifically configured to: determine the input image based on the predicted values ​​of all pixel positions, including: adding the third residual tensor to the predicted values ​​of all pixel positions to obtain an output latent tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor; and performing a synthesis transformation process on the output latent tensor to obtain the input image.

[0190] This application provides a computer-readable storage medium in some embodiments, characterized in that a computer program is stored on the computer-readable storage medium. When executed by a processor, the computer program implements one or more of the image feature processing method, prediction value determination method, and image reconstruction enhancement method described in the above-described method embodiments, and achieves the same technical effect. To avoid repetition, further details are omitted here. In this application, the computer-readable medium includes permanent and non-permanent, removable and non-removable storage media. The storage medium can implement information storage by any method or technology, and the information can be computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0191] Some embodiments of this application provide a computer program product that stores a computer program. When the computer program is executed by a processor, it implements one or more of the image feature processing method, prediction value determination method, and image reconstruction enhancement method in the above-described method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0192] In this application, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0193] In this application, the memory may include non-permanent memory in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM.

[0194] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0195] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image reconstruction and enhancement method, characterized in that, include: Forward wavelet transforms are performed on the first Y component, the first U component, and the first V component of the input image to obtain the Y component transform result, the U component transform result, and the V component transform result, respectively. The transformation results of the Y component, the U component, and the V component are concatenated to obtain the concatenated component. The spliced ​​components are input into an adapter, and the adaptive enhanced components output by the adapter are obtained. The adapter is trained based on a convolutional layer framework. During training, it learns the features of the three components Y, U, and V to have adaptive processing capabilities for the three components Y, U, and V. The adapter is used to process the spliced ​​components sequentially through a first 1×1 convolution and a first normalization layer. The output features of the first normalization layer are input into a channel attention block and a spatial attention block, respectively. The output of the first 1×1 convolution, the output of the channel attention block, and the output of the spatial attention block are added and fused through a first addition fusion module. The addition fusion result of the first addition fusion module is processed sequentially through a second normalization layer and a multilayer perceptron. The addition fusion result and the output of the multilayer perceptron are added and fused through a second addition fusion module. The output of the second addition fusion module is convolved through a second 1×1 convolution to obtain the adaptive enhanced components. The adaptive enhancement component, the Y component transformation result, the U component transformation result, and the V component transformation result are input to the ICCI processing module, and the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module are obtained. Inverse wavelet transforms are performed on the Y-enhanced component, the U-enhanced component, and the V-enhanced component respectively to obtain the output Y-component, the output U-component, and the output V-component; The output image is obtained based on the output Y component, the output U component, and the output V component.

2. The method according to claim 1, characterized in that, The step of performing forward wavelet transforms on the first Y component, first U component, and first V component of the input image to obtain the Y component transform result, U component transform result, and V component transform result includes: Bicubic interpolation is applied to the first U component and the first V component of the input image to obtain a second U component and a second V component with the same shape as the first Y component of the input image; The first Y component, the second U component, and the second V component are respectively subjected to forward wavelet transform to obtain the transform results of the Y component, the U component, and the V component.

3. The method according to claim 1, characterized in that, The step of inputting the adaptive enhancement component, the Y component transformation result, the U component transformation result, and the V component transformation result into the ICCI processing module, and obtaining the Y enhancement component, U enhancement component, and V enhancement component output by the ICCI processing module, includes: The adaptive enhancement component and the Y component transformation result are input into the ICCI processing module to obtain the Y enhancement component output by the ICCI processing module; The adaptive enhancement component and the U component transformation result are input into the ICCI processing module to obtain the U enhancement component output by the ICCI processing module; The adaptive enhancement component and the V component transformation result are input to the ICCI processing module to obtain the V enhancement component output by the ICCI processing module.

4. The method according to claim 1, characterized in that, The training set used to train the framework of the convolutional layer includes the JPEG AI dataset and images compressed at different bitrates corresponding to the JPEG AI data.

5. The method according to claim 1, characterized in that, The ICCI processing module includes: a 3×3 convolution, a batch normalization layer, a ReLU activation function, and Nb 1D residual blocks.

6. The method according to claim 5, characterized in that, The Nb is greater than 8 and less than 13.

7. The method according to claim 1, characterized in that, Before determining the stitching components based on the first Y component, first U component, and first V component of the input image, the method further includes: A 1×1 convolution is performed on the target latent tensor to calculate the predicted value corresponding to the first pixel position in the latent space of the target latent tensor. The target latent tensor is the latent tensor obtained after entropy decoding and super-prior decoding of the first bitstream. The first residual tensor is added to the predicted value corresponding to the first pixel position to obtain the second potential tensor. The first residual tensor is the residual tensor corresponding to the first pixel position in the target residual tensor. The target residual tensor is the residual tensor obtained after entropy decoding and inverse gain processing of the second bitstream. The second latent tensor is convolved using a 3×3 mask convolution to obtain the second contextual features corresponding to the second pixel location. By using entropy probability derivation of the second context feature and the target latent tensor, the predicted values ​​corresponding to the two pixel positions can be obtained. The second residual tensor is added to the predicted values ​​of the two pixel positions to obtain the first potential tensor. The second residual tensor is the residual tensor corresponding to the first pixel position and the second pixel position in the target residual tensor. The first latent tensor is convolved using a 5×5 mask convolution to obtain the first contextual feature corresponding to the third pixel position. By using entropy probability derivation of the first context feature, the second context feature, and the target latent tensor, the predicted values ​​corresponding to all pixel positions can be obtained. In the potential space, the top left corner of each 2×2 block corresponds to the position of the first pixel, the bottom right corner of each 2×2 block corresponds to the position of the second pixel, and the top right and bottom left corners of each 2×2 block correspond to the position of the third pixel. The input image is determined based on the predicted values ​​for all pixel locations.

8. The method according to claim 7, characterized in that, Determining the input image based on the predicted values ​​of all pixel locations includes: The third residual tensor is added to the predicted values ​​of all pixel positions to obtain the output latent tensor, wherein the third residual tensor is the residual tensor corresponding to all positions in the target residual tensor; The output latent tensor is subjected to a synthesis transformation to obtain the input image.

9. A decoding device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the image reconstruction enhancement method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the image reconstruction and enhancement method as described in any one of claims 1 to 8.

Images