Decoding method, device and equipment

By performing feature enhancement and wavelet transformation of the brightness and chrominance components of the image block, combined with the residual block network, the decoding performance and complexity problems in the neural network encoding and decoding methods are solved, and efficient compression is achieved under multiple code rates Defect recovery.

CN119011866AActive Publication Date: 2024-11-22HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411389874.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-03-08
Publication Date
2024-11-22
Estimated Expiration
2043-03-08

AI Technical Summary

Technical Problem

Neural network-based codec methods have shortcomings in decoding performance and complexity, especially in reducing computational complexity and resource requirements, especially for cost- and delay-sensitive devices.

Method used

A decoding method is proposed to enhance the luminance components and chrominance components of the image block, generate adjusted components using Haar wavelet transformation and feature stacking, and enhance feature through residual block network to improve decoding performance and reduce complexity Spend.

Benefits of technology

Improve decoding performance, reduces computational complexity and resource requirements, making this method suitable for cost- and delay-sensitive devices, and can effectively recover compression defects at multiple code rates.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011866A_ABST
    Figure CN119011866A_ABST
Patent Text Reader

Abstract

The present disclosure provides a decoding method, device and equipment, the decoding method comprising: decoding a code stream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block comprising a first initial component and a second initial component, the resolution of the first initial component is greater than or equal to that of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and performing feature enhancement on the adjusted component to obtain a recovered target component corresponding to the second initial component. Through the technical scheme of the invention, the decoding performance can be improved, and the decoding complexity can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of coding and decoding technology, and in particular to a decoding method, device and equipment thereof. Background Art

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy coding, filtering and other processes. For the prediction process, the prediction process can include intra-frame prediction and inter-frame prediction. Inter-frame prediction refers to the use of the correlation in the video time domain to use the pixels of the adjacent encoded images to predict the current pixel, so as to effectively remove the redundancy in the video time domain. Intra-frame prediction refers to the use of the correlation in the video space domain to use the pixels of the encoded blocks of the current frame image to predict the current pixel, so as to remove the redundancy in the video space domain.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification and target detection. Deep learning has also gradually begun to be applied in the field of encoding and decoding, that is, neural networks can be used to encode and decode images. Although the encoding and decoding methods based on neural networks have shown great performance potential, they still have problems such as poor decoding performance and high complexity. Summary of the invention

[0004] In view of this, the present disclosure provides a decoding method, apparatus and device thereof, which can improve decoding performance.

[0005] In a first aspect, the present disclosure provides a decoding method, comprising:

[0006] Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0007] Generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0008] Feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0009] In some embodiments, the reconstructed image block is a reconstructed image block in a YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and a chrominance V component.

[0010] In some embodiments, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes:

[0011] Acquire a first image feature corresponding to the first initial component, wherein the first image feature includes an image feature that has not passed through a neural network;

[0012] Acquire a second image feature corresponding to the second initial component, where the second image feature includes an image feature that has not passed through a neural network;

[0013] The adjusted component is generated based on the first image characteristic and the second image characteristic.

[0014] In some embodiments, the first image feature comprises a Haar wavelet frequency domain feature; and the second image feature comprises a Haar wavelet frequency domain feature.

[0015] In some embodiments, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes:

[0016] Performing at least one Haar wavelet transform on the first initial component to obtain a plurality of transformed frequency bands as a first feature map;

[0017] Performing at least one Haar wavelet transform on the second initial component to obtain a plurality of transformed frequency bands as a second feature map;

[0018] The first characteristic map and the second characteristic map are overlapped to obtain the adjusted component.

[0019] In some embodiments, the stacking operation on the first feature map and the second feature map includes:

[0020] The first feature map and the second feature map are overlapped along the channel dimension.

[0021] In some embodiments, performing at least one Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the transform includes: performing two Haar wavelet transforms on the first initial component to obtain a plurality of frequency bands after the two Haar wavelet transforms;

[0022] The performing at least one Haar wavelet transform on the second initial component to obtain a plurality of frequency bands after the transform includes: performing two Haar wavelet transforms on the second initial component to obtain a plurality of frequency bands after the two Haar wavelet transforms.

[0023] In some embodiments, performing multiple Haar wavelet transforms on the first initial component to obtain multiple frequency bands after the transforms includes: performing two Haar wavelet transforms on the first initial component to obtain multiple frequency bands after the two Haar wavelet transforms;

[0024] The performing Haar wavelet transform on the second initial component multiple times to obtain multiple frequency bands after the transform includes: performing Haar wavelet transform on the second initial component twice to obtain multiple frequency bands after the two Haar wavelet transforms.

[0025] In some embodiments, performing Haar wavelet transform twice on the first initial component to obtain multiple frequency bands after the two Haar wavelet transforms includes: performing a first Haar wavelet transform on the first initial component to obtain multiple frequency bands after the first Haar wavelet transform; performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain the multiple frequency bands after the two Haar wavelet transforms;

[0026] The method of performing two Haar wavelet transforms on the second initial component to obtain multiple frequency bands after the two Haar wavelet transforms includes: performing a first Haar wavelet transform on the second initial component to obtain multiple frequency bands after the first Haar wavelet transform; and performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain multiple frequency bands after the two Haar wavelet transforms.

[0027] In some embodiments, the step of performing feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component includes:

[0028] The adjusted component is feature enhanced through multiple residual block networks to obtain the target component.

[0029] In a second aspect, the present disclosure provides an encoding method, comprising:

[0030] A first initial component and a second initial component of a current image block are encoded to obtain a code stream corresponding to the current image block, wherein a resolution of the first initial component is greater than or equal to a resolution of the second initial component.

[0031] In some embodiments, the current image block is in YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and a chrominance V component.

[0032] In a third aspect, the present disclosure provides a decoding device, including:

[0033] A decoding module, configured to decode a code stream corresponding to a current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0034] a determination module, configured to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0035] A processing module is used to perform feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0036] In a fourth aspect, the present disclosure provides an encoding device, including:

[0037] The encoding module is used to encode the first initial component and the second initial component of the current image block to obtain a code stream corresponding to the current image block, wherein the resolution of the first initial component is greater than or equal to the resolution of the second initial component.

[0038] In a fifth aspect, the present disclosure provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method described in any embodiment of the first aspect.

[0039] In the sixth aspect, the present disclosure provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method described in any embodiment of the second aspect.

[0040] In a seventh aspect, the present disclosure provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method described in any embodiment of the first aspect is implemented, or the method described in any embodiment of the second aspect is implemented.

[0041] In an eighth aspect, the present disclosure provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the method described in any embodiment of the first aspect is implemented, or the method described in any embodiment of the second aspect is implemented.

[0042] It can be seen from the above technical scheme that in the disclosed embodiment, after decoding to obtain a reconstructed image block, the reconstructed image block includes a first initial component and a second initial component, the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, that is, the second initial component is adjusted based on the first initial component to obtain the adjusted component, and then the target component corresponding to the second initial component is determined based on the adjusted component, thereby improving the decoding performance, reducing the decoding complexity, reducing the computational complexity, and requiring less resources. Devices that are sensitive to cost and delay can also use the above algorithm, and propose a low-complexity and efficient post-processing network model framework for compression defect recovery at multiple bit rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present disclosure;

[0044] Figure 2 is a flowchart of a decoding method in one embodiment of the present disclosure;

[0045] Figure 3 is a schematic diagram of a processing process of an encoding end in one embodiment of the present disclosure;

[0046] Figure 4 is a schematic diagram of a processing process of a decoding end in one embodiment of the present disclosure;

[0047] Figure 5A is a schematic diagram of the structure of end-to-end image compression in one embodiment of the present disclosure;

[0048] Figure 5B is a structural schematic diagram of post-processing based on deep learning in one embodiment of the present disclosure;

[0049] Figure 6A-Figure 6I is a schematic diagram of the structure of a post-processing network in one embodiment of the present disclosure;

[0050] Figure 7A-7C is a schematic diagram of upsampling in one embodiment of the present disclosure;

[0051] Fig.7D is a schematic diagram of the structure of a post-processing network in one embodiment of the present disclosure;

[0052] Fig. 7E and Figure 7F is a schematic diagram of feature enhancement in one embodiment of the present disclosure;

[0053] Figure 8 It is a hardware structure diagram of a decoding end device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, rather than for limiting the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "at the time of...", or "when...", or "in response to determination".

[0055] A decoding method is proposed in an embodiment of the present disclosure, and the decoding method may involve the following concepts:

[0056] JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous tone static image compression. The file extension can be .jpg or .jpeg. JPEG is a commonly used image file format. JPEG uses a joint coding method of predictive coding (DPCM), discrete cosine transform (DCT) and entropy coding to remove redundant image and color data. It is a lossy compression format that can compress images in a very small storage space, which will cause damage to image data to a certain extent. In particular, when using too high a compression ratio, the quality of the image restored after the final decompression will be reduced. If you pursue high-quality images, JPEG should not use too high a compression ratio.

[0057] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation that significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, effectively improving performance in image processing and computer vision tasks. JPEG-AI targets a wide range of applications such as cloud storage, vision management, self-driving cars and devices, image acquisition storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a codec solution that needs to significantly improve compression efficiency at the same subjective quality and provide efficient compressed domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires hardware- and software-friendly encoding and decoding, support for 8-bit and 10-bit depths, and efficient encoding and progressive decoding of images with text and graphics.

[0058] Post-processing filter: The encoder compresses the image to form a stream, and the decoder decodes and reconstructs the image from the stream. However, due to the encoding algorithm itself, after the image is decoded and reconstructed from the stream, defects such as block effects, image artifacts and color drift will appear in the image. The role of post-processing filter is to improve the defects of the image caused by encoding compression, restore the image data to the greatest extent, and improve the subjective quality of the decoded and reconstructed image.

[0059] Image Super-Resolution: Super-Resolution mainly refers to restoring a low-resolution image (Low-Resolution) to a high-resolution (High-Resolution) image, while enhancing image details and improving the subjective quality of the image. Image super-resolution is a technology used to enhance image resolution in computer vision image processing.

[0060] YUV Color Space: Each pixel of a color image can usually be described by several independent physical quantities, thus forming a spatial coordinate, which is the color space of the image. YUV color space is a color space in which the pixels of a color image are described by three attributes: Y (brightness), U (chroma), and V (chroma).

[0061] Entropy coding: Entropy coding is the coding process that does not lose any information according to the entropy principle. Information entropy is the average amount of information of the source (a measure of uncertainty). Entropy coding methods may include but are not limited to: Shannon coding, Huffman coding, and arithmetic coding.

[0062] Neural Network (NN): Neural network refers to artificial neural network. Neural network is a computing model composed of a large number of nodes (or neurons) connected to each other. In a neural network, neuron processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. The types of processing units in a neural network can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the outside world; output units realize the output of processing results; hidden units are units between input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and the representation and processing of information are reflected in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamic behavior of neural network, and imitate the information processing function of human brain nervous system to different degrees and levels. In the field of video processing, commonly used neural networks may include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN), fully connected network, etc.

[0063] Convolutional Neural Network (CNN): Convolutional neural network is a feedforward neural network and one of the most representative network structures in deep learning technology. The artificial neurons of convolutional neural network can respond to surrounding units within a certain coverage range and have excellent performance in large image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer). The input of each neuron is connected to the local receptive field of the previous layer and extracts the local features. Once the local feature is extracted, its positional relationship with other features is also determined. The second is the feature mapping layer (also called activation layer). Each computing layer of the neural network consists of multiple feature maps. Each feature map is a plane, and the weights of all neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of the convolutional network. In addition, since neurons on a mapping surface share weights, the number of free parameters of the network can be reduced.

[0064] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid the complex pre-processing of images (extracting artificial features, etc.), and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected, which will result in a huge number of parameters, making network training time-consuming or even difficult to train, while convolutional neural networks avoid this difficulty through local connections, weight sharing and other methods.

[0065] Deconvolution layer: Deconvolution layer is also called transposed convolution layer. The working process of deconvolution layer and convolution layer is very similar. The main difference is that deconvolution layer will make the output larger than the input (or keep it the same) through padding. If stride is 1, it means that the output size is equal to the input size; if stride is N, it means that the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0066] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to fresh samples. The purpose of learning is to learn the rules hidden behind the data pairs. For data outside the learning set with the same rules, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0067] Feature: The feature involved in this disclosure is a three-dimensional feature matrix of C*W*H, see Figure 1 As shown, it is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of a neural network or the output of a neural network.

[0068] Rate-Distortion Optimized: There are two major indicators for evaluating coding efficiency: bit rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of the two. For example, the cost corresponding to the mode: J(mode) = D + λ*R, where D represents Distortion, which can usually be measured using the SSE indicator. SSE refers to the mean square sum of the difference between the reconstructed image block and the source image. In order to achieve cost considerations, the SAD indicator can also be used. SAD refers to the sum of the absolute values ​​of the difference between the reconstructed image block and the source image; λ is the Lagrange multiplier, and R is the actual number of bits required for encoding the image block in this mode, including the sum of bits required for encoding mode information, motion information, residuals, etc. When selecting a mode, if the rate-distortion principle is used to make a comparison decision on the encoding mode, the best encoding performance can usually be guaranteed.

[0069] A large number of coding tools have been proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the coding tools that can achieve the best coding performance are often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bitstream. Although this method brings higher coding complexity, it can adaptively select the optimal mode combination for different content to obtain the best coding performance. The decoding end can obtain relevant mode information by directly parsing the flag information, and the complexity impact is small.

[0070] In the end-to-end image compression technology based on deep learning, the image is converted into the YUV color space, and the UV chromaticity signal is downsampled by half, and then the Y signal and UV signal are compressed and encoded separately. The decoding end decodes and reconstructs the bit stream data to obtain the reconstructed Y signal and UV signal. Since the UV chromaticity signal is compressed and encoded after being downsampled by half, that is, the resolution of the UV chromaticity signal is halved, the resolution of the UV chromaticity signal can also be restored in the post-processing process. However, the computational complexity of the resolution recovery process is high and the demand for resources is relatively large. Some devices that are sensitive to cost and delay cannot restore the resolution of the UV chromaticity signal.

[0071] In view of the above findings, a neural network-based compression defect removal network model method is proposed in this embodiment. In the field of end-to-end image compression, a low-complexity and efficient post-processing network model framework is proposed, which can enhance the UV signal based on the Y signal. This method has low complexity and high performance, and can recover compression defects under various bit rates.

[0072] The decoding method in the embodiment of the present disclosure is described in detail below in conjunction with several specific embodiments.

[0073] Embodiment 1: A decoding method is proposed in the embodiment of the present disclosure, see Figure 2 FIG. 2 is a flow chart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method can include steps 201-203.

[0074] Step 201: Decode the code stream corresponding to the current image block to obtain a reconstructed image block.

[0075] Exemplarily, the reconstructed image block may include a first initial component and a second initial component, the resolution of the second initial component is a downsampled resolution, and the resolution of the first initial component may be greater than or equal to the resolution of the second initial component. For example, the resolution of the first initial component may be a resolution that has not been downsampled (i.e., the original resolution) or a downsampled resolution, but the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and there is no limitation on this.

[0076] Exemplarily, if the reconstructed image block is a reconstructed image block in YUV format, the first initial component may be a luminance component, and the second initial component may be a chrominance U component and / or a chrominance V component; or, if the reconstructed image block is a reconstructed image block in RGB format, the first initial component may be a G component, and the second initial component may be an R component and / or a B component.

[0077] Step 202: Generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component. The reconstruction quality of the adjusted component may be better than the reconstruction quality of the second initial component, the reconstruction quality of the adjusted component may be equal to the reconstruction quality of the second initial component, and the reconstruction quality of the second initial component may also be better than the reconstruction quality of the adjusted component.

[0078] Exemplarily, a first image feature corresponding to a first initial component may be obtained, and the first image feature may include but is not limited to image features that have not passed through a neural network and / or image features that have passed through a neural network; a second image feature corresponding to a second initial component may be obtained, and the second image feature may include but is not limited to image features that have not passed through a neural network and / or image features that have passed through a neural network; an adjusted component may be generated based on the first image feature and the second image feature.

[0079] Exemplarily, if the first image feature includes an image feature that has not been processed by a neural network, the first image feature may include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature; if the second image feature includes an image feature that has not been processed by a neural network, the second image feature may include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The above are just a few examples and are not intended to be limiting.

[0080] If the first image feature includes a first feature map that has passed through a neural network, and the second image feature includes a second feature map that has passed through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component may include but is not limited to: inputting the first initial component to the first neural network to obtain a first feature map; inputting the second initial component to the second neural network to obtain a second feature map; adding the first feature map and the second feature map to obtain an adjusted component; or, stacking the first feature map and the second feature map to obtain an adjusted component.

[0081] Exemplarily, if the first image feature includes a weight coefficient map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component may include but is not limited to: inputting the first initial component to the first neural network to obtain a first feature map corresponding to the first initial component, and performing guided filtering on the first feature map to obtain a weight coefficient map; inputting the second initial component to the second neural network to obtain a second feature map; and generating an adjusted component based on the weight coefficient map and the second feature map.

[0082] Exemplarily, performing guided filtering on the first feature map to obtain a weight coefficient map may include but is not limited to: performing a convolution operation on the first feature map to obtain a convolved feature map; and performing weight mapping on the convolved feature map to obtain a weight coefficient map.

[0083] Exemplarily, weight mapping is performed on the feature map after convolution to obtain a weight coefficient map, including but not limited to: performing a pooling operation on the feature map after convolution to obtain a feature map after pooling operation; performing a full connection operation and a ReLU (Rectified Linear Unit) activation operation on the feature map after pooling operation to obtain a feature map after ReLU activation operation; performing a full connection operation and a Sigmoid (S-shaped curve, i.e., S-shaped function) activation operation on the feature map after ReLU activation operation to obtain a feature map after Sigmoid activation operation; generating a weight coefficient map based on the feature map after convolution and the feature map after Sigmoid activation operation, and multiplying the feature map after convolution and the feature map after Sigmoid activation operation to obtain the weight coefficient map.

[0084] Exemplarily, weight mapping is performed on the convolution feature map to obtain a weight coefficient map, which may include but is not limited to: a convolution operation and a Sigmoid activation operation may be performed on the convolution feature map to obtain a weight coefficient map.

[0085] Exemplarily, generating an adjusted component based on the weight coefficient map and the second feature map may include but is not limited to: multiplying the weight coefficient map and the second feature map to obtain a multiplied feature map; performing a convolution operation on the first feature map to obtain a convolved feature map; adding the multiplied feature map and the convolved feature map to obtain an added feature map; and stacking the added feature map and the convolved feature map to obtain an adjusted component.

[0086] Exemplarily, if the first image feature includes K first feature maps that have passed through a neural network, and the second image feature includes K+1 second feature maps that have passed through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component may include but is not limited to: for the first first feature map and the first second feature map, the first initial component may be input into the neural network to obtain the first first feature map, and the second initial component may be input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, i is any integer greater than or equal to 2 and less than or equal to K, and K may be a positive integer greater than 1, then the i-1th first feature map may be input into the neural network to obtain the i-th first feature map; feature fusion is performed on the i-1th first feature map and the i-1th second feature map to obtain a fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map; after obtaining the last second feature map (i.e., the last second feature map among the K+1 second feature maps), the adjusted component may be generated based on the last second feature map.

[0087] Exemplarily, the first feature map (such as the i-1th first feature map) and the second feature map (such as the i-1th second feature map) are subjected to feature fusion to obtain fused features, which may include but is not limited to: adding the first feature map and the second feature map to obtain fused features; or, superimposing the first feature map and the second feature map to obtain fused features; or, guiding filtering the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating fused features based on the weight coefficient map and the second feature map. The above are just a few examples and are not limited to this.

[0088] Exemplarily, inputting the first initial component into the first neural network to obtain a first feature map may include but is not limited to: extracting M-dimensional image features from the first initial component, where M is a positive integer; stacking the M-dimensional image features to obtain stacked multi-dimensional features; and inputting the multi-dimensional features into the first neural network to obtain a first feature map.

[0089] Exemplarily, the image features of M dimensions may include but are not limited to at least one of the following: image features after convolving the first initial component with a convolution kernel; image features after convolving the first initial component with two convolution kernels; first-order spatial features after processing the first initial component with a sobel operator; second-order spatial features after processing the first initial component with a Laplacian operator; or frequency domain features after Fourier transforming the first initial component. The above are just a few examples of image features and are not limited to them.

[0090] Exemplarily, the first initial component is input into the first neural network to obtain the first feature map, which may include but is not limited to: performing a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform; inputting the multiple frequency bands or part of the multiple frequency bands into the first neural network to obtain the first feature map. The second initial component is input into the second neural network to obtain the second feature map, which may include but is not limited to: performing a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform; inputting the multiple frequency bands or part of the multiple frequency bands into the second neural network to obtain the second feature map.

[0091] When the first initial component is subjected to wavelet transformation to obtain multiple frequency bands after wavelet transformation, the first initial component can be subjected to wavelet transformation once to obtain multiple frequency bands after wavelet transformation, or the first initial component can be subjected to wavelet transformation multiple times to obtain multiple frequency bands after wavelet transformation. When the first initial component is subjected to wavelet transformation multiple times, the first initial component is first subjected to wavelet transformation to obtain multiple frequency bands after wavelet transformation, and then a target frequency band (all or part of the frequency bands in the multiple frequency bands) is selected from the multiple frequency bands, and the target frequency band is subjected to wavelet transformation to obtain multiple frequency bands after wavelet transformation, and so on, until the wavelet transformation is completed to obtain multiple frequency bands after wavelet transformation. When the second initial component is subjected to wavelet transformation to obtain multiple frequency bands after wavelet transformation, the second initial component can be subjected to wavelet transformation once to obtain multiple frequency bands after wavelet transformation, or the second initial component can be subjected to wavelet transformation multiple times to obtain multiple frequency bands after wavelet transformation.

[0092] Exemplarily, before the first initial component is input into the first neural network to obtain the first feature map, the first initial component may be preprocessed to obtain the preprocessed first initial component; wherein the preprocessed first initial component is used to input into the first neural network to obtain the first feature map. Preprocessing the first initial component to obtain the preprocessed first initial component may include but is not limited to: edge enhancement of the first initial component to obtain image features after edge enhancement; multi-scale feature extraction of the image features after edge enhancement to obtain multi-scale features; first initial component preprocessed based on the multi-scale features, such as using the multi-scale features as the preprocessed first initial component.

[0093] Exemplarily, performing multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features may include but is not limited to: performing a convolution operation on the edge-enhanced image features to obtain convolution features; performing a downsampling operation on the convolution features to obtain down-sampled features; performing channel transformation on the down-sampled features to obtain channel-transformed features; performing an upsampling operation on the channel-transformed features to obtain up-sampled features; generating multi-scale features based on the up-sampled features and the channel-transformed features, and then inputting the multi-scale features into the first neural network as the preprocessed first initial component.

[0094] Step 203: Perform feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0095] In a possible implementation, before generating the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the second initial component may be upsampled to obtain the upsampled second initial component, wherein the resolution of the upsampled second initial component may be equal to the resolution of the first initial component.

[0096] In another possible implementation, before feature enhancement is performed on the adjusted component to obtain the restored target component corresponding to the second initial component (i.e., before step 203), the adjusted component may be upsampled to obtain an upsampled adjusted component. The resolution of the upsampled adjusted component may be equal to the resolution of the first initial component.

[0097] In another possible implementation, after the adjusted component is feature enhanced to obtain the restored target component corresponding to the second initial component (i.e., after step 203), the target component may be upsampled to obtain an upsampled target component. The resolution of the upsampled target component may be equal to the resolution of the first initial component.

[0098] Exemplarily, feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component, which may include but is not limited to: feature enhancement is performed on the adjusted component through at least one residual block network to obtain a target component corresponding to the second initial component; or, feature enhancement is performed on the adjusted component through a U-Net (a network with an Encoder-Decoder structure, the first half is feature extraction, and the second half is upsampling) network to obtain a target component corresponding to the second initial component.

[0099] Exemplarily, the above execution order is only for the convenience of describing the examples given. In practical applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in the present disclosure, and the steps included in the method may be more or less than those described in the present disclosure. In addition, a single step described in the present disclosure may be decomposed into multiple steps for description in other embodiments; multiple steps described in the present disclosure may also be combined into a single step for description in other embodiments.

[0100] It can be seen from the above technical scheme that in the disclosed embodiment, after decoding to obtain a reconstructed image block, the reconstructed image block includes a first initial component and a second initial component, the resolution of the second initial component is a downsampled resolution, the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, that is, the second initial component is adjusted based on the first initial component to obtain the adjusted component, and then the target component corresponding to the second initial component is determined based on the adjusted component, thereby improving decoding performance, reducing decoding complexity, reducing computational complexity, and requiring less resources. Devices that are sensitive to cost and delay can also use the above algorithm, and propose a low-complexity and efficient post-processing network model framework for compression defect recovery at multiple bit rates.

[0101] Embodiment 2: For the processing process of the encoding end (the encoding end is also called a video encoder), please refer to Figure 3 As shown, Figure 3 This is just an example of the processing process at the encoding end, and there is no limitation on the processing process at the encoding end.

[0102] After obtaining the current image block x (the current image block x may be the original image block x, i.e., the input image block), the encoder may analyze and transform the current image block x through an analysis transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Among them, performing feature transformation on the current image block x through the analysis transformation network means: transforming the current image block x to an image feature y in the latent domain, thereby facilitating all subsequent processes to operate on the latent domain.

[0103] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0104] After obtaining the image feature y, the encoding end performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs coefficient hyperparameter feature transformation on the image feature y to obtain coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of the hyperparameter encoding network, as long as the coefficient hyperparameter feature transformation can be performed on the image feature y. The image feature y in the latent domain obtains the super prior latent information z after passing through the hyperparameter encoding network.

[0105] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 3 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain Bitstream#1 (i.e., the first bitstream) corresponding to the current image block, i.e. Figure 3 The AE operation in represents the encoding process, such as the entropy encoding process. Alternatively, the encoding end can also directly encode the coefficient hyperparameter feature z to obtain the Bitstream#1 corresponding to the current image block. Among them, the hyperparameter quantization feature or coefficient hyperparameter feature z carried in Bitstream#1 is mainly used to obtain the parameters of the mean and probability distribution model.

[0106] After obtaining the Bitstream#1 corresponding to the current image block, the encoder may send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the decoder for the Bitstream#1 corresponding to the current image block, refer to the subsequent embodiments.

[0107] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 3 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 3 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0108] For the encoding process of Bitstream#1, the encoding method of the fixed probability density model can be adopted, and for the decoding process of Bitstream#1, the decoding method of the fixed probability density model can be adopted. There is no restriction on this encoding and decoding process.

[0109] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain the residual by subtracting from the original feature and to obtain the reconstructed y by adding the decoded residual.

[0110] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 3 The dashed box in indicates that the mean prediction network is optional.

[0111] After obtaining the image feature y, the encoder can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature y, the encoder can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0112] After obtaining the image feature s, the encoder can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 3 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e. Figure 3 The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block without involving the quantization process of the image feature s.

[0113] After obtaining Bitstream#2 corresponding to the current image block, the encoder may send Bitstream#2 corresponding to the current image block to the decoder. For the processing of Bitstream#2 corresponding to the current image block by the decoder, refer to the subsequent embodiments.

[0114] After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 3 AD in the figure represents the decoding process. Then, the encoder can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 3 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0115] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process, and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the prediction value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the prediction value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0116] After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0117] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encode the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes Bitstream#2, it also needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0118] To obtain the probability distribution model, continue to refer to Figure 3 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input to the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network, and it is sufficient to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0119] In a possible implementation, the processing process at the encoding end may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without any limitation on the encoding process.

[0120] Embodiment 3: For the processing process of the decoding end (the decoding end is also called a video decoder), please refer to Figure 4 As shown, Figure 4 This is just an example of the processing process at the decoding end, and there is no limitation on the processing process at the decoding end.

[0121] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 4 AD in it represents the decoding process. Then, the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat can be the same as or different from the coefficient hyperparameter feature z. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoder can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0122] For the decoding process of Bitstream#1, a decoding method of a fixed probability density model may be used, and there is no restriction on this.

[0123] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0124] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in the subsequent embodiments) to obtain the prediction value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0125] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0126] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 The IQ operation in is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoder can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0127] After obtaining the image feature s', the decoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoder determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, and the image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, and the prediction value mu is provided by the mean prediction network. Alternatively, after obtaining the image feature s', the decoder can perform feature recovery on the image feature s' to obtain the image feature y_hat, which is the same as or different from the image feature y. In this case, it is not necessary to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0128] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0129] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probability hyperparameter decoding network, and the probability hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probability hyperparameter decoding network can be a trained neural network, and there is no restriction on the training process of this probability hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0130] In a possible implementation, the processing process at the above decoding end can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, without limiting the decoding process.

[0131] Example 4: See Figure 5AAs shown in the figure, it is a schematic diagram of the structure of end-to-end image compression based on deep learning. At the encoding end, the image will be converted into the YUV color space, and the UV signal will be downsampled by half. Then the Y signal and the UV signal will be compressed and encoded respectively, and the Y signal code stream and the UV signal code stream will be sent to the decoding end. At the decoding end, the Y signal code stream is decoded and reconstructed to obtain the reconstructed Y signal, and the UV signal code stream is decoded and reconstructed to obtain the reconstructed UV signal. Since the UV signal is compressed and encoded after being downsampled by half, that is, the resolution of the UV signal is halved, therefore, the resolution of the reconstructed UV signal is halved, that is, the resolution of the reconstructed Y signal is H*W, and the resolution of the reconstructed UV signal is H / 2*W / 2, that is, the resolution of the reconstructed UV signal is half of the resolution of the reconstructed Y signal.

[0132] Since the resolution of the reconstructed UV signal is half of the resolution of the reconstructed Y signal, the resolution of the UV signal can be restored in the post-processing process, that is, the resolution of the UV signal is restored to H*W. When restoring the resolution of the UV signal, the computational complexity of the resolution restoration process is high and the demand for resources is relatively large. Some devices that are sensitive to cost and delay cannot restore the resolution of the UV chromaticity signal.

[0133] In response to the above findings, in the embodiment of the present disclosure, in the field of end-to-end image compression, a low-complexity and efficient post-processing method is proposed. During the post-processing process, the UV signal can be enhanced based on the Y signal. This method has low complexity and high performance, can improve decoding performance, reduce decoding complexity, reduce computational complexity, and has relatively low resource requirements. Cost- and delay-sensitive devices can also use the above algorithm, and can recover from compression defects at a variety of bit rates.

[0134] In the disclosed embodiment, after receiving the code stream corresponding to the current image block, the decoding end may decode the code stream corresponding to the current image block to obtain a reconstructed image block. For example, the decoding end may adopt the processing flow of Embodiment 3 to decode the code stream corresponding to the current image block to obtain a reconstructed image block, and there is no restriction on the decoding process of the reconstructed image block.

[0135] After obtaining the reconstructed image block, if the reconstructed image block is a reconstructed image block in YUV format, the chrominance U component (i.e., U component) can be auxiliary enhanced based on the luminance component (i.e., Y component), the chrominance V component (i.e., V component) can be auxiliary enhanced based on the Y component, and the U component and the V component can be auxiliary enhanced based on the Y component. If the reconstructed image block is a reconstructed image block in RGB format, the R component can be auxiliary enhanced based on the G component, the B component can be auxiliary enhanced based on the G component, and the R component and the B component can be auxiliary enhanced based on the G component. For the convenience of description, in this embodiment, the reconstructed image block in YUV format is used as an example for explanation. Therefore, the reconstructed image block can include the Y component, the U component, and the V component. The resolution of the Y component can be greater than the resolution of the U component. For example, the resolution of the U component is a resolution that has not been downsampled, and the resolution of the Y component can be the original resolution or the resolution that has been downsampled, but the resolution of the Y component is greater than the resolution of the U component. The resolution of the Y component can be greater than the resolution of the V component. For example, if the resolution of the V component is not downsampled, the resolution of the Y component can be the original resolution or the downsampled resolution, but the resolution of the Y component is greater than the resolution of the V component. For example, assuming that the resolution of the Y component is H*W, the resolution of the U component is H / 2*W / 2, and the resolution of the V component is H / 2*W / 2.

[0136] See also Figure 5B As shown, it is a structural diagram of post-processing based on deep learning. The decoding end can decode the code stream corresponding to the current image block to obtain a reconstructed image block. If the reconstructed image block is a reconstructed image block in YUV format, the reconstructed image block may include a Y component, a U component and a V component. The Y component is recorded as the Y initial component (i.e., the first initial component of the above embodiment), the U component is recorded as the U initial component (i.e., the second initial component of the above embodiment), and the V component is recorded as the V initial component (i.e., the second initial component of the above embodiment). Based on this, the U initial component can be auxiliary enhanced based on the Y initial component, the V initial component can be auxiliary enhanced based on the Y initial component, and the U initial component and the V initial component can be auxiliary enhanced based on the Y initial component. Among them, the resolution of the Y initial component can be greater than the resolution of the U initial component, and the resolution of the Y initial component can be greater than the resolution of the U initial component. For example, see Figure 5B As shown, the resolution of the Y initial component is H*W, the resolution of the U initial component is H / 2*W / 2, and the resolution of the V initial component is H / 2*W / 2.

[0137] In this embodiment, see Figure 5BAs shown, a preprocessing module is involved, and the preprocessing module is an optional module. After decoding to obtain the Y initial component, if the preprocessing module is deployed, the Y initial component can be preprocessed by the preprocessing module to obtain the preprocessed Y initial component, and the preprocessed Y initial component is input to the Y auxiliary UV module and the YUV signal enhancement module. If the preprocessing module is not deployed, the Y initial component is directly input to the Y auxiliary UV module and the YUV signal enhancement module. In subsequent embodiments, the Y initial component obtained by the Y auxiliary UV module and the YUV signal enhancement module can be the preprocessed Y initial component or the original Y initial component obtained by decoding, and there is no limitation on this.

[0138] Since there is a certain correlation between the three YUV components, although the compression loss of the Y component at the encoding end is smaller than that of the UV component, the Y component can also be enhanced to a certain extent in order to better enhance the UV component. Therefore, the Y initial component can be preprocessed by the preprocessing module to enhance the features of the Y initial component.

[0139] In this embodiment, see Figure 5B As shown, it involves the Y auxiliary UV module. After decoding to obtain the Y initial component, the U initial component and the V initial component, the U initial component and the V initial component suffer more losses, while the Y initial component retains relatively more image detail information, and the U initial component and the V initial component have a certain correlation with the Y initial component. Therefore, making full use of the information of the Y initial component can help improve the reconstruction quality of the U initial component and the V initial component.

[0140] Based on this, the Y auxiliary UV module can use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component, and can also use the information of the Y initial component to assist in improving the reconstruction quality of the V initial component, and can also use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component and the reconstruction quality of the V initial component.

[0141] In this embodiment, see Figure 5BAs shown, it involves a resolution conversion module. Since the resolution of the U initial component is smaller than the resolution of the Y initial component, such as the resolution of the U initial component is the resolution after 2 times downsampling, the resolution of the U initial component is half of the original signal resolution, such as H / 2*W / 2. Therefore, the resolution conversion module upsamples the resolution of the U initial component to restore the U component with the original signal resolution, that is, upsamples the resolution of the U initial component to obtain a U component with a resolution of H*W. Since the resolution of the V initial component is smaller than the resolution of the Y initial component, such as the resolution of the V initial component is the resolution after 2 times downsampling, the resolution of the V initial component is half of the original signal resolution, such as H / 2*W / 2. Therefore, the resolution conversion module upsamples the resolution of the V initial component to restore the V component with the original signal resolution, that is, upsamples the resolution of the V initial component to obtain a V component with a resolution of H*W.

[0142] The position of the resolution conversion module may be position 1, that is, the resolution conversion module upsamples the resolution of the U initial component to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the Y auxiliary UV module. The resolution conversion module upsamples the resolution of the V initial component to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the Y auxiliary UV module. Alternatively, the position of the resolution conversion module may be position 2, that is, the resolution conversion module upsamples the resolution of the U component output by the Y auxiliary UV module to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the YUV signal enhancement module. The resolution conversion module upsamples the resolution of the V component output by the Y auxiliary UV module to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the YUV signal enhancement module. Alternatively, the position of the resolution conversion module can be position 3, that is, the resolution conversion module upsamples the resolution of the U component output by the YUV signal enhancement module to obtain a U component with a resolution of H*W, and finally outputs the H*W U component. The resolution conversion module upsamples the resolution of the V component output by the YUV signal enhancement module to obtain a V component with a resolution of H*W, and finally outputs the H*W V component.

[0143] For the resolution transformation module, on the one hand, the resolution transformation module can appropriately reduce the resolution of the feature map, thereby effectively reducing the computational complexity of the network. On the other hand, in the JPEG-AI framework, the resolution transformation module is used to upsample the resolution of the reconstructed UV component, thereby restoring a reconstructed U / V signal with the same resolution as the original signal.

[0144] In this embodiment, see Figure 5BAs shown, a YUV signal enhancement module is involved. In the process of image coding compression, in addition to reducing the resolution of the UV signal, the compression algorithm and the convolution process will cause additional information loss to the YUV signal. Therefore, the Y component can be enhanced by the YUV signal enhancement module, so that the reconstructed Y component is close to the original signal before compression to improve the subjective quality. The U component can be enhanced by the YUV signal enhancement module, so that the reconstructed U component is close to the original signal before compression to improve the subjective quality. The V component can be enhanced by the YUV signal enhancement module, so that the reconstructed V component is close to the original signal before compression to improve the subjective quality.

[0145] For example, see Figure 5B As shown, the three network branches (the three network branches process the Y component, the U component, and the V component respectively) can be merged into one network branch, that is, a single network is used to process the Y component, the U component, and the V component at the same time, and output the reconstructed Y component, U component, and V component at the same time, thereby reducing the network calculation complexity.

[0146] Embodiment 5: In Embodiment 4, the Y auxiliary UV module can use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component and / or the V initial component, for example, based on the Y initial component and the U initial component, an adjusted component corresponding to the U initial component is generated, and / or, based on the Y initial component and the V initial component, an adjusted component corresponding to the V initial component is generated. Among them, the reconstruction quality of the adjusted component corresponding to the U initial component can be better than the reconstruction quality of the U initial component, thereby improving the reconstruction quality of the U initial component, and the reconstruction quality of the adjusted component corresponding to the U initial component may also be equal to or lower than the reconstruction quality of the U initial component, and there is no limitation on this. The reconstruction quality of the adjusted component corresponding to the V initial component can be better than the reconstruction quality of the V initial component, thereby improving the reconstruction quality of the V initial component, and the reconstruction quality of the adjusted component corresponding to the V initial component may also be equal to or lower than the reconstruction quality of the V initial component, and there is no limitation on this.

[0147] In a possible implementation, the Y auxiliary UV module may obtain a first image feature corresponding to the Y initial component, and the first image feature may be an image feature that has not been subjected to a neural network, such as the first image feature may include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module may obtain a second image feature corresponding to the U initial component, and the second image feature may be an image feature that has not been subjected to a neural network, such as the second image feature may include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module may generate an adjusted component based on the first image feature and the second image feature. For example, a texture fusion feature may be generated based on the texture feature corresponding to the Y initial component and the texture feature corresponding to the U initial component, and the texture fusion feature may be used to replace the texture feature corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. For another example, a subjective fusion feature may be generated based on the subjective feature corresponding to the Y initial component and the subjective feature corresponding to the U initial component, and the subjective fusion feature may be used to replace the subjective feature corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. For another example, a frequency domain fusion feature can be generated based on the frequency domain features corresponding to the Y initial component and the frequency domain features corresponding to the U initial component, and the frequency domain fusion feature is used to replace the frequency domain features corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. For another example, a histogram fusion feature can be generated based on the histogram features corresponding to the Y initial component and the histogram features corresponding to the U initial component, and the histogram fusion feature is used to replace the histogram features corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. For another example, a texture fusion feature can be generated based on the texture features corresponding to the Y initial component and the texture features corresponding to the U initial component, a frequency domain fusion feature can be generated based on the frequency domain features corresponding to the Y initial component and the frequency domain features corresponding to the U initial component, the texture fusion feature is used to replace the texture feature corresponding to the U initial component, and the frequency domain fusion feature is used to replace the frequency domain feature corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. The above are just a few examples and there is no limitation to this.

[0148] The Y auxiliary UV module can obtain the first image feature corresponding to the Y initial component, and the first image feature can be an image feature that has not been through the neural network, such as the first image feature includes but is not limited to at least one of the following: texture feature, subjective feature, frequency domain feature, or histogram feature. The Y auxiliary UV module can obtain the second image feature corresponding to the V initial component, and the second image feature can be an image feature that has not been through the neural network, such as the second image feature includes but is not limited to at least one of the following: texture feature, subjective feature, frequency domain feature, or histogram feature. The Y auxiliary UV module can generate an adjusted component corresponding to the V initial component based on the first image feature and the second image feature, and the generation method refers to the processing process of the U initial component.

[0149] In another possible implementation, the Y auxiliary UV module may obtain a first image feature corresponding to the Y initial component, and the first image feature may be an image feature that has passed through a neural network, that is, an image feature output by a neural network, such as the first image feature may include but is not limited to a first feature map, a weight coefficient map, etc. The Y auxiliary UV module may obtain a second image feature corresponding to the U initial component, and the second image feature may be an image feature that has passed through a neural network, that is, an image feature output by a neural network, such as the second image feature may include but is not limited to a second feature map, etc. The Y auxiliary UV module may generate an adjusted component corresponding to the U initial component based on the first image feature and the second image feature. For example, the adjusted component corresponding to the U initial component may be generated based on the first feature map corresponding to the Y initial component and the second feature map corresponding to the U initial component. For another example, the adjusted component corresponding to the U initial component may be generated based on the weight coefficient map corresponding to the Y initial component and the second feature map corresponding to the U initial component. The above are just a few examples and are not limited to this.

[0150] The Y auxiliary UV module can obtain the first image feature corresponding to the Y initial component. The first image feature can be an image feature that has been processed by a neural network, that is, an image feature output by a neural network. For example, the first image feature can include but is not limited to a first feature map, a weight coefficient map, etc. The Y auxiliary UV module can obtain the second image feature corresponding to the V initial component. The second image feature can be an image feature that has been processed by a neural network, that is, an image feature output by a neural network. For example, the second image feature can include but is not limited to a second feature map, etc. The Y auxiliary UV module can generate an adjusted component corresponding to the V initial component based on the first image feature and the second image feature. The generation method refers to the processing process of the U initial component.

[0151] Embodiment 6: In order to enable the UV component to fully utilize the information of the Y component, at the input stage of the post-processing network, the Y component is convolved to obtain a feature map with multiple channels. These feature maps contain different types of features of the Y component. These different types of features are added (Add) or concatenated (Concat) with the feature map obtained by convolving the UV component, so that the subsequent neural network can learn multiple features of the Y component, thereby helping the UV component to improve the reconstruction quality.

[0152] For example, see Fig. 6AAs shown in the figure, it is a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component and the V initial component, the Y auxiliary UV module can input the Y initial component to the first neural network (such as CNN1) to obtain the first feature map, which can also be called the Y feature map (Y Feature Map). The U initial component can be input to the second neural network (such as CNN2) to obtain the second feature map corresponding to the U initial component, which can also be called the U feature map (UFeature Map). The V initial component can be input to the second neural network (such as CNN2) to obtain the second feature map corresponding to the V initial component, which can also be called the V feature map (V Feature Map).

[0153] After obtaining the first feature map and the second feature map corresponding to the U initial component, the adjusted component corresponding to the U initial component can be generated based on the first feature map and the second feature map. For example, the first feature map and the second feature map are added together. Fig. 6A In represents an addition operation of the feature map, and the feature map after the addition operation is input to the third neural network (such as CNN3) to obtain the adjusted component corresponding to the initial component U. Alternatively, the first feature map and the second feature map are overlapped, Fig. 6A In Represents a stacking operation of the feature map, and inputs the feature map after the stacking operation to the third neural network (such as CNN3) to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the adjusted component corresponding to the V initial component can be generated based on the first feature map and the second feature map. For example, the first feature map and the second feature map are added, and the feature map after the addition operation is input to the third neural network to obtain the adjusted component corresponding to the V initial component. Alternatively, the first feature map and the second feature map are stacked, and the feature map after the stacking operation is input to the third neural network to obtain the adjusted component corresponding to the V initial component.

[0154] Performing a splicing operation on the first feature map and the second feature map refers to performing a splicing operation on the first feature map and the second feature map along a channel dimension.

[0155] Exemplarily, in order to correctly implement the above operations, the number of convolution kernels of the second neural network can be the same as the number of convolution kernels of the first neural network, such as CNN1 and CNN2 have the same number of convolution kernels. Fig. 6A The processing shown can also be expressed as: f CNN represents a convolutional neural network, Indicates the addition or stacking operation of the feature map. Y ​​represents the initial component of Y, fCNN1 (Y) represents the first feature map, U represents the initial component of U, and f CNN2 (U) represents the second feature map, V represents the initial component of V, and f CNN2 (V) represents the second feature map, U rec represents the adjusted component corresponding to the initial component U, V rec Indicates the adjusted component corresponding to the initial component of V.

[0156] Example 7: In order to utilize the information of each feature map of the Y component, guided filtering can be introduced, that is, the features generated by the Y component guide the UV component to generate a feature map with stronger representation ability. Consider using the feature map of the Y component as a guiding variable to weight the pixel values ​​in the UV feature map by attention, thereby strengthening the important features in the UV component and suppressing the general features in the UV component, thereby achieving efficient conversion and fusion of the features of the Y component and the UV component.

[0157] For example, see Figure 6B As shown in the figure, it is a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component and the V initial component, the Y auxiliary UV module can input the Y initial component to the first neural network (such as CNN1) to obtain the first feature map, which can also be called the Y feature map (Y Feature Map). The U initial component can be input to the second neural network (such as CNN2) to obtain the second feature map corresponding to the U initial component, which can also be called the U feature map (UFeature Map). The V initial component can be input to the second neural network (such as CNN2) to obtain the second feature map corresponding to the V initial component, which can also be called the V feature map (V Feature Map).

[0158] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map can also be guided filtered to obtain a weight coefficient map, and the adjusted component corresponding to the U initial component is generated based on the weight coefficient map and the second feature map. For example, an output feature map (Output Feature Map) is obtained by performing an operation based on the weight coefficient map and the second feature map, such as at least one of a multiplication operation, an addition operation, or a splicing operation, such as a multiplication operation based on the weight coefficient map and the second feature map to obtain an output feature map, or a multiplication operation and an addition operation can be performed based on the weight coefficient map and the second feature map to obtain an output feature map, or a multiplication operation, an addition operation and a splicing operation can be performed based on the weight coefficient map and the second feature map to obtain an output feature map. After obtaining the output feature map, the output feature map can be input into a third neural network (such as CNN3) to obtain an adjusted component corresponding to the U initial component.

[0159] After obtaining the first feature map and the second feature map corresponding to the initial component of V, the first feature map may be subjected to guided filtering to obtain a weight coefficient map, and an adjusted component corresponding to the initial component of V may be generated based on the weight coefficient map and the second feature map. For example, an output feature map (Output Feature Map) may be obtained by performing an operation based on the weight coefficient map and the second feature map, such as at least one of a multiplication operation, an addition operation, or a stacking operation. After obtaining the output feature map, the output feature map may be input into a third neural network (such as CNN3) to obtain an adjusted component corresponding to the initial component of V.

[0160] For example, Figure 6B The processing process shown can also be expressed as: rec / V rec =f CNN3 (f G (f CNN1 (Y), f CNN2 (U / V))), f CNN represents the convolutional neural network, f G represents guided filtering, which takes the feature maps of the Y component and the UV component as input and outputs a fused feature map after guided weighting. Y represents the initial component of Y, and f CNN1 (Y) represents the first feature map, U represents the initial component of U, and f CNN2 (U) represents the second feature map, V represents the initial component of V, f CNN2 (V) represents the second feature map, U rec represents the adjusted component corresponding to the initial component U, V rec Indicates the adjusted component corresponding to the initial component of V.

[0161] In a possible implementation, the implementation of guided filtering can be found in Figure 6C As shown, the first feature map (Y Feature Map) can be convolved to obtain a convolved feature map (hereinafter referred to as a convolved feature map A). ​​For example, the first feature map is input into a CNN network, and the CNN network performs a convolution operation on the first feature map to obtain a convolved feature map A. Then, the convolved feature map A is weight mapped to obtain a weight coefficient map after weight mapping.

[0162] When generating the adjusted component corresponding to the U initial component based on the weight coefficient map and the second feature map corresponding to the U initial component, the weight coefficient map and the second feature map (U Feature Map) corresponding to the U initial component can be multiplied to obtain the multiplied feature map. Then, the multiplied feature map and the convolution feature map A are added to obtain the added feature map. Then, the added feature map and the convolution feature map A are overlapped to obtain the output feature map (Output Feature Map), and the output feature map is input to the third neural network (such as CNN3) to obtain the adjusted component corresponding to the U initial component.

[0163] When generating an adjusted component corresponding to the V initial component based on the weight coefficient map and the second feature map corresponding to the V initial component, the weight coefficient map and the second feature map (V Feature Map) corresponding to the V initial component can be multiplied to obtain the multiplied feature map. Then, the multiplied feature map and the convolution feature map A are added to obtain the added feature map. Then, the added feature map and the convolution feature map A are overlapped to obtain the output feature map (Output Feature Map), and the output feature map is input into the third neural network (such as CNN3) to obtain the adjusted component corresponding to the V initial component.

[0164] See also Figure 6C As shown, after the first feature map of the Y component passes through several convolution layers, the convolution feature map A is obtained, and the convolution feature map A is mapped to a specific weight factor, that is, a weight coefficient map, through weight mapping. The weight coefficient map is multiplied with the second feature map of the UV component to obtain the feature guide map of the UV component, and then the convolution feature map A is added to the feature guide map of the UV component, and finally overlapped with the convolution feature map A to output the fusion feature map (Output Feature Map). For example, the guided filtering process is expressed as: Out = Concat ((U / V) * f w (f CNN (Y))+f CNN (Y),f CNN (Y)), f w represents weight mapping, Concat(·) represents the stacking operation, and f CNN (Y) represents the feature map A after convolution, and Out is the output feature map.

[0165] In a possible implementation, during the weight mapping process, weight mapping is performed on the convolution feature map A to obtain a weight coefficient map after weight mapping. The implementation of weight mapping can be found in Fig.6D As shown, Fig.6DTwo weight mapping methods are shown in . In the first weight mapping method, a pooling operation is performed on the convolution feature map A to obtain a pooled feature map. For example, a pooling operation (Pooling) is performed on the convolution feature map A of size C*H*W to obtain a pooled feature map of size C*1*1, that is, the resolution of the feature map is reduced to 1*1 by the "Pooling" method.

[0166] Then, the feature map after the pooling operation is fully connected and activated by ReLU to obtain the feature map after the ReLU activation operation. For example, the feature map after the pooling operation with a size of C*1*1 is fully connected (i.e., FC operation) and ReLU activation operation (i.e., ReLU operation) are performed to obtain the feature map after the ReLU activation operation with a size of C*1*1. The feature map after the ReLU activation operation is fully connected and activated by Sigmoid to obtain the feature map after the Sigmoid activation operation. For example, the feature map after the ReLU activation operation with a size of C*1*1 is fully connected (i.e., FC operation) and Sigmoid activation operation (i.e., Sigmoid operation) are performed to obtain the feature map after the Sigmoid activation operation with a size of C*1*1.

[0167] Then, a weight coefficient map is generated based on the post-convolution feature map A and the post-Sigmoid activation feature map. For example, the post-convolution feature map A and the post-Sigmoid activation feature map are multiplied to obtain a weight coefficient map. For example, the post-convolution feature map A with a size of C*H*W and the post-Sigmoid activation feature map with a size of C*1*1 are multiplied to obtain a weight coefficient map with a size of C*H*W. Obviously, after the resolution of the feature map is reduced to 1*1 by the "Pooling" method, it passes through several fully connected layers and finally obtains the weight factor of each channel in the original input feature map through the Sigmoid activation function. These weight factors will give different weights to the channels of the original input feature map, and finally obtain a weight coefficient map (also called a weight feature map).

[0168] In the second weight mapping method, the convolution operation can be performed on the convolution feature map A to obtain the convolution feature map B. For example, the convolution feature map A of size C*H*W is input to the CNN network, and the CNN network performs convolution operations on the convolution feature map A through several convolution layers to obtain the convolution feature map B of size C*H*W. Then, the convolution feature map B is activated by Sigmoid to obtain the weight coefficient map. For example, the convolution feature map B of size C*H*W is activated by the Sigmoid activation function to obtain the weight coefficient map of size C*H*W. Obviously, by passing the original input feature map through several convolution layers and then through the Sigmoid activation function, the weight coefficient map is finally obtained.

[0169] The difference between the first weight mapping method and the second weight mapping method is that in the first weight mapping method, different weights are assigned to channels, and in the second weight mapping method, different weights are assigned to each pixel of each channel feature map.

[0170] Embodiment 8: In Embodiment 6 and Embodiment 7, feature extraction and information fusion of the Y component are performed at the input stage of the post-processing network. The difference from Embodiment 6 and Embodiment 7 is that in Embodiment 8, feature extraction and information fusion of the Y component are distributed in various stages of the post-processing network, and the features of the UV component at different stages are fused with the features of the Y component, thereby improving the enhancement capability of the post-processing network for the UV component. For example, see Fig. 6E As shown in the figure, it is a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component and the V initial component, the Y auxiliary UV module distributes the feature extraction and information fusion of the Y component in each stage of the post-processing network. B The feature fusion process can adopt the implementation of Example 6 or Example 7, by fusing the information of the Y component and the UV component to output a fused feature map, and then after passing through several convolution layers, it is used as the input of the next feature fusion process. Correspondingly, the Y component will also pass through several convolution layers as the input of the next feature fusion process. In Example 8, multiple feature fusion processes are involved, so as to fuse the feature map of the Y component with the feature map of the UV component at each stage of the network.

[0171] In a possible implementation, since the feature map of the Y component is fused with the feature map of the UV component at each stage of the network, multiple first feature maps and multiple second feature maps are involved. Taking K first feature maps and K+1 second feature maps as an example, K can be a positive integer greater than 1. Based on this, after obtaining the Y initial component and the U initial component, for the first first feature map and the first second feature map, the Y initial component is input into the neural network to obtain the first first feature map, and the U initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, the value range of i is 2 to K, the i-1th first feature map is input into the neural network to obtain the i-th first feature map, the i-1th first feature map and the i-1th second feature map are feature fused to obtain the fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map (i.e., the K+1th second feature map), the adjusted component corresponding to the U initial component can be generated based on the last second feature map.

[0172] For example, taking K as 3, the initial component Y is input to the neural network a1 to obtain the first feature map b1, that is, the neural network a1 performs several convolution operations on the initial component Y to obtain the first feature map b1. The initial component U is input to the neural network c1 to obtain the second feature map d1, that is, the neural network c1 performs several convolution operations on the initial component U to obtain the second feature map d1.

[0173] The first feature map b1 is input to the neural network a2 to obtain the first feature map b2, that is, the neural network a2 performs several convolution operations on the first feature map b1 to obtain the first feature map b2. The first feature map b1 and the second feature map d1 are subjected to feature fusion to obtain fused features, and the fused features are input to the neural network c2 to obtain the second feature map d2.

[0174] The first feature map b2 is input to the neural network a3 to obtain the first feature map b3, that is, the neural network a3 performs several convolution operations on the first feature map b2 to obtain the first feature map b3. The first feature map b2 and the second feature map d2 are subjected to feature fusion to obtain fused features, and the fused features are input to the neural network c3 to obtain the second feature map d3.

[0175] Then, the first feature map b3 and the second feature map d3 are feature fused to obtain fused features, and the fused features are input to the neural network c4 to obtain the second feature map d4. After obtaining the second feature map d4, the adjusted component corresponding to the U initial component can be generated based on the second feature map d4, such as using the second feature map d4 as the adjusted component corresponding to the U initial component, or the second feature map d4 can be operated to obtain the adjusted component corresponding to the U initial component.

[0176] In the above process, the first feature map and the second feature map are subjected to feature fusion to obtain the fused feature, which may include but is not limited to: adding the first feature map and the second feature map to obtain the fused feature; or, superimposing the first feature map and the second feature map to obtain the fused feature; or, guiding the first feature map to obtain the weight coefficient map corresponding to the first feature map, and generating the fused feature based on the weight coefficient map and the second feature map. For the implementation of feature fusion, please refer to Example 6 and Example 7, which will not be repeated here.

[0177] After obtaining the Y initial component and the V initial component, for the first first feature map and the first second feature map, the Y initial component is input into the neural network to obtain the first first feature map, and the V initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, the value range of i is 2-K, the i-1th first feature map is input into the neural network to obtain the i-th first feature map, the i-1th first feature map and the i-1th second feature map are feature fused to obtain the fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map, the adjusted component corresponding to the V initial component can be generated based on the last second feature map. The method for obtaining the adjusted component corresponding to the V initial component can refer to the processing process of the U initial component.

[0178] Example 9: In Example 6, Example 7 and Example 8, the fusion method of the Y component guide map is designed. Different from Example 6, Example 7 and Example 8, in Example 9, the Y component is subjected to detailed feature extraction from multiple dimensions and then fused into the post-processing network, thereby improving the network's ability to enhance the UV component.

[0179] See also Fig. 6F As shown in FIG. 1 , it is a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component and the V initial component, the Y auxiliary UV module first performs multi-dimensional feature extraction on the Y initial component. For example, M-dimensional image features are extracted from the Y initial component, where M is a positive integer. Then, the M-dimensional image features are stacked to obtain the stacked multi-dimensional features. After obtaining the multi-dimensional features, the multi-dimensional features can be input into the first neural network to obtain a first feature map (Y Feature Map). In addition, the Y auxiliary UV module can also input the U initial component into the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component, which can also be called a U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component, which can also be called a V feature map (V Feature Map).

[0180] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the V initial component. For the feature fusion process, please refer to Example 6, Example 7 and Example 8.

[0181] For example, Fig. 6F The processing process shown can also be expressed as: rec / V rec =f CNN2 (f G (f A (Y),f CNN1 (U / V))), f G represents the feature fusion process, f A It represents the multi-dimensional feature extraction process of the Y component. In the multi-dimensional feature extraction process, the initial component Y is taken as input and output as multiple feature maps of multiple dimensions, and each branch represents the features of one dimension.

[0182] For example, see Figure 6G As shown, it is a schematic diagram of the multi-dimensional feature extraction process, with the Y initial component as input, and each branch feature extraction structure is used to extract the features of the Y initial component. For example, the Y initial component is input to the branch 1 feature extraction structure, and the branch 1 feature extraction structure extracts the features of the first dimension of the Y initial component. The Y initial component is input to the branch 2 feature extraction structure, and the branch 2 feature extraction structure extracts the features of the second dimension of the Y initial component. By analogy, the Y initial component is input to the branch M feature extraction structure, and the branch M feature extraction structure extracts the features of the Mth dimension of the Y initial component. In summary, image features of M dimensions can be extracted from the Y initial component, and the image features of M dimensions can be stacked to obtain the stacked multi-dimensional features. After obtaining the stacked multi-dimensional features, the multi-dimensional features can be used as the first feature map, or the multi-dimensional features can be input to the first neural network to obtain the first feature map.

[0183] exist Figure 6G In FIG. 1 , a schematic diagram of a multi-dimensional feature extraction process is shown. First, the Y component is passed through different feature extraction channels to obtain feature information on each dimension, and finally these feature information are concat to the output as the Y component feature map. The multi-dimensional feature extraction process can be expressed as: Out = Concat (f ch-1 (Y),f ch-2 (Y),…,f ch-M (Y)), where f ch- It represents the i-th feature extraction structure, which is used to extract the features of the i-th dimension. Concat(·) represents the stacking operation.

[0184] In a possible implementation, the image features of M dimensions may include but are not limited to at least one of the following: image features after convolving the initial component Y with a convolution kernel; image features after convolving the initial component Y with two convolution kernels; first-order spatial features after processing the initial component Y with a sobel operator; second-order spatial features after processing the initial component Y with a Laplacian operator; or frequency domain features after Fourier transforming the initial component Y. The above image features are just a few examples, and the type of image features is not limited in this embodiment.

[0185] See also Figure 6H As shown in FIG. 1 , it is a schematic diagram of a multi-dimensional feature extraction structure. The feature extraction structure of branch 1 (i.e., the first branch) uses a 3*3 convolution to obtain feature information. For example, a 3*3 convolution kernel is used to convolve the initial component Y to obtain the image feature after convolution. The feature extraction structure of branch 2 (i.e., the second branch) uses a 1*1 and 3*3 convolution cascade to obtain feature information after channel expansion and compression. For example, a 1*1 and 3*3 convolution kernel is used to convolve the initial component Y to obtain the image feature after convolution cascade. The feature extraction structure of branch 3 (i.e., the third branch) uses the sobel operator to obtain the first-order spatial domain feature. For example, the sobel operator is used to process the initial component Y to obtain the first-order spatial domain feature. The feature extraction structure of branch 4 (i.e., the fourth branch) uses the Laplacian operator to obtain the second-order spatial domain feature. For example, the Laplacian operator is used to process the initial component Y to obtain the second-order spatial domain feature. The feature extraction structure of branch 5 (i.e., the fifth branch) uses Fourier transform to obtain frequency domain features. For example, Fourier transform is performed on the initial component Y to obtain frequency domain features. After obtaining the features of the above five dimensions, these features can be stacked to obtain multi-dimensional features.

[0186] Example 10: In Example 6 to Example 9, the main focus is on which Y component features to select and how to merge the Y component features with the UV component features. Unlike Example 6 to Example 9, in Example 10, the focus is on how to enhance the features of the Y component. For example, after the YUV components of the image are encoded and compressed, the image details of the Y component are also lost to a certain extent. Therefore, the Y component and the features of the Y component can also be enhanced to a certain extent. For example, since there is a certain correlation between the three YUV components, although the compression loss of the Y component at the encoding end is smaller than that of the UV component, the Y component can also be enhanced to a certain extent in order to better enhance the UV component. Therefore, the Y initial component can be preprocessed by the preprocessing module to enhance the features of the Y initial component.

[0187] For example, after obtaining the Y initial component, the preprocessing module can first perform edge enhancement on the Y initial component to obtain the image features after edge enhancement; then, perform multi-scale feature extraction on the edge enhanced image features to obtain multi-scale features. After obtaining the multi-scale features, the multi-scale features can be used as the preprocessed Y initial component, and the preprocessed Y initial component is input to the Y auxiliary UV module, and the Y auxiliary UV module inputs the preprocessed Y initial component to the first neural network to obtain a first feature map (Y Feature Map). In addition, the Y auxiliary UV module can also input the U initial component to the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component, and the second feature map can also be called a U feature map (U Feature Map). The V initial component can be input to the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component, and the second feature map can also be called a V feature map (V Feature Map).

[0188] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the V initial component. For the feature fusion process, please refer to Example 6, Example 7 and Example 8.

[0189] When the preprocessing module performs multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, it can perform a convolution operation on the edge-enhanced image features to obtain convolution features; perform a downsampling operation on the convolution features to obtain down-sampled features; perform channel transformation on the down-sampled features to obtain channel-transformed features; perform an upsampling operation on the channel-transformed features to obtain up-sampled features; and generate multi-scale features based on the up-sampled features and the channel-transformed features.

[0190] See also Fig.6I As shown in the figure, it is a schematic diagram of the structure of the post-processing network, which lists a multi-scale enhancement method of the Y component. In the downsampling part, from D1 to D3, the resolution of the feature map gradually decreases, and the number of channels gradually increases. In the upsampling part, from U3 to U1, the resolution of the feature map gradually increases, and the number of channels remains unchanged. After channel transformation, D3, D2, and D1 are added to the upsampled feature maps of U3, U2, and U1 respectively to obtain multi-scale features after multi-scale enhancement.

[0191] First, edge enhancement is performed on the Y initial component to obtain image features after edge enhancement. For example, image sharpening enhancement operation is performed on the Y initial component to obtain image features after edge enhancement. There is no restriction on this edge enhancement method.

[0192] Taking 2 downsampling operations and 2 upsampling operations as an example, the number of downsampling operations can be more, and the number of upsampling operations can be more, and there is no limit to this. After obtaining the edge-enhanced image features, the edge-enhanced image features are output to CNN to obtain a feature D1 with a scale of C*H*W, and a downsampling operation is performed on the feature D1 to obtain a feature D2 with a scale of 4C*H / 2*W / 2, and a downsampling operation is performed on the feature D2 to obtain a feature D3 with a scale of 16C*H / 4*W / 4.

[0193] Perform channel transformation on feature D3 to obtain feature U3 with a scale of T*H / 4*W / 4, perform upsampling on feature U3, perform channel transformation on feature D2, add feature U3 after upsampling and feature D2 after channel transformation to obtain feature U2 with a scale of T*H / 2*W / 2. Perform upsampling on feature U2, perform channel transformation on feature D1, add feature U2 after upsampling and feature D1 after channel transformation to obtain feature U1 with a scale of T*H*W. Obviously, feature U1 is a multi-scale feature after multi-scale enhancement, that is, the preprocessed Y initial component, and the preprocessing module will output this preprocessed Y initial component. By performing multi-scale feature extraction on the Y component, the first feature map of the final output integrates feature information at different scales, which helps to enhance the UV component.

[0194] Embodiment 11: In embodiment 4, a resolution conversion module is involved, and the resolution conversion module is used to upsample the resolution of the U component to restore the U component with the original signal resolution, such as obtaining a U component with a resolution of H*W. The resolution conversion module is used to upsample the resolution of the V initial component to restore the V component with the original signal resolution, such as obtaining a V component with a resolution of H*W. For example, in the JPEG-AI framework, the original UV component will be downsampled by 2 times and then compressed and encoded by the encoding end. Therefore, the resolution of the UV component reconstructed by the decoding end is half the resolution of the original signal. In the post-processing network, there is a step of upsampling the UV component to restore the reconstructed UV component with the same resolution as the original signal. Exemplarily, when the resolution conversion module upsamples the resolution of the U component and / or the V component, the upsampling method may include but is not limited to: interpolation sampling, pixel-shuffle upsampling, deconvolution DeConv, etc. The above are just a few examples, and this upsampling method is not limited in this embodiment.

[0195] For example, since upsampling the resolution of the UV component will affect the computational complexity of the network model, that is, an increase in resolution leads to an increase in computational complexity, the position of the resolution transformation module will affect the computational complexity of the network model. Therefore, the position of the resolution transformation module can be involved to reduce the computational complexity of the network model.

[0196] Case 1: See Figure 5B As shown, the position of the resolution conversion module can be position 1, that is, the resolution conversion module upsamples the resolution of the U initial component to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the Y auxiliary UV module. The resolution conversion module upsamples the resolution of the V initial component to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the Y auxiliary UV module. On this basis, for Embodiments 5 to 10, the U initial components are all upsampled U components, and the V initial components are all upsampled V components. The resolution of the upsampled U component can be equal to the resolution of the Y initial component, and the resolution of the upsampled V component can be equal to the resolution of the Y initial component. In this way, Embodiments 5 to 10 are performed based on the Y initial component and the upsampled U component, and Embodiments 5 to 10 are performed based on the Y initial component and the upsampled V component.

[0197] In summary, in case 1, upsampling can be performed at the input of the network, see Fig. 7A As shown, it is a schematic diagram of upsampling at the input end of the network, that is, at the input end of the network, the U initial component can be upsampled to obtain the upsampled U component, such as the U component with a resolution of H*W, and at the input end of the network, the V initial component can be upsampled to obtain the upsampled V component, such as the V component with a resolution of H*W.

[0198] Case 2: See Figure 5BAs shown, the position of the resolution conversion module can be position 2, that is, the resolution conversion module upsamples the resolution of the U component output by the Y auxiliary UV module to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the YUV signal enhancement module. The resolution conversion module upsamples the resolution of the V component output by the Y auxiliary UV module to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the YUV signal enhancement module. On this basis, for Embodiments 5 to 10, the U initial component is the U component before upsampling, and the V initial component is the V component before upsampling. The resolution of the U component before upsampling can be less than the resolution of the Y initial component, and the resolution of the V component before upsampling can be less than the resolution of the Y initial component. In this way, Embodiments 5 to 10 are performed based on the Y initial component and the U component before upsampling, and Embodiments 5 to 10 are performed based on the Y initial component and the V component before upsampling. After obtaining the adjusted components corresponding to the U component / V component based on Examples 5 to 10, the adjusted components can also be upsampled to obtain the upsampled adjusted components, and the upsampled adjusted components are input to the YUV signal enhancement module. The resolution of the upsampled adjusted components is equal to the resolution of the Y initial components.

[0199] In summary, in case 2, the network can be progressively upsampled layer by layer, see Figure 7B As shown, it is a schematic diagram of the network's progressive upsampling layer by layer, that is, at multiple network layers, the U initial component can be upsampled to obtain the upsampled U component, and the resolution of the upsampled U component can be equal to the resolution of the Y initial component, such as a U component with a resolution of H*W. At multiple network layers, the V initial component can be upsampled to obtain the upsampled V component, and the resolution of the upsampled V component can be equal to the resolution of the Y initial component, such as a V component with a resolution of H*W.

[0200] Case 3: See Figure 5BAs shown, the position of the resolution conversion module can be position 3, that is, the resolution conversion module upsamples the resolution of the U component output by the YUV signal enhancement module to obtain a U component with a resolution of H*W, and finally outputs the H*W U component. The resolution conversion module upsamples the resolution of the V component output by the YUV signal enhancement module to obtain a V component with a resolution of H*W, and finally outputs the H*W V component. On this basis, for Embodiments 5 to 10, the U initial component is the U component before upsampling, the V initial component is the V component before upsampling, the resolution of the U component before upsampling is less than the resolution of the Y initial component, and the resolution of the V component before upsampling is less than the resolution of the Y initial component. In this way, Embodiments 5 to 10 are performed based on the Y initial component and the U component before upsampling, and Embodiments 5 to 10 are performed based on the Y initial component and the V component before upsampling. After obtaining the adjusted component corresponding to the U component / V component based on Example 5-Example 10, the adjusted component is input to the YUV signal enhancement module, and the adjusted component is the U component / V component before upsampling, and the U component / V component before upsampling is less than the resolution of the Y initial component, so that the YUV signal enhancement module performs signal enhancement based on the U component / V component before upsampling to obtain the U component / V component after signal enhancement. After obtaining the U component / V component after signal enhancement, the U component / V component is upsampled to obtain the upsampled U component / V component, and the resolution of the upsampled U component / V component is equal to the resolution of the Y initial component.

[0201] In summary, in case 3, upsampling can be performed at the output of the network, see Figure 7C As shown, it is a schematic diagram of upsampling at the output end of the network, that is, at the output end of the network, the U component can be upsampled to obtain the upsampled U component, and the resolution of the upsampled U component can be equal to the resolution of the Y initial component, such as the U component with a resolution of H*W. At the output end of the network, the V component can be upsampled to obtain the upsampled V component, and the resolution of the upsampled V component can be equal to the resolution of the Y initial component, such as the V component with a resolution of H*W.

[0202] Embodiment 12: In Embodiment 4, a resolution transformation module is involved. On the one hand, the resolution transformation module can appropriately reduce the resolution of the feature map by using a resolution transformation without information loss, thereby effectively reducing the computational complexity of the network. On the other hand, the resolution transformation module can upsample the resolution of the reconstructed U component and V component, thereby restoring the U component and V component with the same resolution as the original signal. For the function of the second aspect, see Embodiment 11.

[0203] The function of the first aspect can be implemented by a resolution conversion module or by a Y auxiliary UV module. For example, by reducing the resolution of the YUV component or the intermediate component, the network complexity can be effectively reduced, and the resolution can be increased at the output position of the network to restore the original resolution. For example, a wavelet transform (such as Haar wavelet transform, etc.) can be used to halve the resolution of the original image. In this way, the computational complexity of the network can be reduced by almost four times. When the hardware has high requirements for computational complexity, it is also possible to consider using the wavelet transform multiple times in a row so that the resolution can be further reduced. For several frequency bands after the wavelet transform, all of them can be used or some of them can be selected.

[0204] For example, see Fig.7D As shown in the figure, it is a schematic diagram of the structure of the post-processing network, which can perform wavelet transform on the initial component of Y ( Fig.7D The DWT transform in the above example is used to obtain multiple frequency bands after wavelet transform. The multiple frequency bands or part of the multiple frequency bands (i.e., Y subbands) can be input into the first neural network to obtain the first feature map (Y Feature Map). The U initial component can be subjected to wavelet transform ( Fig.7D DWT transform in), multiple frequency bands after wavelet transform are obtained, and multiple frequency bands or part of the frequency bands in the multiple frequency bands (i.e., U sub-bands) can be input into the second neural network to obtain a second feature map (UFeature Map). The V initial component can be subjected to wavelet transform to obtain multiple frequency bands after wavelet transform, and multiple frequency bands or part of the frequency bands in the multiple frequency bands (i.e., V sub-bands) can be input into the second neural network to obtain a second feature map (V Feature Map). After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be feature fused to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be feature fused to obtain an adjusted component corresponding to the V initial component. For the feature fusion process, please refer to Example 6, Example 7, and Example 8.

[0205] It should be noted that when the first feature map and the second feature map are subjected to feature fusion to obtain the adjusted component corresponding to the U initial component, Embodiment 6, Embodiment 7 and Embodiment 8 are first used to obtain the fused feature, and then the fused feature is input to the neural network (CNN3) to obtain the output feature of the neural network (U subband), and then the output feature can also be subjected to an inverse wavelet transform (i.e., the inverse operation of the wavelet transform, Fig.7DIDWT transformation in the example) to obtain the adjusted component of the U component. When the first feature map and the second feature map are subjected to feature fusion to obtain the adjusted component corresponding to the initial component of V, firstly, Embodiment 6, Embodiment 7 and Embodiment 8 are used to obtain the fused feature, and then the fused feature is input to the neural network (CNN3) to obtain the output feature of the neural network (V subband), and then, the output feature can also be subjected to inverse wavelet transformation (i.e., the inverse operation of wavelet transformation, Fig.7D IDWT transformation in ), and obtain the adjusted component of the V component.

[0206] Embodiment 13: In Embodiment 4, a YUV signal enhancement module is involved. The YUV signal enhancement module is used to enhance the signal of the Y component so that the reconstructed Y component is close to the original signal before compression. The YUV signal enhancement module is used to enhance the signal of the U component so that the reconstructed U component is close to the original signal before compression. The YUV signal enhancement module is used to enhance the signal of the V component so that the reconstructed V component is close to the original signal before compression. On this basis, the YUV signal enhancement module can perform feature enhancement on the initial Y component to obtain the target component corresponding to the Y component, and the target component is the Y component after signal recovery. The YUV signal enhancement module can perform feature enhancement on the adjusted component of the U component to obtain the target component corresponding to the U component, and the target component is the U component after signal recovery. The YUV signal enhancement module can perform feature enhancement on the adjusted component of the V component to obtain the target component corresponding to the V component, and the target component is the V component after signal recovery.

[0207] In a possible implementation, the feature enhancement of the initial component Y can be performed through at least one residual block network to obtain a target component corresponding to the Y component. The feature enhancement of the adjusted component U can be performed through at least one residual block network to obtain a target component corresponding to the U component. The feature enhancement of the adjusted component V can be performed through at least one residual block network to obtain a target component corresponding to the V component. Fig. 7E As shown, it is a schematic diagram of using a residual block cascade network to enhance the Y component, U component and V component, and there is no restriction on the structure of the residual block cascade network.

[0208] In a possible implementation, the feature enhancement of the initial component Y can be performed through a U-Net network to obtain a target component corresponding to the component Y. The feature enhancement of the adjusted component U can be performed through a U-Net network to obtain a target component corresponding to the component U. The feature enhancement of the adjusted component V can be performed through a U-Net network to obtain a target component corresponding to the component V. Figure 7FAs shown, it is a schematic diagram of using a U-Net network to enhance the Y component, the U component, and the V component. The U-Net network includes multiple downsampling network layers and multiple upsampling network layers, and there is no restriction on the structure of the U-Net network.

[0209] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 13 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 13 can be implemented in combination.

[0210] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0211] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present disclosure, and the decoding device is applied to the decoding end. The decoding device may include: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding method in the above embodiments 1 to 13, that is, the processing flow of the decoding end.

[0212] For example, in one possible implementation, the decoder is configured to implement:

[0213] Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0214] Generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0215] Feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0216] Exemplarily, if the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a brightness component, and the second initial component is a chrominance U component and / or a chrominance V component; or, if the reconstructed image block is a reconstructed image block in RGB format, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0217] Exemplarily, the decoder is configured to further implement: obtaining a first image feature corresponding to the first initial component, the first image feature including image features that have not passed through the neural network and / or image features that have passed through the neural network; obtaining a second image feature corresponding to the second initial component, the second image feature including image features that have not passed through the neural network and / or image features that have passed through the neural network; and generating the adjusted component based on the first image feature and the second image feature.

[0218] Exemplarily, the decoder is configured to further implement: if the first image feature includes a first feature map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the first initial component is input into the first neural network to obtain the first feature map; the second initial component is input into the second neural network to obtain the second feature map; the first feature map and the second feature map are added to obtain the adjusted component; or the first feature map and the second feature map are overlapped to obtain the adjusted component.

[0219] Exemplarily, the decoder is configured to further implement: if the first image feature includes a weight coefficient map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the first initial component is input into the first neural network to obtain a first feature map corresponding to the first initial component, and the first feature map is guided filtered to obtain the weight coefficient map; the second initial component is input into the second neural network to obtain the second feature map; and the adjusted component is generated based on the weight coefficient map and the second feature map.

[0220] Exemplarily, the decoder is configured to further implement: performing a convolution operation on the first feature map to obtain a convolved feature map; and performing weight mapping on the convolved feature map to obtain the weight coefficient map.

[0221] Exemplarily, the decoder is configured to further implement: performing a pooling operation on the feature map after the convolution operation; performing a full connection operation and a ReLU activation operation on the feature map after the pooling operation; performing a full connection operation and a Sigmoid activation operation on the feature map after the ReLU activation operation; and generating the weight coefficient map based on the feature map after the convolution operation and the feature map after the Sigmoid activation operation.

[0222] Exemplarily, the decoder is configured to further implement: performing a convolution operation and a Sigmoid activation operation on the convolution feature map to obtain the weight coefficient map.

[0223] Exemplarily, the decoder is configured to further implement: multiplying the weight coefficient map and the second feature map to obtain a multiplied feature map; performing a convolution operation on the first feature map to obtain a convolved feature map; adding the multiplied feature map and the convolved feature map to obtain an added feature map; and stacking the added feature map and the convolved feature map to obtain the adjusted component.

[0224] Exemplarily, the decoder is configured to further implement: if the first image feature includes K first feature maps passed through the neural network, and the second image feature includes K+1 second feature maps passed through the neural network, for the first first feature map and the first second feature map, the first initial component is input into the neural network to obtain the first first feature map, and the second initial component is input into the neural network to obtain the first second feature map; for the i-th first feature map and the i-th second feature map, the value range of i is 2-K, K is a positive integer greater than 1, the i-1-th first feature map is input into the neural network to obtain the i-th first feature map; the i-1-th first feature map and the i-1-th second feature map are feature fused to obtain a fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map; after obtaining the last second feature map, the adjusted component is generated based on the last second feature map.

[0225] Exemplarily, the decoder is configured to further implement: extracting M-dimensional image features from the first initial component, where M is a positive integer; stacking the M-dimensional image features to obtain stacked multi-dimensional features; and inputting the multi-dimensional features into the first neural network to obtain the first feature map.

[0226] Exemplarily, the decoder is configured to further implement: performing a wavelet transform on the first initial component to obtain a plurality of frequency bands after the wavelet transform; inputting the plurality of frequency bands or part of the plurality of frequency bands into the first neural network to obtain the first feature map; performing a wavelet transform on the second initial component to obtain a plurality of frequency bands after the wavelet transform; inputting the plurality of frequency bands or part of the plurality of frequency bands into the second neural network to obtain the second feature map.

[0227] Exemplarily, the decoder is configured to further implement: preprocessing the first initial component to obtain a preprocessed first initial component; wherein the preprocessed first initial component is used to input into the first neural network to obtain the first feature map; the decoder is configured to further implement: edge enhancement on the first initial component to obtain edge-enhanced image features; multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features; and determining the preprocessed first initial component based on the multi-scale features.

[0228] Exemplarily, the decoder is configured to further implement: performing a convolution operation on the edge-enhanced image features to obtain convolution features; performing a downsampling operation on the convolution features to obtain downsampled features; performing a channel transformation on the downsampled features to obtain channel transformed features; performing an upsampling operation on the channel transformed features to obtain upsampled features; and generating the multi-scale features based on the upsampled features and the channel transformed features.

[0229] Exemplarily, a decoder is configured to further implement: before generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, upsampling the second initial component to obtain an upsampled second initial component; the resolution of the upsampled second initial component is equal to the resolution of the first initial component.

[0230] Exemplarily, the decoder is configured to further implement: before performing feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, upsampling the adjusted component to obtain the upsampled adjusted component; wherein the resolution of the upsampled adjusted component is equal to the resolution of the first initial component.

[0231] Exemplarily, the decoder is configured to further implement: after performing feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, upsampling the target component to obtain the upsampled target component; wherein the resolution of the upsampled target component is equal to the resolution of the first initial component.

[0232] Exemplarily, the decoder is configured to further implement: performing feature enhancement on the adjusted component through at least one residual block network to obtain a target component corresponding to the second initial component; or performing feature enhancement on the adjusted component through a U-Net network to obtain a target component corresponding to the second initial component.

[0233] Based on the same application concept as the above method, the decoding end device (also referred to as a video decoder) provided in the embodiment of the present disclosure, from the hardware level, its hardware architecture diagram can be specifically referred to as Figure 8 The system comprises: a processor 811 and a machine-readable storage medium 812, wherein the machine-readable storage medium 812 stores machine-executable instructions that can be executed by the processor 811; the processor 811 is used to execute the machine-executable instructions to implement the decoding method of the above-mentioned embodiments 1-12 of the present disclosure.

[0234] For example, in one possible implementation, the processor 811 executes the machine executable instructions to implement:

[0235] Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0236] Generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0237] Feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0238] Based on the same application concept as the above method, an embodiment of the present disclosure provides an electronic device, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the decoding method of the above embodiments 1-12 of the present disclosure.

[0239] Based on the same application concept as the above method, an embodiment of the present disclosure also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present disclosure can be implemented, such as the decoding method in the above embodiments.

[0240] Based on the same application concept as the above method, an embodiment of the present disclosure further provides a computer application, which, when executed by a processor, can implement the decoding method disclosed in the above example of the present disclosure.

[0241] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present disclosure. The decoding device can be applied to a decoding end (also called a video decoder). The decoding device may include:

[0242] A decoding module is used to decode the code stream corresponding to the current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, and the resolution of the first initial component is greater than or equal to the resolution of the second initial component; a determination module is used to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and a processing module is used to perform feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0243] Exemplarily, if the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a brightness component, and the second initial component is a chrominance U component and / or a chrominance V component; or, if the reconstructed image block is a reconstructed image block in RGB format, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0244] Exemplarily, when the determination module generates an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, it is specifically used to: obtain a first image feature corresponding to the first initial component, the first image feature includes an image feature that has not passed through the neural network and / or an image feature that has passed through the neural network; obtain a second image feature corresponding to the second initial component, the second image feature includes an image feature that has not passed through the neural network and / or an image feature that has passed through the neural network; and generate the adjusted component based on the first image feature and the second image feature.

[0245] Exemplarily, if the first image feature includes image features that have not passed through a neural network, the first image feature includes at least one of the following: texture features, subjective features, frequency domain features, or histogram features; if the second image feature includes image features that have not passed through a neural network, the second image feature includes at least one of the following: texture features, subjective features, frequency domain features, or histogram features.

[0246] Exemplarily, if the first image feature includes a first feature map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the determination module generates an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and is specifically used to: input the first initial component to the first neural network to obtain the first feature map; input the second initial component to the second neural network to obtain the second feature map; add the first feature map and the second feature map to obtain the adjusted component; or overlap the first feature map and the second feature map to obtain the adjusted component.

[0247] Exemplarily, if the first image feature includes a weight coefficient map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the determination module generates an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and is specifically used to: input the first initial component to the first neural network to obtain a first feature map corresponding to the first initial component, and perform guided filtering on the first feature map to obtain the weight coefficient map; input the second initial component to the second neural network to obtain the second feature map; and generate the adjusted component based on the weight coefficient map and the second feature map.

[0248] Exemplarily, the determination module performs guided filtering on the first feature map to obtain the weight coefficient map, which is specifically used for: performing a convolution operation on the first feature map to obtain a convolved feature map; and performing weight mapping on the convolved feature map to obtain the weight coefficient map.

[0249] Exemplarily, the determination module performs weight mapping on the post-convolution feature map, and when obtaining the weight coefficient map, it is specifically used for: performing a pooling operation on the post-convolution feature map; performing a full connection operation and a rectified linear unit ReLU activation operation on the feature map after the pooling operation; performing a full connection operation and an S-curve Sigmoid activation operation on the feature map after the ReLU activation operation; and generating the weight coefficient map based on the post-convolution feature map and the feature map after the Sigmoid activation operation.

[0250] Exemplarily, the determination module performs weight mapping on the post-convolution feature map to obtain the weight coefficient map, which is specifically used to: perform a convolution operation and a Sigmoid activation operation on the post-convolution feature map to obtain the weight coefficient map.

[0251] Exemplarily, when the determination module generates the adjusted component based on the weight coefficient map and the second feature map, it is specifically used to: multiply the weight coefficient map and the second feature map to obtain a multiplied feature map; perform a convolution operation on the first feature map to obtain a convolved feature map; add the multiplied feature map and the convolved feature map to obtain an added feature map; and stack the added feature map and the convolved feature map to obtain the adjusted component.

[0252] Exemplarily, if the first image feature includes K first feature maps that have passed through a neural network, and the second image feature includes K+1 second feature maps that have passed through a neural network, the determination module generates an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and is specifically used to: for the first first feature map and the first second feature map, input the first initial component to the neural network to obtain the first first feature map, and input the second initial component to the neural network to obtain the first second feature map; for the i-th first feature map and the i-th second feature map, the value range of i is 2 to K, K is a positive integer greater than 1, and the i-1-th first feature map is input to the neural network to obtain the i-th first feature map; perform feature fusion on the i-1-th first feature map and the i-1-th second feature map to obtain a fused feature, and input the fused feature to the neural network to obtain the i-th second feature map; after obtaining the last second feature map, generate the adjusted component based on the last second feature map.

[0253] Exemplarily, when the determination module performs feature fusion on the first feature map and the second feature map to obtain the fused feature, it is specifically used to: perform an addition operation on the first feature map and the second feature map to obtain the fused feature; or, perform an overlay operation on the first feature map and the second feature map to obtain the fused feature; or, perform guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generate the fused feature based on the weight coefficient map and the second feature map.

[0254] Exemplarily, the determination module inputs the first initial component to the first neural network, and when obtaining the first feature map, is specifically used to: extract M-dimensional image features from the first initial component, where M is a positive integer; stack the M-dimensional image features to obtain stacked multi-dimensional features; and input the multi-dimensional features to the first neural network to obtain the first feature map.

[0255] Exemplarily, the image features of M dimensions include at least one of the following: image features after convolving the first initial component with a convolution kernel; image features after convolving the first initial component with two convolution kernels; first-order spatial domain features after processing the first initial component with a Sobel operator; second-order spatial domain features after processing the first initial component with a Laplacian operator; or frequency domain features after Fourier transforming the first initial component.

[0256] Exemplarily, when the determination module inputs the first initial component to the first neural network, and obtains the first feature map, it is specifically used to: perform a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform; input the multiple frequency bands or part of the multiple frequency bands to the first neural network to obtain the first feature map; when the determination module inputs the second initial component to the second neural network, and obtains the second feature map, it is specifically used to: perform a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform; input the multiple frequency bands or part of the multiple frequency bands to the second neural network to obtain the second feature map.

[0257] Exemplarily, before the determination module inputs the first initial component to the first neural network to obtain the first feature map, it is also used to: preprocess the first initial component to obtain the preprocessed first initial component; wherein the preprocessed first initial component is used to input the first neural network to obtain the first feature map. When the determination module preprocesses the first initial component to obtain the preprocessed first initial component, it is specifically used to: perform edge enhancement on the first initial component to obtain image features after edge enhancement; perform multi-scale feature extraction on the image features after edge enhancement to obtain multi-scale features; and determine the preprocessed first initial component based on the multi-scale features.

[0258] Exemplarily, the determination module performs multi-scale feature extraction on the edge-enhanced image features, and when obtaining the multi-scale features, it is specifically used for: performing a convolution operation on the edge-enhanced image features to obtain convolution features; performing a downsampling operation on the convolution features to obtain downsampled features; performing a channel transformation on the downsampled features to obtain channel transformed features; performing an upsampling operation on the channel transformed features to obtain upsampled features; and generating the multi-scale features based on the upsampled features and the channel transformed features.

[0259] Exemplarily, before generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the processing module is also used to upsample the second initial component to obtain an upsampled second initial component; wherein the resolution of the upsampled second initial component is equal to the resolution of the first initial component.

[0260] Exemplarily, the processing module is used to upsample the adjusted component to obtain an upsampled adjusted component before performing feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component; wherein the resolution of the upsampled adjusted component is equal to the resolution of the first initial component.

[0261] Exemplarily, the processing module performs feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component, and is further used to upsample the target component to obtain an upsampled target component; wherein the resolution of the upsampled target component is equal to the resolution of the first initial component.

[0262] Exemplarily, when the processing module performs feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, it is specifically used to: perform feature enhancement on the adjusted component through at least one residual block network to obtain the target component; or, perform feature enhancement on the adjusted component through a U-Net network to obtain the target component.

[0263] It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. The present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. The embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure.

[0264] For those skilled in the art, the present disclosure may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure should be included in the scope of the claims of the present disclosure.

Claims

1. An image decoding method, characterized in that: include: Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, and the initial chrominance component includes a first chrominance component and a second chrominance component; Performing Haar wavelet transform processing on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; According to the Haar wavelet frequency domain feature of the initial luminance component and the Haar wavelet frequency domain feature of the first chrominance component, the frequency domain feature of the first chrominance component is adjusted to obtain an adjusted first chrominance component; According to the Haar wavelet frequency domain feature of the initial luminance component and the Haar wavelet frequency domain feature of the second chrominance component, the frequency domain feature of the second chrominance component is adjusted to obtain an adjusted second chrominance component.

2. The method according to claim 1, characterized in that The adjusting the frequency domain characteristics of the first chrominance component according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component to obtain the adjusted first chrominance component includes: Performing a splicing operation on the Haar wavelet frequency domain feature of the initial luminance component and the Haar wavelet frequency domain feature of the first chrominance component to generate a frequency domain fusion feature; based on the first chrominance component, replacing the frequency domain feature corresponding to the first chrominance component with the frequency domain fusion feature to obtain an adjusted first chrominance component; The adjusting the frequency domain characteristics of the second chrominance component according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component to obtain the adjusted second chrominance component includes: The Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the second chrominance component are overlapped to generate frequency domain fusion features; based on the second chrominance component, the frequency domain features corresponding to the second chrominance component are replaced with the frequency domain fusion features to obtain an adjusted second chrominance component.

3. The method according to claim 1, characterized in that The reconstructed image block is a reconstructed image block in a YUV format, the initial luminance component is a Y component, the first chrominance component included in the initial chrominance component is a U component, and the second chrominance component included in the initial chrominance component is a V component; The resolution of the initial luminance component is greater than or equal to the resolution of the first chrominance component, and the resolution of the initial luminance component is greater than or equal to the resolution of the second chrominance component.

4. The method according to claim 1, characterized in that: The performing Haar wavelet transform on the initial brightness component to obtain the Haar wavelet frequency domain features of the initial brightness component comprises: performing at least one Haar wavelet transform on the initial brightness component to obtain a plurality of frequency bands after at least one Haar wavelet transform of the initial brightness component; and determining the Haar wavelet frequency domain features of the initial brightness component based on the plurality of frequency bands after at least one Haar wavelet transform of the initial brightness component; The performing Haar wavelet transform on the first chroma component to obtain the Haar wavelet frequency domain features of the first chroma component includes: performing at least one Haar wavelet transform on the first chroma component to obtain a plurality of frequency bands after the at least one Haar wavelet transform of the first chroma component; and determining the Haar wavelet frequency domain features of the first chroma component based on the plurality of frequency bands after the at least one Haar wavelet transform of the first chroma component; The method of performing Haar wavelet transform on the second chrominance component to obtain Haar wavelet frequency domain characteristics of the second chrominance component includes: performing at least one Haar wavelet transform on the second chrominance component to obtain multiple frequency bands after the at least one Haar wavelet transform of the second chrominance component; and determining the Haar wavelet frequency domain characteristics of the second chrominance component based on the multiple frequency bands after the at least one Haar wavelet transform of the second chrominance component.

5. The method according to claim 4, characterized in that Performing at least one Haar wavelet transform on the initial brightness component, comprising: Performing a first Haar wavelet transform on the initial brightness component to obtain a plurality of frequency bands of the initial brightness component after the first Haar wavelet transform; performing a second Haar wavelet transform on the plurality of frequency bands after the first Haar wavelet transform to obtain a plurality of frequency bands of the initial brightness component after two Haar wavelet transforms; Performing at least one Haar wavelet transform on the first chrominance component, comprising: Performing a first Haar wavelet transform on the first chrominance component to obtain a plurality of frequency bands of the first chrominance component after the first Haar wavelet transform; performing a second Haar wavelet transform on the plurality of frequency bands after the first Haar wavelet transform to obtain a plurality of frequency bands of the first chrominance component after two Haar wavelet transforms; Performing at least one Haar wavelet transform on the second chrominance component, comprising: Perform a first Haar wavelet transform on the second chrominance component to obtain multiple frequency bands of the second chrominance component after the first Haar wavelet transform; perform a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain multiple frequency bands of the second chrominance component after two Haar wavelet transforms.

6. The method according to claim 1, characterized in that The method further comprises: The adjusted first chrominance component is feature enhanced through a first residual block network to obtain a restored target chrominance component corresponding to the first chrominance component; the adjusted second chrominance component is feature enhanced through a second residual block network to obtain a restored target chrominance component corresponding to the second chrominance component.

7. An image encoding method, characterized in that: include: Acquire a reconstructed image block corresponding to the current image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, and the initial chrominance component includes a first chrominance component and a second chrominance component; Performing Haar wavelet transform processing on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; According to the Haar wavelet frequency domain feature of the initial luminance component and the Haar wavelet frequency domain feature of the first chrominance component, the frequency domain feature of the first chrominance component is adjusted to obtain an adjusted first chrominance component; According to the Haar wavelet frequency domain feature of the initial luminance component and the Haar wavelet frequency domain feature of the second chrominance component, the frequency domain feature of the second chrominance component is adjusted to obtain an adjusted second chrominance component.

8. An image decoding device, characterized in that: include: A decoding module, configured to decode a code stream corresponding to a current image block to obtain a reconstructed image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, wherein the initial chrominance component includes a first chrominance component and a second chrominance component; A determination module, configured to perform Haar wavelet transform processing on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; According to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component, the frequency domain characteristics of the first chrominance component are adjusted to obtain an adjusted first chrominance component; according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component, the frequency domain characteristics of the second chrominance component are adjusted to obtain an adjusted second chrominance component.

9. An image encoding device, characterized in that: include: An acquisition module, configured to acquire a reconstructed image block corresponding to the current image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, wherein the initial chrominance component includes a first chrominance component and a second chrominance component; A determination module, configured to perform Haar wavelet transform processing on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; According to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component, the frequency domain characteristics of the first chrominance component are adjusted to obtain an adjusted first chrominance component; according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component, the frequency domain characteristics of the second chrominance component are adjusted to obtain an adjusted second chrominance component.

10. An image decoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 6.

11. An image encoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is configured to execute machine executable instructions to implement the method of claim 7.

12. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by a processor, the method described in any one of claims 1 to 6 is implemented; or, when the computer instructions are executed by a processor, the method described in claim 7 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Method for removing color deviation in the image generated because of high and low brightness

    CN101080026A

  • Band separation filtering / inverse filtering for frame packing / unpacking higher-resolution chroma sampling formats

    CN104982036A

  • Video processing method and device, terminal and storage medium

    CN112565887A

  • Method for eliminating color shift phenomenon that produces due to high or lower brightness in an image

    US20080063268A1

  • Image enhancement method and apparatus

    US20220156890A1