A decoding method, device and equipment

Abstract

The present disclosure provides a decoding method, apparatus, and device thereof. The decoding method comprises: decoding a code stream corresponding to a current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, wherein the resolution of the first initial component is greater than or equal to the resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first and second initial components; and performing feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component. The technical solution disclosed herein can improve decoding performance and reduce decoding complexity.

Description

Technical Field

[0001] The present disclosure relates to the field of coding and decoding technology, and in particular to a decoding method, apparatus and device thereof. Background Art

[0002] To save space, video images are encoded before transmission. Complete video encoding involves prediction, transformation, quantization, entropy coding, and filtering. Prediction can include intra-frame prediction and inter-frame prediction. Inter-frame prediction leverages temporal correlations in the video to predict the current pixel using pixels from adjacent coded images, effectively removing temporal redundancy. Intra-frame prediction leverages spatial correlations in the video to predict the current pixel using pixels from coded blocks in the current frame, effectively removing spatial redundancy.

[0003] With the rapid development of deep learning, which has achieved success in many high-level computer vision problems, such as image classification and object detection, it has also gradually begun to be applied in the field of codecs, where neural networks can be used to encode and decode images. Although neural network-based codecs show great performance potential, they still suffer from problems such as poor decoding performance and high complexity. Summary of the Invention

[0004] In view of this, the present disclosure provides a decoding method, apparatus and device thereof, which can improve decoding performance.

[0005] In a first aspect, the present disclosure provides a decoding method, comprising:

[0006] Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, where the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0007] generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0008] Feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0009] In some embodiments, the reconstructed image block is a reconstructed image block in a YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and a chrominance V component.

[0010] In some embodiments, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes:

[0011] Obtaining a first image feature corresponding to the first initial component, where the first image feature includes an image feature that has not been processed by a neural network;

[0012] Obtaining a second image feature corresponding to the second initial component, where the second image feature includes an image feature that has not been processed by the neural network;

[0013] The adjusted component is generated based on the first image feature and the second image feature.

[0014] In some embodiments, the first image feature includes a Haar wavelet frequency domain feature; and the second image feature includes a Haar wavelet frequency domain feature.

[0015] In some embodiments, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes:

[0016] Performing at least one Haar wavelet transform on the first initial component to obtain a plurality of transformed frequency bands as a first feature map;

[0017] Performing at least one Haar wavelet transform on the second initial component to obtain a plurality of transformed frequency bands as a second feature map;

[0018] Performing a splicing operation on the first feature map and the second feature map to obtain the adjusted component.

[0019] In some embodiments, the stacking operation on the first feature map and the second feature map includes:

[0020] The first feature map and the second feature map are stacked along the channel dimension.

[0021] In some embodiments, performing at least one Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the transform includes: performing two Haar wavelet transforms on the first initial component to obtain a plurality of frequency bands after the two Haar wavelet transforms;

[0022] The performing at least one Haar wavelet transform on the second initial component to obtain a plurality of frequency bands after the transform includes: performing two Haar wavelet transforms on the second initial component to obtain a plurality of frequency bands after the two Haar wavelet transforms.

[0023] In some embodiments, performing multiple Haar wavelet transforms on the first initial component to obtain multiple frequency bands after the transforms includes: performing two Haar wavelet transforms on the first initial component to obtain multiple frequency bands after the two Haar wavelet transforms;

[0024] The performing multiple Haar wavelet transforms on the second initial component to obtain multiple frequency bands after the transforms includes: performing two Haar wavelet transforms on the second initial component to obtain multiple frequency bands after the two Haar wavelet transforms.

[0025] In some embodiments, performing two Haar wavelet transforms on the first initial component to obtain the multiple frequency bands after the two Haar wavelet transforms includes: performing a first Haar wavelet transform on the first initial component to obtain the multiple frequency bands after the first Haar wavelet transform; performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain the multiple frequency bands after the two Haar wavelet transforms;

[0026] The performing two Haar wavelet transforms on the second initial component to obtain multiple frequency bands after the two Haar wavelet transforms includes: performing a first Haar wavelet transform on the second initial component to obtain multiple frequency bands after the first Haar wavelet transform; and performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain multiple frequency bands after the two Haar wavelet transforms.

[0027] In some embodiments, the performing feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component includes:

[0028] The adjusted component is subjected to feature enhancement through multiple residual block networks to obtain the target component.

[0029] In a second aspect, the present disclosure provides an encoding method, comprising:

[0030] A first initial component and a second initial component of a current image block are encoded to obtain a code stream corresponding to the current image block, wherein a resolution of the first initial component is greater than or equal to a resolution of the second initial component.

[0031] In some embodiments, the current image block is in YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and a chrominance V component.

[0032] In a third aspect, the present disclosure provides a decoding device, comprising:

[0033] a decoding module, configured to decode a code stream corresponding to a current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, wherein a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0034] a determining module, configured to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0035] A processing module is used to perform feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0036] In a fourth aspect, the present disclosure provides an encoding device, comprising:

[0037] The encoding module is used to encode the first initial component and the second initial component of the current image block to obtain a code stream corresponding to the current image block, wherein the resolution of the first initial component is greater than or equal to the resolution of the second initial component.

[0038] In a fifth aspect, the present disclosure provides a decoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method described in any embodiment of the first aspect.

[0039] In the sixth aspect, the present disclosure provides an encoding end device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method described in any embodiment of the second aspect.

[0040] In a seventh aspect, the present disclosure provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method described in any embodiment of the first aspect or the method described in any embodiment of the second aspect is implemented.

[0041] In an eighth aspect, the present disclosure provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, it implements the method described in any embodiment of the first aspect, or implements the method described in any embodiment of the second aspect.

[0042] It can be seen from the above technical solution that in the embodiment of the present disclosure, after decoding to obtain a reconstructed image block, the reconstructed image block includes a first initial component and a second initial component, the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, that is, the second initial component is adjusted based on the first initial component to obtain the adjusted component, and then the target component corresponding to the second initial component is determined based on the adjusted component, thereby improving decoding performance, reducing decoding complexity, reducing computational complexity, and requiring relatively few resources. Devices that are sensitive to cost and delay can also use the above algorithm, and propose a low-complexity and efficient post-processing network model framework for compression defect recovery at multiple bit rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present disclosure;

[0044] Figure 2 is a flowchart of a decoding method in one embodiment of the present disclosure;

[0045] Figure 3 is a schematic diagram of a processing process of an encoding end in one embodiment of the present disclosure;

[0046] Figure 4 is a schematic diagram of a processing process of a decoding end in one embodiment of the present disclosure;

[0047] Figure 5A is a schematic structural diagram of end-to-end image compression in one embodiment of the present disclosure;

[0048] Figure 5B is a schematic structural diagram of post-processing based on deep learning in one embodiment of the present disclosure;

[0049] Figures 6A-6I is a schematic structural diagram of a post-processing network in one embodiment of the present disclosure;

[0050] Figures 7A-7C is a schematic diagram of upsampling in one embodiment of the present disclosure;

[0051] Figure 7D is a schematic structural diagram of a post-processing network in one embodiment of the present disclosure;

[0052] Figure 7E and Figure 7F is a schematic diagram of feature enhancement in one embodiment of the present disclosure;

[0053] Figure 8 It is a hardware structure diagram of a decoding end device in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not used to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items. It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, depending on the context. In addition, the word "if" used can be interpreted as "when...", or "when...", or "in response to determination".

[0055] In an embodiment of the present disclosure, a decoding method is proposed, which may involve the following concepts:

[0056] JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous-tone still image compression. File extensions can be .jpg or .jpeg. JPEG is a commonly used image file format. JPEG uses a combination of predictive coding (DPCM), discrete cosine transform (DCT), and entropy coding to remove redundant image and color data. It is a lossy compression format that can compress images into a very small storage space, but this can cause some damage to the image data. Using too high a compression ratio will reduce the quality of the image recovered after decompression. For high-quality images, JPEG should not use too high a compression ratio.

[0057] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact compressed domain representation, significantly improves compression efficiency over commonly used image coding standards at the same subjective quality, and effectively improves performance in image processing and computer vision tasks. JPEG-AI targets a wide range of applications, such as cloud storage, vision management, autonomous vehicles and devices, image acquisition storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a codec solution that needs to significantly improve compression efficiency at the same subjective quality and provide efficient compressed domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires hardware- and software-friendly encoding and decoding, support for 8-bit and 10-bit depths, and efficient encoding and progressive decoding of images with text and graphics.

[0058] Post-processing filtering: The encoder compresses the image to form a stream of bits. The decoder decodes and reconstructs the image from the stream. However, due to the encoding algorithm itself, the image may exhibit defects such as blocking effects, image artifacts, and color drift after decoding and reconstructing the image. The purpose of post-processing filtering is to improve the defects caused by encoding and compression, restore the image data to the greatest extent possible, and improve the subjective quality of the decoded and reconstructed image.

[0059] Image Super-Resolution: Super-Resolution mainly refers to restoring a low-resolution image (Low-Resolution) to a high-resolution image (High-Resolution), while enhancing image details and improving the subjective quality of the image. Image super-resolution is a technology used to enhance image resolution in computer vision image processing.

[0060] YUV Color Space: Each pixel in a color image can typically be described using several independent physical quantities, forming a spatial coordinate system. This is the image's color space. The YUV color space describes the pixels of a color image using three attributes: Y (brightness), U (chrominance), and V (color).

[0061] Entropy encoding: Entropy encoding is a method of encoding that does not lose any information during the encoding process according to the entropy principle. Information entropy is the average amount of information in the source (a measure of uncertainty). Entropy coding methods include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0062] Neural Network (NN): A neural network refers to an artificial neural network. A NN is a computational model composed of a large number of interconnected nodes (or neurons). In a NN, neuronal processing units can represent different objects, such as features, letters, concepts, or some meaningful abstract patterns. Processing units in a NN can be divided into three categories: input units, output units, and hidden units. Input units receive signals and data from the external world; output units output the processed results; and hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the strength of the connections between units, and the representation and processing of information is reflected in the connections between processing units. A NN is a non-programmed, brain-like information processing method. Its essence is to achieve parallel and distributed information processing capabilities through the transformations and dynamics of the neural network, mimicking the information processing capabilities of the human brain to varying degrees and levels. In the field of video processing, commonly used neural networks include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), and fully connected networks.

[0063] Convolutional Neural Network (CNN): A convolutional neural network (CNN) is a feedforward neural network and one of the most representative network structures in deep learning technology. Its artificial neurons can respond to surrounding cells within a certain coverage area, making it an excellent choice for processing large images. The basic structure of a CNN consists of two layers. The first is the feature extraction layer (also known as the convolution layer). Each neuron's input is connected to the local receptive field of the previous layer, extracting local features. Once a local feature is extracted, its positional relationship with other features is determined. The second is the feature mapping layer (also known as the activation layer). Each computational layer of the neural network consists of multiple feature maps. Each feature map is a plane where all neurons have equal weights. The feature map structure can use Sigmoid, ReLU, Leaky-ReLU, PReLU, GDN, and other activation functions. Furthermore, because neurons within a mapping plane share weights, the number of free parameters in the network can be reduced.

[0064] For example, one of the advantages of convolutional neural networks over image processing algorithms is that they avoid complex pre-processing of images (such as extracting artificial features) and can directly input the original image for end-to-end learning. One of the advantages of convolutional neural networks over ordinary neural networks is that ordinary neural networks use a fully connected approach, that is, all neurons from the input layer to the hidden layer are connected. This will result in a huge number of parameters, making network training time-consuming or even difficult to train. Convolutional neural networks avoid this difficulty through methods such as local connections and weight sharing.

[0065] Deconvolution: Deconvolution is also called transposed convolution. It works similarly to convolution, but uses padding to make the output larger than the input (or even smaller). A stride of 1 equals the input size. A stride of N means the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0066] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to new samples. The purpose of learning is to learn the patterns implicit in the data pairs. For data outside the learning set with the same patterns, the trained network can also give appropriate outputs. This ability can be called generalization ability.

[0067] Feature: The feature involved in this disclosure is a three-dimensional feature matrix of C*W*H, see Figure 1 The figure below is a schematic diagram of a three-dimensional feature matrix. In the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input or output of a neural network.

[0068] Rate-Distortion Optimized (RDO): Coding efficiency is evaluated using two key metrics: bit rate and Peak Signal to Noise Ratio (PSNR). A smaller bitrate results in a higher compression ratio, while a higher PSNR indicates better reconstructed image quality. When selecting a mode, the discriminant formula is essentially a comprehensive evaluation of these two metrics. For example, the cost of a mode is: J(mode) = D + λ*R, where D represents distortion. This is typically measured using the SSE metric (mean square sum of squared differences between the reconstructed image block and the source image). For cost considerations, the SAD metric (the sum of absolute differences between the reconstructed image block and the source image) can also be used. λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in that mode, including the bits required for coding mode information, motion information, and residual information. Using the rate-distortion principle to compare coding modes during mode selection generally ensures optimal coding performance.

[0069] Numerous coding tools have been proposed for various modules on the encoding side, and each tool often has multiple modes. Different coding tools often produce the best coding performance for different video sequences. Therefore, during the encoding process, Rate-Distortion Optimization (RDO) is often used to compare the coding performance of different tools or modes to select the optimal mode. After determining the optimal tool or mode, the decision-making information for the tool or mode is conveyed by encoding marker information in the bitstream. While this approach introduces higher coding complexity, it can adaptively select the optimal mode combination for different content, achieving optimal coding performance. The decoding side can obtain relevant mode information by directly parsing the marker information, with minimal complexity impact.

[0070] In end-to-end image compression technology based on deep learning, images are converted to the YUV color space, and the UV chrominance signal is downsampled by half. The Y and UV signals are then compressed and encoded separately. The decoder decodes and reconstructs the bitstream data to obtain the reconstructed Y and UV signals. Because the UV chrominance signal is compressed and encoded after being downsampled by half, its resolution is halved. Therefore, the resolution of the UV chrominance signal can be restored during post-processing. However, this resolution recovery process is computationally complex and resource-intensive. Some cost- and latency-sensitive devices cannot restore the resolution of the UV chrominance signal.

[0071] In response to the above findings, this embodiment proposes a neural network-based compression defect removal network model method. In the field of end-to-end image compression, a low-complexity and efficient post-processing network model framework is proposed, which can enhance the UV signal based on the Y signal. This method has low complexity and high performance, and can recover compression defects under various bit rates.

[0072] The decoding method in the embodiment of the present disclosure is described in detail below with reference to several specific embodiments.

[0073] Example 1: A decoding method is proposed in the embodiment of the present disclosure, see Figure 2 FIG. 2 is a flow chart of the decoding method. The method can be applied to a decoding end (also called a video decoder). The method can include steps 201-203.

[0074] Step 201: Decode the code stream corresponding to the current image block to obtain a reconstructed image block.

[0075] Exemplarily, the reconstructed image block may include a first initial component and a second initial component, the resolution of the second initial component is a downsampled resolution, and the resolution of the first initial component may be greater than or equal to the resolution of the second initial component. For example, the resolution of the first initial component may be a resolution that has not been downsampled (i.e., the original resolution) or a downsampled resolution, but the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and this is not limited.

[0076] Exemplarily, if the reconstructed image block is a reconstructed image block in YUV format, the first initial component may be a luminance component, and the second initial component may be a chrominance U component and / or a chrominance V component; or, if the reconstructed image block is a reconstructed image block in RGB format, the first initial component may be a G component, and the second initial component may be an R component and / or a B component.

[0077] Step 202: Generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component. The reconstruction quality of the adjusted component may be better than the reconstruction quality of the second initial component, the reconstruction quality of the adjusted component may be equal to the reconstruction quality of the second initial component, or the reconstruction quality of the second initial component may also be better than the reconstruction quality of the adjusted component.

[0078] Exemplarily, a first image feature corresponding to a first initial component can be obtained, and the first image feature may include but is not limited to image features that have not passed through a neural network and / or image features that have passed through a neural network; a second image feature corresponding to a second initial component can be obtained, and the second image feature may include but is not limited to image features that have not passed through a neural network and / or image features that have passed through a neural network; an adjusted component can be generated based on the first image feature and the second image feature.

[0079] For example, if the first image feature includes an image feature that has not been processed by a neural network, the first image feature may include, but is not limited to, at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature; if the second image feature includes an image feature that has not been processed by a neural network, the second image feature may include, but is not limited to, at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The above are just a few examples and are not intended to be limiting.

[0080] If the first image feature includes a first feature map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component may include but is not limited to: inputting the first initial component into the first neural network to obtain a first feature map; inputting the second initial component into the second neural network to obtain a second feature map; adding the first feature map and the second feature map to obtain an adjusted component; or, stacking the first feature map and the second feature map to obtain an adjusted component.

[0081] Exemplarily, if the first image feature includes a weight coefficient map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component may include but is not limited to: inputting the first initial component into the first neural network to obtain a first feature map corresponding to the first initial component, and performing guided filtering on the first feature map to obtain a weight coefficient map; inputting the second initial component into the second neural network to obtain a second feature map; and generating the adjusted component based on the weight coefficient map and the second feature map.

[0082] Exemplarily, performing guided filtering on the first feature map to obtain a weight coefficient map may include but is not limited to: performing a convolution operation on the first feature map to obtain a convolved feature map; and performing weight mapping on the convolved feature map to obtain a weight coefficient map.

[0083] Exemplarily, weight mapping is performed on the feature map after convolution to obtain a weight coefficient map, including but not limited to: performing a pooling operation on the feature map after convolution to obtain a feature map after pooling operation; performing a full connection operation and a ReLU (Rectified Linear Unit) activation operation on the feature map after pooling operation to obtain a feature map after ReLU activation operation; performing a full connection operation and a Sigmoid (S-shaped curve, i.e., S-shaped function) activation operation on the feature map after ReLU activation operation to obtain a feature map after Sigmoid activation operation; generating a weight coefficient map based on the feature map after convolution and the feature map after Sigmoid activation operation, and multiplying the feature map after convolution and the feature map after Sigmoid activation operation to obtain the weight coefficient map.

[0084] Exemplarily, weight mapping is performed on the convolution feature map to obtain a weight coefficient map, which may include but is not limited to: performing a convolution operation and a Sigmoid activation operation on the convolution feature map to obtain a weight coefficient map.

[0085] Exemplarily, generating the adjusted component based on the weight coefficient map and the second feature map may include but is not limited to: multiplying the weight coefficient map and the second feature map to obtain a multiplied feature map; performing a convolution operation on the first feature map to obtain a convolved feature map; adding the multiplied feature map and the convolved feature map to obtain an added feature map; and stacking the added feature map and the convolved feature map to obtain the adjusted component.

[0086] Exemplarily, if the first image feature includes K first feature maps that have passed through a neural network, and the second image feature includes K+1 second feature maps that have passed through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component may include, but is not limited to: for the first first feature map and the first second feature map, the first initial component may be input into the neural network to obtain the first first feature map, and the second initial component may be input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, i is any integer greater than or equal to 2 and less than or equal to K, and K may be a positive integer greater than 1, then the i-1th first feature map may be input into the neural network to obtain the i-th first feature map; feature fusion is performed on the i-1th first feature map and the i-1th second feature map to obtain a fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map; after obtaining the last second feature map (i.e., the last second feature map among the K+1 second feature maps), the adjusted component may be generated based on the last second feature map.

[0087] Exemplarily, performing feature fusion on a first feature map (such as the i-1th first feature map) and a second feature map (such as the i-1th second feature map) to obtain a fused feature may include, but is not limited to: performing an addition operation on the first feature map and the second feature map to obtain a fused feature; or, performing a splicing operation on the first feature map and the second feature map to obtain a fused feature; or, performing guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating a fused feature based on the weight coefficient map and the second feature map. The above are just a few examples and are not limiting.

[0088] Exemplarily, inputting the first initial component into the first neural network to obtain a first feature map may include but is not limited to: extracting M-dimensional image features from the first initial component, where M is a positive integer; stacking the M-dimensional image features to obtain stacked multi-dimensional features; and inputting the multi-dimensional features into the first neural network to obtain a first feature map.

[0089] Exemplarily, the M-dimensional image features may include, but are not limited to, at least one of the following: image features obtained by convolving the first initial component with a convolution kernel; image features obtained by convolving the first initial component with two convolution kernels; first-order spatial features obtained by processing the first initial component with a Sobel operator; second-order spatial features obtained by processing the first initial component with a Laplacian operator; or frequency domain features obtained by Fourier transforming the first initial component. The above are only a few examples of image features and are not intended to be limiting.

[0090] Exemplarily, inputting the first initial component into the first neural network to obtain the first feature map may include, but is not limited to: performing a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform; and inputting the multiple frequency bands or portions of the multiple frequency bands into the first neural network to obtain the first feature map. Inputting the second initial component into the second neural network to obtain the second feature map may include, but is not limited to: performing a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform; and inputting the multiple frequency bands or portions of the multiple frequency bands into the second neural network to obtain the second feature map.

[0091] When performing a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform, the first initial component can be subjected to a single wavelet transform to obtain multiple frequency bands after the wavelet transform, or the first initial component can be subjected to multiple wavelet transforms to obtain multiple frequency bands after the wavelet transform. When performing multiple wavelet transforms on the first initial component, the first initial component is first subjected to a wavelet transform to obtain multiple frequency bands after the wavelet transform, then a target frequency band (all or part of the multiple frequency bands) is selected from the multiple frequency bands, and the target frequency band is subjected to a wavelet transform to obtain multiple frequency bands after the wavelet transform, and so on, until the wavelet transform is completed to obtain multiple frequency bands after the wavelet transform. When performing a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform, the second initial component can be subjected to a single wavelet transform to obtain multiple frequency bands after the wavelet transform, or the second initial component can be subjected to multiple wavelet transforms to obtain multiple frequency bands after the wavelet transform.

[0092] Exemplarily, before inputting the first initial component into the first neural network to obtain the first feature map, the first initial component may be preprocessed to obtain a preprocessed first initial component; wherein the preprocessed first initial component is used to input into the first neural network to obtain the first feature map. Preprocessing the first initial component to obtain the preprocessed first initial component may include, but is not limited to: edge enhancement of the first initial component to obtain edge-enhanced image features; multi-scale feature extraction of the edge-enhanced image features to obtain multi-scale features; and using the multi-scale features as the preprocessed first initial component based on the preprocessed first initial component.

[0093] Exemplarily, multi-scale feature extraction is performed on the edge-enhanced image features to obtain multi-scale features, which may include but is not limited to: performing a convolution operation on the edge-enhanced image features to obtain convolution features; performing a downsampling operation on the convolution features to obtain down-sampled features; performing a channel transformation on the down-sampled features to obtain channel-transformed features; performing an upsampling operation on the channel-transformed features to obtain up-sampled features; generating multi-scale features based on the up-sampled features and the channel-transformed features, and then inputting the multi-scale features into the first neural network as the preprocessed first initial component.

[0094] Step 203: Perform feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0095] In one possible implementation, before generating the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the second initial component may be upsampled to obtain an upsampled second initial component. The resolution of the upsampled second initial component may be equal to the resolution of the first initial component.

[0096] In another possible implementation, before performing feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component (i.e., before step 203), the adjusted component may be upsampled to obtain an upsampled adjusted component. The resolution of the upsampled adjusted component may be equal to the resolution of the first initial component.

[0097] In another possible implementation, after feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component (i.e., after step 203), the target component may be upsampled to obtain an upsampled target component. The resolution of the upsampled target component may be equal to the resolution of the first initial component.

[0098] Exemplarily, feature enhancement is performed on the adjusted component to obtain a recovered target component corresponding to the second initial component, which may include but is not limited to: feature enhancement is performed on the adjusted component through at least one residual block network to obtain a target component corresponding to the second initial component; or, feature enhancement is performed on the adjusted component through a U-Net (a network with an Encoder-Decoder structure, the first half is feature extraction, and the second half is upsampling) network to obtain a target component corresponding to the second initial component.

[0099] Exemplarily, the above execution order is only for the convenience of describing the examples given. In actual applications, the execution order between the steps can also be changed, and there is no limitation on this execution order. Moreover, in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this disclosure, and the steps included in the method may be more or less than those described in this disclosure. In addition, a single step described in this disclosure may be decomposed into multiple steps for description in other embodiments; multiple steps described in this disclosure may also be combined into a single step for description in other embodiments.

[0100] It can be seen from the above technical solution that in the embodiment of the present disclosure, after decoding to obtain a reconstructed image block, the reconstructed image block includes a first initial component and a second initial component, the resolution of the second initial component is the downsampled resolution, the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, that is, the second initial component is adjusted based on the first initial component to obtain the adjusted component, and then the target component corresponding to the second initial component is determined based on the adjusted component, thereby improving decoding performance, reducing decoding complexity, reducing computational complexity, and requiring relatively few resources. Devices that are sensitive to cost and delay can also use the above algorithm, and propose a low-complexity and efficient post-processing network model framework for compression defect recovery at multiple bit rates.

[0101] Example 2: For the processing process of the encoding end (the encoding end is also called a video encoder), please refer to Figure 3 As shown, Figure 3 This is just an example of the processing process at the encoding end, and does not limit the processing process at the encoding end.

[0102] After obtaining the current image block x (which can be the original image block x, i.e., the input image block), the encoder can perform an analysis transformation on the current image block x using an analysis transformation network (i.e., a neural network) to obtain the image features y corresponding to the current image block x. Performing feature transformation on the current image block x using the analysis transformation network means transforming the current image block x into image features y in the latent domain, thereby facilitating all subsequent processes to operate in the latent domain.

[0103] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the encoding and decoding process for the image block can also be directly used for the image.

[0104] After obtaining the image feature y, the encoding end performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. For example, the image feature y can be input into a hyperparameter encoding network (i.e., a neural network), and the hyperparameter encoding network performs a coefficient hyperparameter feature transformation on the image feature y to obtain a coefficient hyperparameter feature z. The hyperparameter encoding network can be a trained neural network, and there is no restriction on the training process of the hyperparameter encoding network. It only needs to be able to perform a coefficient hyperparameter feature transformation on the image feature y. The image feature y in the latent domain obtains the super-prior latent information z after passing through the hyperparameter encoding network.

[0105] After obtaining the coefficient hyperparameter feature z, the encoder can quantize the coefficient hyperparameter feature z to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is, Figure 3 The Q operation in is the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the hyperparameter quantization feature is encoded to obtain the Bitstream#1 (i.e., the first code stream) corresponding to the current image block, i.e. Figure 3 The AE operation in [ ] represents an encoding process, such as entropy encoding. Alternatively, the encoder can directly encode the coefficient hyperparameter feature z to obtain Bitstream #1 corresponding to the current image block. The hyperparameter quantization features or coefficient hyperparameter feature z carried in Bitstream #1 are primarily used to obtain the mean and probability distribution model parameters.

[0106] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can send the Bitstream#1 corresponding to the current image block to the decoder. For the processing process of the Bitstream#1 corresponding to the current image block by the decoder, please refer to the subsequent embodiments.

[0107] After obtaining the Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the super-parameter quantization feature, that is, Figure 3 AD in it represents the decoding process. Then, the super-parameter quantization feature is dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. Figure 3 The IQ operation in [1] is the inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the encoder can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0108] For the encoding process of Bitstream#1, an encoding method of a fixed probability density model can be used. For the decoding process of Bitstream#1, a decoding method of a fixed probability density model can be used. There is no restriction on this encoding and decoding process.

[0109] After obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (see the subsequent embodiments for the determination process of the residual feature y_hat) to obtain the predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat. The two are jointly input to obtain a more accurate predicted value mu. The predicted value mu is used to obtain the residual by subtracting from the original feature and adding it to the decoded residual to obtain the reconstructed y.

[0110] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 3 The dashed box in indicates that the mean prediction network is optional.

[0111] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the predicted value mu, such as taking the difference between the image feature y and the predicted value mu as the residual feature r. Then, the residual feature r is feature processed to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, it is necessary to deploy a mean prediction network, and the mean prediction network provides the predicted value mu. Alternatively, after obtaining the image feature y, the encoding end can perform feature processing on the image feature y to obtain the image feature s. There is no restriction on this feature processing process, and it can be any feature processing method. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0112] After obtaining the image feature s, the encoding end can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, that is, Figure 3 The Q operation in is the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoder can encode the image quantization feature to obtain Bitstream#2 (i.e., the second code stream) corresponding to the current image block, i.e. Figure 3 The AE operation in represents an encoding process, such as an entropy encoding process. Alternatively, the encoder can directly encode the image feature s to obtain Bitstream #2 corresponding to the current image block without involving the quantization process of the image feature s.

[0113] After obtaining Bitstream#2 corresponding to the current image block, the encoder can send Bitstream#2 corresponding to the current image block to the decoder. For details about the processing of Bitstream#2 corresponding to the current image block by the decoder, see the subsequent embodiments.

[0114] After obtaining the Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 3 AD in the image represents the decoding process. Then, the encoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 3 The IQ operation in is the inverse quantization process. Alternatively, after obtaining Bitstream#2 corresponding to the current image block, the encoder can also decode Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0115] After obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing). There is no restriction on this feature recovery process and it can be any feature recovery method to obtain the residual feature r_hat. The residual feature r_hat and the residual feature r can be the same or different. After obtaining the residual feature r_hat, the encoder determines the image feature y_hat based on the residual feature r_hat and the predicted value mu. The image feature y_hat and the image feature y can be the same or different. For example, the sum of the residual feature r_hat and the predicted value mu is used as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, which provides the predicted value mu. Alternatively, after obtaining the image feature s', the encoder can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the image feature y_hat. The image feature y_hat and the image feature y can be the same or different. In this case, there is no need to deploy a mean prediction network. The dotted box indicates that the residual process is an optional process.

[0116] After obtaining the image feature y_hat, the encoding end can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0117] In one possible implementation, when the encoder encodes the image quantization feature or image feature s to obtain Bitstream #2 corresponding to the current image block, the encoder needs to first determine a probability distribution model and then encode the image quantization feature or image feature s based on the probability distribution model. Furthermore, when decoding Bitstream #2, the encoder also needs to first determine a probability distribution model and then decode Bitstream #2 based on the probability distribution model.

[0118] To obtain the probability distribution model, continue to refer to Figure 3 As shown, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probabilistic hyperparameter decoding network can be a trained neural network. There is no restriction on the training process of this probabilistic hyperparameter decoding network. It only needs to be able to perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0119] In one possible implementation, the processing at the encoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without any restriction on the encoding process.

[0120] Example 3: For the processing process of the decoding end (the decoding end is also called a video decoder), please refer to Figure 4 As shown, Figure 4 This is just an example of the processing process at the decoding end, and does not limit the processing process at the decoding end.

[0121] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode Bitstream#1 to obtain the super-parameter quantization feature, that is, Figure 4 AD in it represents the decoding process. Then, the super-parameter quantization feature is dequantized to obtain the coefficient super-parameter feature z_hat. The coefficient super-parameter feature z_hat can be the same as or different from the coefficient super-parameter feature z. Figure 4 The IQ operation in [1] is the inverse quantization process. Alternatively, after obtaining Bitstream#1 corresponding to the current image block, the decoder can also decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat without involving the inverse quantization process of the coefficient hyperparameter feature z_hat.

[0122] For the decoding process of Bitstream#1, a decoding method using a fixed probability density model may be used, and there is no restriction on this.

[0123] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be an image, that is, the decoding process for the image block can also be directly used for the image.

[0124] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (see the subsequent embodiments for the determination process of the residual feature y_hat) to obtain the predicted value mu (i.e., mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input to the mean prediction network, and the mean prediction network determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. There is no restriction on this prediction process. Among them, for the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0125] It should be noted that the mean prediction network is an optional neural network, that is, there is no mean prediction network, that is, there is no need to determine the predicted value mu through the mean prediction network. Figure 4 The dashed box in indicates that the mean prediction network is optional.

[0126] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the figure represents the decoding process. Then, the decoding end can dequantize the image quantization feature to obtain the image feature s'. The image feature s' can be the same as or different from the image feature s. Figure 4 The IQ operation in [ 1 ] is the inverse quantization process. Alternatively, after obtaining Bitstream #2 corresponding to the current image block, the decoder can also decode Bitstream #2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0127] After obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' (i.e., the inverse process of feature processing) to obtain the residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoding end determines the image feature y_hat based on the residual feature r_hat and the predicted value mu. The image feature y_hat is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the predicted value mu as the image feature y_hat. In this case, it is necessary to deploy a mean prediction network, which provides the predicted value mu. Alternatively, after obtaining the image feature s', the decoding end can perform feature recovery on the image feature s' to obtain the image feature y_hat. The image feature y_hat can be the same as or different from the image feature y. In this case, there is no need to deploy a mean prediction network, and the dotted box indicates that the residual process is an optional process.

[0128] After obtaining the image feature y_hat, the decoding end can perform a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input into the synthetic transformation network, and the synthetic transformation network performs a synthetic transformation on the image feature y_hat to obtain the reconstructed image block x_hat. At this point, the image reconstruction process is completed.

[0129] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine the probability distribution model, and then decode Bitstream#2 based on the probability distribution model. Figure 4 As shown, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into the probabilistic hyperparameter decoding network, and the probabilistic hyperparameter decoding network performs a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. Among them, the probabilistic hyperparameter decoding network can be a trained neural network. There is no restriction on the training process of this probabilistic hyperparameter decoding network. It can perform a coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0130] In one possible implementation, the processing at the decoding end may be performed by a deep learning model or a neural network model, thereby achieving an end-to-end image compression and encoding process, without limiting the decoding process.

[0131] Example 4: See Figure 5AFigure 2 shows a schematic diagram of the architecture of end-to-end image compression based on deep learning. At the encoder, the image is converted to the YUV color space, and the UV signal is downsampled by half. The Y and UV signals are then compressed and encoded separately, and the Y and UV signal streams are sent to the decoder. At the decoder, the Y signal stream is decoded and reconstructed to obtain the reconstructed Y signal, and the UV signal stream is decoded and reconstructed to obtain the reconstructed UV signal. Since the UV signal is compressed and encoded after being downsampled by half, the resolution of the UV signal is halved. Therefore, the resolution of the reconstructed UV signal is also halved. The resolution of the reconstructed Y signal is H*W, while the resolution of the reconstructed UV signal is H / 2*W / 2. This means that the resolution of the reconstructed UV signal is half that of the reconstructed Y signal.

[0132] Since the resolution of the reconstructed UV signal is half that of the reconstructed Y signal, the resolution of the UV signal can be restored during post-processing, that is, the resolution of the UV signal can be restored to H*W. However, this resolution restoration process is computationally complex and resource-intensive. Therefore, some cost- and latency-sensitive devices cannot restore the resolution of the UV chrominance signal.

[0133] In response to the above findings, in the embodiment of the present disclosure, in the field of end-to-end image compression, a low-complexity and efficient post-processing method is proposed. During the post-processing process, the UV signal can be enhanced based on the Y signal. This method has low complexity and high performance, can improve decoding performance, reduce decoding complexity, reduce computational complexity, and has relatively low resource requirements. Cost- and delay-sensitive devices can also use the above algorithm, and can recover compression defects under various bit rates.

[0134] In the embodiment of the present disclosure, after receiving the code stream corresponding to the current image block, the decoding end can decode the code stream corresponding to the current image block to obtain a reconstructed image block. For example, the decoding end can adopt the processing flow of Example 3 to decode the code stream corresponding to the current image block to obtain a reconstructed image block. There is no restriction on the decoding process of this reconstructed image block.

[0135] After obtaining the reconstructed image block, if the reconstructed image block is in YUV format, the chrominance U component (i.e., U component) can be auxiliary enhanced based on the luminance component (i.e., Y component), the chrominance V component (i.e., V component) can be auxiliary enhanced based on the Y component, and the U component and the V component can be auxiliary enhanced based on the Y component. If the reconstructed image block is in RGB format, the R component can be auxiliary enhanced based on the G component, the B component can be auxiliary enhanced based on the G component, and the R component and the B component can be auxiliary enhanced based on the G component. For the convenience of description, in this embodiment, the reconstructed image block in YUV format is used as an example for explanation. Therefore, the reconstructed image block can include Y component, U component and V component. The resolution of the Y component can be greater than the resolution of the U component. For example, the resolution of the U component is the resolution that has not been downsampled. The resolution of the Y component can be the original resolution or the resolution that has been downsampled, but the resolution of the Y component is greater than the resolution of the U component. The resolution of the Y component can be greater than that of the V component. For example, if the resolution of the V component is not downsampled, the resolution of the Y component can be the original resolution or the downsampled resolution, but the resolution of the Y component is greater than that of the V component. For example, if the resolution of the Y component is H*W, the resolution of the U component is H / 2*W / 2, and the resolution of the V component is H / 2*W / 2.

[0136] See also Figure 5B As shown, it is a structural diagram of post-processing based on deep learning. The decoding end can decode the code stream corresponding to the current image block to obtain a reconstructed image block. If the reconstructed image block is a reconstructed image block in YUV format, the reconstructed image block may include a Y component, a U component, and a V component. The Y component is recorded as the Y initial component (i.e., the first initial component of the above embodiment), the U component is recorded as the U initial component (i.e., the second initial component of the above embodiment), and the V component is recorded as the V initial component (i.e., the second initial component of the above embodiment). Based on this, the U initial component can be auxiliary enhanced based on the Y initial component, the V initial component can be auxiliary enhanced based on the Y initial component, and the U initial component and the V initial component can be auxiliary enhanced based on the Y initial component. Among them, the resolution of the Y initial component can be greater than the resolution of the U initial component, and the resolution of the Y initial component can be greater than the resolution of the U initial component. For example, see Figure 5B As shown, the resolution of the Y initial component is H*W, the resolution of the U initial component is H / 2*W / 2, and the resolution of the V initial component is H / 2*W / 2.

[0137] In this embodiment, see Figure 5BAs shown, a preprocessing module is involved. The preprocessing module is an optional module. After decoding to obtain the Y initial component, if the preprocessing module is deployed, the Y initial component can be preprocessed by the preprocessing module to obtain the preprocessed Y initial component, and the preprocessed Y initial component is input to the Y auxiliary UV module and the YUV signal enhancement module. If the preprocessing module is not deployed, the Y initial component is directly input to the Y auxiliary UV module and the YUV signal enhancement module. In subsequent embodiments, the Y initial component obtained by the Y auxiliary UV module and the YUV signal enhancement module can be the preprocessed Y initial component or the original Y initial component obtained by decoding, without limitation.

[0138] Since there is a certain correlation between the three YUV components, although the compression loss of the Y component at the encoding end is smaller than that of the UV component, the Y component can also be enhanced to a certain extent in order to better enhance the UV component. Therefore, the Y initial component can be preprocessed by the preprocessing module to enhance the features of the Y initial component.

[0139] In this embodiment, see Figure 5B As shown, involving the Y auxiliary UV module, after decoding to obtain the Y initial component, U initial component and V initial component, the U initial component and the V initial component are lost more, while the Y initial component retains relatively more image detail information, and the U initial component and the V initial component have a certain correlation with the Y initial component. Therefore, making full use of the information of the Y initial component can help improve the reconstruction quality of the U initial component and the V initial component.

[0140] Based on this, the Y auxiliary UV module can use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component, and can also use the information of the Y initial component to assist in improving the reconstruction quality of the V initial component, and can also use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component and the reconstruction quality of the V initial component.

[0141] In this embodiment, see Figure 5BAs shown, the resolution conversion module is involved. Since the resolution of the U initial component is smaller than the resolution of the Y initial component, for example, the resolution of the U initial component is the resolution after 2 times downsampling, the resolution of the U initial component is half of the original signal resolution, such as H / 2*W / 2. Therefore, the resolution conversion module upsamples the resolution of the U initial component to restore the U component with the original signal resolution, that is, upsamples the resolution of the U initial component to obtain a U component with a resolution of H*W. Since the resolution of the V initial component is smaller than the resolution of the Y initial component, for example, the resolution of the V initial component is the resolution after 2 times downsampling, the resolution of the V initial component is half of the original signal resolution, such as H / 2*W / 2. Therefore, the resolution conversion module upsamples the resolution of the V initial component to restore the V component with the original signal resolution, that is, upsamples the resolution of the V initial component to obtain a V component with a resolution of H*W.

[0142] The position of the resolution conversion module can be position 1, that is, the resolution conversion module upsamples the resolution of the U initial component to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the Y auxiliary UV module. The resolution conversion module upsamples the resolution of the V initial component to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the Y auxiliary UV module. Alternatively, the position of the resolution conversion module can be position 2, that is, the resolution conversion module upsamples the resolution of the U component output by the Y auxiliary UV module to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the YUV signal enhancement module. The resolution conversion module upsamples the resolution of the V component output by the Y auxiliary UV module to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the YUV signal enhancement module. Alternatively, the position of the resolution conversion module can be position 3, that is, the resolution conversion module upsamples the resolution of the U component output by the YUV signal enhancement module to obtain a U component with a resolution of H*W, and finally outputs this H*W U component. The resolution conversion module upsamples the resolution of the V component output by the YUV signal enhancement module to obtain a V component with a resolution of H*W, and finally outputs this H*W V component.

[0143] Regarding the resolution transformation module, on the one hand, it can appropriately reduce the resolution of the feature map, thereby effectively reducing the computational complexity of the network. On the other hand, in the JPEG-AI framework, the resolution transformation module is used to upsample the resolution of the reconstructed UV component, thereby restoring a reconstructed U / V signal with the same resolution as the original signal.

[0144] In this embodiment, see Figure 5BAs shown, it involves a YUV signal enhancement module. In the image coding and compression process, in addition to reducing the resolution of the UV signal, the compression algorithm and the convolution process will cause additional information loss to the YUV signal. Therefore, the Y component can be enhanced by the YUV signal enhancement module so that the reconstructed Y component is close to the original signal before compression to improve the subjective quality. The U component can be enhanced by the YUV signal enhancement module so that the reconstructed U component is close to the original signal before compression to improve the subjective quality. The V component can be enhanced by the YUV signal enhancement module so that the reconstructed V component is close to the original signal before compression to improve the subjective quality.

[0145] For example, see Figure 5B As shown in FIG, the three network branches (the three network branches process the Y component, the U component, and the V component respectively) can be merged into one network branch, that is, a single network is used to process the Y component, the U component, and the V component at the same time, and output the reconstructed Y component, U component, and V component at the same time, thereby reducing the network calculation complexity.

[0146] Example 5: In Example 4, the Y auxiliary UV module can use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component and / or the V initial component. For example, based on the Y initial component and the U initial component, an adjusted component corresponding to the U initial component is generated, and / or, based on the Y initial component and the V initial component, an adjusted component corresponding to the V initial component is generated. Among them, the reconstruction quality of the adjusted component corresponding to the U initial component can be better than the reconstruction quality of the U initial component, thereby improving the reconstruction quality of the U initial component. The reconstruction quality of the adjusted component corresponding to the U initial component may also be equal to or lower than the reconstruction quality of the U initial component, and there is no limitation on this. The reconstruction quality of the adjusted component corresponding to the V initial component can be better than the reconstruction quality of the V initial component, thereby improving the reconstruction quality of the V initial component. The reconstruction quality of the adjusted component corresponding to the V initial component may also be equal to or lower than the reconstruction quality of the V initial component, and there is no limitation on this.

[0147] In one possible implementation, the Y auxiliary UV module may obtain a first image feature corresponding to the Y initial component. The first image feature may be an image feature that has not been processed through a neural network. For example, the first image feature may include, but is not limited to, at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module may obtain a second image feature corresponding to the U initial component. The second image feature may be an image feature that has not been processed through a neural network. For example, the second image feature may include, but is not limited to, at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module may generate an adjusted component based on the first and second image features. For example, a texture fusion feature may be generated based on the texture feature corresponding to the Y initial component and the texture feature corresponding to the U initial component. The texture fusion feature may be used to replace the texture feature corresponding to the U initial component to obtain an adjusted component corresponding to the U initial component. For another example, a subjective fusion feature may be generated based on the subjective feature corresponding to the Y initial component and the subjective feature corresponding to the U initial component. The subjective fusion feature may be used to replace the subjective feature corresponding to the U initial component to obtain an adjusted component corresponding to the U initial component. For another example, frequency domain fusion features can be generated based on the frequency domain features corresponding to the Y initial component and the frequency domain features corresponding to the U initial component, and the frequency domain fusion features are used to replace the frequency domain features corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. For another example, histogram fusion features can be generated based on the histogram features corresponding to the Y initial component and the histogram features corresponding to the U initial component, and the histogram fusion features are used to replace the histogram features corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. For another example, texture fusion features can be generated based on the texture features corresponding to the Y initial component and the texture features corresponding to the U initial component, frequency domain fusion features can be generated based on the frequency domain features corresponding to the Y initial component and the frequency domain features corresponding to the U initial component, texture fusion features can be used to replace the texture features corresponding to the U initial component, and frequency domain fusion features can be used to replace the frequency domain features corresponding to the U initial component to obtain the adjusted component corresponding to the U initial component. The above are just a few examples and there is no limitation to this.

[0148] The Y auxiliary UV module can obtain a first image feature corresponding to the Y initial component. The first image feature can be an image feature that has not been processed by a neural network, such as including but not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module can obtain a second image feature corresponding to the V initial component. The second image feature can be an image feature that has not been processed by a neural network, such as including but not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module can generate an adjusted component corresponding to the V initial component based on the first and second image features. The generation method is described in the processing of the U initial component.

[0149] In another possible implementation, the Y auxiliary UV module can obtain a first image feature corresponding to the Y initial component. The first image feature can be an image feature that has been processed by a neural network, that is, an image feature output by the neural network. For example, the first image feature can include but is not limited to a first feature map, a weight coefficient map, etc. The Y auxiliary UV module can obtain a second image feature corresponding to the U initial component. The second image feature can be an image feature that has been processed by a neural network, that is, an image feature output by the neural network. For example, the second image feature can include but is not limited to a second feature map, etc. The Y auxiliary UV module can generate an adjusted component corresponding to the U initial component based on the first image feature and the second image feature. For example, the adjusted component corresponding to the U initial component can be generated based on the first feature map corresponding to the Y initial component and the second feature map corresponding to the U initial component. For another example, the adjusted component corresponding to the U initial component can be generated based on the weight coefficient map corresponding to the Y initial component and the second feature map corresponding to the U initial component. The above are just a few examples and are not limiting.

[0150] The Y auxiliary UV module can obtain a first image feature corresponding to the Y initial component. The first image feature can be an image feature processed by a neural network, that is, an image feature output by the neural network. For example, the first image feature can include but is not limited to a first feature map, a weight coefficient map, etc. The Y auxiliary UV module can obtain a second image feature corresponding to the V initial component. The second image feature can be an image feature processed by a neural network, that is, an image feature output by the neural network. For example, the second image feature can include but is not limited to a second feature map, etc. The Y auxiliary UV module can generate an adjusted component corresponding to the V initial component based on the first image feature and the second image feature. The generation method is described in the processing of the U initial component.

[0151] Example 6: In order to enable the UV component to fully utilize the information of the Y component, at the input stage of the post-processing network, the Y component is convolved to obtain feature maps with multiple channels. These feature maps contain different types of features of the Y component. These different types of features are added (Add) or concatenated (Concat) with the feature map obtained by convolving the UV component, so that the subsequent neural network can learn multiple features of the Y component, thereby helping to improve the reconstruction quality of the UV component.

[0152] For example, see Figure 6AAs shown in FIG, it is a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component, and the V initial component, the Y auxiliary UV module can input the Y initial component into the first neural network (such as CNN1) to obtain a first feature map, which can also be called a Y feature map (Y Feature Map). The U initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component, which can also be called a U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component, which can also be called a V feature map (V Feature Map).

[0153] After obtaining the first feature map and the second feature map corresponding to the U initial component, the adjusted component corresponding to the U initial component can be generated based on the first feature map and the second feature map. For example, the first feature map and the second feature map are added together. Figure 6A in The addition operation of the feature map is input to the third neural network (such as CNN3) to obtain the adjusted component corresponding to the initial component U. Alternatively, the first feature map and the second feature map are overlapped. Figure 6A in Represents a splicing operation of the feature map, and inputs the feature map after the splicing operation into the third neural network (such as CNN3) to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the adjusted component corresponding to the V initial component can be generated based on the first feature map and the second feature map. For example, the first feature map and the second feature map are added, and the feature map after the addition operation is input into the third neural network to obtain the adjusted component corresponding to the V initial component. Alternatively, the first feature map and the second feature map are spliced, and the feature map after the splicing operation is input into the third neural network to obtain the adjusted component corresponding to the V initial component.

[0154] Performing a splicing operation on the first feature map and the second feature map refers to performing a splicing operation on the first feature map and the second feature map along a channel dimension.

[0155] Exemplarily, in order to correctly implement the above operations, the number of convolution kernels of the second neural network can be the same as the number of convolution kernels of the first neural network, such as CNN1 and CNN2 have the same number of convolution kernels. Figure 6A The processing shown can also be expressed as: f CNN represents a convolutional neural network, Indicates the addition or stacking operation of the feature map. Y ​​represents the initial component of Y, fCNN1 (Y) represents the first feature map, U represents the initial component of U, f CNN2 (U) represents the second feature map, V represents the initial component of V, f CNN2 (V) represents the second feature map, U rec Indicates the adjusted component corresponding to the initial component U, V rec Indicates the adjusted component corresponding to the initial component of V.

[0156] Example 7: To utilize the information in each feature map of the Y component, guided filtering can be introduced. This involves using the features generated by the Y component to guide the UV component to produce a more powerful feature map. Consider using attention to weight the pixel values ​​in the UV feature map using the feature map of the Y component as a guiding variable. This strengthens important features in the UV component and suppresses general features in the UV component, thereby achieving efficient conversion and fusion of features between the Y and UV components.

[0157] For example, see Figure 6B As shown in FIG, it is a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component, and the V initial component, the Y auxiliary UV module can input the Y initial component into the first neural network (such as CNN1) to obtain a first feature map, which can also be called a Y feature map (Y Feature Map). The U initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component, which can also be called a U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component, which can also be called a V feature map (V Feature Map).

[0158] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map can also be guided filtered to obtain a weight coefficient map, and based on the weight coefficient map and the second feature map, an adjusted component corresponding to the U initial component is generated. For example, an operation is performed based on the weight coefficient map and the second feature map, such as at least one of a multiplication operation, an addition operation, or a splicing operation, to obtain an output feature map (Output Feature Map), such as a multiplication operation is performed based on the weight coefficient map and the second feature map to obtain an output feature map, or a multiplication operation and an addition operation can be performed based on the weight coefficient map and the second feature map to obtain an output feature map, or a multiplication operation, an addition operation and a splicing operation can be performed based on the weight coefficient map and the second feature map to obtain an output feature map. After obtaining the output feature map, the output feature map can be input into a third neural network (such as CNN3) to obtain an adjusted component corresponding to the U initial component.

[0159] After obtaining the first feature map and the second feature map corresponding to the initial component of V, guided filtering can be performed on the first feature map to obtain a weight coefficient map, and an adjusted component corresponding to the initial component of V can be generated based on the weight coefficient map and the second feature map. For example, an operation such as at least one of a multiplication operation, an addition operation, or a stacking operation can be performed based on the weight coefficient map and the second feature map to obtain an output feature map (Output Feature Map). After obtaining the output feature map, the output feature map can be input into a third neural network (such as CNN3) to obtain an adjusted component corresponding to the initial component of V.

[0160] For example, Figure 6B The processing process shown can also be expressed as: rec / V rec =f CNN3 (f G (f CNN1 (Y), f CNN2 (U / V))), f CNN represents the convolutional neural network, f G represents guided filtering, which takes the feature maps of the Y component and UV component as input and outputs a fused feature map after guided weighting. Y represents the initial Y component, f CNN1 (Y) represents the first feature map, U represents the initial component of U, f CNN2 (U) represents the second feature map, V represents the initial component of V, f CNN2 (V) represents the second feature map, U rec Indicates the adjusted component corresponding to the initial component U, V rec Indicates the adjusted component corresponding to the initial component of V.

[0161] In one possible implementation, the implementation of guided filtering can be found in Figure 6C As shown, the first feature map (Y Feature Map) can be convolved to obtain a convolved feature map (hereinafter referred to as the convolved feature map A). ​​For example, the first feature map is input into the CNN network, and the CNN network performs a convolution operation on the first feature map to obtain the convolved feature map A. Then, the convolved feature map A is weight mapped to obtain a weight coefficient map after weight mapping.

[0162] When generating the adjusted component corresponding to the U initial component based on the weight coefficient map and the second feature map corresponding to the U initial component, the weight coefficient map and the second feature map (U Feature Map) corresponding to the U initial component can be multiplied to obtain the multiplied feature map. Then, the multiplied feature map and the convolution feature map A are added to obtain the added feature map. Then, the added feature map and the convolution feature map A are superimposed to obtain the output feature map (Output Feature Map), and the output feature map is input into the third neural network (such as CNN3) to obtain the adjusted component corresponding to the U initial component.

[0163] When generating the adjusted component corresponding to the V initial component based on the weight coefficient map and the second feature map corresponding to the V initial component, the weight coefficient map and the second feature map (V Feature Map) corresponding to the V initial component can be multiplied to obtain the multiplied feature map. Then, the multiplied feature map and the convolution feature map A are added to obtain the added feature map. Then, the added feature map and the convolution feature map A are superimposed to obtain the output feature map (Output Feature Map), and the output feature map is input into the third neural network (such as CNN3) to obtain the adjusted component corresponding to the V initial component.

[0164] See also Figure 6C As shown, after the first feature map of the Y component passes through several convolution layers, the convolution feature map A is obtained. The convolution feature map A is mapped to a specific weight factor, that is, a weight coefficient map, through weight mapping. The weight coefficient map is multiplied with the second feature map of the UV component to obtain the feature guide map of the UV component. Then, the convolution feature map A is added to the feature guide map of the UV component, and finally superimposed with the convolution feature map A to output the fusion feature map (Output Feature Map). For example, the guided filtering process is expressed as: Out = Concat((U / V)*f w (f CNN (Y))+f CNN (Y),f CNN (Y)), f w represents weight mapping, Concat(·) represents the stacking operation, f CNN (Y) represents the feature map A after convolution, and Out is the output feature map.

[0165] In a possible implementation, during the weight mapping process, weight mapping is performed on the convolution feature map A to obtain a weight coefficient map after weight mapping. The implementation of weight mapping can be found in Figure 6D As shown, Figure 6DTwo weight mapping methods are shown in Figure 1. In the first weight mapping method, a pooling operation is performed on the convolution feature map A to obtain a pooled feature map. For example, a pooling operation (Pooling) is performed on the convolution feature map A of size C*H*W to obtain a pooled feature map of size C*1*1. That is, the resolution of the feature map is reduced to 1*1 through "pooling".

[0166] Then, a fully connected operation and a ReLU activation operation are performed on the feature map after the pooling operation to obtain a feature map after the ReLU activation operation. For example, a fully connected operation (i.e., FC operation) and a ReLU activation operation (i.e., ReLU operation) are performed on the feature map after the pooling operation of size C*1*1 to obtain a feature map after the ReLU activation operation of size C*1*1. A fully connected operation and a Sigmoid activation operation are performed on the feature map after the ReLU activation operation to obtain a feature map after the Sigmoid activation operation. For example, a fully connected operation (i.e., FC operation) and a Sigmoid activation operation (i.e., Sigmoid operation) are performed on the feature map after the ReLU activation operation of size C*1*1 to obtain a feature map after the Sigmoid activation operation of size C*1*1.

[0167] Then, a weight coefficient map is generated based on the post-convolution feature map A and the post-Sigmoid activation feature map. For example, the post-convolution feature map A and the post-Sigmoid activation feature map are multiplied to obtain a weight coefficient map. For example, the post-convolution feature map A of size C*H*W and the post-Sigmoid activation feature map of size C*1*1 are multiplied to obtain a weight coefficient map of size C*H*W. Obviously, after the resolution of the feature map is reduced to 1*1 through "Pooling", it passes through several fully connected layers and finally uses the Sigmoid activation function to obtain the weight factor of each channel in the original input feature map. These weight factors will give different weights to the channels of the original input feature map, and finally obtain the weight coefficient map (also called weight feature map).

[0168] In the second weight mapping method, a convolution operation can be performed on the convolution feature map A to obtain the convolution feature map B. For example, the convolution feature map A of size C*H*W is input into the CNN network, which then convolves the convolution feature map A through several convolution layers to obtain the convolution feature map B of size C*H*W. Then, a Sigmoid activation operation is performed on the convolution feature map B to obtain a weight coefficient map. For example, the Sigmoid activation function is used to activate the convolution feature map B of size C*H*W, resulting in a weight coefficient map of size C*H*W. Obviously, by passing the original input feature map through several convolution layers and then the Sigmoid activation function, the weight coefficient map is finally obtained.

[0169] The difference between the first weight mapping method and the second weight mapping method is that in the first weight mapping method, different weights are assigned to channels, and in the second weight mapping method, different weights are assigned to each pixel of each channel feature map.

[0170] Example 8: In both Example 6 and Example 7, the feature extraction and information fusion of the Y component are performed at the input stage of the post-processing network. However, in Example 8, the feature extraction and information fusion of the Y component are distributed at each stage of the post-processing network, and the features of the UV component at different stages are fused with the features of the Y component, thereby improving the enhancement capability of the post-processing network for the UV component. For example, see Figure 6E As shown in the figure, it is a structural diagram of the post-processing network. After obtaining the Y initial component, U initial component and V initial component, the Y auxiliary UV module distributes the feature extraction and information fusion of the Y component in each stage of the post-processing network. B The feature fusion process can adopt the implementation of Example 6 or Example 7, by fusing the information of the Y component and the UV component to output a fused feature map, which then passes through several convolutional layers and serves as the input of the next feature fusion process. Correspondingly, the Y component will also pass through several convolutional layers and serve as the input of the next feature fusion process. In Example 8, multiple feature fusion processes are involved, thereby fusing the feature map of the Y component with the feature maps of the UV component at each stage of the network.

[0171] In a possible embodiment, since the feature map of the Y component is fused with the feature map of the UV component at each stage of the network, multiple first feature maps and multiple second feature maps are involved. Taking K first feature maps and K+1 second feature maps as an example, K can be a positive integer greater than 1. Based on this, after obtaining the Y initial component and the U initial component, for the first first feature map and the first second feature map, the Y initial component is input into the neural network to obtain the first first feature map, and the U initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, the value range of i is 2 to K, the i-1th first feature map is input into the neural network to obtain the i-th first feature map, the i-1th first feature map and the i-1th second feature map are feature fused to obtain the fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map (i.e., the K+1th second feature map), the adjusted component corresponding to the U initial component can be generated based on the last second feature map.

[0172] For example, assuming K is 3, the initial component Y is input to neural network a1 to obtain the first feature map b1. That is, neural network a1 performs several convolution operations on the initial component Y to obtain the first feature map b1. The initial component U is input to neural network c1 to obtain the second feature map d1. That is, neural network c1 performs several convolution operations on the initial component U to obtain the second feature map d1.

[0173] The first feature map b1 is input to neural network a2 to obtain the first feature map b2. That is, neural network a2 performs several convolution operations on the first feature map b1 to obtain the first feature map b2. The first feature map b1 and the second feature map d1 are fused to obtain the fused features, and the fused features are input to neural network c2 to obtain the second feature map d2.

[0174] The first feature map b2 is input to neural network a3 to obtain the first feature map b3. That is, neural network a3 performs several convolution operations on the first feature map b2 to obtain the first feature map b3. The first feature map b2 and the second feature map d2 are fused to obtain the fused features, and the fused features are input to neural network c3 to obtain the second feature map d3.

[0175] Then, the first feature map b3 and the second feature map d3 are fused to obtain a fused feature, and the fused feature is input into the neural network c4 to obtain the second feature map d4. After obtaining the second feature map d4, an adjusted component corresponding to the initial component U can be generated based on the second feature map d4. For example, the second feature map d4 can be used as the adjusted component corresponding to the initial component U, or the second feature map d4 can be operated to obtain the adjusted component corresponding to the initial component U.

[0176] In the above process, the first feature map and the second feature map are subjected to feature fusion to obtain the fused feature, which may include but is not limited to: adding the first feature map and the second feature map to obtain the fused feature; or, superimposing the first feature map and the second feature map to obtain the fused feature; or, performing guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating the fused feature based on the weight coefficient map and the second feature map. Among them, regarding the implementation method of feature fusion, please refer to Example 6 and Example 7, which will not be repeated here.

[0177] After obtaining the Y initial component and the V initial component, for the first first feature map and the first second feature map, the Y initial component is input into the neural network to obtain the first first feature map, and the V initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, the value range of i is 2-K, the i-1th first feature map is input into the neural network to obtain the i-th first feature map, the i-1th first feature map and the i-1th second feature map are feature fused to obtain the fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map, the adjusted component corresponding to the V initial component can be generated based on the last second feature map. The method for obtaining the adjusted component corresponding to the V initial component can refer to the processing process of the U initial component.

[0178] Example 9: In Example 6, Example 7 and Example 8, the fusion method of the Y component guide map is designed. Different from Example 6, Example 7 and Example 8, in Example 9, the Y component is subjected to detailed feature extraction from multiple dimensions and then fused into the post-processing network, thereby improving the network's ability to enhance the UV component.

[0179] See also Figure 6F The figure shows a schematic diagram of the structure of the post-processing network. After obtaining the Y initial component, the U initial component, and the V initial component, the Y auxiliary UV module first performs multi-dimensional feature extraction on the Y initial component. For example, M-dimensional image features are extracted from the Y initial component, where M is a positive integer. Then, the M-dimensional image features are stacked to obtain the stacked multi-dimensional features. After obtaining the multi-dimensional features, the multi-dimensional features can be input into the first neural network to obtain a first feature map (Y Feature Map). In addition, the Y auxiliary UV module can also input the U initial component into the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component. The second feature map can also be called a U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component. The second feature map can also be called a V feature map (V Feature Map).

[0180] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the V initial component. For the feature fusion process, please refer to Example 6, Example 7 and Example 8.

[0181] For example, Figure 6F The processing process shown can also be expressed as: rec / V rec =f CNN2 (f G (f A (Y),f CNN1 (U / V))), f G represents the feature fusion process, f A Represents the multi-dimensional feature extraction process of the Y component. In the multi-dimensional feature extraction process, the initial component Y is taken as input and output as multiple feature maps of multiple dimensions, where each branch represents the features of one dimension.

[0182] For example, see Figure 6G As shown, it is a schematic diagram of the multi-dimensional feature extraction process, with the Y initial component as input, and each branch feature extraction structure is used to extract the features of the Y initial component. For example, the Y initial component is input to the branch 1 feature extraction structure, and the branch 1 feature extraction structure extracts the features of the first dimension of the Y initial component. The Y initial component is input to the branch 2 feature extraction structure, and the branch 2 feature extraction structure extracts the features of the second dimension of the Y initial component. And so on, the Y initial component is input to the branch M feature extraction structure, and the branch M feature extraction structure extracts the features of the Mth dimension of the Y initial component. In summary, image features of M dimensions can be extracted from the Y initial component, and the image features of M dimensions can be stacked to obtain the stacked multi-dimensional features. After obtaining the stacked multi-dimensional features, the multi-dimensional features can be used as the first feature map, or the multi-dimensional features can be input to the first neural network to obtain the first feature map.

[0183] exist Figure 6G In the figure, a schematic diagram of a multi-dimensional feature extraction process is shown. First, the Y component is passed through different feature extraction channels to obtain feature information on each dimension, and finally these feature information are concat to the output as the Y component feature map. The multi-dimensional feature extraction process can be expressed as: Out = Concat (f ch-1 (Y),f ch-2 (Y),…,f ch-M (Y)), where f ch- It represents the i-th feature extraction structure, which is used to extract the features of the i-th dimension. Concat(·) represents the stacking operation.

[0184] In one possible implementation, the M-dimensional image features may include, but are not limited to, at least one of the following: image features resulting from convolution of the initial component Y with a single convolution kernel; image features resulting from convolution of the initial component Y with two convolution kernels; first-order spatial features resulting from processing the initial component Y with a Sobel operator; second-order spatial features resulting from processing the initial component Y with a Laplacian operator; or frequency domain features resulting from Fourier transform of the initial component Y. The above image features are merely examples, and this embodiment does not limit the type of image features.

[0185] See also Figure 6H Figure 2 shows a schematic diagram of a multi-dimensional feature extraction structure. The feature extraction structure of branch 1 (i.e., the first branch) uses a 3*3 convolution to obtain feature information. For example, a 3*3 convolution kernel is used to convolve the initial component Y to obtain the convolved image features. The feature extraction structure of branch 2 (i.e., the second branch) uses a 1*1 and 3*3 convolution cascade to obtain feature information after channel expansion and compression. For example, a 1*1 and 3*3 convolution kernel is used to convolve the initial component Y to obtain the convolved image features. The feature extraction structure of branch 3 (i.e., the third branch) uses the Sobel operator to obtain first-order spatial features. For example, the Sobel operator is used to process the initial component Y to obtain first-order spatial features. The feature extraction structure of branch 4 (i.e., the fourth branch) uses the Laplacian operator to obtain second-order spatial features. For example, the Laplacian operator is used to process the initial component Y to obtain second-order spatial features. The feature extraction structure of branch 5 (i.e., the fifth branch) uses Fourier transform to obtain frequency domain features. For example, the initial component Y is Fourier transformed to obtain frequency domain features. After obtaining the features of the above five dimensions, these features can be stacked to obtain multi-dimensional features.

[0186] Example 10: In Example 6 to Example 9, the main focus is on which Y component features to select and how to fuse the Y component features with the UV component features. Unlike Example 6 to Example 9, in Example 10, the emphasis is on how to enhance the features of the Y component. For example, after the YUV components of the image are encoded and compressed, the image details of the Y component are also lost to a certain extent. Therefore, the Y component and the features of the Y component can also be enhanced to a certain extent. For example, since there is a certain correlation between the three YUV components, although the compression loss of the Y component at the encoding end is smaller than that of the UV component, the Y component can also be enhanced to a certain extent in order to better enhance the UV component. Therefore, the Y initial component can be preprocessed by the preprocessing module to enhance the features of the Y initial component.

[0187] For example, after obtaining the Y initial component, the preprocessing module can first perform edge enhancement on the Y initial component to obtain edge-enhanced image features; then, perform multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features. After obtaining the multi-scale features, the multi-scale features can be used as the preprocessed Y initial component, and the preprocessed Y initial component can be input into the Y auxiliary UV module, which then inputs the preprocessed Y initial component into the first neural network to obtain a first feature map (Y Feature Map). In addition, the Y auxiliary UV module can also input the U initial component into the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component, which can also be called a U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component, which can also be called a V feature map (V Feature Map).

[0188] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be feature fused to obtain the adjusted component corresponding to the V initial component. For the feature fusion process, please refer to Example 6, Example 7 and Example 8.

[0189] When the preprocessing module performs multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, it can perform a convolution operation on the edge-enhanced image features to obtain convolution features; perform a downsampling operation on the convolution features to obtain downsampling features; perform channel transformation on the downsampling features to obtain channel-transformed features; perform an upsampling operation on the channel-transformed features to obtain upsampling features; and generate multi-scale features based on the upsampling features and the channel-transformed features.

[0190] See also Figure 6I Figure 2 shows the structure of the post-processing network, illustrating a multi-scale enhancement method for the Y component. In the downsampling phase, from D1 to D3, the resolution of the feature map decreases while the number of channels increases. In the upsampling phase, from U3 to U1, the resolution of the feature map increases while the number of channels remains unchanged. After channel transformation, D3, D2, and D1 are added to the upsampled feature maps of U3, U2, and U1, respectively, to obtain the multi-scale features after multi-scale enhancement.

[0191] First, edge enhancement is performed on the Y initial component to obtain edge-enhanced image features. For example, image sharpening enhancement operation is performed on the Y initial component to obtain edge-enhanced image features. There is no restriction on the edge enhancement method.

[0192] Taking two downsampling operations and two upsampling operations as an example, the number of downsampling operations can be greater, and the number of upsampling operations can be greater, and there is no limit to this. After obtaining the edge-enhanced image features, the edge-enhanced image features are output to the CNN to obtain feature D1 of scale C*H*W. Feature D1 is downsampled to obtain feature D2 of scale 4C*H / 2*W / 2. Feature D2 is downsampled to obtain feature D3 of scale 16C*H / 4*W / 4.

[0193] Feature D3 is channel-transformed to obtain feature U3 of scale T*H / 4*W / 4. Feature U3 is upsampled, feature D2 is channel-transformed, and the upsampled feature U3 and the channel-transformed feature D2 are added together to obtain feature U2 of scale T*H / 2*W / 2. Feature U2 is upsampled, feature D1 is channel-transformed, and the upsampled feature U2 and the channel-transformed feature D1 are added together to obtain feature U1 of scale T*H*W. Clearly, feature U1 is a multi-scale feature after multi-scale enhancement, namely, the preprocessed initial Y component. The preprocessing module outputs this preprocessed initial Y component. By performing multi-scale feature extraction on the Y component, the final output, the first feature map, integrates feature information at different scales, making it helpful in enhancing the UV component.

[0194] Example 11: In Example 4, a resolution conversion module is involved. The resolution conversion module is used to upsample the resolution of the U component to restore the U component with the original signal resolution, such as obtaining a U component with a resolution of H*W. The resolution conversion module is used to upsample the resolution of the initial V component to restore the V component with the original signal resolution, such as obtaining a V component with a resolution of H*W. For example, in the JPEG-AI framework, the original UV component is downsampled by 2 times and then compressed and encoded at the encoder. Therefore, the resolution of the UV component reconstructed by the decoder is half the resolution of the original signal. In the post-processing network, there is a step of upsampling the UV component to restore the reconstructed UV component with the same resolution as the original signal. Exemplarily, when the resolution conversion module upsamples the resolution of the U component and / or the V component, the upsampling method may include but is not limited to: interpolation sampling, pixel-shuffle upsampling, deconvolution DeConv, etc. The above are just a few examples, and this embodiment does not limit the upsampling method.

[0195] For example, since upsampling the resolution of the UV component will affect the computational complexity of the network model, that is, the increase in resolution leads to an increase in computational complexity, the position of the resolution transformation module will affect the computational complexity of the network model. Therefore, the position of the resolution transformation module can be involved to reduce the computational complexity of the network model.

[0196] Case 1: See Figure 5B As shown, the position of the resolution conversion module can be position 1, that is, the resolution conversion module upsamples the resolution of the U initial component to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the Y auxiliary UV module. The resolution conversion module upsamples the resolution of the V initial component to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the Y auxiliary UV module. On this basis, for Examples 5 to 10, the U initial components are all upsampled U components, and the V initial components are all upsampled V components. The resolution of the upsampled U components can be equal to the resolution of the Y initial components, and the resolution of the upsampled V components can be equal to the resolution of the Y initial components. In this way, Examples 5 to 10 are performed based on the Y initial components and the upsampled U components, and Examples 5 to 10 are performed based on the Y initial components and the upsampled V components.

[0197] In summary, we can see that in case 1, upsampling can be performed at the input of the network, see Figure 7A As shown, it is a schematic diagram of upsampling at the input end of the network, that is, at the input end of the network, the U initial component can be upsampled to obtain the upsampled U component, such as the U component with a resolution of H*W. At the input end of the network, the V initial component can be upsampled to obtain the upsampled V component, such as the V component with a resolution of H*W.

[0198] Case 2: See Figure 5BAs shown, the position of the resolution conversion module can be position 2, that is, the resolution conversion module upsamples the resolution of the U component output by the Y auxiliary UV module to obtain a U component with a resolution of H*W, and inputs the U component with a resolution of H*W to the YUV signal enhancement module. The resolution conversion module upsamples the resolution of the V component output by the Y auxiliary UV module to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the YUV signal enhancement module. On this basis, for Examples 5 to 10, the U initial component is the U component before upsampling, and the V initial component is the V component before upsampling. The resolution of the U component before upsampling can be smaller than the resolution of the Y initial component, and the resolution of the V component before upsampling can be smaller than the resolution of the Y initial component. In this way, Examples 5 to 10 are performed based on the Y initial component and the U component before upsampling, and Examples 5 to 10 are performed based on the Y initial component and the V component before upsampling. After obtaining the adjusted components corresponding to the U component / V component based on Examples 5-10, the adjusted components can also be upsampled to obtain the upsampled adjusted components, and the upsampled adjusted components are input to the YUV signal enhancement module. The resolution of the upsampled adjusted components is equal to the resolution of the Y initial components.

[0199] In summary, we can see that in case 2, we can gradually upsample the network layer by layer, see Figure 7B As shown in FIG, it is a schematic diagram of the progressive upsampling of the network layer by layer, that is, at multiple network layers, the U initial component can be upsampled to obtain the upsampled U component, and the resolution of the upsampled U component can be equal to the resolution of the Y initial component, such as the U component with a resolution of H*W. At multiple network layers, the V initial component can be upsampled to obtain the upsampled V component, and the resolution of the upsampled V component can be equal to the resolution of the Y initial component, such as the V component with a resolution of H*W.

[0200] Case 3: See Figure 5BAs shown, the position of the resolution conversion module can be position 3, that is, the resolution conversion module upsamples the resolution of the U component output by the YUV signal enhancement module to obtain a U component with a resolution of H*W, and finally outputs this H*W U component. The resolution conversion module upsamples the resolution of the V component output by the YUV signal enhancement module to obtain a V component with a resolution of H*W, and finally outputs this H*W V component. On this basis, for Examples 5 to 10, the U initial component is the U component before upsampling, and the V initial component is the V component before upsampling. The resolution of the U component before upsampling is smaller than the resolution of the Y initial component, and the resolution of the V component before upsampling is smaller than the resolution of the Y initial component. In this way, Examples 5 to 10 are performed based on the Y initial component and the U component before upsampling, and Examples 5 to 10 are performed based on the Y initial component and the V component before upsampling. After obtaining the adjusted components corresponding to the U component / V component based on Examples 5-10, the adjusted components are input to the YUV signal enhancement module, and the adjusted components are the U component / V component before upsampling, and the resolution of the U component / V component before upsampling is less than the resolution of the initial Y component. In this way, the YUV signal enhancement module performs signal enhancement based on the U component / V component before upsampling to obtain the signal-enhanced U component / V component. After obtaining the signal-enhanced U component / V component, the U component / V component is upsampled to obtain the upsampled U component / V component, and the resolution of the upsampled U component / V component is equal to the resolution of the initial Y component.

[0201] In summary, we can see that in case 3, upsampling can be performed at the output of the network, see Figure 7C As shown in FIG, it is a schematic diagram of upsampling at the output end of the network, that is, at the output end of the network, the U component can be upsampled to obtain the upsampled U component, and the resolution of the upsampled U component can be equal to the resolution of the initial Y component, such as the U component with a resolution of H*W. At the output end of the network, the V component can be upsampled to obtain the upsampled V component, and the resolution of the upsampled V component can be equal to the resolution of the initial Y component, such as the V component with a resolution of H*W.

[0202] Example 12: In Example 4, the resolution transformation module is involved. First, the resolution transformation module can appropriately reduce the resolution of the feature map by using a resolution transformation without information loss, thereby effectively reducing the computational complexity of the network. Second, the resolution transformation module can upsample the resolution of the reconstructed U and V components to restore U and V components with the same resolution as the original signal. The function of this second aspect is described in Example 11.

[0203] The function of the first aspect can be implemented by a resolution conversion module or by a Y auxiliary UV module. For example, by reducing the resolution of the YUV component or the intermediate component, the network complexity can be effectively reduced, and the resolution can be increased at the output of the network to restore the original resolution. For example, a wavelet transform (such as Haar wavelet transform, etc.) can be used to halve the resolution of the original image. In this way, the computational complexity of the network can be reduced by almost four times. When the hardware has high requirements for computational complexity, it is also possible to consider using the wavelet transform multiple times in succession so that the resolution can be further reduced. For several frequency bands after the wavelet transform, all of them can be used or some of them can be selected.

[0204] For example, see Figure 7D As shown in the figure, it is a structural diagram of the post-processing network, which can perform wavelet transform on the initial component of Y ( Figure 7D The DWT transform in the DWT transform is used to obtain multiple frequency bands after wavelet transform. The multiple frequency bands or part of the frequency bands in the multiple frequency bands (i.e., Y sub-bands) can be input into the first neural network to obtain the first feature map (Y Feature Map). The U initial component can be subjected to wavelet transform ( Figure 7D DWT transform in), multiple frequency bands after wavelet transform are obtained, and multiple frequency bands or part of the frequency bands in the multiple frequency bands (i.e., U sub-bands) can be input into the second neural network to obtain a second feature map (UFeature Map). The V initial component can be subjected to wavelet transform to obtain multiple frequency bands after wavelet transform, and multiple frequency bands or part of the frequency bands in the multiple frequency bands (i.e., V sub-bands) can be input into the second neural network to obtain a second feature map (V Feature Map). After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be feature fused to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be feature fused to obtain an adjusted component corresponding to the V initial component. For the feature fusion process, please refer to Example 6, Example 7, and Example 8.

[0205] It should be noted that when the first feature map and the second feature map are subjected to feature fusion to obtain the adjusted component corresponding to the U initial component, the fused features are first obtained by using Embodiment 6, Embodiment 7 and Embodiment 8. Then, the fused features are input to the neural network (CNN3) to obtain the output features of the neural network (U subband). Then, the output features can also be subjected to an inverse wavelet transform (i.e., the inverse operation of the wavelet transform). Figure 7DIDWT transform in ), to obtain the adjusted component of the U component. When performing feature fusion on the first feature map and the second feature map to obtain the adjusted component corresponding to the initial component of V, first use Example 6, Example 7 and Example 8 to obtain the fused feature, then input the fused feature into the neural network (CNN3) to obtain the output feature of the neural network (V subband), and then, the output feature can also be subjected to an inverse wavelet transform (i.e., the inverse operation of the wavelet transform, Figure 7D The IDWT transformation in is used to obtain the adjusted component of the V component.

[0206] Example 13: In Example 4, a YUV signal enhancement module is involved. The YUV signal enhancement module is used to enhance the signal of the Y component so that the reconstructed Y component is close to the original signal before compression. The YUV signal enhancement module is used to enhance the signal of the U component so that the reconstructed U component is close to the original signal before compression. The YUV signal enhancement module is used to enhance the signal of the V component so that the reconstructed V component is close to the original signal before compression. On this basis, the YUV signal enhancement module can perform feature enhancement on the initial Y component to obtain the target component corresponding to the Y component, and the target component is the Y component after signal recovery. The YUV signal enhancement module can perform feature enhancement on the adjusted component of the U component to obtain the target component corresponding to the U component, and the target component is the U component after signal recovery. The YUV signal enhancement module can perform feature enhancement on the adjusted component of the V component to obtain the target component corresponding to the V component, and the target component is the V component after signal recovery.

[0207] In one possible implementation, at least one residual block network can be used to perform feature enhancement on the initial component Y to obtain a target component corresponding to the Y component. At least one residual block network can be used to perform feature enhancement on the adjusted component U to obtain a target component corresponding to the U component. At least one residual block network can be used to perform feature enhancement on the adjusted component V to obtain a target component corresponding to the V component. Figure 7E As shown in FIG, a schematic diagram of enhancing the Y component, U component and V component by using a residual block cascade network, and there is no restriction on the structure of the residual block cascade network.

[0208] In one possible implementation, the feature enhancement of the initial component Y can be performed using a U-Net network to obtain the target component corresponding to the Y component. The feature enhancement of the adjusted component U can be performed using a U-Net network to obtain the target component corresponding to the U component. The feature enhancement of the adjusted component V can be performed using a U-Net network to obtain the target component corresponding to the V component. Figure 7FAs shown in FIG, a schematic diagram of using a U-Net network to enhance the Y component, U component, and V component. The U-Net network includes multiple downsampling network layers and multiple upsampling network layers. There is no restriction on the structure of the U-Net network.

[0209] Illustratively, the above-mentioned embodiments can be implemented individually or in combination. For example, each embodiment in Embodiment 1 to Embodiment 13 can be implemented individually, and at least two embodiments in Embodiment 1 to Embodiment 13 can be implemented in combination.

[0210] Illustratively, in the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0211] Based on the same application concept as the above method, a decoding device is also proposed in the embodiment of the present disclosure. The decoding device is applied to the decoding end. The decoding device may include: a memory, which is configured to store video data; a decoder, which is configured to implement the decoding methods in the above embodiments 1 to 13, that is, the processing flow of the decoding end.

[0212] For example, in one possible implementation, a decoder is configured to implement:

[0213] Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, where the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0214] generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0215] Feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0216] Exemplarily, if the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and / or a chrominance V component; or, if the reconstructed image block is a reconstructed image block in RGB format, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0217] Exemplarily, the decoder is configured to further implement: obtaining a first image feature corresponding to the first initial component, the first image feature including an image feature that has not passed through the neural network and / or an image feature that has passed through the neural network; obtaining a second image feature corresponding to the second initial component, the second image feature including an image feature that has not passed through the neural network and / or an image feature that has passed through the neural network; and generating the adjusted component based on the first image feature and the second image feature.

[0218] Exemplarily, the decoder is configured to further implement: if the first image feature includes a first feature map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the first initial component is input into the first neural network to obtain the first feature map; the second initial component is input into the second neural network to obtain the second feature map; the first feature map and the second feature map are added to obtain the adjusted component; or the first feature map and the second feature map are superimposed to obtain the adjusted component.

[0219] Exemplarily, the decoder is configured to further implement: if the first image feature includes a weight coefficient map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the first initial component is input into the first neural network to obtain a first feature map corresponding to the first initial component, and the first feature map is guided filtered to obtain the weight coefficient map; the second initial component is input into the second neural network to obtain the second feature map; and the adjusted component is generated based on the weight coefficient map and the second feature map.

[0220] Exemplarily, the decoder is configured to further implement: performing a convolution operation on the first feature map to obtain a convolved feature map; and performing weight mapping on the convolved feature map to obtain the weight coefficient map.

[0221] Exemplarily, the decoder is configured to further implement: performing a pooling operation on the feature map after the convolution operation; performing a full connection operation and a ReLU activation operation on the feature map after the pooling operation; performing a full connection operation and a Sigmoid activation operation on the feature map after the ReLU activation operation; and generating the weight coefficient map based on the feature map after the convolution operation and the feature map after the Sigmoid activation operation.

[0222] Exemplarily, the decoder is configured to further implement: performing a convolution operation and a Sigmoid activation operation on the convolution feature map to obtain the weight coefficient map.

[0223] Exemplarily, the decoder is configured to further implement: multiplying the weight coefficient map and the second feature map to obtain a multiplied feature map; performing a convolution operation on the first feature map to obtain a convolved feature map; adding the multiplied feature map and the convolved feature map to obtain an added feature map; and stacking the added feature map and the convolved feature map to obtain the adjusted component.

[0224] Exemplarily, the decoder is configured to further implement: if the first image feature includes K first feature maps passed through the neural network, and the second image feature includes K+1 second feature maps passed through the neural network, for the first first feature map and the first second feature map, the first initial component is input into the neural network to obtain the first first feature map, and the second initial component is input into the neural network to obtain the first second feature map; for the i-th first feature map and the i-th second feature map, the value range of i is 2-K, K is a positive integer greater than 1, the i-1-th first feature map is input into the neural network to obtain the i-th first feature map; the i-1-th first feature map and the i-1-th second feature map are feature fused to obtain a fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map; after obtaining the last second feature map, the adjusted component is generated based on the last second feature map.

[0225] Exemplarily, the decoder is configured to further implement: extracting M-dimensional image features from the first initial component, where M is a positive integer; stacking the M-dimensional image features to obtain stacked multi-dimensional features; and inputting the multi-dimensional features into the first neural network to obtain the first feature map.

[0226] Exemplarily, the decoder is configured to further implement: performing a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform; inputting the multiple frequency bands or part of the multiple frequency bands into the first neural network to obtain the first feature map; performing a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform; inputting the multiple frequency bands or part of the multiple frequency bands into the second neural network to obtain the second feature map.

[0227] Exemplarily, the decoder is configured to further implement: preprocessing the first initial component to obtain a preprocessed first initial component; wherein the preprocessed first initial component is used to input into the first neural network to obtain the first feature map; the decoder is configured to further implement: edge enhancement on the first initial component to obtain edge-enhanced image features; multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features; and determining the preprocessed first initial component based on the multi-scale features.

[0228] Exemplarily, the decoder is configured to further implement: performing a convolution operation on the edge-enhanced image features to obtain convolution features; performing a downsampling operation on the convolution features to obtain downsampling features; performing a channel transformation on the downsampling features to obtain channel-transformed features; performing an upsampling operation on the channel-transformed features to obtain upsampling features; and generating the multi-scale features based on the upsampling features and the channel-transformed features.

[0229] Exemplarily, the decoder is configured to further implement: before generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, upsampling the second initial component to obtain an upsampled second initial component; the resolution of the upsampled second initial component is equal to the resolution of the first initial component.

[0230] Exemplarily, the decoder is configured to further implement: before performing feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, upsampling the adjusted component to obtain the upsampled adjusted component; wherein the resolution of the upsampled adjusted component is equal to the resolution of the first initial component.

[0231] Exemplarily, the decoder is configured to further implement: after performing feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, upsampling the target component to obtain the upsampled target component; wherein the resolution of the upsampled target component is equal to the resolution of the first initial component.

[0232] Exemplarily, the decoder is configured to further implement: performing feature enhancement on the adjusted component through at least one residual block network to obtain a target component corresponding to the second initial component; or performing feature enhancement on the adjusted component through a U-Net network to obtain a target component corresponding to the second initial component.

[0233] Based on the same application concept as the above method, the decoding end device (also called video decoder) provided in the embodiment of the present disclosure has a hardware architecture diagram, which can be specifically referred to as Figure 8 The system comprises: a processor 811 and a machine-readable storage medium 812, wherein the machine-readable storage medium 812 stores machine-executable instructions that can be executed by the processor 811; the processor 811 is configured to execute the machine-executable instructions to implement the decoding methods of the above-mentioned embodiments 1-12 of the present disclosure.

[0234] For example, in one possible implementation, the processor 811 executes the machine-executable instructions to implement:

[0235] Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, where the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component;

[0236] generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0237] Feature enhancement is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0238] Based on the same application concept as the above-mentioned method, an embodiment of the present disclosure provides an electronic device comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the decoding methods of embodiments 1-12 of the present disclosure.

[0239] Based on the same application concept as the above method, an embodiment of the present disclosure also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present disclosure can be implemented, such as the decoding method in the above embodiments.

[0240] Based on the same application concept as the above method, an embodiment of the present disclosure further provides a computer application, which, when executed by a processor, can implement the decoding method disclosed in the above example of the present disclosure.

[0241] Based on the same application concept as the above method, the present disclosure further provides a decoding device, which can be applied to a decoding end (also referred to as a video decoder). The decoding device may include:

[0242] A decoding module is used to decode the code stream corresponding to the current image block to obtain a reconstructed image block, where the reconstructed image block includes a first initial component and a second initial component, and the resolution of the first initial component is greater than or equal to the resolution of the second initial component; a determination module is used to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and a processing module is used to perform feature enhancement on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0243] Exemplarily, if the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and / or a chrominance V component; or, if the reconstructed image block is a reconstructed image block in RGB format, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0244] Exemplarily, when the determination module generates the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, it is specifically used to: obtain a first image feature corresponding to the first initial component, the first image feature includes an image feature that has not passed through the neural network and / or an image feature that has passed through the neural network; obtain a second image feature corresponding to the second initial component, the second image feature includes an image feature that has not passed through the neural network and / or an image feature that has passed through the neural network; and generate the adjusted component based on the first image feature and the second image feature.

[0245] Exemplarily, if the first image feature includes image features that have not passed through a neural network, the first image feature includes at least one of the following: texture features, subjective features, frequency domain features, or histogram features; if the second image feature includes image features that have not passed through a neural network, the second image feature includes at least one of the following: texture features, subjective features, frequency domain features, or histogram features.

[0246] Exemplarily, if the first image feature includes a first feature map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the determination module generates an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and is specifically used to: input the first initial component to the first neural network to obtain the first feature map; input the second initial component to the second neural network to obtain the second feature map; perform an addition operation on the first feature map and the second feature map to obtain the adjusted component; or perform a splicing operation on the first feature map and the second feature map to obtain the adjusted component.

[0247] Exemplarily, if the first image feature includes a weight coefficient map passed through a neural network, and the second image feature includes a second feature map passed through a neural network, the determination module generates an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and is specifically used to: input the first initial component to the first neural network to obtain a first feature map corresponding to the first initial component, and perform guided filtering on the first feature map to obtain the weight coefficient map; input the second initial component to the second neural network to obtain the second feature map; and generate the adjusted component based on the weight coefficient map and the second feature map.

[0248] Exemplarily, the determination module performs guided filtering on the first feature map to obtain the weight coefficient map, which is specifically used to: perform a convolution operation on the first feature map to obtain a convolved feature map; and perform weight mapping on the convolved feature map to obtain the weight coefficient map.

[0249] Exemplarily, the determination module performs weight mapping on the post-convolution feature map, and when obtaining the weight coefficient map, it is specifically used to: perform a pooling operation on the post-convolution feature map; perform a full connection operation and a rectified linear unit ReLU activation operation on the feature map after the pooling operation; perform a full connection operation and an S-shaped curve Sigmoid activation operation on the feature map after the ReLU activation operation; and generate the weight coefficient map based on the post-convolution feature map and the feature map after the Sigmoid activation operation.

[0250] Exemplarily, the determination module performs weight mapping on the post-convolution feature map to obtain the weight coefficient map, which is specifically used to: perform a convolution operation and a Sigmoid activation operation on the post-convolution feature map to obtain the weight coefficient map.

[0251] Exemplarily, when the determination module generates the adjusted component based on the weight coefficient map and the second feature map, it is specifically used to: multiply the weight coefficient map and the second feature map to obtain a multiplied feature map; perform a convolution operation on the first feature map to obtain a convolved feature map; perform an addition operation on the multiplied feature map and the convolved feature map to obtain an added feature map; and perform a splicing operation on the added feature map and the convolved feature map to obtain the adjusted component.

[0252] Exemplarily, if the first image feature includes K first feature maps that have passed through the neural network, and the second image feature includes K+1 second feature maps that have passed through the neural network, the determination module generates the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and is specifically used to: for the first first feature map and the first second feature map, input the first initial component to the neural network to obtain the first first feature map, and input the second initial component to the neural network to obtain the first second feature map; for the i-th first feature map and the i-th second feature map, the value range of i is 2 to K, K is a positive integer greater than 1, input the i-1th first feature map to the neural network to obtain the i-th first feature map; perform feature fusion on the i-1th first feature map and the i-1th second feature map to obtain a fused feature, and input the fused feature to the neural network to obtain the i-th second feature map; after obtaining the last second feature map, generate the adjusted component based on the last second feature map.

[0253] Exemplarily, when the determination module performs feature fusion on the first feature map and the second feature map to obtain the fused feature, it is specifically used to: perform an addition operation on the first feature map and the second feature map to obtain the fused feature; or, perform a splicing operation on the first feature map and the second feature map to obtain the fused feature; or, perform guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generate the fused feature based on the weight coefficient map and the second feature map.

[0254] Exemplarily, the determination module inputs the first initial component into the first neural network, and when obtaining the first feature map, is specifically used to: extract M-dimensional image features from the first initial component, where M is a positive integer; stack the M-dimensional image features to obtain stacked multi-dimensional features; and input the multi-dimensional features into the first neural network to obtain the first feature map.

[0255] Exemplarily, the image features of the M dimensions include at least one of the following: image features obtained by convolving the first initial component with a convolution kernel; image features obtained by convolving and cascading the first initial component with two convolution kernels; first-order spatial domain features obtained by processing the first initial component with a Sobel operator; second-order spatial domain features obtained by processing the first initial component with a Laplacian operator; or frequency domain features obtained by Fourier transforming the first initial component.

[0256] Exemplarily, when the determination module inputs the first initial component into the first neural network, and obtains the first feature map, it is specifically used to: perform a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform; input the multiple frequency bands or part of the multiple frequency bands into the first neural network to obtain the first feature map; when the determination module inputs the second initial component into the second neural network, and obtains the second feature map, it is specifically used to: perform a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform; input the multiple frequency bands or part of the multiple frequency bands into the second neural network to obtain the second feature map.

[0257] Exemplarily, before inputting the first initial component into the first neural network to obtain the first feature map, the determination module is further configured to: preprocess the first initial component to obtain a preprocessed first initial component; wherein the preprocessed first initial component is input into the first neural network to obtain the first feature map. When the determination module preprocesses the first initial component to obtain the preprocessed first initial component, it is specifically configured to: perform edge enhancement on the first initial component to obtain edge-enhanced image features; perform multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features; and determine the preprocessed first initial component based on the multi-scale features.

[0258] Exemplarily, the determination module performs multi-scale feature extraction on the edge-enhanced image features, and when obtaining the multi-scale features, is specifically used to: perform a convolution operation on the edge-enhanced image features to obtain convolution features; perform a downsampling operation on the convolution features to obtain downsampled features; perform a channel transformation on the downsampled features to obtain channel-transformed features; perform an upsampling operation on the channel-transformed features to obtain upsampled features; and generate the multi-scale features based on the upsampled features and the channel-transformed features.

[0259] Exemplarily, before generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the processing module is also used to upsample the second initial component to obtain an upsampled second initial component; wherein the resolution of the upsampled second initial component is equal to the resolution of the first initial component.

[0260] Exemplarily, the processing module is used to upsample the adjusted component to obtain an upsampled adjusted component before performing feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component; wherein the resolution of the upsampled adjusted component is equal to the resolution of the first initial component.

[0261] Exemplarily, the processing module performs feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, and is also used to upsample the target component to obtain the upsampled target component; wherein the resolution of the upsampled target component is equal to the resolution of the first initial component.

[0262] Exemplarily, when the processing module performs feature enhancement on the adjusted component to obtain the restored target component corresponding to the second initial component, it is specifically used to: perform feature enhancement on the adjusted component through at least one residual block network to obtain the target component; or, perform feature enhancement on the adjusted component through a U-Net network to obtain the target component.

[0263] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. The present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. The embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The foregoing is merely an embodiment of the present disclosure and is not intended to limit the present disclosure.

[0264] For those skilled in the art, the present disclosure may be subject to various modifications and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure should be included in the scope of the claims of the present disclosure.

Claims

1. An image decoding method, characterized in that: include: Decoding a code stream corresponding to the current image block to obtain a reconstructed image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, wherein the initial chrominance component includes a first chrominance component and a second chrominance component; Performing Haar wavelet transform on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; Adjusting the frequency domain characteristics of the first chrominance component according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component to obtain an adjusted first chrominance component; According to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component, the frequency domain characteristics of the second chrominance component are adjusted to obtain an adjusted second chrominance component.

2. The method according to claim 1, characterized in that The adjusting the frequency domain characteristics of the first chrominance component according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component to obtain the adjusted first chrominance component includes: performing a splicing operation on the Haar wavelet frequency domain feature of the initial luminance component and the Haar wavelet frequency domain feature of the first chrominance component to generate a frequency domain fusion feature; based on the first chrominance component, replacing the frequency domain feature corresponding to the first chrominance component with the frequency domain fusion feature to obtain an adjusted first chrominance component; The adjusting the frequency domain characteristics of the second chrominance component according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component to obtain the adjusted second chrominance component includes: The Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the second chrominance component are superimposed to generate frequency domain fusion features; based on the second chrominance component, the frequency domain features corresponding to the second chrominance component are replaced with the frequency domain fusion features to obtain an adjusted second chrominance component.

3. The method according to claim 1, characterized in that The reconstructed image block is a reconstructed image block in a YUV format, the initial luminance component is a Y component, the first chroma component included in the initial chroma components is a U component, and the second chroma component included in the initial chroma components is a V component; The resolution of the initial luminance component is greater than or equal to the resolution of the first chrominance component, and the resolution of the initial luminance component is greater than or equal to the resolution of the second chrominance component.

4. The method according to claim 1, wherein The performing Haar wavelet transform on the initial brightness component to obtain the Haar wavelet frequency domain features of the initial brightness component includes: performing at least one Haar wavelet transform on the initial brightness component to obtain a plurality of frequency bands after the at least one Haar wavelet transform of the initial brightness component; and determining the Haar wavelet frequency domain features of the initial brightness component based on the plurality of frequency bands after the at least one Haar wavelet transform of the initial brightness component; The performing Haar wavelet transform on the first chroma component to obtain the Haar wavelet frequency domain characteristics of the first chroma component includes: performing at least one Haar wavelet transform on the first chroma component to obtain a plurality of frequency bands after the at least one Haar wavelet transform of the first chroma component; and determining the Haar wavelet frequency domain characteristics of the first chroma component based on the plurality of frequency bands after the at least one Haar wavelet transform of the first chroma component; The performing Haar wavelet transform on the second chroma component to obtain the Haar wavelet frequency domain characteristics of the second chroma component includes: performing at least one Haar wavelet transform on the second chroma component to obtain multiple frequency bands after the at least one Haar wavelet transform of the second chroma component; and determining the Haar wavelet frequency domain characteristics of the second chroma component based on the multiple frequency bands after the at least one Haar wavelet transform of the second chroma component.

5. The method according to claim 4, characterized in that Performing at least one Haar wavelet transform on the initial luminance component includes: Performing a first Haar wavelet transform on the initial brightness component to obtain multiple frequency bands of the initial brightness component after the first Haar wavelet transform; performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain multiple frequency bands of the initial brightness component after two Haar wavelet transforms; Performing at least one Haar wavelet transform on the first chrominance component includes: Performing a first Haar wavelet transform on the first chrominance component to obtain multiple frequency bands of the first chrominance component after the first Haar wavelet transform; performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain multiple frequency bands of the first chrominance component after two Haar wavelet transforms; Performing at least one Haar wavelet transform on the second chrominance component, comprising: Performing a first Haar wavelet transform on the second chroma component to obtain multiple frequency bands of the second chroma component after the first Haar wavelet transform; performing a second Haar wavelet transform on the multiple frequency bands after the first Haar wavelet transform to obtain multiple frequency bands of the second chroma component after two Haar wavelet transforms.

6. The method according to claim 1, characterized in that The method further comprises: The adjusted first chroma component is feature enhanced through a first residual block network to obtain a restored target chroma component corresponding to the first chroma component; the adjusted second chroma component is feature enhanced through a second residual block network to obtain a restored target chroma component corresponding to the second chroma component.

7. An image coding method, characterized in that: include: Obtain a reconstructed image block corresponding to the current image block, the reconstructed image block including an initial luminance component and an initial chrominance component, the initial chrominance component including a first chrominance component and a second chrominance component; Performing Haar wavelet transform on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; Adjusting the frequency domain characteristics of the first chrominance component according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component to obtain an adjusted first chrominance component; According to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component, the frequency domain characteristics of the second chrominance component are adjusted to obtain an adjusted second chrominance component.

8. An image decoding device, characterized in that: include: A decoding module, configured to decode a code stream corresponding to a current image block to obtain a reconstructed image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, wherein the initial chrominance component includes a first chrominance component and a second chrominance component; a determination module, configured to perform Haar wavelet transform processing on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; According to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component, the frequency domain characteristics of the first chrominance component are adjusted to obtain an adjusted first chrominance component; according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component, the frequency domain characteristics of the second chrominance component are adjusted to obtain an adjusted second chrominance component.

9. An image coding device, characterized in that include: An acquisition module, configured to acquire a reconstructed image block corresponding to the current image block, wherein the reconstructed image block includes an initial luminance component and an initial chrominance component, wherein the initial chrominance component includes a first chrominance component and a second chrominance component; a determination module, configured to perform Haar wavelet transform processing on the initial brightness component to obtain Haar wavelet frequency domain features of the initial brightness component; Performing Haar wavelet transform processing on the first chrominance component to obtain Haar wavelet frequency domain features of the first chrominance component, and performing Haar wavelet transform processing on the second chrominance component to obtain Haar wavelet frequency domain features of the second chrominance component; According to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the first chrominance component, the frequency domain characteristics of the first chrominance component are adjusted to obtain an adjusted first chrominance component; according to the Haar wavelet frequency domain characteristics of the initial luminance component and the Haar wavelet frequency domain characteristics of the second chrominance component, the frequency domain characteristics of the second chrominance component are adjusted to obtain an adjusted second chrominance component.

10. An image decoding device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 6.

11. An image encoding device, characterized in that include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of claim 7.

12. A machine-readable storage medium, characterized in that The machine-readable storage medium stores a plurality of computer instructions, and when the computer instructions are executed by the processor, the method according to any one of claims 1 to 6 is implemented, or when the computer instructions are executed by the processor, the method according to claim 7 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images