A decoding method, apparatus and device

Abstract

The present disclosure provides a decoding method, device and equipment thereof, the decoding method comprising: decoding a code stream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block comprising a first initial component and a second initial component, wherein the resolution of the first initial component is greater than or equal to the resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and performing feature enhancement on the adjusted component to obtain a recovered target component corresponding to the second initial component. Through the technical solution of the present disclosure, the decoding performance can be improved, and the decoding complexity can be reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of coding technology, and in particular, to a decoding method, device and equipment thereof. BACKGROUND

[0002] In order to save space, video images are transmitted after being encoded. Complete video encoding can include prediction, transformation, quantization, entropy encoding, filtering and the like. For the prediction process, the prediction process can include intra prediction and inter prediction. Inter prediction refers to using the correlation in the time domain of a video to predict the current pixel using the pixels of the adjacent coded image, so as to effectively remove the temporal redundancy of the video. Intra prediction refers to using the correlation in the spatial domain of a video to predict the current pixel using the pixels of the coded block of the current frame image, so as to remove the spatial redundancy of the video.

[0003] With the rapid development of deep learning, deep learning has achieved success in many high-level computer vision problems, such as image classification, target detection, etc. Deep learning has also gradually begun to be applied in the field of coding and decoding, that is, a neural network can be used to encode and decode images. Although the coding and decoding method based on the neural network shows great performance potential, the coding and decoding method based on the neural network still has problems such as poor decoding performance and high complexity. SUMMARY

[0004] Therefore, the present disclosure provides a decoding method, device and equipment thereof, which can improve the decoding performance.

[0005] In a first aspect, the present disclosure provides a decoding method, comprising:

[0006] decoding a code stream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block comprising a first initial component and a second initial component, the resolution of the first initial component being greater than or equal to the resolution of the second initial component;

[0007] generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0008] performing feature enhancement on the adjusted component to obtain a recovered target component corresponding to the second initial component.

[0009] In some embodiments, the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a luminance component, and the second initial component is a chroma U component and a chroma V component.

[0010] In some embodiments, the generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component comprises:

[0011] obtain a first image feature corresponding to the first initial component, the first image feature comprising an image feature not passing through a neural network;

[0012] obtain a second image feature corresponding to the second initial component, the second image feature comprising an image feature not passing through a neural network;

[0013] generate the adjusted component based on the first image feature and the second image feature.

[0014] In some embodiments, the first image feature comprises a Haar wavelet frequency domain feature; and the second image feature comprises a Haar wavelet frequency domain feature.

[0015] In some embodiments, the generating the adjusted component corresponding to the second initial component based on the first initial component and the second initial component comprises:

[0016] performing at least one Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the transform as a first feature map;

[0017] performing at least one Haar wavelet transform on the second initial component to obtain a plurality of frequency bands after the transform as a second feature map;

[0018] performing a stacking operation on the first feature map and the second feature map to obtain the adjusted component.

[0019] In some embodiments, the performing the stacking operation on the first feature map and the second feature map comprises:

[0020] performing the stacking operation on the first feature map and the second feature map along a channel dimension.

[0021] In some embodiments, the performing at least one Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the transform comprises: performing twice Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the twice Haar wavelet transform.

[0022] The performing at least one Haar wavelet transform on the second initial component to obtain a plurality of frequency bands after the transform comprises: performing twice Haar wavelet transform on the second initial component to obtain a plurality of frequency bands after the twice Haar wavelet transform.

[0023] In some embodiments, the performing at least one Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the transform comprises: performing twice Haar wavelet transform on the first initial component to obtain a plurality of frequency bands after the twice Haar wavelet transform.

[0024] The multiple times of Haar wavelet transform on the second initial component to obtain the multiple frequency bands after the multiple times of Haar wavelet transform includes: twice of Haar wavelet transform on the second initial component to obtain the multiple frequency bands after the twice of Haar wavelet transform.

[0025] In some embodiments, the twice of Haar wavelet transform on the first initial component to obtain the multiple frequency bands after the twice of Haar wavelet transform includes: first time of Haar wavelet transform on the first initial component to obtain the multiple frequency bands after the first time of Haar wavelet transform; second time of Haar wavelet transform on the multiple frequency bands after the first time of Haar wavelet transform to obtain the multiple frequency bands after the twice of Haar wavelet transform.

[0026] The twice of Haar wavelet transform on the second initial component to obtain the multiple frequency bands after the twice of Haar wavelet transform includes: first time of Haar wavelet transform on the second initial component to obtain the multiple frequency bands after the first time of Haar wavelet transform; second time of Haar wavelet transform on the multiple frequency bands after the first time of Haar wavelet transform to obtain the multiple frequency bands after the twice of Haar wavelet transform.

[0027] In some embodiments, the feature enhancement on the adjusted component to obtain the second initial component corresponding to the recovered target component includes:

[0028] The feature enhancement on the adjusted component to obtain the target component is performed by multiple residual block networks.

[0029] In a second aspect, the present disclosure provides an encoding method, including:

[0030] Encoding a first initial component and a second initial component of a current image block to obtain a code stream corresponding to the current image block, wherein a resolution of the first initial component is greater than or equal to a resolution of the second initial component.

[0031] In some embodiments, the current image block is in YUV format, the first initial component is a luminance component, and the second initial component is a chroma U component and a chroma V component.

[0032] In a third aspect, the present disclosure provides a decoding device, including:

[0033] A decoding module is configured to decode a code stream corresponding to a current image block to obtain a reconstructed image block, wherein the reconstructed image block includes a first initial component and a second initial component, and a resolution of the first initial component is greater than or equal to a resolution of the second initial component.

[0034] determining a second initial component corresponding to the first initial component and the second initial component;

[0035] processing the adjusted component to obtain a restored target component corresponding to the second initial component.

[0036] In a fourth aspect, the present disclosure provides an encoding device, comprising:

[0037] an encoding module configured to encode a first initial component and a second initial component of a current image block to obtain a bitstream corresponding to the current image block, wherein a resolution of the first initial component is greater than or equal to a resolution of the second initial component.

[0038] In a fifth aspect, the present disclosure provides a decoding device, comprising: a processor and a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions which can be executed by the processor; and the processor is configured to execute the machine executable instructions to implement the method in any one of the first aspect.

[0039] In a sixth aspect, the present disclosure provides an encoding device, comprising: a processor and a machine readable storage medium, wherein the machine readable storage medium stores machine executable instructions which can be executed by the processor; and the processor is configured to execute the machine executable instructions to implement the method in any one of the second aspect.

[0040] In a seventh aspect, the present disclosure provides a machine readable storage medium, wherein the machine readable storage medium stores a plurality of computer instructions, and the computer instructions are executed by a processor to implement the method in any one of the first aspect, or implement the method in any one of the second aspect.

[0041] In an eighth aspect, the present disclosure provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the method in any one of the first aspect, or implement the method in any one of the second aspect.

[0042] As can be seen from the above technical solutions, in this embodiment of the present disclosure, after decoding to obtain the reconstructed image block, the reconstructed image block includes a first initial component and a second initial component. The resolution of the first initial component is greater than or equal to the resolution of the second initial component. An adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component. That is, the second initial component is adjusted based on the first initial component to obtain the adjusted component. Then, the target component corresponding to the second initial component is determined based on the adjusted component. This can improve decoding performance, reduce decoding complexity, reduce computational complexity, require fewer resources, and can be used by cost- and latency-sensitive devices. A low-complexity and efficient post-processing network model framework is proposed to address compression defect recovery under various bitrates. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a three-dimensional feature matrix in one embodiment of the present disclosure;

[0044] Figure 2 This is a flowchart of a decoding method in one embodiment of the present disclosure;

[0045] Figure 3 This is a schematic diagram of the processing procedure at the encoding end in one embodiment of this disclosure;

[0046] Figure 4 This is a schematic diagram of the processing procedure at the decoding end in one embodiment of this disclosure;

[0047] Figure 5A This is a schematic diagram of the structure of end-to-end image compression in one embodiment of this disclosure;

[0048] Figure 5B This is a schematic diagram of the post-processing structure based on deep learning in one embodiment of this disclosure;

[0049] Figures 6A-6I This is a schematic diagram of the structure of a post-processing network in one embodiment of the present disclosure;

[0050] Figures 7A-7C This is a schematic diagram of upsampling in one embodiment of the present disclosure;

[0051] Figure 7D This is a schematic diagram of the structure of a post-processing network in one embodiment of the present disclosure;

[0052] Figure 7E and Figure 7F This is a schematic diagram of feature enhancement in one embodiment of the present disclosure;

[0053] Figure 8 This is a hardware structure diagram of a decoding device in one embodiment of the present disclosure. Detailed Implementation

[0054] The terminology used in the disclosure of the embodiments herein is for the purpose of describing particular embodiments only and is not intended to be limiting thereof. The use of the terms "a" and "an" and "the" and "said" herein is intended to include the plural, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, objects, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, objects, and / or components thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are used only to distinguish one from another different information. For example, a first information could be termed a second information, and, similarly, a second information could be termed a first information, without departing from the scope of the present disclosure. Furthermore, the use of the term "if" can be interpreted as meaning "when" or "upon" or "in response to determining" depending on the context.

[0055] A decoding method is proposed in the embodiments of the disclosure, which can involve the following concepts:

[0056] JPEG (Joint Photographic Experts Group): JPEG is a standard for compressing continuous-tone still images, and the file suffix can be.jpg or.jpeg. JPEG is a commonly used image file format. JPEG is a joint coding method using predictive coding (DPCM), discrete cosine transform (DCT) and entropy coding to remove redundant image and color data, which belongs to a lossy compression format, can compress the image into a very small storage space, and will cause damage to the image data to some extent. Especially when using too high compression ratio, the quality of the image recovered after final decompression will be reduced. If high-quality images are pursued, JPEG should not use too high compression ratio.

[0057] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to create a learning-based image coding standard that provides a single-stream, compact, compressed-domain representation, significantly improving compression efficiency over commonly used image coding standards at equivalent subjective quality, effectively improving performance in image processing and computer vision tasks. JPEG-AI targets a wide range of applications such as cloud storage, visual management, autonomous vehicles and devices, image acquisition, storage and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design a coding solution that significantly improves compression efficiency at the same subjective quality, providing efficient compressed-domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI requires encoding and decoding that is friendly to hardware and software implementation, supports 8-bit and 10-bit depth, and efficiently encodes and progressively decodes images using text and graphics.

[0058] Post Filter: The encoding end forms a stream of codes by encoding and compressing images, and the decoding end decodes and reconstructs images from the code stream. However, due to the reasons of the encoding algorithm itself, after decoding and reconstructing the image from the code stream, the image will have defects such as blockiness, image artifacts, and color drift. The role of post-filtering is to improve the defects caused by encoding and compression, maximize the recovery of image data, and improve the subjective quality of the decoded and reconstructed image.

[0059] Image Super-Resolution: Super-Resolution mainly refers to restoring low-resolution images to high-resolution images, while enhancing image details and improving image subjective quality. Image Super-Resolution is a technology used to enhance image resolution in computer vision image processing.

[0060] YUV Color Space: Each pixel of a color image can usually be described by several independent physical quantities, thus forming a spatial coordinate, which is the color space of the image. YUV color space is a color space in which the pixels of a color image are described by Y (luminance), U (chrominance), and V (chrominance) attributes.

[0061] Entropy Encoding: Entropy encoding refers to an encoding process that does not lose any information according to the principle of entropy. The entropy of information is the average amount of information of the source (a measure of uncertainty). The encoding method of entropy encoding can include but is not limited to: Shannon encoding, Huffman encoding and arithmetic encoding.

[0062] Neural Network (NN): Neural network refers to artificial neural network. Neural network is an operation model composed of a large number of nodes (or neurons) connected with each other. In neural network, neuron processing unit can represent different objects such as features, letters, concepts or some meaningful abstract patterns. The types of processing units in neural network can be divided into three categories: input unit, output unit and hidden unit. The input unit accepts signals and data from the external world; the output unit realizes the output of processing result; the hidden unit is between the input and output units and cannot be observed from outside the system. The connection weight between neurons reflects the connection strength between units, and the representation and processing of information are embodied in the connection relationship of processing units. Neural network is a non-programmed, brain-like information processing method. The essence of neural network is to obtain a parallel distributed information processing function through the transformation and dynamics of neural network, and to simulate the information processing function of human brain nervous system at different levels. In the field of video processing, commonly used neural networks can include but are not limited to: convolutional neural network (CNN), recurrent neural network (RNN) and fully connected network.

[0063] Convolutional Neural Network (CNN): Convolutional neural network is a kind of feedforward neural network, which is one of the most representative network structures in deep learning technology. The artificial neuron of convolutional neural network can respond to a part of the surrounding units in the coverage range, and has excellent performance for large image processing. The basic structure of convolutional neural network includes two layers. One is the feature extraction layer (also called convolution layer), the input of each neuron is connected with the local receptive field of the previous layer, and the local feature is extracted. Once the local feature is extracted, the positional relationship between the local feature and other features is also determined. The second is the feature mapping layer (also called activation layer). Each calculation layer of neural network is composed of multiple feature mappings. Each feature mapping is a plane, and all the weights of the neurons on the plane are equal. The feature mapping structure can use Sigmoid function, ReLU function, Leaky-ReLU function, PReLU function, GDN function, etc. as the activation function of convolutional network. In addition, since the neurons on a mapping plane share weights, the number of free parameters of the network can be reduced.

[0064] For example, one of the advantages of the convolutional neural network over the image processing algorithm is that the complex pre-processing process of the image (extracting artificial features, etc.) is avoided, and the original image can be directly inputted for end-to-end learning. One of the advantages of the convolutional neural network over the general neural network is that the general neural network adopts full connection, i.e., all the neurons from the input layer to the hidden layer are connected, which will result in a large number of parameters, and the network training will be time-consuming or even difficult to train. The convolutional neural network avoids this difficulty through local connection and weight sharing.

[0065] Deconvolution: The deconvolution layer is also called the transpose convolution layer. The working process of the deconvolution layer is similar to that of the convolution layer, and the main difference is that the deconvolution layer will be padded to make the output larger than the input (or the same). If the stride is 1, the output size is equal to the input size. If the stride is N, the width of the output feature is N times the width of the input feature, and the height of the output feature is N times the height of the input feature.

[0066] Generalization Ability: The generalization ability can refer to the adaptability of the machine learning algorithm to new samples. The purpose of learning is to learn the rule hidden in the data pair. The trained network can also give a suitable output for the data outside the learning set with the same rule, and this ability can be called the generalization ability.

[0067] Feature: The feature involved in the present disclosure is a three-dimensional feature matrix of C*W*H. Referring to FIG. 1, which is a schematic diagram of a three-dimensional feature matrix, in the three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input of the neural network or the output of the neural network. Figure 1

[0068] ​Rate-Distortion Optimized: There are two indicators to evaluate the coding efficiency: code rate and PSNR (Peak Signal to Noise Ratio). The smaller the bit stream, the greater the compression rate, and the greater the PSNR, the better the quality of the reconstructed image. In mode selection, the evaluation formula is essentially a comprehensive evaluation of the two. For example, the cost of the mode: J (mode) = D + λ * R, where D represents Distortion (distortion), which can usually be measured using the SSE indicator. SSE refers to the sum of the squares of the differences between the reconstructed image block and the source image. In order to achieve cost consideration, the SAD indicator can also be used. SAD is the sum of the absolute values of the differences between the reconstructed image block and the source image. λ is the Lagrange multiplier, and R is the actual number of bits required for image block coding under this mode, including the total number of bits required for coding mode information, motion information, and residual error. In mode selection, if the rate-distortion principle is used to compare and decide the coding mode, the coding performance can usually be guaranteed to be optimal.

[0069] A large number of encoding tools are proposed for each module of the encoding end, and each tool often has multiple modes. For different video sequences, the encoding tool that can obtain the optimal coding performance is often different. Therefore, in the encoding process, RDO (Rate-Distortion Optimize) is usually used to compare the coding performance of different tools or modes to select the best mode. After determining the optimal tool or mode, the decision information of the tool or mode is transmitted by encoding the marker information in the bit stream. Although this method brings higher encoding complexity, it can adaptively select the optimal mode combination for different content to obtain the optimal coding performance. The decoding end can obtain the relevant mode information by directly parsing the flag information, and the complexity is less affected.

[0070] In the end-to-end image compression technology based on deep learning, the image is converted to YUV color space, and the UV chrominance signal is subsampled by one-half. Then the Y signal and the UV signal are compressed and encoded respectively. The decoding end decodes and reconstructs according to the bit stream data to obtain the reconstructed Y signal and UV signal. Since the UV chrominance signal is compressed and encoded after being subsampled by one-half, that is, the resolution of the UV chrominance signal is halved, the resolution of the UV chrominance signal can also be restored in the post-processing process. However, the calculation complexity of the resolution restoration process is high, and the demand for resources is large, so some cost and delay sensitive devices cannot restore the resolution of the UV chrominance signal.

[0071] In view of the above finding, the embodiment proposes a neural network-based defect removal network model method. In the field of end-to-end image compression, a low-complexity and efficient post-processing network model framework is proposed, which can enhance the UV signal based on the Y signal. The method has low complexity and high performance, and can recover compression defects at various code rates.

[0072] The decoding method in the embodiment of the disclosure is described in detail below in combination with several specific embodiments.

[0073] Embodiment 1: The embodiment proposes a decoding method, as shown in Figure 2 The method can be applied to a decoding end (also referred to as a video decoder), and can include steps 201-203.

[0074] Step 201: Decoding the code stream corresponding to the current image block to obtain a reconstructed image block.

[0075] For example, the reconstructed image block can include a first initial component and a second initial component, the resolution of the second initial component is a down-sampled resolution, and the resolution of the first initial component can be greater than or equal to the resolution of the second initial component. For example, the resolution of the first initial component can be an un-down-sampled resolution (i.e., an original resolution), or a down-sampled resolution, but the resolution of the first initial component is greater than or equal to the resolution of the second initial component, which is not limited.

[0076] For example, if the reconstructed image block is a YUV format reconstructed image block, the first initial component can be a luminance component, and the second initial component can be a chroma U component and / or a chroma V component; or, if the reconstructed image block is an RGB format reconstructed image block, the first initial component can be a G component, and the second initial component can be an R component and / or a B component.

[0077] Step 202: Generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component. The reconstruction quality of the adjusted component can be better than that of the second initial component, the reconstruction quality of the adjusted component can be equal to that of the second initial component, and the reconstruction quality of the second initial component can also be better than that of the adjusted component.

[0078] Exemplarily, a first image feature corresponding to the first initial component can be acquired, which can include but is not limited to an image feature not passing through a neural network and / or an image feature passing through a neural network; a second image feature corresponding to the second initial component can be acquired, which can include but is not limited to an image feature not passing through a neural network and / or an image feature passing through a neural network; and an adjusted component can be generated based on the first image feature and the second image feature.

[0079] Exemplarily, if the first image feature includes an image feature not passing through a neural network, the first image feature can include but is not limited to at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature; and if the second image feature includes an image feature not passing through a neural network, the second image feature can include but is not limited to at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The above are only a few examples, and the present disclosure is not limited in this regard.

[0080] If the first image feature includes a first feature map passing through a neural network and the second image feature includes a second feature map passing through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component can include but is not limited to: inputting the first initial component into a first neural network to obtain the first feature map; inputting the second initial component into a second neural network to obtain the second feature map; performing an addition operation on the first feature map and the second feature map to obtain the adjusted component; or performing a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0081] Exemplarily, if the first image feature includes a weight coefficient map passing through a neural network and the second image feature includes a second feature map passing through a neural network, generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component can include but is not limited to: inputting the first initial component into a first neural network to obtain a first feature map corresponding to the first initial component, and performing guided filtering on the first feature map to obtain the weight coefficient map; inputting the second initial component into a second neural network to obtain the second feature map; and generating the adjusted component based on the weight coefficient map and the second feature map.

[0082] Exemplarily, performing guided filtering on the first feature map to obtain the weight coefficient map can include but is not limited to: performing a convolution operation on the first feature map to obtain a convolution feature map; and performing weight mapping on the convolution feature map to obtain the weight coefficient map.

[0083] Exemplarily, the weight mapping is performed on the convolutional feature map to obtain a weight coefficient map, including but not limited to: performing a pooling operation on the convolutional feature map to obtain a pooled feature map; performing a full connection operation and a ReLU (Rectified Linear Unit) activation operation on the pooled feature map to obtain a ReLU activated feature map; performing a full connection operation and a Sigmoid (S-shaped curve, i.e., S-shaped function) activation operation on the ReLU activated feature map to obtain a Sigmoid activated feature map; and generating the weight coefficient map based on the convolutional feature map and the Sigmoid activated feature map, and multiplying the convolutional feature map and the Sigmoid activated feature map to obtain the weight coefficient map.

[0084] Exemplarily, the weight mapping is performed on the convolutional feature map to obtain a weight coefficient map, including but not limited to: performing a convolution operation and a Sigmoid activation operation on the convolutional feature map to obtain the weight coefficient map.

[0085] Exemplarily, the adjusted component is generated based on the weight coefficient map and the second feature map, including but not limited to: performing a multiplication operation on the weight coefficient map and the second feature map to obtain a multiplied feature map; performing a convolution operation on the first feature map to obtain a convolutional feature map; performing an addition operation on the multiplied feature map and the convolutional feature map to obtain an added feature map; and performing a concatenation operation on the added feature map and the convolutional feature map to obtain the adjusted component.

[0086] Exemplarily, if the first image feature includes K first feature maps passing through the neural network, and the second image feature includes K+1 second feature maps passing through the neural network, the adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, including but not limited to: for the first first feature map and the first second feature map, the first initial component is input into the neural network to obtain the first first feature map, and the second initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, i is any integer greater than or equal to 2 and less than or equal to K, K can be a positive integer greater than 1, then the (i-1)-th first feature map is input into the neural network to obtain the i-th first feature map; the (i-1)-th first feature map and the (i-1)-th second feature map are fused to obtain a fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map; after obtaining the last second feature map (i.e., the last second feature map in the K+1 second feature maps), the adjusted component is generated based on the last second feature map.

[0087] Exemplarily, the feature fusion of the first feature map (e.g., the i-1th first feature map) and the second feature map (e.g., the i-1th second feature map) to obtain the fused feature can include but is not limited to: performing an addition operation on the first feature map and the second feature map to obtain the fused feature; or performing a concatenation operation on the first feature map and the second feature map to obtain the fused feature; or performing guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating the fused feature based on the weight coefficient map and the second feature map. The above are only a few examples, and this is not limited.

[0088] Exemplarily, inputting the first initial component into the first neural network to obtain the first feature map can include but is not limited to: extracting M-dimensional image features from the first initial component, M being a positive integer; performing concatenation on the M-dimensional image features to obtain multi-dimensional features after concatenation; and inputting the multi-dimensional features into the first neural network to obtain the first feature map.

[0089] Exemplarily, the M-dimensional image features can include but are not limited to at least one of: image features after convolution of the first initial component by using one convolution kernel; image features after convolution cascade of the first initial component by using two convolution kernels; first-order spatial domain features after processing the first initial component by using a sobel operator; second-order spatial domain features after processing the first initial component by using a Laplacian operator; or frequency domain features after Fourier transform of the first initial component. The above are only a few examples of image features, and this is not limited.

[0090] Exemplarily, inputting the first initial component into the first neural network to obtain the first feature map can include but is not limited to: performing wavelet transform on the first initial component to obtain a plurality of frequency bands after wavelet transform; and inputting the plurality of frequency bands or part of the plurality of frequency bands into the first neural network to obtain the first feature map. Inputting the second initial component into the second neural network to obtain the second feature map can include but is not limited to: performing wavelet transform on the second initial component to obtain a plurality of frequency bands after wavelet transform; and inputting the plurality of frequency bands or part of the plurality of frequency bands into the second neural network to obtain the second feature map.

[0091] When the first initial component is wavelet transformed to obtain a plurality of frequency bands after wavelet transformation, the first initial component can be wavelet transformed once to obtain a plurality of frequency bands after wavelet transformation, or the first initial component can be wavelet transformed multiple times to obtain a plurality of frequency bands after wavelet transformation. When the first initial component is wavelet transformed multiple times, the first initial component is first wavelet transformed to obtain a plurality of frequency bands after wavelet transformation, then, a target frequency band (all or part of the plurality of frequency bands) is selected from the plurality of frequency bands, and the target frequency band is wavelet transformed to obtain a plurality of frequency bands after wavelet transformation, and so on, until the wavelet transformation is completed to obtain a plurality of frequency bands after wavelet transformation. When the second initial component is wavelet transformed to obtain a plurality of frequency bands after wavelet transformation, the second initial component can be wavelet transformed once to obtain a plurality of frequency bands after wavelet transformation, or the second initial component can be wavelet transformed multiple times to obtain a plurality of frequency bands after wavelet transformation.

[0092] Exemplarily, before the first initial component is input to the first neural network to obtain the first feature map, the first initial component can be preprocessed to obtain a preprocessed first initial component; wherein the preprocessed first initial component is used to input to the first neural network to obtain the first feature map. Wherein the preprocessing of the first initial component to obtain the preprocessed first initial component can include but is not limited to: performing edge enhancement on the first initial component to obtain an edge-enhanced image feature; performing multi-scale feature extraction on the edge-enhanced image feature to obtain a multi-scale feature; and taking the multi-scale feature as the preprocessed first initial component based on the multi-scale feature preprocessed first initial component.

[0093] Exemplarily, the multi-scale feature extraction on the edge-enhanced image feature to obtain a multi-scale feature can include but is not limited to: performing convolution operation on the edge-enhanced image feature to obtain a convolution feature; performing down-sampling operation on the convolution feature to obtain a down-sampled feature; performing channel transformation on the down-sampled feature to obtain a channel-transformed feature; performing up-sampling operation on the channel-transformed feature to obtain an up-sampled feature; and generating the multi-scale feature based on the up-sampled feature and the channel-transformed feature, and then inputting the multi-scale feature as the preprocessed first initial component to the first neural network.

[0094] Step 203, performing feature enhancement on the adjusted component to obtain a second initial component corresponding to a recovered target component.

[0095] In a possible implementation, before the second initial component corresponding to the adjusted component is generated based on the first initial component and the second initial component, the second initial component can be up-sampled to obtain an up-sampled second initial component. The resolution of the up-sampled second initial component can be equal to the resolution of the first initial component.

[0096] In another possible implementation, before the feature enhancement is performed on the adjusted component to obtain the recovered target component corresponding to the second initial component (i.e., before step 203), the adjusted component can also be up-sampled to obtain an up-sampled adjusted component. The resolution of the up-sampled adjusted component can be equal to the resolution of the first initial component.

[0097] In another possible implementation, after the feature enhancement is performed on the adjusted component to obtain the recovered target component corresponding to the second initial component (i.e., after step 203), the target component can also be up-sampled to obtain an up-sampled target component. The resolution of the up-sampled target component can be equal to the resolution of the first initial component.

[0098] Exemplarily, the feature enhancement performed on the adjusted component to obtain the recovered target component corresponding to the second initial component can include, but is not limited to, performing the feature enhancement on the adjusted component by at least one residual block network to obtain the target component corresponding to the second initial component, or performing the feature enhancement on the adjusted component by a U-Net (Encoder-Decoder structure network, the first half is feature extraction, and the second half is up-sampling) network to obtain the target component corresponding to the second initial component.

[0099] Exemplarily, the above execution order is only an example given for the convenience of description, and in actual application, the execution order between steps can also be changed, and the execution order is not limited. Moreover, in other embodiments, the steps of the corresponding method can not necessarily be executed in the order shown and described in the present disclosure, and the steps included in the method can be more or less than those described in the present disclosure. In addition, a single step described in the present disclosure can be divided into multiple steps for description in other embodiments; multiple steps described in the present disclosure can also be combined into a single step for description in other embodiments.

[0100] It can be seen from the above technical solutions that, in the embodiment of the disclosure, after the reconstructed image block is decoded, the reconstructed image block includes a first initial component and a second initial component, the resolution of the second initial component is a resolution after downsampling, the resolution of the first initial component is greater than or equal to the resolution of the second initial component, and the second initial component is adjusted based on the first initial component and the second initial component to generate an adjusted component corresponding to the second initial component, that is, the second initial component is adjusted based on the first initial component to obtain an adjusted component, and then the target component corresponding to the second initial component is determined based on the adjusted component, so that the decoding performance can be improved, the decoding complexity can be reduced, the calculation complexity can be reduced, the resource demand is relatively small, and the algorithm can be used for cost and delay sensitive devices. A low complexity and efficient post-processing network model framework is proposed to recover compression defects under multiple code rates.

[0101] In an embodiment, the processing process of the encoding end (also referred to as a video encoder) can refer to FIG. 2. Figure 3 Figure 3 The processing process of the encoding end is only an example, and the processing process of the encoding end is not limited.

[0102] After the encoding end obtains the current image block x (the current image block x can be the original image block x, that is, the input image block), the current image block x can be analyzed and transformed by an analysis transformation network (that is, a neural network) to obtain the image feature y corresponding to the current image block x. Wherein, the feature transformation of the current image block x by the analysis transformation network refers to: transforming the current image block x into the image feature y in the latent domain, so as to facilitate the operation of all subsequent processes in the latent domain.

[0103] The image can be divided into one image block, or can be divided into multiple image blocks. If the image is divided into one image block, the current image block x can also be the image, that is, the coding and decoding process of the image block can also be directly used for the image.

[0104] After the encoding end obtains the image feature y, the image feature y is subjected to coefficient hyperparameter feature transformation to obtain the coefficient hyperparameter feature z. For example, the image feature y can be input to a hyperparameter coding network (that is, a neural network) to perform coefficient hyperparameter feature transformation on the image feature y by the hyperparameter coding network to obtain the coefficient hyperparameter feature z. The hyperparameter coding network can be a trained neural network, and the training process of the hyperparameter coding network is not limited, as long as the image feature y can be subjected to coefficient hyperparameter feature transformation. The image feature y in the latent domain is subjected to the hyperparameter coding network to obtain the hyperprior latent information z.

[0105] After the encoding end obtains the coefficient hyperparameter feature z, the coefficient hyperparameter feature z can be quantized to obtain the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, that is,​Figure 3 Q in the equation represents a quantization process. After obtaining the hyper-parameter quantization feature corresponding to the coefficient hyper-parameter feature z, the hyper-parameter quantization feature is encoded to obtain the Bitstream#1 (i.e., the first code stream) corresponding to the current image block, that is, Figure 3 The AE in the equation represents an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the coefficient hyper-parameter feature z to obtain the Bitstream#1 corresponding to the current image block. The hyper-parameter quantization feature or the coefficient hyper-parameter feature z carried in the Bitstream#1 is mainly used to obtain the parameters of the mean value and the probability distribution model.

[0106] After obtaining the Bitstream#1 corresponding to the current image block, the encoding end can send the Bitstream#1 corresponding to the current image block to the decoding end. For the processing process of the decoding end for the Bitstream#1 corresponding to the current image block, see the subsequent embodiments.

[0107] After obtaining the Bitstream#1 corresponding to the current image block, the encoding end can also decode the Bitstream#1 to obtain the hyper-parameter quantization feature, that is, Figure 3 AD in the equation represents a decoding process, and then the hyper-parameter quantization feature is dequantized to obtain the coefficient hyper-parameter feature z_hat. The coefficient hyper-parameter feature z_hat can be the same as or different from the coefficient hyper-parameter feature z, Figure 3 The IQ in the equation represents a dequantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the encoding end can also decode the Bitstream#1 to obtain the coefficient hyper-parameter feature z_hat, without involving the dequantization process of the coefficient hyper-parameter feature z_hat.

[0108] For the encoding process of the Bitstream#1, a fixed probability density model encoding method can be used. For the decoding process of the Bitstream#1, a fixed probability density model decoding method can be used. The encoding and decoding processes are not limited.

[0109] After obtaining the coefficient hyper-parameter feature z_hat, the encoding end can perform context-based prediction based on the coefficient hyper-parameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in subsequent embodiments) to obtain the prediction value mu (i.e., the mean value mu) corresponding to the current image block. For example, the coefficient hyper-parameter feature z_hat and the residual feature y_hat are input into the mean value prediction network, and the mean value prediction network determines the prediction value mu based on the coefficient hyper-parameter feature z_hat and the residual feature y_hat. The prediction process is not limited in this regard. For the context-based prediction process, the input includes the coefficient hyper-parameter feature z_hat and the decoded residual feature y_hat, which are jointly input to obtain a more accurate prediction value mu. The prediction value mu is used to obtain the residual by subtracting the original feature and to obtain the reconstruction y by adding the decoded residual.

[0110] It should be noted that the mean value prediction network is an optional neural network, i.e., there can be no mean value prediction network, i.e., the prediction value mu does not need to be determined by the mean value prediction network. Figure 3 The dashed box in the above formula indicates that the mean value prediction network is optional.

[0111] After obtaining the image feature y, the encoding end can determine the residual feature r based on the image feature y and the prediction value mu, such as taking the difference between the image feature y and the prediction value mu as the residual feature r. Then, the residual feature r is processed to obtain the image feature s, and the feature processing process is not limited and can be any feature processing manner. In this case, the mean value prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature y, the encoding end can process the image feature y to obtain the image feature s, and the feature processing process is not limited and can be any feature processing manner. In this case, the mean value prediction network is not required, and the residual process is optional, as indicated by the dashed box.

[0112] After obtaining the image feature s, the encoding end can quantize the image feature s to obtain the image quantization feature corresponding to the image feature s, i.e., Figure 3 The Q operation in the above formula is a quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding end can encode the image quantization feature to obtain the Bitstream#2 (i.e., the second code stream) corresponding to the current image block, i.e., Figure 3 The AE operation in the above formula indicates an encoding process, such as an entropy encoding process. Alternatively, the encoding end can directly encode the image feature s to obtain the Bitstream#2 corresponding to the current image block, without involving the quantization process of the image feature s.

[0113] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can send the Bitstream#2 corresponding to the current image block to the decoding end. For the processing process of the decoding end for the Bitstream#2 corresponding to the current image block, see the subsequent embodiments.

[0114] After obtaining the Bitstream#2 corresponding to the current image block, the encoding end can also decode the Bitstream#2 to obtain the image quantization feature, that is, Figure 3 AD in the above formula represents the decoding process. Then, the encoding end can perform inverse quantization on the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 3 The IQ operation in the above formula is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the encoding end can also decode the Bitstream#2 to obtain the image feature s' without involving the inverse quantization process of the image quantization feature.

[0115] After obtaining the image feature s', the encoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' without limitation. Any feature restoration manner can be used to obtain the residual feature r_hat, which can be the same as or different from the residual feature r. After obtaining the residual feature r_hat, the encoding end determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, which can be the same as or different from the image feature y. For example, the sum of the residual feature r_hat and the prediction value mu is taken as the image feature y_hat. In this case, a mean prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature s', the encoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' to obtain the image feature y_hat, which can be the same as or different from the image feature y. In this case, the mean prediction network does not need to be deployed, and the residual process is optional, which is represented by the dashed box.

[0116] After obtaining the image feature y_hat, the encoding end can perform synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input to a synthesis transformation network, and the synthesis transformation network performs synthesis transformation on the image feature y_hat to obtain the reconstructed image block x_hat. Thus, the image reconstruction process is completed.

[0117] In a possible implementation, when the encoding end encodes the image quantization feature or the image feature s to obtain the Bitstream#2 corresponding to the current image block, the encoding end needs to first determine the probability distribution model, and then encodes the image quantization feature or the image feature s based on the probability distribution model. In addition, when the encoding end decodes the Bitstream#2, the encoding end also needs to first determine the probability distribution model, and then decodes the Bitstream#2 based on the probability distribution model.

[0118] To obtain the probability distribution model, continue to refer to FIG. 2. Figure 3 As shown in FIG. 2, after obtaining the coefficient hyperparameter feature z_hat, the encoding end can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat is input into a probability hyperparameter decoding network, and the probability hyperparameter decoding network performs coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, the probability distribution model can be generated based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network, and the training process of the probability hyperparameter decoding network is not limited. The probability hyperparameter decoding network can only perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0119] In a possible implementation, the processing process of the encoding end described above can be performed by a deep learning model or a neural network model, so as to realize an end-to-end image compression and encoding process, and the encoding process is not limited.

[0120] Embodiment 3: The processing process of the decoding end (also referred to as a video decoder) can be referred to as FIG. 3. Figure 4 As shown in FIG. 3, Figure 4 This is only an example of the processing process of the decoding end, and the processing process of the decoding end is not limited.

[0121] After obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode the Bitstream#1 to obtain the hyperparameter quantization feature, that is, Figure 4 AD in FIG. 3 represents the decoding process, and then the hyperparameter quantization feature is dequantized to obtain the coefficient hyperparameter feature z_hat, which can be the same as or different from the coefficient hyperparameter feature z, Figure 4 The IQ operation in FIG. 3 is the dequantization process. Alternatively, after obtaining the Bitstream#1 corresponding to the current image block, the decoding end can also decode the Bitstream#1 to obtain the coefficient hyperparameter feature z_hat, without involving the dequantization process of the coefficient hyperparameter feature z_hat.

[0122] For the decoding process of Bitstream#1, a decoding method of fixed probability density model can be adopted, and no limitation is made.

[0123] The image can be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can also be the image, that is, the decoding process of the image block can also be directly used for the image.

[0124] After obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (the determination process of the residual feature y_hat is described in subsequent embodiments), to obtain the prediction value mu (i.e., the mean mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat are input into the mean prediction network, and the mean prediction network determines the prediction value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. The prediction process is not limited. For the context-based prediction process, the input includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and the two are jointly input to obtain a more accurate prediction value mu.

[0125] It should be noted that the mean prediction network is an optional neural network, that is, there can be no mean prediction network, that is, the prediction value mu does not need to be determined by the mean prediction network. Figure 4 The dashed box in the above formula indicates that the mean prediction network is optional.

[0126] After obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode the Bitstream#2 to obtain the image quantization feature, that is, Figure 4 AD in the above formula indicates the decoding process. Then, the decoding end can perform inverse quantization on the image quantization feature to obtain the image feature s', which can be the same as or different from the image feature s, Figure 4 The IQ operation in the above formula is the inverse quantization process. Alternatively, after obtaining the Bitstream#2 corresponding to the current image block, the decoding end can also decode the Bitstream#2 to obtain the image feature s', without involving the inverse quantization process of the image quantization feature.

[0127] After obtaining the image feature s', the decoding end can perform feature restoration (i.e., the inverse process of feature processing) on the image feature s' to obtain a residual feature r_hat, which is the same as or different from the residual feature r. After obtaining the residual feature r_hat, the decoding end determines the image feature y_hat based on the residual feature r_hat and the prediction value mu, which is the same as or different from the image feature y, such as taking the sum of the residual feature r_hat and the prediction value mu as the image feature y_hat. In this case, a mean prediction network needs to be deployed to provide the prediction value mu. Alternatively, after obtaining the image feature s', the decoding end can perform feature restoration on the image feature s' to obtain the image feature y_hat, which can be the same as or different from the image feature y. In this case, a mean prediction network does not need to be deployed, and the residual process is represented by a dashed box.

[0128] After obtaining the image feature y_hat, the decoding end can perform synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x, such as inputting the image feature y_hat to a synthesis transformation network to perform synthesis transformation on the image feature y_hat by the synthesis transformation network to obtain the reconstructed image block x_hat, thereby completing the image reconstruction process.

[0129] In a possible implementation, when decoding Bitstream#2, the decoding end needs to first determine a probability distribution model, and then decode Bitstream#2 based on the probability distribution model. To obtain the probability distribution model, continue to refer to FIG. 8. Figure 4 As shown in FIG. 8, after obtaining the coefficient hyperparameter feature z_hat, the decoding end can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain a probability distribution parameter p, such as inputting the coefficient hyperparameter feature z_hat to a probability hyperparameter decoding network to perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat by the probability hyperparameter decoding network to obtain the probability distribution parameter p, and after obtaining the probability distribution parameter p, a probability distribution model can be generated based on the probability distribution parameter p. The probability hyperparameter decoding network can be a trained neural network, and the training process of the probability hyperparameter decoding network is not limited. The probability hyperparameter decoding network can only perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0130] In a possible implementation, the processing process of the decoding end described above can be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, and the decoding process is not limited.

[0131] Embodiment 4: refer to FIG. 8. Figure 5AAs shown, it is a structural diagram of end-to-end image compression based on deep learning. At the encoding end, the image is converted to the YUV color space, and the UV signal is down-sampled by one-half. Then, the Y signal and the UV signal are compressed and encoded respectively, and the Y signal stream and the UV signal stream are sent to the decoding end. At the decoding end, the Y signal stream is decoded and reconstructed to obtain the reconstructed Y signal, and the UV signal stream is decoded and reconstructed to obtain the reconstructed UV signal. Since the UV signal is compressed and encoded after being down-sampled by one-half, that is, the resolution of the UV signal is halved, therefore, the resolution of the reconstructed UV signal is halved, that is, the resolution of the reconstructed Y signal is H*W, and the resolution of the reconstructed UV signal is H / 2*W / 2, that is, the resolution of the reconstructed UV signal is half of the resolution of the reconstructed Y signal.

[0132] Since the resolution of the reconstructed UV signal is half of the resolution of the reconstructed Y signal, in the post-processing process, the resolution of the UV signal can also be restored, that is, the resolution of the UV signal is restored to H*W. When the resolution of the UV signal is restored, the calculation complexity of the resolution restoration process is high, and the demand for resources is large, and some cost and delay sensitive devices cannot restore the resolution of the UV chroma signal.

[0133] In view of the above finding, in the field of end-to-end image compression, a low-complexity and efficient post-processing method is proposed in the embodiments of the present disclosure, which can enhance the UV signal based on the Y signal in the post-processing process. The method has low complexity and high performance, can improve the decoding performance, reduce the decoding complexity, reduce the calculation complexity, has less demand for resources, and can be used by cost and delay sensitive devices. The algorithm can be used to restore compression defects under multiple code rates.

[0134] In the embodiments of the present disclosure, after the decoding end receives the code stream corresponding to the current image block, the decoding end can decode the code stream corresponding to the current image block to obtain a reconstructed image block. For example, the decoding end can use the processing procedure of Embodiment 3 to decode the code stream corresponding to the current image block to obtain a reconstructed image block. The decoding process of the reconstructed image block is not limited.

[0135] After obtaining the reconstructed image patch, if the reconstructed image patch is in YUV format, the chromaticity U component (i.e., U component) can be enhanced based on the luminance component (Y component), the chromaticity V component (i.e., V component) can be enhanced based on the Y component, or both the U and V components can be enhanced based on the Y component simultaneously. If the reconstructed image patch is in RGB format, the R component can be enhanced based on the G component, the B component can be enhanced based on the G component, or both the R and B components can be enhanced based on the G component simultaneously. For ease of description, this embodiment uses a YUV format reconstructed image patch as an example. Therefore, the reconstructed image patch can include Y, U, and V components. The resolution of the Y component can be greater than the resolution of the U component. For example, the resolution of the U component is the unsampled resolution, and the resolution of the Y component can be the original resolution or the downsampled resolution, but the resolution of the Y component is greater than the resolution of the U component. The resolution of the Y component can be greater than that of the V component. For example, the resolution of the V component is the unsampled resolution, while the resolution of the Y component can be either the original resolution or the downsampled resolution, but the resolution of the Y component is greater than that of the V component. For instance, suppose the resolution of the Y component is H*W, the resolution of the U component is H / 2*W / 2, and the resolution of the V component is H / 2*W / 2.

[0136] See Figure 5B The diagram illustrates the structure of a deep learning-based post-processing architecture. The decoder can decode the bitstream corresponding to the current image block to obtain a reconstructed image block. If the reconstructed image block is in YUV format, it can include a Y component, a U component, and a V component. The Y component is designated as the initial Y component (i.e., the first initial component in the above embodiment), the U component as the initial U component (i.e., the second initial component in the above embodiment), and the V component as the initial V component (i.e., the second initial component in the above embodiment). Based on this, auxiliary enhancement can be performed on the initial U component based on the initial Y component, or on the initial V component based on the initial Y component, or on both the initial U and initial V components based on the initial Y component. The resolution of the initial Y component can be greater than the resolution of the initial U component. For example, see [link to documentation]. Figure 5B As shown, the resolution of the initial Y component is H*W, the resolution of the initial U component is H / 2*W / 2, and the resolution of the initial V component is H / 2*W / 2.

[0137] In this embodiment, see Figure 5BAs shown, the pre-processing module is an optional module. After obtaining the Y initial component, if the pre-processing module is deployed, the Y initial component can be pre-processed by the pre-processing module to obtain a pre-processed Y initial component, and the pre-processed Y initial component is input to the Y auxiliary UV module and the YUV signal enhancement module. If the pre-processing module is not deployed, the Y initial component is directly input to the Y auxiliary UV module and the YUV signal enhancement module. In subsequent embodiments, the Y initial component obtained by the Y auxiliary UV module and the YUV signal enhancement module can be the pre-processed Y initial component or the original Y initial component obtained by decoding, which is not limited.

[0138] Since there is a certain correlation between the three YUV components, although the compression loss of the Y component is smaller than that of the UV component at the encoding end, the Y component can also be enhanced to better enhance the UV component. Therefore, the Y initial component can be pre-processed by the pre-processing module to enhance the characteristics of the Y initial component.

[0139] In this embodiment, referring to Figure 5B As shown, the Y auxiliary UV module is used. After obtaining the Y initial component, the U initial component and the V initial component, the U initial component and the V initial component have more loss, while the Y initial component retains relatively more image detail information, and the U initial component and the V initial component have a certain correlation with the Y initial component. Therefore, making full use of the information of the Y initial component can help improve the reconstruction quality of the U initial component and the V initial component.

[0140] Based on this, the Y auxiliary UV module can use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component, or use the information of the Y initial component to assist in improving the reconstruction quality of the V initial component, or use the information of the Y initial component to assist in improving the reconstruction quality of the U initial component and the V initial component.

[0141] In this embodiment, referring to Figure 5BAs shown, it relates to the resolution conversion module, since the resolution of the U initial component is less than the resolution of the Y initial component, such as the resolution of the U initial component is 2 times the resolution after downsampling, the resolution of the U initial component is half of the original signal resolution, such as H / 2*W / 2, therefore, the resolution conversion module up-samples the resolution of the U initial component, restores the U component with the original signal resolution, that is, up-samples the resolution of the U initial component to obtain the U component with the resolution of H*W. Since the resolution of the V initial component is less than the resolution of the Y initial component, such as the resolution of the V initial component is 2 times the resolution after downsampling, the resolution of the V initial component is half of the original signal resolution, such as H / 2*W / 2, therefore, the resolution conversion module up-samples the resolution of the V initial component, restores the V component with the original signal resolution, that is, up-samples the resolution of the V initial component to obtain the V component with the resolution of H*W.

[0142] The position of the resolution conversion module can be position 1, that is, the resolution conversion module up-samples the resolution of the U initial component to obtain the U component with the resolution of H*W, inputs the U component with the resolution of H*W to the Y auxiliary UV module, up-samples the resolution of the V initial component to obtain the V component with the resolution of H*W, and inputs the V component with the resolution of H*W to the Y auxiliary UV module. Or, the position of the resolution conversion module can be position 2, that is, the resolution conversion module up-samples the resolution of the U component output by the Y auxiliary UV module to obtain the U component with the resolution of H*W, inputs the U component with the resolution of H*W to the YUV signal enhancement module, up-samples the resolution of the V component output by the Y auxiliary UV module to obtain the V component with the resolution of H*W, and inputs the V component with the resolution of H*W to the YUV signal enhancement module. Or, the position of the resolution conversion module can be position 3, that is, the resolution conversion module up-samples the resolution of the U component output by the YUV signal enhancement module to obtain the U component with the resolution of H*W, and finally outputs the U component with the resolution of H*W, up-samples the resolution of the V component output by the YUV signal enhancement module to obtain the V component with the resolution of H*W, and finally outputs the V component with the resolution of H*W.

[0143] For the resolution conversion module, on the one hand, the resolution conversion module can appropriately reduce the resolution of the feature map, so as to effectively reduce the calculation complexity of the network. On the other hand, in the JPEG-AI framework, the resolution conversion module is used to up-sample the resolution of the reconstructed UV component, so as to restore the reconstructed U / V signal with the same resolution as the original signal.

[0144] In this embodiment, referring to Figure 5BAs shown, a YUV signal enhancement module is involved. During image encoding and compression, in addition to reducing the resolution of the UV signals, the compression algorithm and convolution process will cause additional information loss to the YUV signals. Therefore, the YUV signal enhancement module can be used to enhance the Y component, making the reconstructed Y component close to the original signal before compression, thereby improving the subjective quality. The YUV signal enhancement module can also be used to enhance the U component, making the reconstructed U component close to the original signal before compression, thereby improving the subjective quality. Finally, the YUV signal enhancement module can be used to enhance the V component, making the reconstructed V component close to the original signal before compression, thereby improving the subjective quality.

[0145] For example, see Figure 5B As shown, the three network branches (which process the Y, U, and V components respectively) can be merged into one network branch, that is, a single network can be used to process the Y, U, and V components simultaneously and output the reconstructed Y, U, and V components, thereby reducing the computational complexity of the network.

[0146] Example 5: In Example 4, the Y-assisted UV module can utilize information from the initial Y component to improve the reconstruction quality of the initial U and / or initial V components. For example, it can generate an adjusted component corresponding to the initial U component based on the initial Y and initial U components, and / or generate an adjusted component corresponding to the initial V component based on the initial Y and initial V components. The reconstruction quality of the adjusted component corresponding to the initial U component can be better than the reconstruction quality of the initial U component, thereby improving the reconstruction quality of the initial U component. The reconstruction quality of the adjusted component corresponding to the initial U component may also be equal to or lower than the reconstruction quality of the initial U component; this is not limited. Similarly, the reconstruction quality of the adjusted component corresponding to the initial V component can be better than the reconstruction quality of the initial V component, thereby improving the reconstruction quality of the initial V component. The reconstruction quality of the adjusted component corresponding to the initial V component may also be equal to or lower than the reconstruction quality of the initial V component; this is not limited.

[0147] In a possible implementation, the Y-assisted UV module can obtain a first image feature corresponding to the Y initial component. The first image feature can be an image feature that has not passed through a neural network. For example, the first image feature can include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y-assisted UV module can obtain a second image feature corresponding to the U initial component. The second image feature can be an image feature that has not passed through a neural network. For example, the second image feature can include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y-assisted UV module can generate an adjusted component corresponding to the U initial component based on the first image feature and the second image feature. For example, a texture fusion feature can be generated based on the texture feature corresponding to the Y initial component and the texture feature corresponding to the U initial component, the texture fusion feature is used to replace the texture feature corresponding to the U initial component, and an adjusted component corresponding to the U initial component is obtained. For another example, a subjective fusion feature can be generated based on the subjective feature corresponding to the Y initial component and the subjective feature corresponding to the U initial component, the subjective fusion feature is used to replace the subjective feature corresponding to the U initial component, and an adjusted component corresponding to the U initial component is obtained. For another example, a frequency domain fusion feature can be generated based on the frequency domain feature corresponding to the Y initial component and the frequency domain feature corresponding to the U initial component, the frequency domain fusion feature is used to replace the frequency domain feature corresponding to the U initial component, and an adjusted component corresponding to the U initial component is obtained. For another example, a histogram fusion feature can be generated based on the histogram feature corresponding to the Y initial component and the histogram feature corresponding to the U initial component, the histogram fusion feature is used to replace the histogram feature corresponding to the U initial component, and an adjusted component corresponding to the U initial component is obtained. For another example, a texture fusion feature can be generated based on the texture feature corresponding to the Y initial component and the texture feature corresponding to the U initial component, a frequency domain fusion feature can be generated based on the frequency domain feature corresponding to the Y initial component and the frequency domain feature corresponding to the U initial component, the texture fusion feature is used to replace the texture feature corresponding to the U initial component, and the frequency domain fusion feature is used to replace the frequency domain feature corresponding to the U initial component, and an adjusted component corresponding to the U initial component is obtained. The above are only a few examples, and the disclosure is not limited in this regard.

[0148] The Y-assisted UV module can obtain a first image feature corresponding to the Y initial component. The first image feature can be an image feature that has not passed through a neural network. For example, the first image feature can include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y-assisted UV module can obtain a second image feature corresponding to the V initial component. The second image feature can be an image feature that has not passed through a neural network. For example, the second image feature can include but is not limited to at least one of the following: a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y-assisted UV module can generate an adjusted component corresponding to the V initial component based on the first image feature and the second image feature. The generation manner can refer to the processing process of the U initial component.

[0149] In another possible implementation, the Y-assisted UV module can obtain a first image feature corresponding to the Y initial component, the first image feature can be an image feature after a neural network, i.e., an image feature output by the neural network, for example, the first image feature can include but is not limited to a first feature map, a weight coefficient map, etc. The Y-assisted UV module can obtain a second image feature corresponding to the U initial component, the second image feature can be an image feature after a neural network, i.e., an image feature output by the neural network, for example, the second image feature can include but is not limited to a second feature map, etc. The Y-assisted UV module can generate an adjusted component corresponding to the U initial component based on the first image feature and the second image feature. For example, an adjusted component corresponding to the U initial component can be generated based on the first feature map corresponding to the Y initial component and the second feature map corresponding to the U initial component. For another example, an adjusted component corresponding to the U initial component can be generated based on the weight coefficient map corresponding to the Y initial component and the second feature map corresponding to the U initial component. The above are only a few examples, which are not limited in this regard.

[0150] The Y-assisted UV module can obtain a first image feature corresponding to the Y initial component, the first image feature can be an image feature after a neural network, i.e., an image feature output by the neural network, for example, the first image feature can include but is not limited to a first feature map, a weight coefficient map, etc. The Y-assisted UV module can obtain a second image feature corresponding to the V initial component, the second image feature can be an image feature after a neural network, i.e., an image feature output by the neural network, for example, the second image feature can include but is not limited to a second feature map, etc. The Y-assisted UV module can generate an adjusted component corresponding to the V initial component based on the first image feature and the second image feature, for details of the generation method, see the processing process of the U initial component.

[0151] Embodiment 6: In order to make the UV component fully utilize the information of the Y component, in the input stage of the post-processing network, the Y component is convolved to obtain a feature map with multiple channels, these feature maps contain different types of features of the Y component, and these different types of features are added or concatenated with the feature map obtained by convolving the UV component, so that the subsequent neural network can learn multiple features of the Y component, thereby helping the UV component to improve the reconstruction quality.

[0152] For example, see Figure 6AAs shown in the figure, the structure diagram of the post-processing network, after obtaining the Y initial component, the U initial component and the V initial component, the Y initial component can be input into the first neural network (such as CNN1) to obtain the first feature map, which can also be called Y feature map (Y Feature Map). The U initial component can be input into the second neural network (such as CNN2) to obtain the second feature map corresponding to the U initial component, which can also be called U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain the second feature map corresponding to the V initial component, which can also be called V feature map (V Feature Map).

[0153] After obtaining the first feature map and the second feature map corresponding to the U initial component, the U initial component corresponding to the adjusted component can be generated based on the first feature map and the second feature map. For example, the first feature map and the second feature map are added, Figure 6A indicates the addition operation of the feature map, and the feature map after the addition operation is input into the third neural network (such as CNN3) to obtain the U initial component corresponding to the adjusted component. Or, the first feature map and the second feature map are stacked, Figure 6A indicates the stacking operation of the feature map, and the feature map after the stacking operation is input into the third neural network (such as CNN3) to obtain the U initial component corresponding to the adjusted component. After obtaining the first feature map and the second feature map corresponding to the V initial component, the V initial component corresponding to the adjusted component can be generated based on the first feature map and the second feature map. For example, the first feature map and the second feature map are added,

[0154] The stacking operation of the first feature map and the second feature map means that the first feature map and the second feature map are stacked along the channel dimension.

[0155] For example, in order to correctly implement the above operation, the number of convolution kernels of the second neural network can be the same as the number of convolution kernels of the first neural network, such as CNN1 and CNN2 having the same number of convolution kernels. Figure 6A The processing process shown can also be represented as: f CNN indicates the convolutional neural network, indicates the addition operation or stacking operation of the feature map. Y represents the Y initial component, f​​CNN1 (Y) represents the first feature map, U represents the initial component of U, and f CNN2 (U) represents the second feature map, V represents the initial component of V, and f CNN2 (V) represents the second feature map, U rec V represents the adjusted component corresponding to the initial component of U. rec This represents the adjusted component corresponding to the initial component of V.

[0156] Example 7: To utilize the information from each feature map of the Y component, guided filtering can be introduced. This involves using the features generated by the Y component to guide the generation of more representative feature maps for the UV components. Consider using attention to weight the pixel values ​​in the UV feature maps as guiding variables, thereby strengthening important features in the UV components and suppressing general features, thus achieving efficient feature conversion and fusion between the Y and UV components.

[0157] For example, see Figure 6B The diagram shows the structure of the post-processing network. After obtaining the initial components of Y, U, and V, the Y-assisted UV module can input the initial Y component into the first neural network (such as CNN1) to obtain the first feature map, which can also be called the Y feature map. The initial U component can be input into the second neural network (such as CNN2) to obtain the second feature map corresponding to the initial U component, which can also be called the U feature map. Similarly, the initial V component can be input into the second neural network (such as CNN2) to obtain the second feature map corresponding to the initial V component, which can also be called the V feature map.

[0158] After obtaining the first feature map and the second feature map corresponding to the initial U component, guided filtering can be applied to the first feature map to obtain a weight coefficient map. Based on this weight coefficient map and the second feature map, an adjusted component corresponding to the initial U component can be generated. For example, operations such as multiplication, addition, or concatenation can be performed on the weight coefficient map and the second feature map to obtain an output feature map. Specifically, multiplication of the weight coefficient map and the second feature map yields the output feature map; or multiplication and addition of the weight coefficient map and the second feature map yield the output feature map; or multiplication, addition, and concatenation of the weight coefficient map and the second feature map yield the output feature map. After obtaining the output feature map, it can be input into a third neural network (such as CNN3) to obtain the adjusted component corresponding to the initial U component.

[0159] After obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map can also be guided filtering to obtain a weight coefficient map, and based on the weight coefficient map and the second feature map, a component corresponding to the V initial component is generated after adjustment. For example, based on the weight coefficient map and the second feature map, at least one of the operations such as multiplication operation, addition operation, or splicing operation is performed to obtain an output feature map. After obtaining the output feature map, the output feature map can be input to a third neural network (such as CNN3) to obtain the component corresponding to the V initial component after adjustment.

[0160] Exemplary, Figure 6B The processing process shown can also be represented as: U rec / V rec = f CNN3 (f G (f CNN1 (Y), f CNN2 (U / V)), f CNN represents a convolutional neural network, f G represents guided filtering, and the guided filtering takes the feature maps of the Y component and the UV component as input and outputs a fused feature map after guided weighting. Y represents the Y initial component, f CNN1 (Y) represents the first feature map, U represents the U initial component, f CNN2 (U) represents the second feature map, V represents the V initial component, f CNN2 (V) represents the second feature map, U rec represents the component corresponding to the U initial component after adjustment, V rec represents the component corresponding to the V initial component after adjustment.

[0161] In one possible implementation, the implementation of guided filtering can refer to Figure 6C As shown, the first feature map (Y Feature Map) can be subjected to convolution operation to obtain a convolution feature map (hereinafter referred to as convolution feature map A). For example, the first feature map is input to the CNN network, and the CNN network is used to perform convolution operation on the first feature map to obtain the convolution feature map A. Then, the convolution feature map A is subjected to weight mapping to obtain a weight coefficient map after weight mapping.

[0162] When generating the adjusted component corresponding to the initial component U based on the weight coefficient map and the second feature map corresponding to the initial component U, the weight coefficient map and the second feature map (U Feature Map) corresponding to the initial component U can be multiplied to obtain the multiplied feature map. Then, the multiplied feature map and the convolutional feature map A are added to obtain the summed feature map. Then, the summed feature map and the convolutional feature map A are concatenated to obtain the output feature map, which is then input into a third neural network (such as CNN3) to obtain the adjusted component corresponding to the initial component U.

[0163] When generating the adjusted component corresponding to the initial V component based on the weight coefficient map and the second feature map corresponding to the initial V component, the weight coefficient map and the second feature map (V Feature Map) corresponding to the initial V component can be multiplied to obtain the multiplied feature map. Then, the multiplied feature map and the convolutional feature map A are added to obtain the summed feature map. Then, the summed feature map and the convolutional feature map A are concatenated to obtain the output feature map, which is then input into a third neural network (such as CNN3) to obtain the adjusted component corresponding to the initial V component.

[0164] See Figure 6C As shown, after passing the first feature map of the Y component through several convolutional layers, a convolutional feature map A is obtained. This convolutional feature map A is then mapped to specific weight factors, i.e., a weight coefficient map, through weight mapping. The weight coefficient map is multiplied by the second feature map of the UV component to obtain the guided feature map of the UV component. Then, the convolutional feature map A is added to the guided feature map of the UV component, and finally, the two maps are concatenated to output a fused feature map. For example, the guided filtering process can be represented as: Out = Concat((U / V)*f w (f CNN (Y))+f CNN (Y),f CNN (Y)), f w This represents a weight mapping, Concat(·) represents the concatenation operation, and f CNN (Y) represents the convolutional feature map A, and Out is the output feature map.

[0165] In one possible implementation, during the weight mapping process, the convolutional feature map A is weighted to obtain a weighted coefficient map. The implementation method for weight mapping can be found in [reference needed]. Figure 6D As shown, Figure 6DTwo weight mapping manners are shown in the middle. In the first weight mapping manner, the post-convolution feature map A is subjected to a pooling operation to obtain a post-pooling feature map. For example, the post-convolution feature map A with a size of C*H*W is subjected to a pooling operation (Pooling) to obtain a post-pooling feature map with a size of C*1*1, that is, the resolution of the feature map is reduced to 1*1 by the “Pooling” manner.

[0166] Then, the post-pooling feature map is subjected to a full connection operation and a ReLU activation operation to obtain a post- ReLU activation feature map. For example, the post-pooling feature map with a size of C*1*1 is subjected to a full connection operation (FC operation) and a ReLU activation operation (ReLU operation) to obtain a post- ReLU activation feature map with a size of C*1*1. The post- ReLU activation feature map is subjected to a full connection operation and a Sigmoid activation operation to obtain a post-Sigmoid activation feature map. For example, the post- ReLU activation feature map with a size of C*1*1 is subjected to a full connection operation (FC operation) and a Sigmoid activation operation (Sigmoid operation) to obtain a post-Sigmoid activation feature map with a size of C*1*1.

[0167] Then, the weight coefficient map is generated based on the post-convolution feature map A and the post-Sigmoid activation feature map, for example, the post-convolution feature map A and the post-Sigmoid activation feature map are multiplied to obtain the weight coefficient map. For example, the post-convolution feature map A with a size of C*H*W and the post-Sigmoid activation feature map with a size of C*1*1 are subjected to a multiplication operation to obtain a weight coefficient map with a size of C*H*W. Obviously, after the resolution of the feature map is reduced to 1*1 by the “Pooling” manner, the weight factors of each channel in the original input feature map are obtained through a plurality of full connection layers and a Sigmoid activation function, and these weight factors will give different weights to the original input feature map channels, and finally the weight coefficient map (also referred to as a weight feature map) is obtained.

[0168] In the second weight mapping manner, the post-convolution feature map A can be subjected to a convolution operation to obtain a post-convolution feature map B, for example, the post-convolution feature map A with a size of C*H*W is input into a CNN network, and the post-convolution feature map A is subjected to a convolution operation by a plurality of convolution layers of the CNN network to obtain a post-convolution feature map B with a size of C*H*W. Then, the post-convolution feature map B is subjected to a Sigmoid activation operation to obtain a weight coefficient map. For example, the post-convolution feature map B with a size of C*H*W is subjected to an activation operation by a Sigmoid activation function to obtain a weight coefficient map with a size of C*H*W. Obviously, the weight coefficient map is finally obtained by the Sigmoid activation function after the original input feature map passes through a plurality of convolution layers.

[0169] The difference between the first and second weight mapping methods is that the first method assigns different weights to the channels, while the second method assigns different weights to each pixel of each channel feature map.

[0170] Example 8: In Examples 6 and 7, feature extraction and information fusion of the Y component are performed at the input stage of the post-processing network. Unlike Examples 6 and 7, in Example 8, feature extraction and information fusion of the Y component are distributed across various stages of the post-processing network. Features of the UV components from different stages are fused with the features of the Y component, thereby improving the post-processing network's ability to enhance the UV component. For example, see... Figure 6E The diagram shows the structure of the post-processing network. After obtaining the initial Y, U, and V components, the Y-assisted UV module distributes the feature extraction and information fusion of the Y components across various stages of the post-processing network. B The feature fusion process can be implemented as described in Embodiment 6 or Embodiment 7. A fused feature map is output by fusing information from the Y component and UV components. After passing through several convolutional layers, this map serves as the input for the next feature fusion process. Similarly, the Y component also passes through several convolutional layers before serving as the input for the next feature fusion process. In Embodiment 8, multiple feature fusion processes are involved, thereby fusing the feature map of the Y component with the feature maps of the UV components from various stages of the network.

[0171] In one possible implementation, since the feature map of the Y component is fused with the feature maps of the UV components at each stage of the network, multiple first feature maps and multiple second feature maps are involved. For example, taking K first feature maps and K+1 second feature maps, where K can be a positive integer greater than 1, after obtaining the initial Y component and the initial U component, for the first first feature map and the first second feature map, the initial Y component is input into the neural network to obtain the first first feature map, and the initial U component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, where i ranges from 2 to K, the (i-1)-th first feature map is input into the neural network to obtain the i-th first feature map. Feature fusion is performed on the (i-1)-th first feature map and the (i-1)-th second feature map to obtain the fused feature, and the fused feature is input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map (i.e., the (K+1)-th second feature map), the adjusted component corresponding to the initial U component can be generated based on the last second feature map.

[0172] For example, taking K as 3 as an example, the Y initial component is input into the neural network a1 to obtain the first feature map b1, that is, the neural network a1 performs a plurality of convolution operations on the Y initial component to obtain the first feature map b1. The U initial component is input into the neural network c1 to obtain the second feature map d1, that is, the neural network c1 performs a plurality of convolution operations on the U initial component to obtain the second feature map d1.

[0173] The first feature map b1 is input into the neural network a2 to obtain the first feature map b2, that is, the neural network a2 performs a plurality of convolution operations on the first feature map b1 to obtain the first feature map b2. The first feature map b1 and the second feature map d1 are fused to obtain fused features, and the fused features are input into the neural network c2 to obtain the second feature map d2.

[0174] The first feature map b2 is input into the neural network a3 to obtain the first feature map b3, that is, the neural network a3 performs a plurality of convolution operations on the first feature map b2 to obtain the first feature map b3. The first feature map b2 and the second feature map d2 are fused to obtain fused features, and the fused features are input into the neural network c3 to obtain the second feature map d3.

[0175] Then, the first feature map b3 and the second feature map d3 are fused to obtain fused features, and the fused features are input into the neural network c4 to obtain the second feature map d4. After obtaining the second feature map d4, the adjusted component corresponding to the U initial component can be generated based on the second feature map d4, such as taking the second feature map d4 as the adjusted component corresponding to the U initial component, or performing operations on the second feature map d4 to obtain the adjusted component corresponding to the U initial component.

[0176] In the above process, the first feature map and the second feature map are fused to obtain fused features, which can include but is not limited to: performing an addition operation on the first feature map and the second feature map to obtain fused features; or performing a splicing operation on the first feature map and the second feature map to obtain fused features; or performing guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating fused features based on the weight coefficient map and the second feature map. Wherein, for the implementation of feature fusion, please refer to embodiments 6 and 7, which will not be repeated here.

[0177] After obtaining the Y initial component and the V initial component, the Y initial component is input into the neural network to obtain the first first feature map, and the V initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, i is in the range of 2-K, the (i-1)-th first feature map is input into the neural network to obtain the i-th first feature map, feature fusion is performed on the (i-1)-th first feature map and the (i-1)-th second feature map to obtain fused features, and the fused features are input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map, an adjusted component corresponding to the V initial component can be generated based on the last second feature map. The method for obtaining the adjusted component corresponding to the V initial component can refer to the processing process of the U initial component.

[0178] In embodiments 6, 7 and 8, the fusion method of the Y component guide map is designed. Different from embodiments 6, 7 and 8, in embodiment 9, the Y component is extracted from multiple dimensions for detailed feature extraction, and then fused into the post-processing network, so as to improve the enhancement capability of the network for UV components.

[0179] Referring to Figure 6F As shown in FIG. 9, it is a structural schematic diagram of the post-processing network. After obtaining the Y initial component, the U initial component and the V initial component, the Y auxiliary UV module first performs multi-dimensional feature extraction on the Y initial component, such as extracting M-dimensional image features from the Y initial component, M being a positive integer, and then stacking the M-dimensional image features to obtain the stacked multi-dimensional features. After obtaining the multi-dimensional features, the multi-dimensional features can be input into the first neural network to obtain the first feature map (Y Feature Map). In addition, the Y auxiliary UV module can also input the U initial component into the second neural network (such as CNN2) to obtain the second feature map corresponding to the U initial component, which can also be referred to as the U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain the second feature map corresponding to the V initial component, which can also be referred to as the V feature map (V Feature Map).

[0180] After obtaining the first feature map and the second feature map corresponding to the U initial component, feature fusion can be performed on the first feature map and the second feature map to obtain the adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, feature fusion can be performed on the first feature map and the second feature map to obtain the adjusted component corresponding to the V initial component. The feature fusion process can refer to embodiments 6, 7 and 8.

[0181] For example,Figure 6F The processing process shown can also be represented as: U rec / V rec = f CNN2 (f G (f A (Y), f CNN1 (U / V), f G represents a feature fusion process, f A represents a multi-dimensional feature extraction process of the Y component, in which the Y initial component is taken as input, and a plurality of feature maps of multiple dimensions are output, and each branch represents a feature of one dimension.

[0182] For example, referring to Figure 6G shown, a schematic diagram of a multi-dimensional feature extraction process is shown, in which the Y initial component is taken as input, and each branch feature extraction structure is used to extract the features of the Y initial component. For example, the Y initial component is input to the branch 1 feature extraction structure, and the first dimension feature of the Y initial component is extracted by the branch 1 feature extraction structure, the Y initial component is input to the branch 2 feature extraction structure, and the second dimension feature of the Y initial component is extracted by the branch 2 feature extraction structure, and so on, the Y initial component is input to the branch M feature extraction structure, and the Mth dimension feature of the Y initial component is extracted by the branch M feature extraction structure. In summary, M-dimensional image features can be extracted from the Y initial component, and the M-dimensional image features are stacked to obtain the stacked multi-dimensional features. After obtaining the stacked multi-dimensional features, the multi-dimensional features can be taken as the first feature map, or the multi-dimensional features can be input to the first neural network to obtain the first feature map.

[0183] In Figure 6G , a schematic diagram of a multi-dimensional feature extraction process is shown, in which the Y component is first passed through different feature extraction channels to obtain feature information on each dimension, and finally these feature information is concatenated to the output as the Y component feature map, and the multi-dimensional feature extraction process can be represented as: Out = Concat(f ch-1 (Y), f ch-2 (Y), …, f ch-M (Y)), where f ch- represents the i-th feature extraction structure for extracting the i-th dimension feature, and Concat(·) represents the stacking operation.

[0184] In a possible implementation, the image features of the M dimensions can include, but are not limited to, at least one of the following: image features obtained by convolving the Y initial component with a convolution kernel; image features obtained by cascading the Y initial component with two convolution kernels; first-order spatial domain features obtained by processing the Y initial component with a sobel operator; second-order spatial domain features obtained by processing the Y initial component with a Laplacian operator; or frequency domain features obtained by performing Fourier transform on the Y initial component. The above image features are only a few examples, and the type of the image features is not limited in this embodiment.

[0185] Referring to Figure 6H As shown in FIG. 10, which is a schematic diagram of a multi-dimensional feature extraction structure, the branch 1 feature extraction structure (i.e., the first branch) obtains feature information by using a 3*3 convolution, for example, obtains image features by convolving the Y initial component with a 3*3 convolution kernel. The branch 2 feature extraction structure (i.e., the second branch) obtains channel expansion compressed feature information by using a 1*1 and 3*3 convolution cascade, for example, obtains image features by cascading the Y initial component with a 1*1 and 3*3 convolution kernel. The branch 3 feature extraction structure (i.e., the third branch) obtains first-order spatial domain features by using a sobel operator, for example, obtains first-order spatial domain features by processing the Y initial component with a sobel operator. The branch 4 feature extraction structure (i.e., the fourth branch) obtains second-order spatial domain features by using a Laplacian operator, for example, obtains second-order spatial domain features by processing the Y initial component with a Laplacian operator. The branch 5 feature extraction structure (i.e., the fifth branch) obtains frequency domain features by using Fourier transform, for example, obtains frequency domain features by performing Fourier transform on the Y initial component. After obtaining the above five-dimensional features, the features can be spliced to obtain multi-dimensional features.

[0186] Embodiment 10: In Embodiments 6-9, the focus is on selecting which Y component features and how to fuse the Y component features with the UV component features. Unlike Embodiments 6-9, in Embodiment 10, the focus is on how to strengthen the features of the Y component. For example, after the YUV components of an image are compressed, the image details of the Y component are also lost to a certain extent, and therefore, the Y component and the features of the Y component can be enhanced. For example, although the compression loss of the Y component is smaller than that of the UV components, because there is a certain correlation between the YUV three components, the Y component can be enhanced to better enhance the UV components, and therefore, the Y initial component can be preprocessed by a preprocessing module to enhance the features of the Y initial component.

[0187] For example, after obtaining the Y initial component, the pre-processing module can first perform edge enhancement on the Y initial component to obtain an edge-enhanced image feature; then, multi-scale feature extraction is performed on the edge-enhanced image feature to obtain a multi-scale feature. After obtaining the multi-scale feature, the multi-scale feature can be taken as the pre-processed Y initial component, and the pre-processed Y initial component is input into the Y auxiliary UV module, and the Y auxiliary UV module inputs the pre-processed Y initial component into the first neural network to obtain a first feature map (Y Feature Map). In addition, the Y auxiliary UV module can also input the U initial component into the second neural network (such as CNN2) to obtain a second feature map corresponding to the U initial component, which can also be referred to as a U feature map (U Feature Map). The V initial component can be input into the second neural network (such as CNN2) to obtain a second feature map corresponding to the V initial component, which can also be referred to as a V feature map (V Feature Map).

[0188] After obtaining the first feature map and the second feature map corresponding to the U initial component, the first feature map and the second feature map can be fused to obtain an adjusted component corresponding to the U initial component, and after obtaining the first feature map and the second feature map corresponding to the V initial component, the first feature map and the second feature map can be fused to obtain an adjusted component corresponding to the V initial component. For the feature fusion process, please refer to embodiments 6, 7 and 8.

[0189] When the pre-processing module performs multi-scale feature extraction on the edge-enhanced image feature to obtain a multi-scale feature, a convolution operation can be performed on the edge-enhanced image feature to obtain a convolution feature; a down-sampling operation is performed on the convolution feature to obtain a down-sampled feature; a channel transformation is performed on the down-sampled feature to obtain a channel-transformed feature; an up-sampling operation is performed on the channel-transformed feature to obtain an up-sampled feature; and the multi-scale feature is generated based on the up-sampled feature and the channel-transformed feature.

[0190] Referring to Figure 6I As shown in FIG. 6, it is a structural schematic diagram of the post-processing network, which lists a multi-scale enhancement mode of Y component. In the down-sampling part, from D1 to D3, the resolution of the feature map gradually decreases, and the number of channels gradually increases. In the up-sampling part, from U3 to U1, the resolution of the feature map gradually increases, and the number of channels remains unchanged. D3, D2 and D1 are added to the up-sampled feature maps of U3, U2 and U1 respectively through channel transformation to obtain a multi-scale feature after multi-scale enhancement.

[0191] First, the Y initial component is edge enhanced to obtain an edge-enhanced image feature. For example, the Y initial component is subjected to image sharpening enhancement to obtain an edge-enhanced image feature. The edge enhancement method is not limited.

[0192] Taking 2 times of downsampling operation and 2 times of upsampling operation as an example, the number of downsampling operations can be more, and the number of upsampling operations can be more, and the same is not limited. After obtaining the edge-enhanced image feature, the edge-enhanced image feature is output to the CNN to obtain a feature D1 with a size of C*H*W. The feature D1 is subjected to a downsampling operation to obtain a feature D2 with a size of 4C*H / 2*W / 2. The feature D2 is subjected to a downsampling operation to obtain a feature D3 with a size of 16C*H / 4*W / 4.

[0193] The feature D3 is subjected to channel transformation to obtain a feature U3 with a size of T*H / 4*W / 4. The feature U3 is subjected to an upsampling operation. The feature D2 is subjected to channel transformation. The upsampling operation is performed on the feature U3 and the channel transformation is performed on the feature D2 to obtain a feature U2 with a size of T*H / 2*W / 2. The feature U2 is subjected to an upsampling operation. The feature D1 is subjected to channel transformation. The upsampling operation is performed on the feature U2 and the channel transformation is performed on the feature D1 to obtain a feature U1 with a size of T*H*W. Obviously, the feature U1 is a multi-scale feature after multi-scale enhancement, that is, the Y initial component after preprocessing. The preprocessing module will output the Y initial component after preprocessing. Through multi-scale feature extraction on the Y component, the first feature map output finally integrates the feature information at different scales, which helps to enhance the UV component.

[0194] In embodiment 11, the resolution conversion module is used to upsample the resolution of the U component to restore the U component with the original signal resolution, such as obtaining the U component with a resolution of H*W. The resolution conversion module is used to upsample the resolution of the V initial component to restore the V component with the original signal resolution, such as obtaining the V component with a resolution of H*W. For example, in the JPEG-AI framework, the original UV component is subjected to 2 times of downsampling processing, and then compressed and encoded by the encoding end. Therefore, the resolution of the UV component reconstructed by the decoding end is half of the original signal resolution. In the post-processing network, there is a step of upsampling the UV component, so as to restore the reconstructed UV component with the same resolution as the original signal. For example, when the resolution conversion module upsample the resolution of the U component and / or the V component, the upsample method can include but is not limited to: interpolation sampling, Pixel-shuffle upsample, DeConv, etc. The above are only a few examples, and the upsample method is not limited in this embodiment.

[0195] For example, since the resolution of the UV component is up-sampled, the computational complexity of the network model is affected, that is, the resolution increases, which leads to an increase in computational complexity, and the position of the resolution conversion module affects the computational complexity of the network model. Therefore, the position of the resolution conversion module can be involved to reduce the computational complexity of the network model.

[0196] Case 1: Referring to FIG. 6A, Figure 5B The position of the resolution conversion module can be position 1, that is, the resolution conversion module up-samples the resolution of the U initial component to obtain a U component with a resolution of H*W, inputs the U component with a resolution of H*W to the Y auxiliary UV module, up-samples the resolution of the V initial component to obtain a V component with a resolution of H*W, and inputs the V component with a resolution of H*W to the Y auxiliary UV module. On this basis, for embodiments 5-10, the U initial component is the up-sampled U component, the V initial component is the up-sampled V component, the resolution of the up-sampled U component can be equal to the resolution of the Y initial component, and the resolution of the up-sampled V component can be equal to the resolution of the Y initial component. In this way, embodiments 5-10 are performed based on the Y initial component and the up-sampled U component, and embodiments 5-10 are performed based on the Y initial component and the up-sampled V component.

[0197] As can be seen from the above, in case 1, up-sampling can be performed at the input end of the network, referring to FIG. 6B, Figure 7A that is, at the input end of the network, the U initial component can be up-sampled to obtain an up-sampled U component, such as a U component with a resolution of H*W, and at the input end of the network, the V initial component can be up-sampled to obtain an up-sampled V component, such as a V component with a resolution of H*W.

[0198] Case 2: Referring to FIG. 7A, Figure 5BAs shown, the position of the resolution conversion module can be position 2, that is, the resolution conversion module up-samples the resolution of the U component output by the Y auxiliary UV module to obtain a U component with a resolution of H*W, inputs the U component with the resolution of H*W to the YUV signal enhancement module, and up-samples the resolution of the V component output by the Y auxiliary UV module to obtain a V component with a resolution of H*W, and inputs the V component with the resolution of H*W to the YUV signal enhancement module. On this basis, for embodiments 5-10, the U initial component is the U component before up-sampling, and the V initial component is the V component before up-sampling. The resolution of the U component before up-sampling can be less than the resolution of the Y initial component, and the resolution of the V component before up-sampling can be less than the resolution of the Y initial component. In this way, embodiments 5-10 are performed based on the Y initial component and the U component before up-sampling, and embodiments 5-10 are performed based on the Y initial component and the V component before up-sampling. After obtaining the adjusted components corresponding to the U component / V component based on embodiments 5-10, the adjusted components can be up-sampled to obtain up-sampled adjusted components, and the up-sampled adjusted components are input to the YUV signal enhancement module. The resolution of the up-sampled adjusted components is equal to the resolution of the Y initial component.

[0199] As can be seen from the above, in case 2, the network can be progressively up-sampled layer by layer, see Figure 7B As shown, it is a schematic diagram of the network progressively up-sampling layer by layer, that is, at multiple network layers, the U initial component can be up-sampled to obtain an up-sampled U component, and the resolution of the up-sampled U component can be equal to the resolution of the Y initial component, such as the U component with the resolution of H*W. At multiple network layers, the V initial component can be up-sampled to obtain an up-sampled V component, and the resolution of the up-sampled V component can be equal to the resolution of the Y initial component, such as the V component with the resolution of H*W.

[0200] Case 3: see Figure 5BAs shown, the position of the resolution conversion module can be position 3, that is, the resolution conversion module up-samples the resolution of the U component output by the YUV signal enhancement module to obtain a U component with a resolution of H*W, and finally outputs this H*W U component, and up-samples the resolution of the V component output by the YUV signal enhancement module to obtain a V component with a resolution of H*W, and finally outputs this H*W V component. On this basis, for embodiments 5-10, the U initial component is the U component before up-sampling, the V initial component is the V component before up-sampling, the resolution of the U component before up-sampling is less than the resolution of the Y initial component, and the resolution of the V component before up-sampling is less than the resolution of the Y initial component. Therefore, embodiments 5-10 are performed based on the Y initial component and the U component before up-sampling, and embodiments 5-10 are performed based on the Y initial component and the V component before up-sampling. After obtaining the adjusted components corresponding to the U component / V component based on embodiments 5-10, the adjusted components are input to the YUV signal enhancement module, and the adjusted components are the U component / V component before up-sampling, which is less than the resolution of the Y initial component. Therefore, the YUV signal enhancement module performs signal enhancement based on the U component / V component before up-sampling to obtain the signal-enhanced U component / V component. After obtaining the signal-enhanced U component / V component, the U component / V component is up-sampled to obtain the up-sampled U component / V component, and the resolution of the up-sampled U component / V component is equal to the resolution of the Y initial component.

[0201] As can be seen from the above, in case 3, up-sampling can be performed at the output end of the network, as shown in Figure 7C As shown, it is a schematic diagram of up-sampling at the output end of the network, that is, at the output end of the network, the U component can be up-sampled to obtain an up-sampled U component, and the resolution of the up-sampled U component can be equal to the resolution of the Y initial component, such as a U component with a resolution of H*W, and at the output end of the network, the V component can be up-sampled to obtain an up-sampled V component, and the resolution of the up-sampled V component can be equal to the resolution of the Y initial component, such as a V component with a resolution of H*W.

[0202] Embodiment 12: In embodiment 4, the resolution conversion module is involved. In a first aspect, the resolution conversion module can appropriately reduce the resolution of the feature map by using a resolution conversion without information loss, thereby effectively reducing the computational complexity of the network. In a second aspect, the resolution conversion module can up-sample the resolution of the reconstructed U component and V component, thereby recovering the U component and V component with the same resolution as the original signal. The function of the second aspect is described in embodiment 11.

[0203] Regarding the first aspect of functionality, it can be implemented by a resolution transformation module or a Y-assisted UV module. For example, by reducing the resolution of the YUV components or intermediate components, the network complexity can be effectively reduced. A resolution upscaling operation can then be performed at the network output to restore the original resolution. For instance, wavelet transform (such as Haar wavelet transform) can be used to halve the resolution of the original image, thus reducing the network's computational complexity by almost four times. When the hardware has high computational complexity requirements, wavelet transform can be used multiple times consecutively to further reduce the resolution. For the several frequency bands resulting from the wavelet transform, all bands can be used, or only a portion of the bands can be selected.

[0204] For example, see Figure 7D The diagram shows the structure of the post-processing network, which can perform wavelet transform on the initial Y component. Figure 7D The DWT transform in the wavelet transform is used to obtain multiple frequency bands. These multiple frequency bands, or a portion of them (i.e., Y sub-bands), can be input into the first neural network to obtain the first feature map (Y Feature Map). Wavelet transform can be performed on the initial U components (…). Figure 7D The first feature map (U-feature map) is obtained by performing a wavelet transform on the initial V component to obtain multiple frequency bands. These multiple frequency bands, or a portion thereof (i.e., the U-subband), can be input into the second neural network to obtain the second feature map (V-feature map). After obtaining the first feature map and the second feature map corresponding to the initial U component, feature fusion can be performed to obtain the adjusted component corresponding to the initial U component. For details on this feature fusion process, please refer to Examples 6, 7, and 8.

[0205] It should be noted that when fusing the first and second feature maps to obtain the adjusted components corresponding to the initial U component, Examples 6, 7, and 8 are first used to obtain the fused features. Then, the fused features are input into the neural network (CNN3) to obtain the output features (U subband). Furthermore, an inverse wavelet transform (i.e., the inverse operation of the wavelet transform) can be performed on these output features. Figure 7DThe IDWT transform in the model is used to obtain the adjusted component of the U component. When fusing the first and second feature maps to obtain the adjusted component corresponding to the initial V component, Examples 6, 7, and 8 are first used to obtain the fused features. Then, the fused features are input into the neural network (CNN3) to obtain the output features (V subband). Furthermore, this output feature can be subjected to an inverse wavelet transform (i.e., the inverse operation of the wavelet transform). Figure 7D The IDWT transform in the process yields the adjusted V component.

[0206] Example 13: In Example 4, a YUV signal enhancement module is involved. This module enhances the Y component, making the reconstructed Y component approach the original signal before compression. Similarly, it enhances the U component, making the reconstructed U component approach the original signal before compression, and enhances the V component, making the reconstructed V component approach the original signal before compression. Based on this, the YUV signal enhancement module can perform feature enhancement on the initial Y component to obtain the target component corresponding to the Y component. The target component is the Y component after signal recovery. The YUV signal enhancement module can also perform feature enhancement on the adjusted U component to obtain the target component corresponding to the U component. The target component is the U component after signal recovery. Finally, the YUV signal enhancement module can perform feature enhancement on the adjusted V component to obtain the target component corresponding to the V component. The target component is the V component after signal recovery.

[0207] In one possible implementation, the initial component of Y can be feature-enhanced using at least one residual block network to obtain the target component corresponding to the Y component. Similarly, the adjusted component of U can be feature-enhanced using at least one residual block network to obtain the target component corresponding to the U component. Likewise, the adjusted component of V can be feature-enhanced using at least one residual block network to obtain the target component corresponding to the V component. See also Figure 7E The diagram shown illustrates the use of a residual block cascaded network to enhance the Y, U, and V components. The structure of this residual block cascaded network is not restricted.

[0208] In one possible implementation, the initial Y component can be feature-enhanced using a U-Net network to obtain the target component corresponding to the Y component. Similarly, the adjusted U component can be feature-enhanced using a U-Net network to obtain the target component corresponding to the U component. See also... Figure 7FAs shown, it is a schematic diagram of enhancing Y component, U component and V component by using U-Net network, the U-Net network includes a plurality of down-sampling network layers and a plurality of up-sampling network layers, and the structure of the U-Net network is not limited.

[0209] For example, each of the above embodiments can be implemented alone, or can be implemented in combination, for example, each of the embodiments 1-13 can be implemented alone, and at least two of the embodiments 1-13 can be implemented in combination.

[0210] For example, in each of the above embodiments, the content of the encoding end can also be applied to the decoding end, that is, the decoding end can be processed in the same way, and the content of the decoding end can also be applied to the encoding end, that is, the encoding end can be processed in the same way.

[0211] Based on the same application concept as the above method, the disclosure embodiments also propose a decoding device, which is applied to a decoding end, and can include a memory configured to store video data, and a decoder configured to implement the decoding method in the above embodiments 1-13, that is, the processing flow of the decoding end.

[0212] For example, in a possible implementation, the decoder is configured to implement:

[0213] decode a code stream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, the resolution of the first initial component being greater than or equal to the resolution of the second initial component;

[0214] generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0215] perform feature enhancement on the adjusted component to obtain a recovered target component corresponding to the second initial component.

[0216] For example, if the reconstructed image block is a YUV format reconstructed image block, the first initial component is a luminance component, and the second initial component is a chroma U component and / or a chroma V component; or, if the reconstructed image block is an RGB format reconstructed image block, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0217] In an example, the decoder is further configured to obtain a first image feature corresponding to the first initial component, the first image feature comprising a non-neural network image feature and / or a neural network image feature; obtain a second image feature corresponding to the second initial component, the second image feature comprising a non-neural network image feature and / or a neural network image feature; and generate the adjusted component based on the first image feature and the second image feature.

[0218] In an example, the decoder is further configured to, if the first image feature comprises a first feature map of a neural network and the second image feature comprises a second feature map of a neural network, input the first initial component into a first neural network to obtain the first feature map; input the second initial component into a second neural network to obtain the second feature map; perform an addition operation on the first feature map and the second feature map to obtain the adjusted component; or perform a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0219] In an example, the decoder is further configured to, if the first image feature comprises a weight coefficient map of a neural network and the second image feature comprises a second feature map of a neural network, input the first initial component into a first neural network to obtain a first feature map corresponding to the first initial component, and perform guided filtering on the first feature map to obtain the weight coefficient map; input the second initial component into a second neural network to obtain the second feature map; and generate the adjusted component based on the weight coefficient map and the second feature map.

[0220] In an example, the decoder is further configured to perform a convolution operation on the first feature map to obtain a convolutional feature map; and perform weight mapping on the convolutional feature map to obtain the weight coefficient map.

[0221] In an example, the decoder is further configured to perform a pooling operation on the convolutional feature map; perform a fully connected operation and a ReLU activation operation on the feature map after the pooling operation; perform a fully connected operation and a Sigmoid activation operation on the feature map after the ReLU activation operation; and generate the weight coefficient map based on the convolutional feature map and the feature map after the Sigmoid activation operation.

[0222] In an example, the decoder is further configured to perform a convolution operation and a Sigmoid activation operation on the convolutional feature map to obtain the weight coefficient map.

[0223] The decoder is further configured to perform a multiplication operation on the weight coefficient map and the second feature map to obtain a multiplied feature map, perform a convolution operation on the first feature map to obtain a convoluted feature map, perform an addition operation on the multiplied feature map and the convoluted feature map to obtain an added feature map, and perform a concatenation operation on the added feature map and the convoluted feature map to obtain the adjusted component.

[0224] The decoder is further configured to, if the first image feature includes K first feature maps obtained by a neural network and the second image feature includes K+1 second feature maps obtained by the neural network, input a first initial component into the neural network to obtain a first first feature map and input a second initial component into the neural network to obtain a first second feature map, for an i-th first feature map and an i-th second feature map, i is in a range of 2-K, K is a positive integer greater than 1, input an (i-1)-th first feature map into the neural network to obtain the i-th first feature map, perform feature fusion on the (i-1)-th first feature map and an (i-1)-th second feature map to obtain fused feature, and input the fused feature into the neural network to obtain the i-th second feature map, and generate the adjusted component based on a last second feature map after obtaining the last second feature map.

[0225] The decoder is further configured to extract M-dimensional image features from the first initial component, M is a positive integer, perform concatenation on the M-dimensional image features to obtain concatenated multi-dimensional features, and input the multi-dimensional features into a first neural network to obtain the first feature map.

[0226] The decoder is further configured to perform wavelet transform on the first initial component to obtain a plurality of wavelet-transformed frequency bands, input the plurality of wavelet-transformed frequency bands or part of the plurality of wavelet-transformed frequency bands into a first neural network to obtain the first feature map, perform wavelet transform on the second initial component to obtain a plurality of wavelet-transformed frequency bands, and input the plurality of wavelet-transformed frequency bands or part of the plurality of wavelet-transformed frequency bands into a second neural network to obtain the second feature map.

[0227] The decoder is further configured to perform preprocessing on the first initial component to obtain preprocessed first initial component, wherein the preprocessed first initial component is used to be input into the first neural network to obtain the first feature map, and the decoder is further configured to perform edge enhancement on the first initial component to obtain edge-enhanced image features, perform multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, and determine the preprocessed first initial component based on the multi-scale features.

[0228] In an example, the decoder is further configured to perform a convolution operation on the edge-enhanced image feature to obtain a convolutional feature, perform a down-sampling operation on the convolutional feature to obtain a down-sampled feature, perform a channel transformation on the down-sampled feature to obtain a channel-transformed feature, perform an up-sampling operation on the channel-transformed feature to obtain an up-sampled feature, and generate the multi-scale feature based on the up-sampled feature and the channel-transformed feature.

[0229] In an example, the decoder is further configured to perform an up-sampling operation on the second initial component to obtain an up-sampled second initial component before generating the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, and wherein a resolution of the up-sampled second initial component is equal to a resolution of the first initial component.

[0230] In an example, the decoder is further configured to perform an up-sampling operation on the adjusted component to obtain an up-sampled adjusted component before performing feature enhancement on the adjusted component to obtain the recovered target component corresponding to the second initial component, and wherein a resolution of the up-sampled adjusted component is equal to a resolution of the first initial component.

[0231] In an example, the decoder is further configured to perform an up-sampling operation on the target component to obtain an up-sampled target component after performing feature enhancement on the adjusted component to obtain the recovered target component corresponding to the second initial component, and wherein a resolution of the up-sampled target component is equal to a resolution of the first initial component.

[0232] In an example, the decoder is further configured to perform feature enhancement on the adjusted component by at least one residual block network to obtain the target component corresponding to the second initial component, or perform feature enhancement on the adjusted component by a U-Net network to obtain the target component corresponding to the second initial component.

[0233] Based on the same application concept as the above method, the decoding end device (which can also be referred to as a video decoder) provided by the embodiments of the present disclosure can be specifically referred to the hardware architecture diagram shown in FIG. 8. Figure 8 The decoding end device includes a processor 811 and a machine readable storage medium 812. The machine readable storage medium 812 stores machine executable instructions that can be executed by the processor 811. The processor 811 is configured to execute the machine executable instructions to implement the decoding method of the above embodiments 1-12 of the present disclosure.

[0234] For example, in a possible implementation, the processor 811, when executing the machine executable instructions, is configured to implement:

[0235] decode a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block comprising a first initial component and a second initial component, a resolution of the first initial component being greater than or equal to a resolution of the second initial component;

[0236] generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component;

[0237] perform feature enhancement on the adjusted component to obtain a recovered target component corresponding to the second initial component.

[0238] Based on the same application concept as the above method, the embodiment of the disclosure provides an electronic device. The electronic device comprises a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor; the processor is used to execute the machine executable instructions to implement the decoding method of the above embodiments 1-12 of the disclosure.

[0239] Based on the same application concept as the above method, the embodiment of the disclosure also provides a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, the computer instructions are executed by the processor, and the decoding method in the above embodiments can be implemented.

[0240] Based on the same application concept as the above method, the embodiment of the disclosure also provides a computer application program, the computer application program is executed by the processor, and the decoding method disclosed in the above embodiments can be implemented.

[0241] Based on the same application concept as the above method, the embodiment of the disclosure also proposes a decoding device, the decoding device can be applied to a decoding end (also called a video decoder), and the decoding device can comprise:

[0242] a decoding module, configured to decode a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block comprising a first initial component and a second initial component, a resolution of the first initial component being greater than or equal to a resolution of the second initial component; a determining module, configured to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and a processing module, configured to perform feature enhancement on the adjusted component to obtain a recovered target component corresponding to the second initial component.

[0243] For example, if the reconstructed image block is a YUV format reconstructed image block, the first initial component is a luminance component, and the second initial component is a chroma U component and / or a chroma V component; or, if the reconstructed image block is an RGB format reconstructed image block, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0244] For example, when the determining module generates the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the determining module specifically: obtains a first image feature corresponding to the first initial component, the first image feature including an image feature not passing through a neural network and / or an image feature passing through a neural network; obtains a second image feature corresponding to the second initial component, the second image feature including an image feature not passing through a neural network and / or an image feature passing through a neural network; and generates the adjusted component based on the first image feature and the second image feature.

[0245] For example, if the first image feature includes an image feature not passing through a neural network, the first image feature includes at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature; and if the second image feature includes an image feature not passing through a neural network, the second image feature includes at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature.

[0246] For example, if the first image feature includes a first feature map passing through a neural network, and the second image feature includes a second feature map passing through a neural network, when the determining module generates the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the determining module specifically: inputs the first initial component into a first neural network to obtain the first feature map; inputs the second initial component into a second neural network to obtain the second feature map; and performs an addition operation on the first feature map and the second feature map to obtain the adjusted component; or performs a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0247] For example, if the first image feature includes a weight coefficient map passing through a neural network, and the second image feature includes a second feature map passing through a neural network, when the determining module generates the adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the determining module specifically: inputs the first initial component into a first neural network to obtain a first feature map corresponding to the first initial component, and performs guided filtering on the first feature map to obtain the weight coefficient map; inputs the second initial component into a second neural network to obtain the second feature map; and generates the adjusted component based on the weight coefficient map and the second feature map.

[0248] Illustratively, the determining module is specifically configured to: perform convolution operation on the first feature map to obtain a convolutional feature map; and perform weight mapping on the convolutional feature map to obtain the weight coefficient map.

[0249] Illustratively, the determining module is specifically configured to: perform pooling operation on the convolutional feature map; perform full connection operation and rectified linear unit (ReLU) activation operation on a feature map after the pooling operation; perform full connection operation and sigmoid activation operation on a feature map after the ReLU activation operation; and generate the weight coefficient map based on the convolutional feature map and a feature map after the sigmoid activation operation.

[0250] Illustratively, the determining module is specifically configured to: perform convolution operation and sigmoid activation operation on the convolutional feature map to obtain the weight coefficient map.

[0251] Illustratively, the determining module is specifically configured to: perform multiplication operation on the weight coefficient map and the second feature map to obtain a multiplied feature map; perform convolution operation on the first feature map to obtain a convolutional feature map; perform addition operation on the multiplied feature map and the convolutional feature map to obtain an added feature map; and perform concatenation operation on the added feature map and the convolutional feature map to obtain the adjusted component.

[0252] Illustratively, if the first image feature includes K first feature maps after passing through a neural network and the second image feature includes K+1 second feature maps after passing through the neural network, the determining module is specifically configured to: for a first first feature map and a first second feature map, input the first initial component into the neural network to obtain the first first feature map and input the second initial component into the neural network to obtain the first second feature map; for an i-th first feature map and an i-th second feature map, i is in a range of 2 to K, K is a positive integer greater than 1, input an (i-1)-th first feature map into the neural network to obtain the i-th first feature map; perform feature fusion on the (i-1)-th first feature map and an (i-1)-th second feature map to obtain a fused feature, and input the fused feature into the neural network to obtain the i-th second feature map; and after obtaining a last second feature map, generate the adjusted component based on the last second feature map.

[0253] Illustratively, the determining module is specifically configured to: perform addition operation on the first feature map and the second feature map to obtain the fused feature; or perform concatenation operation on the first feature map and the second feature map to obtain the fused feature; or perform guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generate the fused feature based on the weight coefficient map and the second feature map.

[0254] Illustratively, the determining module is specifically configured to: extract image features of M dimensions from the first initial component, where M is a positive integer; perform concatenation on the image features of M dimensions to obtain multi-dimensional features after concatenation; and input the multi-dimensional features into the first neural network to obtain the first feature map.

[0255] Illustratively, the image features of M dimensions include at least one of: image features after convolution of the first initial component by using one convolution kernel; image features after convolution cascade of the first initial component by using two convolution kernels; first-order spatial domain features after processing of the first initial component by using a Sobel operator; second-order spatial domain features after processing of the first initial component by using a Laplacian operator; or frequency domain features after Fourier transform of the first initial component.

[0256] Illustratively, the determining module is specifically configured to: perform wavelet transform on the first initial component to obtain a plurality of frequency bands after wavelet transform; and input the plurality of frequency bands or part of the plurality of frequency bands into the first neural network to obtain the first feature map. The determining module is specifically configured to: perform wavelet transform on the second initial component to obtain a plurality of frequency bands after wavelet transform; and input the plurality of frequency bands or part of the plurality of frequency bands into the second neural network to obtain the second feature map.

[0257] Illustratively, before the determining module inputs the first initial component into the first neural network to obtain the first feature map, the determining module is further configured to: pre-process the first initial component to obtain a pre-processed first initial component; and input the pre-processed first initial component into the first neural network to obtain the first feature map. When pre-processing the first initial component, the determining module is specifically configured to: perform edge enhancement on the first initial component to obtain an edge-enhanced image feature; perform multi-scale feature extraction on the edge-enhanced image feature to obtain a multi-scale feature; and determine the pre-processed first initial component based on the multi-scale feature.

[0258] Illustratively, when the determining module performs multi-scale feature extraction on the edge-enhanced image feature to obtain a multi-scale feature, the determining module is specifically configured to: perform convolution operation on the edge-enhanced image feature to obtain a convolutional feature; perform down-sampling operation on the convolutional feature to obtain a down-sampled feature; perform channel transformation on the down-sampled feature to obtain a channel-transformed feature; perform up-sampling operation on the channel-transformed feature to obtain an up-sampled feature; and generate the multi-scale feature based on the up-sampled feature and the channel-transformed feature.

[0259] Illustratively, before the processing module generates the second initial component corresponding to the adjusted component based on the first initial component and the second initial component, the processing module is further configured to: up-sample the second initial component to obtain an up-sampled second initial component; and wherein a resolution of the up-sampled second initial component is equal to a resolution of the first initial component.

[0260] Illustratively, before the processing module performs feature enhancement on the adjusted component to obtain the second initial component corresponding to the restored target component, the processing module is further configured to: up-sample the adjusted component to obtain an up-sampled adjusted component; and wherein a resolution of the up-sampled adjusted component is equal to a resolution of the first initial component.

[0261] Illustratively, after the processing module performs feature enhancement on the adjusted component to obtain the second initial component corresponding to the restored target component, the processing module is further configured to: up-sample the target component to obtain an up-sampled target component; and wherein a resolution of the up-sampled target component is equal to a resolution of the first initial component.

[0262] Illustratively, when the processing module performs feature enhancement on the adjusted component to obtain the recovered target component corresponding to the second initial component, the processing module is specifically configured to: perform feature enhancement on the adjusted component by using at least one residual block network to obtain the target component; or perform feature enhancement on the adjusted component by using a U-Net network to obtain the target component.

[0263] Those skilled in the art will appreciate that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. The embodiments of the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. The above is only an embodiment of the present disclosure and is not intended to limit the present disclosure.

[0264] The present disclosure can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. An image decoding method, characterized in that, include: The bitstream corresponding to the current image block is decoded to obtain a reconstructed image block. The reconstructed image block includes an initial luminance component and an initial chrominance component. The initial chrominance component includes a first chrominance component and a second chrominance component. The reconstructed image block is a YUV format reconstructed image block. The initial luminance component is the Y component. The first chrominance component included in the initial chrominance component is the U component. The second chrominance component included in the initial chrominance component is the V component. The initial luminance component is subjected to Haar wavelet transform processing to obtain the Haar wavelet frequency domain features of the initial luminance component; the first chrominance component is subjected to Haar wavelet transform processing to obtain the Haar wavelet frequency domain features of the first chrominance component. Based on the Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chrominance component, the frequency domain features of the first chrominance component are adjusted to obtain the adjusted first chrominance component.

2. The method according to claim 1, characterized in that, The step of adjusting the frequency domain features of the first chromaticity component based on the Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chromaticity component to obtain the adjusted first chromaticity component includes: The Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chrominance component are concatenated to generate a frequency domain fusion feature; based on the first chrominance component, the frequency domain features corresponding to the first chrominance component are replaced with the frequency domain fusion feature to obtain the adjusted first chrominance component.

3. The method according to claim 1, characterized in that, in, The resolution of the initial luminance component is greater than or equal to the resolution of the first chrominance component, and the resolution of the initial luminance component is greater than or equal to the resolution of the second chrominance component.

4. The method according to claim 1, characterized in that, The step of performing Haar wavelet transform on the initial luminance component to obtain the Haar wavelet frequency domain features of the initial luminance component includes: performing at least one Haar wavelet transform on the initial luminance component to obtain multiple frequency bands after at least one Haar wavelet transform of the initial luminance component; and determining the Haar wavelet frequency domain features of the initial luminance component based on the multiple frequency bands after at least one Haar wavelet transform of the initial luminance component. The step of performing Haar wavelet transform on the first chromaticity component to obtain the Haar wavelet frequency domain features of the first chromaticity component includes: performing at least one Haar wavelet transform on the first chromaticity component to obtain multiple frequency bands after at least one Haar wavelet transform of the first chromaticity component; and determining the Haar wavelet frequency domain features of the first chromaticity component based on the multiple frequency bands after at least one Haar wavelet transform of the first chromaticity component.

5. The method according to claim 4, characterized in that, Perform at least one Haar wavelet transform on the initial brightness component, including: A first Haar wavelet transform is performed on the initial luminance component to obtain multiple frequency bands of the initial luminance component after the first Haar wavelet transform; a second Haar wavelet transform is performed on the multiple frequency bands of the initial luminance component after the first Haar wavelet transform to obtain multiple frequency bands of the initial luminance component after the two Haar wavelet transforms. Perform at least one Haar wavelet transform on the first chromaticity component, including: Perform a first Haar wavelet transform on the first chromaticity component to obtain multiple frequency bands of the first chromaticity component after the first Haar wavelet transform; perform a second Haar wavelet transform on the multiple frequency bands of the first Haar wavelet transform to obtain multiple frequency bands of the first chromaticity component after the two Haar wavelet transforms.

6. The method according to claim 1, characterized in that, After adjusting the frequency domain features of the first chromaticity component based on the Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chromaticity component to obtain the adjusted first chromaticity component, the method further includes: The adjusted first chromaticity component is enhanced using a residual block network to obtain the recovered target chromaticity component corresponding to the first chromaticity component.

7. An image encoding method, characterized in that, include: Obtain the reconstructed image block corresponding to the current image block. The reconstructed image block includes an initial luminance component and an initial chrominance component. The initial chrominance component includes a first chrominance component and a second chrominance component. The reconstructed image block is a YUV format reconstructed image block. The initial luminance component is the Y component. The first chrominance component included in the initial chrominance component is the U component. The second chrominance component included in the initial chrominance component is the V component. The initial luminance component is subjected to Haar wavelet transform processing to obtain the Haar wavelet frequency domain features of the initial luminance component; the first chrominance component is subjected to Haar wavelet transform processing to obtain the Haar wavelet frequency domain features of the first chrominance component. Based on the Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chrominance component, the frequency domain features of the first chrominance component are adjusted to obtain the adjusted first chrominance component.

8. An image decoding device, characterized in that, include: The decoding module is used to decode the bitstream corresponding to the current image block to obtain a reconstructed image block. The reconstructed image block includes an initial luminance component and an initial chrominance component. The initial chrominance component includes a first chrominance component and a second chrominance component. The reconstructed image block is a YUV format reconstructed image block. The initial luminance component is the Y component. The first chrominance component included in the initial chrominance component is the U component. The second chrominance component included in the initial chrominance component is the V component. The determining module is used to perform Haar wavelet transform processing on the initial luminance component to obtain the Haar wavelet frequency domain features of the initial luminance component; to perform Haar wavelet transform processing on the first chrominance component to obtain the Haar wavelet frequency domain features of the first chrominance component; and to adjust the frequency domain features of the first chrominance component based on the Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chrominance component to obtain the adjusted first chrominance component.

9. An image encoding device, characterized in that, include: The acquisition module is used to acquire the reconstructed image block corresponding to the current image block. The reconstructed image block includes an initial luminance component and an initial chrominance component. The initial chrominance component includes a first chrominance component and a second chrominance component. The reconstructed image block is a YUV format reconstructed image block. The initial luminance component is the Y component. The first chrominance component included in the initial chrominance component is the U component. The second chrominance component included in the initial chrominance component is the V component. The determining module is used to perform Haar wavelet transform processing on the initial luminance component to obtain the Haar wavelet frequency domain features of the initial luminance component; to perform Haar wavelet transform processing on the first chrominance component to obtain the Haar wavelet frequency domain features of the first chrominance component; and to adjust the frequency domain features of the first chrominance component based on the Haar wavelet frequency domain features of the initial luminance component and the Haar wavelet frequency domain features of the first chrominance component to obtain the adjusted first chrominance component.

10. An image decoding device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1-6.

11. An image encoding device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of claim 7.

12. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores a plurality of computer instructions, which, when executed by a processor, implement the method described in any one of claims 1-6, or, when executed by a processor, implement the method described in claim 7.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images