Encoding and decoding method, device, and equipment

Abstract

The present invention provides a decoding method, apparatus, and device thereof, which includes the steps of: decoding a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, where the resolution of the first initial component is equal to or greater than the resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component. The technical solution of the present invention can improve decoding performance and reduce decoding complexity.

Description

[Technical Field]

[0001] The present invention relates to the technical field of encoding and decoding, and in particular to an encoding and decoding method, apparatus and device thereof. [Background technology]

[0002] To achieve the purpose of space saving, all video images are coded before transmission, and the complete video coding process may include prediction, transformation, quantization, entropy coding, filtering, etc. The prediction process may include intra-prediction and inter-prediction. Inter-prediction utilizes the temporal correlation of video and predicts a current pixel using pixels of adjacent coded images to achieve the purpose of efficiently removing redundancy in the temporal domain of video. Intra-prediction utilizes the spatial correlation of video and predicts a current pixel using pixels of coded blocks of a current frame image to achieve the purpose of removing redundancy in the spatial domain of video.

[0003] With the rapid development of deep learning, deep learning has been successfully applied to many high-level computer vision tasks, such as image classification and object detection, and has also been applied to the coding and decoding fields. That is, neural networks can be used to code and decode images. However, although neural network-based coding and decoding methods show high performance potential, they still suffer from problems such as poor decoding performance and high computational complexity. Summary of the Invention

[0004] In view of the above circumstances, the present invention provides an encoding and decoding method, an apparatus, and an apparatus therefor that can improve decoding performance.

[0005] According to a first aspect, the present invention provides a decoding method, the method comprising: decoding a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, and a resolution of the first initial component is equal to or greater than a resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0006] In some embodiments, the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a luma component, and the second initial component is a chroma U component and a chroma V component.

[0007] In some embodiments, the step of generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes: obtaining a first image feature corresponding to the first initial component; obtaining a second image feature corresponding to the second initial component; generating the adjusted component based on the first image feature and the second image feature; the first image features include image features that have not been processed by a neural network; The second image features include image features that have not been processed by a neural network.

[0008] In some embodiments, the first image features include Haar wavelet frequency domain features and the second image features include Haar wavelet frequency domain features.

[0009] In some embodiments, the step of generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes: performing at least one Haar wavelet transform on the first initial component, and acquiring a plurality of frequency bands after the transform as a first feature map; performing at least one Haar wavelet transform on the second initial component, and acquiring a plurality of frequency bands after the transform as a second feature map; performing a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0010] In some embodiments, performing a concatenation operation on the first feature map and the second feature map comprises performing a concatenation operation on the first feature map and the second feature map along a channel dimension.

[0011] In some embodiments, performing at least one Haar wavelet transform on the first initial component to obtain a plurality of transformed frequency bands includes performing two Haar wavelet transforms on the first initial component to obtain a plurality of frequency bands obtained by the two Haar wavelet transforms; Performing at least one Haar wavelet transform on the second initial component and obtaining a plurality of frequency bands after the transform includes performing two Haar wavelet transforms on the second initial component and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms.

[0012] In some embodiments, performing a Haar wavelet transform on the first initial component multiple times to obtain a plurality of frequency bands after the transform includes performing a Haar wavelet transform on the first initial component twice to obtain a plurality of frequency bands obtained by the two Haar wavelet transforms; Performing a Haar wavelet transform multiple times on the second initial component and obtaining multiple frequency bands after the transform includes performing a Haar wavelet transform twice on the second initial component and obtaining multiple frequency bands obtained by the two Haar wavelet transforms.

[0013] In some embodiments, performing two Haar wavelet transforms on the first initial component and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms includes performing a first Haar wavelet transform on the first initial component and obtaining a plurality of frequency bands obtained by the first Haar wavelet transform, and performing a second Haar wavelet transform on the plurality of frequency bands obtained by the first Haar wavelet transform and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms; Performing two Haar wavelet transforms on the second initial component and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms includes performing a first Haar wavelet transform on the second initial component and obtaining a plurality of frequency bands obtained by the first Haar wavelet transform, and performing a second Haar wavelet transform on the plurality of frequency bands obtained by the first Haar wavelet transform and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms.

[0014] In some embodiments, performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component includes performing a feature enhancement process on the adjusted component using a plurality of residual block networks to obtain the target component.

[0015] According to a second aspect, the present invention provides an encoding method, the method comprising: encoding a first initial component and a second initial component of a current image block to obtain a bitstream corresponding to the current image block; The resolution of the first initial component is equal to or greater than the resolution of the second initial component.

[0016] In some embodiments, the current image block is in YUV format, the first initial component is a luma component, and the second initial component is a chroma U component and a chroma V component.

[0017] According to a third aspect, the present invention provides a decoding device, the device comprising: a decoding module configured to decode a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, wherein a resolution of the first initial component is equal to or greater than a resolution of the second initial component; a determination module configured to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and a processing module configured to perform a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0018] According to a fourth aspect, the present invention provides an encoding device, the device comprising: an encoding module configured to encode a first initial component and a second initial component of a current image block to obtain a bitstream corresponding to the current image block; The resolution of the first initial component is equal to or greater than the resolution of the second initial component.

[0019] According to a fifth aspect, the present invention provides a decoding device comprising a processor and a machine-readable storage medium having stored thereon machine-executable instructions executable by the processor, the processor being configured to perform a method according to any of the embodiments of the first aspect by executing the machine-executable instructions.

[0020] According to a sixth aspect, the present invention provides an encoding side device comprising a processor and a machine-readable storage medium having stored thereon machine-executable instructions executable by said processor, said processor being configured to perform a method according to any of the embodiments of the second aspect by executing said machine-executable instructions.

[0021] According to a seventh aspect, the present invention provides a machine-readable storage medium having stored thereon a number of computer instructions which, when executed by a processor, perform a method according to any embodiment of the first aspect, or perform a method according to any embodiment of the second aspect.

[0022] According to an eighth aspect, the present invention provides a computer program product comprising a computer program which, when executed by a processor, performs a method according to any embodiment of the first aspect, or performs a method according to any embodiment of the second aspect. [Effects of the Invention]

[0023] As can be seen from the above technical solution, in an embodiment of the present invention, after obtaining a reconstructed image block through decoding, the reconstructed image block includes a first initial component and a second initial component, the resolution of the first initial component is equal to or greater than the resolution of the second initial component, and an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, i.e., the second initial component is additionally adjusted based on the first initial component to obtain the adjusted component, and a target component corresponding to the second initial component is determined based on the adjusted component, thereby improving decoding performance, reducing decoding complexity, and reducing computational complexity, providing a low-complexity and highly efficient post-processing network model framework with relatively low resource requirements and applicable to cost- and delay-sensitive devices, and recovering compression defects under multiple bit rates. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 2 is a diagram of a three-dimensional feature matrix in one embodiment of the present invention. [Figure 2] 1 is a flowchart of a decoding method according to an embodiment of the present invention. [Figure 3] FIG. 10 is a schematic diagram of the encoding process in one embodiment of the present invention. [Figure 4] FIG. 10 is a schematic diagram of a decoding process in one embodiment of the present invention. [Figure 5A] FIG. 1 is a structural diagram of end-to-end image compression in one embodiment of the present invention. [Figure 5B] FIG. 1 is a structural diagram of deep learning-based post-processing in one embodiment of the present invention. [Figure 6A] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6B] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6C] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6D] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6E]FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6F] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6G] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6H] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 6I] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 7A] FIG. 1 is a structural diagram of upsampling in one embodiment of the present invention. [Figure 7B] FIG. 1 is a structural diagram of upsampling in one embodiment of the present invention. [Figure 7C] FIG. 1 is a structural diagram of upsampling in one embodiment of the present invention. [Figure 7D] FIG. 2 is a structural diagram of a post-processing network in one embodiment of the present invention. [Figure 7E] FIG. 1 is a structural diagram of a feature enhancement process in one embodiment of the present invention. [Figure 7F] FIG. 1 is a structural diagram of a feature enhancement process in one embodiment of the present invention. [Figure 8] FIG. 2 is a hardware structural diagram of a decoding device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0025] The terms used in the embodiments of the present invention are merely for the purpose of describing particular embodiments and are not intended to limit the present invention. As used in the embodiments and claims of the present invention, the singular forms "a," "the," and "said" are intended to include the plural forms unless the context clearly dictates otherwise. The term "and / or" as used herein should be understood to mean any and all possible combinations of one or more of the associated listed items. While the embodiments of the present invention may use terms such as "first," "second," and "third" to describe various pieces of information, it should be understood that such information should not be limited to these terms. These terms are used only to distinguish between pieces of information of the same type. For example, depending on the context, first information may be referred to as second information, and similarly, second information may be referred to as first information, without departing from the scope of the embodiments of the present invention. Furthermore, the word "if" may be interpreted as "when," "if," or "in response to determining."

[0026] An embodiment of the present invention provides a decoding method, which relates to the following concept.

[0027] JPEG (Joint Photographic Experts Group): JPEG is a standard for continuous-tone still image compression, and the file extensions may be .jpg or .jpeg. JPEG is a common image file format. JPEG uses an integrated coding method that combines predictive coding (DPCM), discrete cosine transform (DCT), and entropy coding to remove redundant image and color data. It is a lossy compression format, allowing images to be compressed into small storage spaces, but this results in degradation of image data. In particular, applying an excessively high compression rate can result in degradation of the quality of the final decompressed image. Therefore, using a high compression rate is not recommended when pursuing high-quality images.

[0028] JPEG-AI (Joint Photographic Experts Group Artificial Intelligence): The scope of JPEG-AI is to develop a learning-based image coding standard that provides a single-stream, compact compressed-domain representation. It significantly improves compression efficiency over existing image coding standards at comparable subjective image quality, effectively improving performance in image processing and computer vision tasks. JPEG-AI is targeted at a wide range of applications, including cloud storage, visual management, autonomous vehicles and equipment, image collection, storage, and management, real-time management of visual data, and media distribution. The goal of JPEG-AI is to design an encoding / decoding solution that significantly improves compression efficiency while maintaining the same subjective image quality, providing effective compressed-domain processing for machine learning-based image processing and computer vision tasks. JPEG-AI must achieve encoding and decoding suitable for hardware and software implementation, support 8-bit and 10-bit depth, and perform highly efficient encoding and progressive decoding for images with text and graphics.

[0029] Post-processing filter: The encoding side encodes and compresses an image to form a series of bitstreams, and the decoding side decodes the bitstreams to reconstruct the image. However, due to the characteristics of the encoding algorithm, defects such as block artifacts, image artifacts, and chromaticity shifts occur in the image after decoding the bitstream and reconstructing the image. The post-processing filter aims to improve defects caused by image encoding and compression, restore image data to the maximum extent possible, and improve the subjective image quality when the image is decoded and reconstructed.

[0030] Image Super-Resolution: Super-resolution is a technology that restores low-resolution images to high-resolution images, enhances image details, and improves subjective image quality. Image super-resolution is a technology used to improve image resolution in computer vision image processing.

[0031] YUV Color Space: Each pixel in a color image can usually be described by several independent physical quantities, which form spatial coordinates, which is the color space of the image. YUV color space is a color space in which a pixel in a color image is described by three attributes: Y (luminance), U (chromaticity), and V (chrominance).

[0032] Entropy coding is a coding method that does not involve information loss during the coding process according to the entropy principle, and information entropy represents the average amount of information (a measure of uncertainty) of the information source. Entropy coding methods include, but are not limited to, Shannon coding, Huffman coding, and arithmetic coding.

[0033] Neural network (NN): A neural network refers to an artificial neural network. It is a computational model composed of numerous interconnected nodes (also called neurons). In a neural network, neuron processing units can represent different objects, such as features, alphabets, concepts, or several meaningful abstractions. The processing units in a neural network can be divided into three types: input units, output units, and hidden units. Input units receive external signals and data, output units output the processing results, and hidden units are located between the input and output units and cannot be observed from outside the system. The connection weights between neurons reflect the connection strength between units, and information representation and processing are reflected in the connection relationships between processing units. A neural network is a non-programmable, brain-like information processing method. The essence of a neural network is to obtain parallel, distributed information processing functions through the transformation and dynamic behavior of neural networks, imitating the information processing functions of the human brain and nervous system to different degrees and levels. In the video processing field, commonly used neural networks include, but are not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), fully connected networks, etc.

[0034] Convolutional Neural Network (CNN): A convolutional neural network is a forward propagation neural network and is one of the most common network structures in deep learning technology. The artificial neurons of a convolutional neural network can respond to surrounding units within a limited coverage range and have excellent representation for large-scale image processing. The basic structure of a convolutional neural network includes two layers. The first is a feature extraction layer (also called a convolutional layer), in which the input of each neuron is connected to the local receptive field of the previous layer to extract the local feature. Once the local feature is extracted, its positional relationship with other features is also determined accordingly. The second is a feature mapping layer (also called an activation layer), in which each computational layer of a neural network consists of multiple feature mappings. Each feature mapping is a plane, and the weights of all neurons in the plane are equal. The feature mapping structure can use Sigmoid function, ReLU (Rectified Linear Unit) function, Leaky-ReLU function, PReLU (Parametric ReLU) function, GDN (Generalized Divisive Normalization) function, etc. as the activation function of the convolutional network. In addition, since neurons on one mapping surface share weights, the number of free parameters of the network can be reduced.

[0035] For example, compared to image processing algorithms, one advantage of convolutional neural networks is that they can avoid complex image preprocessing processes (such as artificial feature extraction) and directly input the original image, allowing for end-to-end learning.Compared to general neural networks, one advantage of convolutional neural networks is that general neural networks all adopt a fully connected method, that is, all neurons from the input layer to the hidden layer are interconnected, which results in a huge number of parameters, making network training time-consuming and difficult.Convolutional neural networks avoid this difficulty by using methods such as local connections and weight value sharing.

[0036] Deconvolution: The deconvolution layer is also called the transposed convolution layer. The operation process of the deconvolution layer and the convolution layer is similar, the main difference is that the output size of the deconvolution layer is larger than the input size (it can be kept the same size) due to padding. A stride of 1 indicates that the output size is equal to the input size, and a stride of N indicates that the width of the output feature is N times the width of the input feature and the height of the output feature is N times the height of the input feature.

[0037] Generalization Ability: Generalization ability refers to the ability of a machine learning algorithm to adapt to untrained samples. The goal of learning is to learn the rules behind data pairs, so that the trained network can also generate appropriate outputs for data other than the training set that have the same rules. This ability may also be called generalization ability.

[0038] Feature: A feature according to the present invention is a three-dimensional feature matrix of C×W×H. As shown in Figure 1, which is a diagram of a three-dimensional feature matrix, C represents the number of channels, H represents the feature height, and W represents the feature width. The three-dimensional feature matrix can be the input or output of a neural network.

[0039] Rate-Distortion Optimized: There are two metrics for evaluating coding efficiency: bit rate and PSNR (Peak Signal-to-Noise Ratio). The smaller the bitstream, the greater the compression rate. The higher the PSNR, the better the reconstructed image quality. During mode selection, the cost function essentially evaluates both metrics. For example, the cost corresponding to a mode is J(mode) = D + λ × R, where D represents distortion and is typically evaluated using the SSE metric, which is the root mean square of the differences between the reconstructed image block and the source image. To consider the cost, the SAD metric can also be used, where SAD is the sum of the absolute values ​​of the differences between the reconstructed image block and the source image, λ is the Lagrange multiplier, and R is the actual number of bits required to encode the image block in the mode, including the sum of the bits required to encode mode information, motion information, residuals, etc. During mode selection, using rate-distortion optimization to compare and select coding modes typically ensures optimal coding performance.

[0040] Each module on the encoding side has a variety of encoding tools, each with multiple modes. Different encoding tools often provide optimal encoding performance for different video sequences. Therefore, during the encoding process, rate-distortion optimization (RDO) is typically used to compare the encoding performance of different tools or modes and select the optimal mode. After determining the optimal tool or mode, the tool or mode determination information is transmitted by encoding marker information into the bitstream. While this method results in high encoding complexity, it allows adaptive selection of optimal mode combinations for different content and achieves optimal encoding performance. The decoding side can obtain related mode information by directly analyzing the marker information, resulting in a low impact on complexity.

[0041] In deep learning-based end-to-end image compression technology, images are converted to the YUV color space, and the UV color signal is downsampled by a factor of 2. The Y and UV signals are then compressed and encoded separately. The decoding side performs decoding and reconstruction based on the bitstream data, obtaining the reconstructed Y and UV signals. The UV color signal is downsampled by a factor of 2 before being compressed and encoded. This means that the resolution of the UV color signal is halved. Therefore, the resolution of the UV color signal can be restored in post-processing. However, the resolution restoration process requires high computational complexity and resource requirements, making it impossible to restore the resolution of the UV color signal on cost- and latency-sensitive devices.

[0042] Based on the above findings, this embodiment proposes a neural network-based compression defect correction method. A low-complexity and efficient post-processing network model framework is established for the end-to-end image compression field. This framework can enhance the UV signal based on the Y signal, has low complexity, high performance, and can restore compression defects under multiple bit rates.

[0043] The decoding method according to the embodiment of the present invention will be described in detail below by combining several specific examples.

[0044] Embodiment 1: An embodiment of the present invention provides a decoding method, and as shown in FIG. 2, is a flowchart of the decoding method, which may be applied to a decoding side (also called a video decoder), and may include steps 201 to 203.

[0045] In step 201, the bitstream corresponding to the current image block is decoded to obtain a reconstructed image block.

[0046] For example, the reconstructed image block may include a first initial component and a second initial component, where the resolution of the second initial component is a resolution after downsampling and the resolution of the first initial component may be equal to or greater than the resolution of the second initial component. For example, the resolution of the first initial component may be, but is not limited to, a resolution before downsampling (i.e., original resolution) or a resolution after downsampling, as long as it is equal to or greater than the resolution of the second initial component.

[0047] For example, if the reconstructed image block is a YUV format reconstructed image block, the first initial component may be a luma component, and the second initial component may be a chroma U component and / or a chroma V component. Alternatively, if the reconstructed image block is an RGB format reconstructed image block, the first initial component may be a G component, and the second initial component may be an R component and / or a B component.

[0048] In step 202, an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, where the reconstruction quality of the adjusted component may be better than the reconstruction quality of the second initial component, the reconstruction quality of the adjusted component may be equivalent to the reconstruction quality of the second initial component, or the reconstruction quality of the second initial component may be better than the reconstruction quality of the adjusted component.

[0049] For example, a first image feature corresponding to a first initial component may be obtained, and the first image feature may include, but is not limited to, image features not processed by a neural network and / or image features obtained by a neural network. A second image feature corresponding to a second initial component may be obtained, and the second image feature may include, but is not limited to, image features not processed by a neural network and / or image features obtained by a neural network. An adjusted component may be generated based on the first image feature and the second image feature.

[0050] For example, if the first image feature includes an image feature that has not been processed by a neural network, the first image feature may include, but is not limited to, at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. If the second image feature includes an image feature that has not been processed by a neural network, the second image feature may include, but is not limited to, at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The above are merely a few examples and are not limiting.

[0051] When the first image feature includes a first feature map obtained by a neural network and the second image feature includes a second feature map obtained by a neural network, the step of generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes, but is not limited to, inputting the first initial component into a first neural network to obtain a first feature map, inputting the second initial component into a second neural network to obtain a second feature map, and performing an addition operation on the first feature map and the second feature map to obtain the adjusted component, or performing a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0052] Exemplarily, when the first image feature includes a weight coefficient map obtained by a neural network and the second image feature includes a second feature map obtained by a neural network, the step of generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes, but is not limited to, inputting the first initial component to a first neural network, obtaining a first feature map corresponding to the first initial component, performing guided filtering on the first feature map to obtain a weight coefficient map, inputting the second initial component to a second neural network, obtaining a second feature map, and generating an adjusted component based on the weight coefficient map and the second feature map.

[0053] Exemplarily, performing guided filtering on the first feature map to obtain the weight coefficient map may include, but is not limited to, performing a convolution operation on the first feature map to obtain the convolved feature map, and performing weight mapping on the convolved feature map to obtain the weight coefficient map.

[0054] For example, performing weight mapping on the convoluted feature map to obtain a weight coefficient map includes, but is not limited to, performing a pooling operation on the convoluted feature map to obtain a pooled feature map; performing a full connection operation and a ReLU (Rectified Linear Unit) activation operation on the pooled feature map to obtain a ReLU activation feature map; performing a full connection operation and a Sigmoid activation operation on the ReLU activation feature map to obtain a Sigmoid activation feature map; generating a weight coefficient map based on the convoluted feature map and the Sigmoid activation feature map, and multiplying the convoluted feature map and the Sigmoid activation feature map to obtain a weight coefficient map.

[0055] For example, performing weight mapping on the convolved feature map to obtain the weight coefficient map may include, but is not limited to, performing a convolution operation and a sigmoid activation operation on the convolved feature map to obtain the weight coefficient map.

[0056] For example, generating an adjusted component based on the weighting factor map and the second feature map may include, but is not limited to, performing a multiplication operation on the weighting factor map and the second feature map to obtain a multiplied feature map; performing a convolution operation on the first feature map to obtain a convolved feature map; performing an addition operation on the multiplied feature map and the convolved feature map to obtain an added feature map; and performing a concatenation operation on the added feature map and the convolved feature map to obtain an adjusted component.

[0057] For example, when the first image feature includes K first feature maps obtained by a neural network and the second image feature includes K+1 second feature maps obtained by a neural network, the step of generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component includes, but is not limited to, for the first first feature map and the first second feature map, inputting the first initial component into the neural network to obtain the first first feature map, and inputting the second initial component into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, i may be any integer between 2 and K, inclusive, and K may be a positive integer greater than 1. In this case, the i-1-th first feature map may be input to a neural network to obtain the i-th first feature map, feature fusion may be performed on the i-1-th first feature map and the i-1-th second feature map to obtain fused features, the fused features may be input to the neural network to obtain the i-th second feature map, and the final second feature map (i.e., the final second feature map among the K+1 second feature maps) may be obtained, and then adjusted components may be generated based on the final second feature map.

[0058] For example, performing feature fusion on a first feature map (e.g., the (i-1)th first feature map) and a second feature map (e.g., the (i-1)th second feature map) to obtain a fused feature may include, but is not limited to, performing an addition operation on the first feature map and the second feature map to obtain a fused feature, or performing a concatenation operation on the first feature map and the second feature map to obtain a fused feature, or performing guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating a fused feature based on the weight coefficient map and the second feature map. The above are merely some examples, and are not limited thereto.

[0059] Exemplarily, inputting the first initial component into the first neural network to obtain the first feature map includes, but is not limited to, extracting M-dimensional (M is a positive integer) image features from the first initial component, concatenating the M-dimensional image features to obtain concatenated multidimensional features, and inputting the multidimensional features into the first neural network to obtain the first feature map.

[0060] Illustratively, the M-dimensional image features include, but are not limited to, at least one of: image features after convolution of the first initial component with one convolution kernel, image features after cascade convolution of the first initial component with two convolution kernels, first-order spatial domain features after processing the first initial component with a Sobel operator, second-order spatial domain features after processing the first initial component with a Laplacian operator, and frequency domain features after Fourier transform of the first initial component. The above are only some examples of image features, and are not limited thereto.

[0061] For example, inputting the first initial component into the first neural network to obtain the first feature map includes, but is not limited to, performing a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform, and inputting the multiple frequency bands or a portion of the multiple frequency bands into the first neural network to obtain the first feature map. Inputting the second initial component into the second neural network to obtain the second feature map includes, but is not limited to, performing a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform, and inputting the multiple frequency bands or a portion of the multiple frequency bands into the second neural network to obtain the second feature map.

[0062] When performing a wavelet transform on a first initial component to obtain multiple frequency bands after the wavelet transform, the first initial component may be subjected to a single wavelet transform to obtain multiple frequency bands after the wavelet transform, or the first initial component may be subjected to multiple wavelet transforms to obtain multiple frequency bands after the wavelet transform. When performing a wavelet transform on a first initial component multiple times, the first initial component is first subjected to a wavelet transform to obtain multiple frequency bands after the wavelet transform, and then a target frequency band (all or some of the multiple frequency bands) is selected from the multiple frequency bands, and a wavelet transform is performed on the target frequency band to obtain multiple frequency bands after the wavelet transform. This process is repeated until multiple frequency bands after the wavelet transform are obtained. When performing a wavelet transform on a second initial component to obtain multiple frequency bands after the wavelet transform, the second initial component may be subjected to a single wavelet transform to obtain multiple frequency bands after the wavelet transform, or the second initial component may be subjected to multiple wavelet transforms to obtain multiple frequency bands after the wavelet transform.

[0063] For example, before inputting the first initial component into the first neural network to obtain the first feature map, the first initial component may be preprocessed to obtain the preprocessed first initial component. Here, the preprocessed first initial component is used to input the first neural network to obtain the first feature map. Here, preprocessing the first initial component to obtain the preprocessed first initial component includes, but is not limited to, performing edge enhancement on the first initial component to obtain edge-enhanced image features, performing multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, and determining the preprocessed first initial component based on the multi-scale features, for example, using the multi-scale features as the preprocessed first initial component.

[0064] Exemplarily, performing multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features includes, but is not limited to, performing a convolution operation on the edge-enhanced image features to obtain convolved features; performing a downsampling operation on the convolved features to obtain downsampled features; performing a channel transformation on the downsampled features to obtain channel-transformed features; performing an upsampling operation on the channel-transformed features to obtain upsampled features; generating multi-scale features based on the upsampling features and the channel-transformed features; and then inputting the multi-scale features into a first neural network as a first initial component after preprocessing.

[0065] In step 203, a feature enhancement process is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0066] In one possible embodiment, before generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the second initial component may be upsampled to obtain an upsampled second initial component, where the resolution of the upsampled second initial component may be equal to the resolution of the first initial component.

[0067] In another embodiment, the adjusted component may be subjected to a feature enhancement process and upsampled to obtain an upsampled adjusted component before obtaining a restored target component corresponding to the second initial component (i.e., before step 203). The resolution of the upsampled adjusted component may be equal to the resolution of the first initial component.

[0068] In another embodiment, after performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component (i.e., after step 203), the target component may be upsampled to obtain an upsampled target component, where the resolution of the upsampled target component may be equal to the resolution of the first initial component.

[0069] For example, performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component includes, but is not limited to, performing a feature enhancement process on the adjusted component using at least one residual block network to obtain a target component corresponding to the second initial component, or performing a feature enhancement process on the adjusted component using a U-Net (a network with an Encoder-Decoder structure, the first half of which is feature extraction and the second half of which is upsampling) network to obtain a target component corresponding to the second initial component.

[0070] For illustrative purposes, the above-described execution order is merely an example provided for the convenience of explanation, and in actual applications, the execution order between steps may be changed and is not limited to this execution order. Furthermore, in other embodiments, the steps of the corresponding method may not necessarily be performed in the order shown and described in the present invention, and the steps included in the method may be more or less than those described in the present invention. Furthermore, a single step described in the present invention may be decomposed into multiple steps and described in other embodiments, and multiple steps described in the present invention may be integrated into a single step and described in other embodiments.

[0071] As can be seen from the above technical solution, in an embodiment of the present invention, after obtaining a reconstructed image block through decoding, the reconstructed image block includes a first initial component and a second initial component, the resolution of the second initial component is the resolution after downsampling and is equal to or greater than the resolution of the second initial component, and an adjusted component corresponding to the second initial component is generated based on the first initial component and the second initial component, i.e., the second initial component is additionally adjusted based on the first initial component to obtain the adjusted component, and a target component corresponding to the second initial component is determined based on the adjusted component, thereby improving decoding performance, reducing decoding complexity, and reducing computational complexity, and providing a low-complexity and highly efficient post-processing network model framework with relatively low resource requirements and applicable to cost- and delay-sensitive devices, and recovering compression defects under multiple bit rates.

[0072] Example 2: For the processing process on the encoding side (the encoding side is also called a video encoder), please refer to Fig. 3. Fig. 3 is merely an example of the processing process on the encoding side, and the processing process is not limited to this.

[0073] The encoding side may obtain a current image block x (the current image block x may be the original image block x, i.e., the input image block), and then perform feature transformation on the current image block x using an analysis and transformation network (i.e., a neural network) to obtain an image feature y corresponding to the current image block x. Here, performing feature transformation on the current image block x using an analysis and transformation network refers to transforming the current image block x into an image feature y in the latent domain, so that all subsequent processes can be operated in the latent domain.

[0074] An image may be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can be regarded as the image itself, i.e., the encoding process for the image block can be applied to the image directly.

[0075] After acquiring image feature y, the encoding side performs coefficient hyperparameter feature transformation on image feature y to acquire coefficient hyperparameter feature z. For example, image feature y may be input to a hyperparameter coding network (i.e., a neural network), and the hyperparameter coding network may perform coefficient hyperparameter feature transformation on image feature y to acquire coefficient hyperparameter feature z. Here, the hyperparameter coding network may be a trained neural network, and its learning process is not limited as long as it can perform coefficient hyperparameter feature transformation on image feature y. Here, image feature y in the latent domain is processed by the hyperparameter coding network to obtain hyper prior latent information z.

[0076] After obtaining the coefficient hyperparameter feature z, the encoding side may quantize the coefficient hyperparameter feature z to obtain a hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z. That is, the Q operation in FIG. 3 represents the quantization process. After obtaining the hyperparameter quantization feature corresponding to the coefficient hyperparameter feature z, the encoding side encodes the hyperparameter quantization feature to obtain Bitstream#1 (i.e., a first bitstream) corresponding to the current image block. That is, the AE operation in FIG. 3 represents an encoding process such as an entropy encoding process. Alternatively, the encoding side may directly encode the coefficient hyperparameter feature z to obtain Bitstream#1 corresponding to the current image block. Here, the hyperparameter quantization feature or the coefficient hyperparameter feature z included in Bitstream#1 is mainly used to obtain the mean value and parameters of the probability distribution model.

[0077] After the encoding side obtains the Bitstream #1 corresponding to the current image block, the encoding side may send the Bitstream #1 corresponding to the current image block to the decoding side. For the processing process of the Bitstream #1 corresponding to the current image block on the decoding side, please refer to the subsequent embodiments.

[0078] The encoding side may obtain Bitstream #1 corresponding to the current image block, and then decode Bitstream #1 to obtain the hyperparameter quantization feature; that is, AD in FIG. 3 represents the decoding process. Next, the encoding side may perform inverse quantization on the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat may be the same as or different from the coefficient hyperparameter feature z. The IQ operation in FIG. 3 is the inverse quantization process. Alternatively, the encoding side may obtain Bitstream #1 corresponding to the current image block, and then decode Bitstream #1 to obtain the coefficient hyperparameter feature z_hat; in this case, the inverse quantization process of the coefficient hyperparameter feature z_hat is not performed.

[0079] A fixed probability density model encoding method may be adopted for the encoding process of Bitstream#1, and a fixed probability density model decoding method may be adopted for the decoding process of Bitstream#1, and the encoding and decoding processes are not limited.

[0080] After obtaining the coefficient hyperparameter feature z_hat, the encoding side may perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (see the following examples for the process of determining the residual feature y_hat). The encoding side may obtain a predicted value mu (i.e., the average value mu) corresponding to the current image block. For example, the coefficient hyperparameter feature z_hat and the residual feature y_hat may be input to a mean value prediction network, which may then determine the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. This prediction process is not limited to this. Here, the input for the context-based prediction process includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and a more accurate predicted value mu is obtained by integrating and inputting the two. The predicted value mu is then subtracted from the original feature to obtain a residual, which is then added to the decoded residual to obtain a reconstructed y.

[0081] Note that the mean value prediction network is a selectable neural network, i.e., the mean value prediction network is not necessary. In other words, the predicted value mu does not need to be determined by the mean value prediction network. The dashed box in Figure 3 indicates that the mean value prediction network is selectable.

[0082] After acquiring image feature y, the encoding side may determine residual feature r based on image feature y and predicted value mu. For example, the difference between image feature y and predicted value mu is defined as residual feature r. Then, feature processing is performed on residual feature r to acquire image feature s. This feature processing process is not limited and may be any feature processing method. In this case, a mean value prediction network must be provided, and the predicted value mu is provided by the mean value prediction network. Alternatively, after acquiring image feature y, the encoding side may perform feature processing on image feature y to acquire image feature s. This feature processing process is not limited and may be any feature processing method. In this case, there is no need to provide a mean value prediction network. The dashed frame indicates that the residual process is a selectable process.

[0083] After obtaining the image feature s, the encoding side may quantize the image feature s to obtain an image quantization feature corresponding to the image feature s, i.e., the Q operation in Figure 3 represents the quantization process. After obtaining the image quantization feature corresponding to the image feature s, the encoding side may encode the image quantization feature to obtain Bitstream#2 (i.e., the second bitstream) corresponding to the current image block, i.e., the AE operation in Figure 3 represents an encoding process such as an entropy encoding process. Alternatively, the encoding side may directly encode the image feature s to obtain Bitstream#2 corresponding to the current image block, in which case the quantization process of the image feature s is not performed.

[0084] After obtaining the Bitstream #2 corresponding to the current image block, the encoding side may send the Bitstream #2 corresponding to the current image block to the decoding side. For the processing process of the Bitstream #2 corresponding to the current image block on the encoding side, please refer to the subsequent embodiments.

[0085] The encoding side may obtain Bitstream #2 corresponding to the current image block, and then decode Bitstream #2 to obtain image quantization features, i.e., AD in FIG. 3 represents the decoding process. Next, the encoding side may dequantize the image quantization features to obtain image feature s'. Image feature s' may be the same as or different from image feature s. The IQ operation in FIG. 3 is the dequantization process. Alternatively, the encoding side may obtain Bitstream #2 corresponding to the current image block, and then decode Bitstream #2 to obtain image feature s', in which case the dequantization process of the image quantization features is not performed.

[0086] After obtaining image feature s', the encoding side may perform feature reconstruction (i.e., the inverse process of feature processing) on ​​image feature s' to obtain residual feature r_hat. This feature reconstruction process is not limited and may be any feature reconstruction method, and residual feature r_hat may be the same as or different from residual feature r. After obtaining residual feature r_hat, the encoding side determines image feature y_hat based on residual feature r_hat and predicted value mu. Image feature y_hat may be the same as or different from image feature y. For example, image feature y_hat is the sum of residual feature r_hat and predicted value mu. In this case, a mean value prediction network is required, and the predicted value mu is provided by the mean value prediction network. Alternatively, after obtaining image feature s', the encoding side may perform feature reconstruction (i.e., the inverse process of feature processing) on ​​image feature s' to obtain image feature y_hat. Image feature y_hat may be the same as or different from image feature y. In this case, a mean value prediction network is not required. The dashed frame indicates that the residual process is an optional process.

[0087] After obtaining the image feature y_hat, the encoding side may perform a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input to a synthesis transformation network, and the synthesis transformation network performs a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat, thus completing the image reconstruction process.

[0088] In one embodiment, when the encoding side encodes the image quantization feature or image feature s to obtain Bitstream#2 corresponding to the current image block, it must first determine a probability distribution model, and then encode the image quantization feature or image feature s based on the probability distribution model. Also, when the encoding side decodes Bitstream#2, it must first determine a probability distribution model, and then decode Bitstream#2 based on the probability distribution model.

[0089] To obtain the probability distribution model, as shown in Figure 3, the encoding side may obtain the coefficient hyperparameter feature z_hat and then perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat may be input to a probability hyperparameter decoding network, which may then perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, the probability distribution model may be generated based on the probability distribution parameter p. Here, the probability hyperparameter decoding network may be a trained neural network, and the learning process of this probability hyperparameter decoding network is not limited as long as it can perform coefficient hyperparameter feature inverse transformation on the coefficient hyperparameter feature z_hat.

[0090] In one possible embodiment, the encoding process may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process, but the encoding process is not limited thereto.

[0091] Example 3: For the processing process on the decoding side (also called a video decoder), please refer to Fig. 4. Fig. 4 is merely an example of the processing process on the decoding side, and the processing process is not limited to this.

[0092] The decoding side may obtain Bitstream#1 corresponding to the current image block, and then decode Bitstream#1 to obtain the hyperparameter quantization feature; that is, AD in FIG. 4 represents the decoding process. Next, the decoding side may dequantize the hyperparameter quantization feature to obtain the coefficient hyperparameter feature z_hat. The coefficient hyperparameter feature z_hat may be the same as or different from the coefficient hyperparameter feature z. The IQ operation in FIG. 4 is the dequantization process. Alternatively, the decoding side may obtain Bitstream#1 corresponding to the current image block, and then decode Bitstream#1 to obtain the coefficient hyperparameter feature z_hat; in this case, the coefficient hyperparameter feature z_hat does not undergo the dequantization process.

[0093] The decoding process of Bitstream#1 may use, but is not limited to, a decoding method based on a fixed probability density model.

[0094] The image may be divided into one image block or multiple image blocks. If the image is divided into one image block, the current image block x can be regarded as the image itself, i.e., the decoding process for the image block can be directly applied to the image.

[0095] After obtaining the coefficient hyperparameter feature z_hat, the decoding side may perform context-based prediction based on the coefficient hyperparameter feature z_hat of the current image block and the residual feature y_hat of the previous image block (see the following example for the process of determining the residual feature y_hat). This may involve inputting the coefficient hyperparameter feature z_hat and the residual feature y_hat into a mean value prediction network, which then determines the predicted value mu based on the coefficient hyperparameter feature z_hat and the residual feature y_hat. This prediction process is not limited to this. Here, the input for the context-based prediction process includes the coefficient hyperparameter feature z_hat and the decoded residual feature y_hat, and a more accurate predicted value mu is obtained by integrating the two.

[0096] Note that the mean value prediction network is a selectable neural network, i.e., the mean value prediction network is not necessary. In other words, the predicted value mu does not need to be determined by the mean value prediction network. The dashed box in Figure 4 indicates that the mean value prediction network is selectable.

[0097] The decoding side may obtain Bitstream #2 corresponding to the current image block, and then decode Bitstream #2 to obtain image quantization features, i.e., AD in FIG. 4 represents the decoding process. Then, the decoding side may dequantize the image quantization features to obtain image feature s'. Image feature s' may be the same as or different from image feature s. The IQ operation in FIG. 4 is the dequantization process. Alternatively, the decoding side may obtain Bitstream #2 corresponding to the current image block, and then decode Bitstream #2 to obtain image feature s', in which case the dequantization process of the image quantization features is not performed.

[0098] After acquiring image feature s', the decoding side may perform feature reconstruction (i.e., the reverse process of feature processing) on ​​image feature s' to obtain residual feature r_hat. Residual feature r_hat may be the same as or different from residual feature r. After acquiring residual feature r_hat, the decoding side determines image feature y_hat based on residual feature r_hat and predicted value mu. Image feature y_hat may be the same as or different from image feature y. For example, image feature y_hat is the sum of residual feature r_hat and predicted value mu. In this case, a mean value prediction network must be deployed, and the predicted value mu is provided by the mean value prediction network. Alternatively, after acquiring image feature s', the decoding side may perform feature reconstruction on image feature s' to obtain image feature y_hat. Image feature y_hat may be the same as or different from image feature y. In this case, there is no need to deploy a mean value prediction network. The dashed frame indicates that the residual process is an optional process.

[0099] After obtaining the image feature y_hat, the decoding side may perform a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat corresponding to the current image block x. For example, the image feature y_hat is input to a synthesis transformation network, and the synthesis transformation network performs a synthesis transformation on the image feature y_hat to obtain a reconstructed image block x_hat, thus completing the image reconstruction process.

[0100] In one embodiment, when decoding Bitstream #2, the decoding side must first determine a probability distribution model and then decode Bitstream #2 based on the probability distribution model. To obtain the probability distribution model, as shown in FIG. 4, the decoding side may obtain a coefficient hyperparameter feature z_hat and then perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. For example, the coefficient hyperparameter feature z_hat may be input to a probability hyperparameter decoding network, which then performs a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p. After obtaining the probability distribution parameter p, a probability distribution model may be generated based on the probability distribution parameter p. Here, the probability hyperparameter decoding network may be a trained neural network. The learning process of this probability hyperparameter decoding network is not limited, as long as it is possible to perform a coefficient hyperparameter feature inverse transform on the coefficient hyperparameter feature z_hat to obtain the probability distribution parameter p.

[0101] In one possible embodiment, the decoding process may be performed by a deep learning model or a neural network model, thereby realizing an end-to-end image compression and encoding process. The decoding process is not limited.

[0102] Example 4: As shown in Figure 5A, this is a structural diagram of end-to-end image compression based on deep learning. On the encoding side, the image is converted to YUV color space, and the UV signal is downsampled by 2. Then, the Y signal and UV signal are respectively compression-encoded, and the Y signal bitstream and UV signal bitstream are sent to the decoding side. On the decoding side, the Y signal bitstream is decoded and reconstructed to obtain the reconstructed Y signal, and the UV signal bitstream is decoded and reconstructed to obtain the reconstructed UV signal. The UV signal is downsampled by 2 and then compression-encoded, which means that the resolution of the UV signal is halved. Therefore, the resolution of the reconstructed UV signal is halved. That is, the resolution of the reconstructed Y signal is H × W, and the resolution of the reconstructed UV signal is (H / 2) × (W / 2). That is, the resolution of the reconstructed UV signal is half the resolution of the reconstructed Y signal.

[0103] Because the resolution of the reconstructed UV signal is half that of the reconstructed Y signal, the resolution of the UV signal can be restored in the post-processing process, i.e., the resolution of the UV signal can be restored to H × W. When restoring the resolution of the UV signal, the computational complexity of the resolution restoration process is high and the resource requirements are large, so cost-sensitive / delay-sensitive devices cannot restore the resolution of the UV color signal.

[0104] Based on the above findings, an embodiment of the present invention provides a low-complexity and efficient post-processing method in the field of end-to-end image compression. In the post-processing process, the UV signal can be enhanced based on the Y signal, which has low complexity and high performance, improves decoding performance, reduces decoding complexity, reduces computational complexity, has low resource requirements, and can be used by cost-sensitive / delay-sensitive devices, and can restore compression defects under multiple bit rates.

[0105] In an embodiment of the present invention, after receiving a bitstream corresponding to a current image block, the decoding side can decode the bitstream corresponding to the current image block to obtain a reconstructed image block. For example, the decoding side can use the processing flow of embodiment 3 to decode the bitstream corresponding to the current image block to obtain a reconstructed image block, and the decoding process of this reconstructed image block is not limited.

[0106] After obtaining the reconstructed image block, if the reconstructed image block is a YUV format reconstructed image block, auxiliary enhancement may be performed on the chroma U component (i.e., the U component) based on the luma component (i.e., the Y component), auxiliary enhancement may be performed on the chroma V component (i.e., the V component) based on the Y component, or auxiliary enhancement may be performed on the U component and the V component simultaneously based on the Y component. If the reconstructed image block is an RGB format reconstructed image block, auxiliary enhancement may be performed on the R component based on the G component, auxiliary enhancement may be performed on the B component based on the G component, or auxiliary enhancement may be performed on the R component and the B component simultaneously based on the G component. For convenience of explanation, this embodiment takes a YUV format reconstructed image block as an example, so the reconstructed image block may include a Y component, a U component, and a V component, and the resolution of the Y component may be greater than the resolution of the U component. For example, the resolution of the U component is the resolution before downsampling, and the resolution of the Y component may be the original resolution or the resolution after downsampling. In either case, the resolution of the Y component is greater than the resolution of the U component. The resolution of the Y component may be greater than the resolution of the V component, for example, the resolution of the V component is the resolution before downsampling, and the resolution of the Y component may be the original resolution or the resolution after downsampling, but in either case the resolution of the Y component is greater than the resolution of the V component. For example, the resolution of the Y component is H×W, the resolution of the U component is (H / 2)×(W / 2), and the resolution of the V component is (H / 2)×(W / 2).

[0107] 5B is a structural diagram of deep learning-based post-processing. The decoding side may decode a bitstream corresponding to a current image block to obtain a reconstructed image block. If the reconstructed image block is a YUV format reconstructed image block, the reconstructed image block may include a Y component, a U component, and a V component, where the Y component is represented as a Y initial component (i.e., the first initial component in the above embodiment), the U component is represented as a U initial component (i.e., the second initial component in the above embodiment), and the V component is represented as a V initial component (i.e., the second initial component in the above embodiment). Based on this, the U initial component may be supplementarily enhanced based on the Y initial component, the V initial component may be supplementarily enhanced based on the Y initial component, or the U initial component and the V initial component may be supplementarily enhanced based on the Y initial component. Here, the resolution of the Y initial component may be greater than the resolution of the U initial component, and the resolution of the Y initial component may be greater than the resolution of the U initial component. For example, as shown in FIG. 5B, the resolution of the Y initial component is H×W, the resolution of the U initial component is (H / 2)×(W / 2), and the resolution of the V initial component is (H / 2)×(W / 2).

[0108] 5B , the pre-processing module is an optional module. After obtaining the Y initial component by decoding, if the pre-processing module is provided, the Y initial component can be pre-processed by the pre-processing module to obtain the pre-processed Y initial component, which can then be input to the Y auxiliary UV module and the YUV signal enhancement module. If the pre-processing module is not provided, the Y initial component can be directly input to the Y auxiliary UV module and the YUV signal enhancement module. In subsequent embodiments, the Y initial component obtained by the Y auxiliary UV module and the YUV signal enhancement module can be the pre-processed Y initial component or the original Y initial component obtained by decoding, and is not limited thereto.

[0109] Since there is a certain correlation between the three components of YUV, the compression loss of the Y component is smaller than that of the UV component in the encoding side, but a certain enhancement may be performed on the Y component to better enhance the UV component. Therefore, the pre-processing module may perform pre-processing on the Y initial component and perform feature enhancement processing on the Y initial component.

[0110] In this embodiment, as shown in FIG. 5B , for the Y auxiliary UV module, after decoding to obtain the Y initial component, U initial component, and V initial component, the U initial component and V initial component have large loss, while the Y initial component retains more image detail information, and the U initial component and V initial component have a certain correlation with the Y initial component, so by utilizing the information of the Y initial component, the reconstruction quality of the U initial component and V initial component can be improved.

[0111] Based on this, the Y auxiliary UV module may utilize information of the Y initial component to improve the reconstruction quality of the U initial component, may utilize information of the Y initial component to improve the reconstruction quality of the V initial component, or may utilize information of the Y initial component to improve the reconstruction quality of the U initial component and the reconstruction quality of the V initial component.

[0112] 5B, in the present embodiment, the resolution conversion module, the resolution of the U initial component is smaller than that of the Y initial component, for example, the resolution of the U initial component is a resolution obtained by applying 2x downsampling, and the resolution of the U initial component is half that of the original signal, for example, (H / 2) × (W / 2), so the resolution conversion module upsamples the resolution of the U initial component to restore the U component with the resolution of the original signal, that is, upsamples the resolution of the U initial component to obtain a U component with a resolution of H × W. The resolution of the V initial component is smaller than that of the Y initial component, for example, the resolution of the V initial component is a resolution obtained by applying 2x downsampling, and the resolution of the V initial component is half that of the original signal, for example, (H / 2) × (W / 2), so the resolution conversion module upsamples the resolution of the V initial component to restore the V component with the resolution of the original signal, that is, upsamples the resolution of the V initial component to obtain a V component with a resolution of H × W.

[0113] The position of the resolution conversion module may be position 1, i.e., the resolution conversion module upsamples the resolution of the U initial component to obtain a U component with a resolution of H×W and inputs the U component with a resolution of H×W to the Y auxiliary UV module, the resolution conversion module upsamples the resolution of the V initial component to obtain a V component with a resolution of H×W and inputs the V component with a resolution of H×W to the Y auxiliary UV module, or the position of the resolution conversion module may be position 2, i.e., the resolution conversion module upsamples the resolution of the U component output from the Y auxiliary UV module to obtain a U component with a resolution of H×W and inputs the U component with a resolution of H×W to the YUV signal enhancement module, and the resolution conversion module upsamples the resolution of the V component output from the Y auxiliary UV module to obtain a V component with a resolution of H×W and inputs the V component with a resolution of H×W to the YUV signal enhancement module. Alternatively, the position of the resolution conversion module may be position 3, that is, the resolution conversion module upsamples the resolution of the U component output from the YUV signal enhancement module to obtain a U component with a resolution of H×W, and finally outputs this H×W U component; the resolution conversion module upsamples the resolution of the V component output from the YUV signal enhancement module to obtain a V component with a resolution of H×W, and finally outputs this H×W V component.

[0114] Regarding the resolution conversion module, the resolution conversion module can appropriately reduce the resolution of the feature map and effectively reduce the computational complexity of the network. Meanwhile, in the JPEG-AI framework, the resolution conversion module upsamples the resolution of the reconstructed UV components to restore a reconstructed U / V signal with the same resolution as the original signal.

[0115] In this embodiment, as shown in Figure 5B, with regard to the YUV signal enhancement module, in addition to the resolution reduction of the UV signal during the image encoding compression process, the compression algorithm and the convolution process both cause additional information loss to the YUV signal. Therefore, the YUV signal enhancement module can perform signal enhancement on the Y component to make the reconstructed Y component closer to the original signal before compression, thereby improving the subjective quality; the YUV signal enhancement module can perform signal enhancement on the U component to make the reconstructed U component closer to the original signal before compression, thereby improving the subjective quality; and the YUV signal enhancement module can perform signal enhancement on the V component to make the reconstructed V component closer to the original signal before compression, thereby improving the subjective quality.

[0116] For example, as shown in Figure 5B, three network branches (each processing a Y component, a U component, and a V component) can be integrated into one network branch, i.e., a single network is used to process the Y component, the U component, and the V component simultaneously, and the reconstructed Y component, the U component, and the V component simultaneously output, thereby reducing the computational complexity of the network.

[0117] Example 5: In Example 4, the Y auxiliary UV module can utilize information of the Y initial component to auxiliary improve the reconstruction quality of the U initial component and / or the V initial component, for example, generate an adjusted component corresponding to the U initial component based on the Y initial component and the U initial component, and / or generate an adjusted component corresponding to the V initial component based on the Y initial component and the V initial component. Here, the reconstruction quality of the adjusted component corresponding to the U initial component can be better than the reconstruction quality of the U initial component and can improve the reconstruction quality of the U initial component, and the reconstruction quality of the adjusted component corresponding to the U initial component can be equal to or lower than the reconstruction quality of the U initial component, but is not limited thereto. The reconstruction quality of the adjusted component corresponding to the V initial component can be better than the reconstruction quality of the V initial component and can improve the reconstruction quality of the V initial component, and the reconstruction quality of the adjusted component corresponding to the V initial component can be equal to or lower than the reconstruction quality of the V initial component, but is not limited thereto.

[0118] In one possible embodiment, the Y auxiliary UV module may acquire first image features corresponding to the Y primary component, where the first image features may be image features not processed by a neural network. For example, the first image features may include, but are not limited to, at least one of texture features, subjective features, frequency domain features, or histogram features. The Y auxiliary UV module may acquire second image features corresponding to the U primary component, where the second image features may be image features not processed by a neural network. For example, the second image features may include, but are not limited to, at least one of texture features, subjective features, frequency domain features, or histogram features. The Y auxiliary UV module may generate an adjusted component based on the first image feature and the second image feature. For example, a texture blending feature may be generated based on a texture feature corresponding to the Y primary component and a texture feature corresponding to the U primary component, and the texture blending feature may be substituted for the texture feature corresponding to the U primary component to obtain an adjusted component corresponding to the U primary component. Alternatively, for example, a subjective fusion feature may be generated based on a subjective feature corresponding to the Y initial component and a subjective feature corresponding to the U initial component, and the subjective feature corresponding to the U initial component may be replaced with the subjective fusion feature to obtain an adjusted component corresponding to the U initial component. Alternatively, for example, a frequency domain fusion feature may be generated based on a frequency domain feature corresponding to the Y initial component and a frequency domain feature corresponding to the U initial component, and the adjusted component corresponding to the U initial component may be replaced with the frequency domain fusion feature. Alternatively, for example, a histogram fusion feature may be generated based on a histogram feature corresponding to the Y initial component and a histogram feature corresponding to the U initial component, and the adjusted component corresponding to the U initial component may be replaced with the histogram fusion feature.Alternatively, for example, a texture fusion feature may be generated based on a texture feature corresponding to the Y initial component and a texture feature corresponding to the U initial component, a frequency domain fusion feature may be generated based on a frequency domain feature corresponding to the Y initial component and a frequency domain feature corresponding to the U initial component, the texture feature corresponding to the U initial component may be replaced with the texture fusion feature, and the frequency domain feature corresponding to the U initial component may be replaced with the frequency domain fusion feature, thereby obtaining an adjusted component corresponding to the U initial component. The above are merely a few examples, and are not limited thereto.

[0119] The Y auxiliary UV module may acquire a first image feature corresponding to the Y initial component, where the first image feature may be an image feature not processed by a neural network. For example, the first image feature may include, but is not limited to, at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module may acquire a second image feature corresponding to the V initial component, where the second image feature may be an image feature not processed by a neural network. For example, the second image feature may include, but is not limited to, at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature. The Y auxiliary UV module may generate an adjusted component corresponding to the V initial component based on the first image feature and the second image feature. For a generation method, refer to the processing process for the U initial component.

[0120] In another embodiment, the Y auxiliary UV module may acquire first image features corresponding to the Y primary component, and the first image features may be image features acquired by a neural network, i.e., image features output by the neural network. For example, the first image features may include, but are not limited to, a first feature map, a weight coefficient map, etc. The Y auxiliary UV module may acquire second image features corresponding to the U primary component, and the second image features may be image features acquired by a neural network, i.e., image features output by the neural network. For example, the second image features may include, but are not limited to, a second feature map, etc. The Y auxiliary UV module may generate an adjusted component corresponding to the U primary component based on the first image features and the second image features. For example, the adjusted component corresponding to the U primary component may be generated based on a first feature map corresponding to the Y primary component and a second feature map corresponding to the U primary component. For example, the adjusted component corresponding to the U primary component may be generated based on a weight coefficient map corresponding to the Y primary component and a second feature map corresponding to the U primary component. The above are merely a few examples, and are not limiting.

[0121] The Y auxiliary UV module may acquire first image features corresponding to the Y initial component, where the first image features may be image features acquired by a neural network, i.e., image features output by the neural network. For example, the first image features may include, but are not limited to, a first feature map, a weight coefficient map, etc. The Y auxiliary UV module may acquire second image features corresponding to the V initial component, where the second image features may be image features acquired by a neural network, i.e., image features output by the neural network. For example, the second image features may include, but are not limited to, a second feature map, etc. The Y auxiliary UV module may generate an adjusted component corresponding to the V initial component based on the first image features and the second image features. For the generation method, refer to the processing process of the U initial component.

[0122] Example 6: To enable the UV component to fully utilize the information of the Y component, at the input stage of the post-processing network, convolution is performed on the Y component to obtain feature maps with multiple channels, and these feature maps contain various types of features of the Y component. By adding or concatenating these various types of features with the feature map obtained by convolving the UV component, the subsequent neural network can learn the various features of the Y component, thereby improving the reconstruction quality of the UV component.

[0123] For example, see the structural diagram of the post-processing network shown in FIG. 6A. After the Y auxiliary UV module obtains the Y initial component, the U initial component, and the V initial component, it may input the Y initial component into a first neural network (e.g., CNN 1) to obtain a first feature map, which may be referred to as a Y feature map. The U initial component may be input into a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the U initial component, which may be referred to as a U feature map. The V initial component may be input into a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the V initial component, which may be referred to as a V feature map.

[0124] After obtaining the first feature map and the second feature map corresponding to the U initial component, an adjusted component corresponding to the U initial component may be generated based on the first feature map and the second feature map. For example, an addition operation is performed on the first feature map and the second feature map, and the adjusted component in FIG. 6A is generated.

number

number

[0125] Performing a concatenation operation on the first feature map and the second feature map refers to performing a concatenation operation on the first feature map and the second feature map along the channel dimension.

[0126] For example, to correctly implement the above operation, the number of convolution kernels of the second neural network may be the same as the number of convolution kernels of the first neural network. For example, CNN 1 and CNN 2 have the same number of convolution kernels. The processing process shown in Figure 6A is as follows:

number

number

number

number

number

number

number

number

[0127] Example 7: Guided filtering can be introduced to utilize the information of each feature map of the Y component. That is, the features generated by the Y component are used to guide the UV component to generate a feature map with higher expressive power. By using the attention mechanism to weight the pixel values ​​in the UV feature map using the Y component feature map as a guide variable, important features in the UV component are enhanced and general features in the UV component are suppressed, thereby realizing efficient feature conversion and fusion between the Y component and UV component.

[0128] For example, see the structural diagram of the post-processing network shown in FIG. 6B. After the Y auxiliary UV module obtains the Y initial component, the U initial component, and the V initial component, it may input the Y initial component into a first neural network (e.g., CNN 1) to obtain a first feature map, which may be referred to as a Y feature map. It may input the U initial component into a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the U initial component, which may be referred to as a U feature map. It may input the V initial component into a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the V initial component, which may be referred to as a V feature map.

[0129] After obtaining the first feature map and the second feature map corresponding to the U initial component, guided filtering may be performed on the first feature map to obtain a weight coefficient map, and an adjusted component corresponding to the U initial component may be generated based on the weight coefficient map and the second feature map. For example, at least one of a multiplication operation, an addition operation, and a concatenation operation may be performed based on the weight coefficient map and the second feature map to obtain an output feature map. For example, a multiplication operation may be performed based on the weight coefficient map and the second feature map to obtain the output feature map, or both a multiplication operation and an addition operation may be performed based on the weight coefficient map and the second feature map to obtain the output feature map, or all of a multiplication operation, an addition operation, and a concatenation operation may be performed based on the weight coefficient map and the second feature map to obtain the output feature map. After obtaining the output feature map, the output feature map may be input to a third neural network (e.g., CNN 3) to obtain an adjusted component corresponding to the U initial component.

[0130] After obtaining the first feature map and the second feature map corresponding to the V initial component, guided filtering may be performed on the first feature map to obtain a weight coefficient map, and an adjusted component corresponding to the V initial component may be generated based on the weight coefficient map and the second feature map. For example, at least one of a multiplication operation, an addition operation, or a concatenation operation may be performed based on the weight coefficient map and the second feature map to obtain an output feature map. After obtaining the output feature map, the output feature map may be input to a third neural network (e.g., CNN 3) to obtain an adjusted component corresponding to the V initial component.

[0131] Illustratively, the process shown in FIG. 6B is as follows:

number

number

number

number

number

number

number

number

[0132] In one embodiment, an embodiment of guided filtering is as shown in Figure 6C. Specifically, a convolution operation is performed on a first feature map (Y Feature Map) to obtain a convolved feature map (hereinafter referred to as convolved feature map A). ​​For example, the first feature map may be input to a CNN network, and the CNN network may perform a convolution operation on the first feature map to obtain convolved feature map A. Then, weight mapping may be performed on the convolved feature map A to obtain a weight coefficient map after weight mapping.

[0133] When generating an adjusted component corresponding to the U initial component based on the weight coefficient map and the second feature map corresponding to the U initial component, a multiplication operation may be performed on the weight coefficient map and the second feature map (U Feature Map) corresponding to the U initial component to obtain a multiplied feature map. Then, an addition operation is performed on the multiplied feature map and the convolution feature map A to obtain a summed feature map. After that, a concatenation operation is performed on the summed feature map and the convolution feature map A to obtain an output feature map. The output feature map is input to a third neural network (e.g., CNN 3) to obtain an adjusted component corresponding to the U initial component.

[0134] When generating an adjusted component corresponding to the V initial component based on the weighting coefficient map and the second feature map corresponding to the V initial component, a multiplication operation may be performed on the weighting coefficient map and the second feature map (V Feature Map) corresponding to the V initial component to obtain a multiplied feature map. Then, an addition operation is performed on the multiplied feature map and the convolved feature map A to obtain a summed feature map. After that, a concatenation operation is performed on the summed feature map and the convolved feature map A to obtain an output feature map. The output feature map is input to a third neural network (e.g., CNN 3) to obtain an adjusted component corresponding to the V initial component.

[0135] As shown in Figure 6C, after processing the first feature map of the Y component through several convolution layers, a convolution feature map A is obtained, and the convolution feature map A is mapped to a certain weight coefficient, i.e., a weight coefficient map, through weight mapping. The weight coefficient map is multiplied by the second feature map of the UV component to obtain a feature guide map of the UV component. Then, the convolution feature map A and the feature guide map of the UV component are added, and finally concatenated with the convolution feature map A to output a fusion feature map (Output Feature Map). For example, the guide filtering process can be

number

number

number

number

number

[0136] In one possible embodiment, in the weight mapping process, weight mapping is performed on the convoluted feature map A to obtain a weight coefficient map after weight mapping. An embodiment of weight mapping is as shown in FIG. 6D, which illustrates two weight mapping methods. In the first weight mapping method, a pooling operation is performed on the convoluted feature map A to obtain a pooled feature map. For example, a pooling operation is performed on the convoluted feature map A with a size of C×H×W to obtain a pooled feature map with a size of C×1×1. That is, the "pooling" method reduces the resolution of the feature map to 1×1.

[0137] Next, a full connection operation and a ReLU activation operation are performed on the feature map after the pooling operation to obtain a feature map after the ReLU activation operation. For example, a full connection operation (i.e., an FC operation) and a ReLU activation operation (i.e., a ReLU operation) are performed on a feature map after the pooling operation that has a size of C×1×1 to obtain a feature map after the ReLU activation operation that has a size of C×1×1. A full connection operation and a sigmoid activation operation are performed on the feature map after the ReLU activation operation to obtain a feature map after the sigmoid activation operation. For example, a full connection operation (i.e., an FC operation) and a sigmoid activation operation (i.e., a sigmoid operation) are performed on a feature map after the ReLU activation operation that has a size of C×1×1 to obtain a feature map after the sigmoid activation operation that has a size of C×1×1.

[0138] Then, a weight coefficient map is generated based on the convolution feature map A and the feature map after Sigmoid activation. For example, the weight coefficient map is obtained by multiplying the convolution feature map A and the feature map after Sigmoid activation. For example, the convolution feature map A, which has a size of C×H×W, is multiplied by the feature map after Sigmoid activation, which has a size of C×1×1, to obtain a weight coefficient map of size C×H×W. Obviously, after reducing the resolution of the feature map to 1×1 using the "pooling" method, it passes through several fully connected layers, and finally obtains the weight coefficients of each channel in the original input feature map using the Sigmoid activation function. These weight coefficients are then used to assign different weights to the channels in the original input feature map, and finally the weight coefficient map (also called the weight feature map) is obtained.

[0139] In the second type of weight mapping method, a convolution operation may be performed on a convolved feature map A to obtain a convolved feature map B. For example, the convolved feature map A having a size of C×H×W is input to a CNN network, and the CNN network performs a convolution operation on the convolved feature map A through several convolution layers to obtain a convolved feature map B having a size of C×H×W. Then, a sigmoid activation operation is performed on the convolved feature map B to obtain a weight coefficient map. For example, a sigmoid activation function may be used to perform an activation operation on the convolved feature map B having a size of C×H×W to obtain a weight coefficient map having a size of C×H×W. Obviously, after the original input feature map is processed through several convolution layers, the weight coefficient map is finally obtained using the sigmoid activation function.

[0140] The difference between the first and second weight mapping schemes is that the first weight mapping scheme assigns different weights to the channels, while the second weight mapping scheme assigns different weights to each pixel in each channel feature map.

[0141] Example 8: In Examples 6 and 7, feature extraction and information fusion of the Y component are performed at the input stage of the post-processing network. However, in Example 8, feature extraction and information fusion of the Y component are distributed to each stage of the post-processing network, and UV component features from different stages are fused with Y component features, thereby improving the post-processing network's ability to enhance the UV component. For example, see the structural diagram of the post-processing network shown in Figure 6E. The Y auxiliary UV module obtains the Y initial component, U initial component, and V initial component, and then distributes feature extraction and information fusion of the Y component to each stage of the post-processing network. Here,

number

[0142] In one possible embodiment, multiple first feature maps and multiple second feature maps are involved in fusing the feature map of the Y component with the feature map of the UV component at each stage of the network. Take K first feature maps and K+1 second feature maps as an example, where K is a positive integer greater than 1. In this case, after obtaining the Y initial component and the U initial component, for the first first feature map and the first second feature map, the Y initial component is input into the neural network to obtain the first first feature map, and the U initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, where i ranges from 2 to K, the i-1-th first feature map is input into the neural network to obtain the i-th first feature map, and feature fusion is performed on the i-1-th first feature map and the i-1-th second feature map to obtain fused features, and the fused features are input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map (i.e., the K+1th second feature map), an adjusted component corresponding to the U initial component may be generated based on the last second feature map.

[0143] For example, assume that K is 3. The Y initial component is input to neural network a1 to obtain a first feature map b1. That is, neural network a1 performs several convolution operations on the Y initial component to obtain the first feature map b1. The U initial component is input to neural network c1 to obtain a second feature map d1. That is, neural network c1 performs several convolution operations on the U initial component to obtain the second feature map d1.

[0144] The first feature map b1 is input to the neural network a2 to obtain the first feature map b2, i.e., the neural network a2 performs some convolution operations on the first feature map b1 to obtain the first feature map b2, performs feature fusion on the first feature map b1 and the second feature map d1 to obtain the fused features, and inputs the fused features into the neural network c2 to obtain the second feature map d2.

[0145] The first feature map b2 is input to the neural network a3 to obtain the first feature map b3, i.e., the neural network a3 performs some convolution operations on the first feature map b2 to obtain the first feature map b3, performs feature fusion on the first feature map b2 and the second feature map d2 to obtain the fused features, and inputs the fused features into the neural network c3 to obtain the second feature map d3.

[0146] Then, feature fusion is performed on the first feature map b3 and the second feature map d3 to obtain fused features, and the fused features are input to a neural network c4 to obtain a second feature map d4. After obtaining the second feature map d4, an adjusted component corresponding to the U initial component may be generated based on the second feature map d4. For example, the second feature map d4 may be used as the adjusted component corresponding to the U initial component, or the adjusted component corresponding to the U initial component may be obtained after performing an operation on the second feature map d4.

[0147] In the above process, performing feature fusion on the first feature map and the second feature map to obtain the fused feature includes, but is not limited to, performing an addition operation on the first feature map and the second feature map to obtain the fused feature, or performing a concatenation operation on the first feature map and the second feature map to obtain the fused feature, or performing guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generating the fused feature based on the weight coefficient map and the second feature map. Regarding the implementation of feature fusion, see Examples 6 and 7, and the description will not be repeated here.

[0148] After obtaining the Y initial component and the V initial component, for the first first feature map and the first second feature map, the Y initial component is input into the neural network to obtain the first first feature map, and the V initial component is input into the neural network to obtain the first second feature map. For the i-th first feature map and the i-th second feature map, where i ranges from 2 to K, the i-1-th first feature map is input into the neural network to obtain the i-th first feature map, and feature fusion is performed on the i-1-th first feature map and the i-1-th second feature map to obtain fused features, which are then input into the neural network to obtain the i-th second feature map. After obtaining the last second feature map, an adjusted component corresponding to the V initial component may be generated based on the last second feature map. For the method of obtaining the adjusted component corresponding to the V initial component, see the processing process for the U initial component.

[0149] Example 9: In Examples 6, 7, and 8, a guide map fusion method for the Y component was designed. However, in Example 9, detailed feature extraction of the Y component is performed from multiple dimensions, and then fused into a post-processing network, thereby improving the network's ability to enhance the UV component.

[0150] See the structural diagram of the post-processing network shown in FIG. 6F. After obtaining the Y initial component, the U initial component, and the V initial component, the Y auxiliary UV module first performs multidimensional feature extraction on the Y initial component. For example, image features of M dimensions (M is a positive integer) are extracted from the Y initial component, and then the M-dimensional image features are concatenated to obtain concatenated multidimensional features. After obtaining the multidimensional features, the multidimensional features may be input to a first neural network to obtain a first feature map (Y Feature Map). Furthermore, the Y auxiliary UV module may input the U initial component to a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the U initial component, which may be referred to as a U Feature Map. Alternatively, the V initial component may be input to a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the V initial component, which may be referred to as a V Feature Map.

[0151] After obtaining the first feature map and the second feature map corresponding to the U initial component, feature fusion may be performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, feature fusion may be performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the V initial component. For the feature fusion process, see Examples 6, 7, and 8.

[0152] Illustratively, the process shown in FIG. 6F is:

number

number

number

[0153] For example, see the structural diagram of the multidimensional feature extraction process shown in FIG. 6G. Taking the Y initial component as input, each branch feature extraction structure is used to extract features of the Y initial component. For example, the Y initial component is input to the branch 1 feature extraction structure, and the first-dimensional feature of the Y initial component is extracted from the branch 1 feature extraction structure. The Y initial component is input to the branch 2 feature extraction structure, and the second-dimensional feature of the Y initial component is extracted from the branch 2 feature extraction structure. In this manner, the Y initial component is input to the branch M feature extraction structure, and the M-th-dimensional feature of the Y initial component is extracted from the branch M feature extraction structure. Through the above process, M-dimensional image features can be extracted from the Y initial component, and the M-dimensional image features can be concatenated to obtain concatenated multidimensional features. After obtaining the concatenated multidimensional features, the multidimensional features can be used as a first feature map, or the multidimensional features can be input to a first neural network to obtain the first feature map.

[0154] Figure 6G shows the structure of the multidimensional feature extraction process. Specifically, first, the Y component is passed through different feature extraction channels to obtain feature information for each dimension, and finally, this feature information is concatenated and output as a Y component feature map. The multidimensional feature extraction process is as follows:

number

number

number

[0155] In one embodiment, the M-dimensional image features include, but are not limited to, at least one of image features after performing convolution on the Y initial component with one convolution kernel, image features after performing cascade convolution on the Y initial component with two convolution kernels, first-order spatial domain features after performing processing on the Y initial component with a Sobel operator, second-order spatial domain features after performing processing on the Y initial component with a Laplacian operator, and frequency domain features after performing Fourier transform on the Y initial component. The above image features are merely some examples, and the type of the image features is not limited in this embodiment.

[0156] Figure 6H shows the structure of the multidimensional feature extraction process. The branch 1 feature extraction structure (i.e., the first branch) obtains feature information using a single 3x3 convolution. For example, it convolves the initial Y component with a single 3x3 convolution kernel to obtain the convolved image feature. The branch 2 feature extraction structure (i.e., the second branch) uses a 1x1 and 3x3 cascade convolution method to obtain channel expansion / compression feature information. For example, it uses a 1x1 and 3x3 convolution kernel to perform cascade convolution on the initial Y component to obtain the convolved image feature. The branch 3 feature extraction structure (i.e., the third branch) obtains first-order spatial domain features using the Sobel operator. For example, it processes the initial Y component using the Sobel operator to obtain first-order spatial domain features. The branch 4 feature extraction structure (i.e., the fourth branch) obtains second-order spatial domain features using the Laplacian operator. For example, it processes the initial Y component using the Laplacian operator to obtain second-order spatial domain features. The branch 5 feature extraction structure (i.e., the fifth branch) uses Fourier transform to obtain frequency domain features. For example, it performs a Fourier transform on the initial component of Y to obtain frequency domain features. After obtaining the above five-dimensional features, these features can be concatenated to obtain multidimensional features.

[0157] Example 10: While Examples 6 to 9 primarily focus on "which Y component features to select and how to combine Y component features and UV component features," Example 10 differs from them in that it focuses on "how to enhance the Y component features." For example, after the YUV components of an image are encoded and compressed, some loss of image details in the Y component occurs. Therefore, some enhancement may be performed on the Y component and its features first. For example, because there is a certain correlation between the three components of YUV, the compression loss of the Y component is smaller than the compression loss of the UV component in the encoding side. However, some enhancement may be performed on the Y component to better enhance the UV component. Therefore, the preprocessing module may perform preprocessing on the Y initial component, and then perform feature enhancement processing on the Y initial component.

[0158] For example, the preprocessing module may obtain a Y initial component, then perform edge enhancement on the Y initial component to obtain edge-enhanced image features, and then perform multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features. After obtaining the multi-scale features, the multi-scale features may be used as the preprocessed Y initial component, and the preprocessed Y initial component may be input to a Y auxiliary UV module, which may then input the preprocessed Y initial component to a first neural network to obtain a first feature map (Y Feature Map). The Y auxiliary UV module may also input the U initial component to a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the U initial component, which may be referred to as a U feature map. The V initial component may also be input to a second neural network (e.g., CNN 2) to obtain a second feature map corresponding to the V initial component, which may be referred to as a V feature map.

[0159] After obtaining the first feature map and the second feature map corresponding to the U initial component, feature fusion may be performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, feature fusion may be performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the V initial component. For the feature fusion process, see Examples 6, 7, and 8.

[0160] When performing multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, the pre-processing module performs a convolution operation on the edge-enhanced image features to obtain convolved features, performs a downsampling operation on the convolved features to obtain downsampled features, performs channel transformation on the downsampled features to obtain channel-transformed features, and performs an upsampling operation on the channel-transformed features to obtain upsampled features, and generates multi-scale features based on the upsampled features and the channel-transformed features.

[0161] See Figure 6I for a structural diagram of the post-processing network. Figure 6I shows an example of a multi-scale enhancement method for the Y component. In the downsampling section, the resolution of the feature map gradually decreases from D1 to D3, and the number of channels gradually increases. In the upsampling section, the resolution of the feature map gradually increases from U3 to U1, and the number of channels remains constant. After channel transformation, D3, D2, and D1 are added to the upsampled U3, U2, and U1 feature maps, respectively, to obtain multi-scale enhanced multi-scale features.

[0162] First, edge enhancement is performed on the Y initial component to obtain image features after edge enhancement. For example, image sharpening is performed on the Y initial component to obtain image features after edge enhancement. The edge enhancement method is not limited to this.

[0163] Two downsampling operations and two upsampling operations are used as an example, but the number of downsampling operations and the number of upsampling operations can be increased and are not limited to this. After obtaining the edge-enhanced image features, the edge-enhanced image features are output to the CNN to obtain feature D1 with a scale of C×H×W. Feature D1 is downsampled to obtain feature D2 with a scale of 4C×(H / 2)×(W / 2). Feature D2 is downsampled to obtain feature D3 with a scale of 16C×H / 4×W / 4.

[0164] Feature D3 is subjected to channel transformation to obtain feature U3 with a scale of T×H / 4×W / 4. Feature U3 is then upsampled and channel transformed. Feature D2 is then subjected to channel transformation. The upsampled feature U3 and the channel-transformed feature D2 are then added together to obtain feature U2 with a scale of T×(H / 2)×(W / 2). Feature U2 is then upsampled and channel transformed. Feature D1 is then subjected to channel transformation. The upsampled feature U2 and the channel-transformed feature D1 are then added together to obtain feature U1 with a scale of T×H×W. Obviously, feature U1 is the multiscale feature after multiscale enhancement, i.e., the preprocessed Y initial component. The preprocessing module outputs this preprocessed Y initial component. By performing multiscale feature extraction on the Y component, the final output first feature map integrates feature information from different scales and contributes to enhancing the UV components.

[0165] Example 11: In Example 4, the resolution conversion module is related to the resolution conversion module. The resolution conversion module upsamples the resolution of the U component to restore the U component with the resolution of the original signal, for example, to obtain a U component with a resolution of H×W. The resolution conversion module upsamples the resolution of the V initial component to restore the V component with the resolution of the original signal, for example, to obtain a V component with a resolution of H×W. For example, in the JPEG-AI framework, the original UV components are downsampled by 2 times and then compression-encoded on the encoding side, so that the resolution of the decoded and reconstructed UV components on the decoding side is half the resolution of the original signal. In the post-processing network, a step of upsampling the UV components is included, so that reconstructed UV components with the same resolution as the original signal are restored. Exemplarily, methods used by the resolution conversion module to upsample the resolution of the U component and / or V component include, but are not limited to, interpolation sampling, pixel-shuffle upsampling, deconvolution (DeConv), etc. The above are merely some examples, and the upsampling method is not limited in this example.

[0166] For example, upsampling the resolution of UV components affects the computational complexity of the network model. That is, increasing the resolution increases the computational complexity. Since the location of the resolution conversion module affects the computational complexity of the network model, the location of the resolution conversion module can be designed to reduce the computational complexity of the network model.

[0167] Case 1: As shown in FIG. 5B, when the position of the resolution conversion module is set to position 1, that is, the resolution conversion module upsamples the resolution of the U initial component to obtain a U component with a resolution of H×W, and inputs the U component with a resolution of H×W to the Y auxiliary UV module. The resolution conversion module upsamples the resolution of the V initial component to obtain a V component with a resolution of H×W, and inputs the V component with a resolution of H×W to the Y auxiliary UV module. In this case, in Examples 5 to 10, the U initial component is the upsampled U component, and the V initial component is the upsampled V component. The resolution of the upsampled U component is equal to the resolution of the Y initial component, and the resolution of the upsampled V component is equal to the resolution of the Y initial component. In this way, Examples 5 to 10 are executed based on the Y initial component and the upsampled U component, and Examples 5 to 10 are executed based on the Y initial component and the upsampled V component.

[0168] As can be seen from the above, in Case 1, upsampling can be performed at the input end of the network, as shown in Figure 7A, which is a structural diagram of upsampling at the input end of the network. That is, at the input end of the network, the U initial component can be upsampled to obtain the upsampled U component, for example, the U component with a resolution of HxW, and at the input end of the network, the V initial component can be upsampled to obtain the upsampled V component, for example, the V component with a resolution of HxW.

[0169] Case 2: As shown in FIG. 5B, when the resolution conversion module is set to position 2, the resolution conversion module upsamples the U component output from the Y auxiliary UV module to obtain a U component with a resolution of H×W. The U component with a resolution of H×W is input to the YUV signal enhancement module. The resolution conversion module upsamples the V component output from the Y auxiliary UV module to obtain a V component with a resolution of H×W. The V component with a resolution of H×W is input to the YUV signal enhancement module. In this case, in Examples 5 to 10, the U initial component is the U component before upsampling, and the V initial component is the V component before upsampling. The resolution of the U component before upsampling is smaller than the resolution of the Y initial component, and the resolution of the V component before upsampling is smaller than the resolution of the Y initial component. In this way, Examples 5 to 10 are executed based on the Y initial component and the U component before upsampling, and Examples 5 to 10 are executed based on the Y initial component and the V component before upsampling. After obtaining the adjusted components corresponding to the U component / V component according to Examples 5 to 10, the adjusted components can be upsampled to obtain the upsampled adjusted components, and the upsampled adjusted components can be input into a YUV signal enhancement module, where the resolution of the upsampled adjusted components is equal to the resolution of the Y initial component.

[0170] As can be seen from the above, in Case 2, stepwise upsampling can be performed between network layers, as shown in Figure 7B, which is a structural diagram of stepwise upsampling between network layers. That is, in multiple network layers, U initial components can be upsampled to obtain upsampled U components, and the resolution of the upsampled U components is equal to the resolution of the Y initial components, for example, a U component with a resolution of H x W. In multiple network layers, V initial components can be upsampled to obtain upsampled V components, and the resolution of the upsampled V components is equal to the resolution of the Y initial components, for example, a V component with a resolution of H x W.

[0171] Case 3: As shown in FIG. 5B, when the resolution conversion module is set to position 3, the resolution conversion module upsamples the U component output from the YUV signal enhancement module to obtain a U component with a resolution of H×W, and finally outputs this H×W U component. The resolution conversion module upsamples the V component output from the YUV signal enhancement module to obtain a V component with a resolution of H×W, and finally outputs this H×W V component. In this case, in Examples 5 to 10, the U initial components are all U components before upsampling, and the V initial components are all V components before upsampling. The resolution of the U component before upsampling is smaller than the resolution of the Y initial component, and the resolution of the V component before upsampling is smaller than the resolution of the Y initial component. In this way, Examples 5 to 10 are executed based on the Y initial component and the U component before upsampling, and Examples 5 to 10 are executed based on the Y initial component and the V component before upsampling. After obtaining adjusted components corresponding to the U component / V component according to Examples 5 to 10, the adjusted components are input to a YUV signal enhancement module, where the adjusted components are the U component / V component before upsampling, and the U component / V component before upsampling has a smaller resolution than the Y initial component. Thus, the YUV signal enhancement module performs signal enhancement based on the U component / V component before upsampling to obtain the enhanced U component / V component. After obtaining the enhanced U component / V component, the U component / V component is upsampled to obtain the upsampled U component / V component, and the resolution of the upsampled U component / V component is equal to the resolution of the Y initial component.

[0172] As can be seen from the above, in Case 3, upsampling can be performed at the output end of the network, as shown in Figure 7C, which is a structural diagram of upsampling at the output end of the network. That is, at the output end of the network, the U component can be upsampled to obtain the upsampled U component, and the resolution of the upsampled U component is equal to the resolution of the Y initial component, for example, a U component with a resolution of H x W. At the output end of the network, the V component can be upsampled to obtain the upsampled V component, and the resolution of the upsampled V component is equal to the resolution of the Y initial component, for example, a V component with a resolution of H x W.

[0173] Example 12: In Example 4, with respect to the resolution conversion module, in a first aspect, the resolution conversion module can appropriately reduce the resolution of the feature map by using resolution conversion without information loss, thereby effectively reducing the computational complexity of the network. In a second aspect, the resolution conversion module can restore the U component and V component with the same resolution as the resolution of the original signal by upsampling the resolution of the reconstructed U component and V component. For the function of the second aspect, please refer to Example 11.

[0174] The function of the first aspect may be realized by a resolution conversion module or a Y auxiliary UV module. For example, to restore the original resolution, a wavelet transform (e.g., Haar wavelet transform) can be used to reduce the resolution of the original image by half, thus reducing the computational complexity of the network by approximately one-fourth. If the requirements for hardware computational complexity are high, multiple successive wavelet transforms can be used to further reduce the resolution. Regarding several frequency bands after wavelet transform, all frequency bands can be used, or only some frequency bands can be selected.

[0175] For example, see the structural diagram of the post-processing network shown in FIG. 7D . A wavelet transform (DWT transform in FIG. 7D ) may be performed on the Y initial component to obtain multiple frequency bands after the wavelet transform, and the multiple frequency bands or a portion of the multiple frequency bands (i.e., Y subbands) may be input to a first neural network to obtain a first feature map (Y Feature Map). A wavelet transform (DWT transform in FIG. 7D ) may be performed on the U initial component to obtain multiple frequency bands after the wavelet transform, and the multiple frequency bands or a portion of the multiple frequency bands (i.e., U subbands) may be input to a second neural network to obtain a second feature map (U Feature Map). A wavelet transform may be performed on the V initial component to obtain multiple frequency bands after the wavelet transform, and the multiple frequency bands or a portion of the multiple frequency bands (i.e., V subbands) may be input to a second neural network to obtain a second feature map (V Feature Map). After obtaining the first feature map and the second feature map corresponding to the U initial component, feature fusion may be performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the U initial component. After obtaining the first feature map and the second feature map corresponding to the V initial component, feature fusion may be performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the V initial component. For the feature fusion process, see Examples 6, 7, and 8.

[0176] In addition, when feature fusion is performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the U initial component, the fused features may be first obtained using Examples 6, 7, and 8, and then the fused features may be input to a neural network (CNN 3) to obtain output features (U subband) of the neural network, and then the output features may be subjected to an inverse wavelet transform (i.e., the inverse operation of the wavelet transform, i.e., the IDWT transform in FIG. 7D ) to obtain an adjusted component of the U component. When feature fusion is performed on the first feature map and the second feature map to obtain an adjusted component corresponding to the V initial component, the fused features may be first obtained using Examples 6, 7, and 8, and then the fused features may be input to a neural network (CNN 3) to obtain output features (V subband) of the neural network, and then the output features may be subjected to an inverse wavelet transform (i.e., the inverse operation of the wavelet transform, i.e., the IDWT transform in FIG. 7D ) to obtain an adjusted component of the V component.

[0177] Embodiment 13: In embodiment 4, the YUV signal enhancement module performs signal enhancement on the Y component to approximate the reconstructed Y component to the original signal before compression; performs signal enhancement on the U component to approximate the reconstructed U component to the original signal before compression; and performs signal enhancement on the V component to approximate the reconstructed V component to the original signal before compression. Based on this, the YUV signal enhancement module may perform feature enhancement processing on the initial Y component to obtain a target component corresponding to the Y component, where the target component is the Y component after signal restoration. The YUV signal enhancement module may perform feature enhancement processing on the adjusted U component to obtain a target component corresponding to the U component, where the target component is the U component after signal restoration. The YUV signal enhancement module may perform feature enhancement processing on the adjusted V component to obtain a target component corresponding to the V component, where the target component is the V component after signal restoration.

[0178] In one embodiment, at least one residual block network may perform a feature enhancement process on the Y initial component to obtain a target component corresponding to the Y component. At least one residual block network may perform a feature enhancement process on the adjusted U component to obtain a target component corresponding to the U component. At least one residual block network may perform a feature enhancement process on the adjusted V component to obtain a target component corresponding to the V component. FIG. 7E is a structural diagram illustrating a residual block cascade network for enhancing the Y component, U component, and V component, and the structure of this residual block cascade network is not limited.

[0179] In one embodiment, a U-Net network may perform a feature enhancement process on the Y initial component to obtain a target component corresponding to the Y component. A U-Net network may perform a feature enhancement process on the adjusted U component to obtain a target component corresponding to the U component. A U-Net network may perform a feature enhancement process on the adjusted V component to obtain a target component corresponding to the V component. FIG. 7F illustrates a structure diagram of a U-Net network for enhancing the Y, U, and V components. The U-Net network includes multiple downsampling network layers and multiple upsampling network layers, and the structure of the U-Net network is not limited.

[0180] For example, the above-described embodiments may be implemented singly or in combination. For example, each of the embodiments 1 to 13 may be implemented singly, and at least two of the embodiments 1 to 13 may be implemented in combination.

[0181] For example, in the above embodiment, the content of the encoding side is applicable to the decoding side, i.e., the decoding side can process it in the same way, and the content of the decoding side is applicable to the encoding side, i.e., the encoding side can process it in the same way.

[0182] Based on the same application concept as the above method, an embodiment of the present invention further provides a decoding device applied to the decoding side, and the decoding device may include a memory configured to store video data and a decoder configured to realize the decoding method in the above embodiments 1 to 13, i.e., the processing flow on the decoding side.

[0183] For example, in one embodiment, the decoder Decoding a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, wherein a resolution of the first initial component is equal to or greater than a resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; The method is configured to perform a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0184] Exemplarily, if the reconstructed image block is a YUV format reconstructed image block, the first initial component is a luminance component, and the second initial component is a chrominance U component and / or a chrominance V component. Alternatively, if the reconstructed image block is an RGB format reconstructed image block, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0185] Illustratively, the decoder is further configured to: acquire first image features corresponding to the first initial component, the first image features including image features not processed by a neural network and / or image features obtained by a neural network; acquire second image features corresponding to the second initial component, the second image features including image features not processed by a neural network and / or image features obtained by a neural network; and generate the adjusted component based on the first image features and the second image features.

[0186] Exemplarily, when the first image feature includes a first feature map obtained by a neural network and the second image feature includes a second feature map obtained by a neural network, the decoder is configured to input the first initial component to a first neural network to obtain the first feature map, input the second initial component to a second neural network to obtain the second feature map, and perform an addition operation on the first feature map and the second feature map to obtain the adjusted component, or perform a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0187] Illustratively, the decoder is further configured, when the first image feature includes a weight coefficient map obtained by a neural network and the second image feature includes a second feature map obtained by a neural network, to input the first initial component to a first neural network to obtain a first feature map corresponding to the first initial component, perform guided filtering on the first feature map to obtain the weight coefficient map, input the second initial component to a second neural network to obtain the second feature map, and generate the adjusted component based on the weight coefficient map and the second feature map.

[0188] Illustratively, the decoder is further configured to perform a convolution operation on the first feature map to obtain a convolved feature map, and perform weight mapping on the convolved feature map to obtain the weight coefficient map.

[0189] Exemplarily, the decoder is configured to perform a pooling operation on the convolved feature map, perform a full connection operation and a ReLU activation operation on the pooled feature map, perform a full connection operation and a sigmoid activation operation on the ReLU activation feature map, and generate the weight coefficient map based on the convolved feature map and the sigmoid activation feature map.

[0190] Illustratively, the decoder is configured to perform a convolution operation and a sigmoid activation operation on the convolved feature map to obtain the weight coefficient map.

[0191] Exemplarily, the decoder is further configured to perform a multiplication operation on the weight coefficient map and the second feature map to obtain a multiplied feature map; perform a convolution operation on the first feature map to obtain a convolved feature map; perform an addition operation on the multiplied feature map and the convolved feature map to obtain an added feature map; and perform a concatenation operation on the added feature map and the convolved feature map to obtain the adjusted component.

[0192] Exemplarily, the decoder is further configured, when the first image feature includes K first feature maps obtained by a neural network and the second image feature includes K+1 second feature maps obtained by a neural network, to input, for a first first feature map and a first second feature map, a first initial component into the neural network to obtain a first first feature map, and to input a second initial component into the neural network to obtain a first second feature map; to input, for an ith first feature map and an ith second feature map, i being an integer greater than or equal to 2 and less than or equal to K, where K is a positive integer greater than 1, the ith-1st first feature map into the neural network to obtain the ith first feature map, to perform feature fusion on the ith-1st first feature map and the ith-1st second feature map to obtain fused features, to input the fused features into the neural network to obtain the ith second feature map, to obtain a final second feature map, and then to generate the adjusted components based on the final second feature map.

[0193] Illustratively, the decoder is further configured to extract M-dimensional (M is a positive integer) image features from the first initial components, concatenate the M-dimensional image features to obtain concatenated multi-dimensional features, and input the multi-dimensional features into a first neural network to obtain the first feature map.

[0194] Illustratively, the decoder is configured to perform a wavelet transform on the first initial components to obtain a plurality of frequency bands after the wavelet transform, input the plurality of frequency bands or a portion of the plurality of frequency bands to a first neural network to obtain the first feature map, perform a wavelet transform on the second initial components to obtain a plurality of frequency bands after the wavelet transform, and input the plurality of frequency bands or a portion of the plurality of frequency bands to a second neural network to obtain the second feature map.

[0195] Exemplarily, the decoder is configured to perform preprocessing on the first initial component to obtain a preprocessed first initial component, which is used as an input to the first neural network to obtain the first feature map, perform edge enhancement on the first initial component to obtain edge-enhanced image features, perform multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, and determine the preprocessed first initial component based on the multi-scale features.

[0196] Illustratively, the decoder is further configured to: perform a convolution operation on the edge-enhanced image features to obtain convolved features; perform a downsampling operation on the convolved features to obtain downsampled features; perform a channel transformation on the downsampled features to obtain channel-transformed features; perform an upsampling operation on the channel-transformed features to obtain upsampled features; and generate the multi-scale features based on the upsampling features and the channel-transformed features.

[0197] For example, the decoder is configured to upsample the second initial component to obtain an upsampled second initial component before generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, where the resolution of the upsampled second initial component is equal to the resolution of the first initial component.

[0198] Illustratively, the decoder is further configured to perform a feature enhancement process on the adjusted component, and, before obtaining a reconstructed target component corresponding to the second initial component, perform upsampling on the adjusted component to obtain an upsampled adjusted component, where a resolution of the upsampled adjusted component is equal to a resolution of the first initial component.

[0199] Illustratively, the decoder is further configured to perform a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component, and then perform upsampling on the target component to obtain an upsampled target component, where a resolution of the upsampled target component is equal to a resolution of the first initial component.

[0200] Illustratively, the decoder is further configured to perform a feature enhancement process on the adjusted component using at least one residual block network to obtain a target component corresponding to the second initial component, or to perform a feature enhancement process on the adjusted component using a U-Net network to obtain a target component corresponding to the second initial component.

[0201] Based on the same application concept as the above methods, a decoding side device (also called a video decoder) according to an embodiment of the present invention, from the hardware perspective, can be specifically seen in Figure 8 for a structural diagram of its hardware architecture. The decoding side device includes a processor 811 and a machine-readable storage medium 812. The machine-readable storage medium 812 stores machine-executable instructions that can be executed by the processor 811. The processor 811 is configured to implement the methods of the above embodiments 1 to 12 of the present invention by executing the machine-executable instructions.

[0202] For example, in one embodiment, processor 811 executes machine-executable instructions to: Decoding a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, wherein a resolution of the first initial component is equal to or greater than a resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; A feature enhancement process is performed on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0203] Based on the same application concept as the above-mentioned methods, embodiments of the present invention provide an electronic device, which includes a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions executable by the processor, and the processor is configured to execute the machine-executable instructions to implement the decoding methods of the above-mentioned embodiments 1 to 12 of the present invention.

[0204] Based on the same application concept as the above method, an embodiment of the present invention further provides a machine-readable storage medium storing some computer instructions, which, when executed by a processor, implements the methods disclosed in the above examples of the present invention, such as the decoding methods in the above embodiments.

[0205] Based on the same application concept as the above method, an embodiment of the present invention further provides a computer program, which, when executed by a processor, implements the decoding method disclosed in the above example of the present invention.

[0206] Based on the same application concept as the above method, an embodiment of the present invention further provides a decoding device applied to a decoding side, the decoding device comprising: a decoding module configured to decode a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, wherein a resolution of the first initial component is equal to or greater than a resolution of the second initial component; a determination module configured to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; and a processing module configured to perform a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component.

[0207] Exemplarily, if the reconstructed image block is a YUV format reconstructed image block, the first initial component is a luminance component, and the second initial component is a chrominance U component and / or a chrominance V component. Alternatively, if the reconstructed image block is an RGB format reconstructed image block, the first initial component is a G component, and the second initial component is an R component and / or a B component.

[0208] Illustratively, when generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the determination module is configured to specifically acquire first image features corresponding to the first initial component, the first image features including image features not processed by a neural network and / or image features acquired by a neural network, and acquire second image features corresponding to the second initial component, the second image features including image features not processed by a neural network and / or image features acquired by a neural network, and generate the adjusted component based on the first image features and the second image features.

[0209] Illustratively, when the first image feature includes an image feature that has not been processed by a neural network, the first image feature includes at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature, and when the second image feature includes an image feature that has not been processed by a neural network, the second image feature includes at least one of a texture feature, a subjective feature, a frequency domain feature, or a histogram feature.

[0210] Exemplarily, when the first image feature includes a first feature map obtained by a neural network and the second image feature includes a second feature map obtained by a neural network, when generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the determination module is configured to specifically input the first initial component into a first neural network to obtain the first feature map, input the second initial component into a second neural network to obtain the second feature map, and perform an addition operation on the first feature map and the second feature map to obtain the adjusted component, or perform a concatenation operation on the first feature map and the second feature map to obtain the adjusted component.

[0211] Exemplarily, when the first image feature includes a weight coefficient map obtained by a neural network and the second image feature includes a second feature map obtained by a neural network, when generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, the determination module is configured to specifically input the first initial component to a first neural network, obtain a first feature map corresponding to the first initial component, perform guided filtering on the first feature map to obtain the weight coefficient map, input the second initial component to a second neural network, obtain the second feature map, and generate the adjusted component based on the weight coefficient map and the second feature map.

[0212] Exemplarily, when performing guided filtering on the first feature map to obtain the weight coefficient map, the determination module is specifically configured to perform a convolution operation on the first feature map to obtain a convolved feature map, and perform weight mapping on the convolved feature map to obtain the weight coefficient map.

[0213] Exemplarily, the determination module is configured to perform weight mapping on the convoluted feature map and obtain the weight coefficient map by, specifically, performing a pooling operation on the convoluted feature map, performing a full connection operation and a ReLU activation operation on the pooled feature map, performing a full connection operation and a Sigmoid activation operation on the ReLU activation feature map, and generating the weight coefficient map based on the convoluted feature map and the Sigmoid activation feature map.

[0214] Exemplarily, when performing weight mapping on the convolved feature map to obtain the weight coefficient map, the determination module is configured to specifically perform a convolution operation and a sigmoid activation operation on the convolved feature map to obtain the weight coefficient map.

[0215] Exemplarily, when generating the adjusted component based on the weighting factor map and the second feature map, the determination module is specifically configured to: perform a multiplication operation on the weighting factor map and the second feature map to obtain a multiplied feature map; perform a convolution operation on the first feature map to obtain a convolved feature map; perform an addition operation on the multiplied feature map and the convolved feature map to obtain an added feature map; and perform a concatenation operation on the added feature map and the convolved feature map to obtain the adjusted component.

[0216] For example, when the first image feature includes K first feature maps obtained by a neural network and the second image feature includes K+1 second feature maps obtained by a neural network, the determining module, when generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, specifically, for a first first feature map and a first second feature map, input the first initial component into the neural network to obtain the first first feature map, and input the second initial component into the neural network to obtain the first second feature map. the i-1th first feature map and the i-1th second feature map, where i is an integer between 2 and K, and K is a positive integer greater than 1; the i-1th first feature map is input to a neural network to obtain the i-th first feature map; feature fusion is performed on the i-1th first feature map and the i-1th second feature map to obtain fused features; the fused features are input to the neural network to obtain the i-th second feature map; and after obtaining the final second feature map, the adjusted components are generated based on the final second feature map.

[0217] Exemplarily, when performing feature fusion on the first feature map and the second feature map to obtain the fused feature, the determination module is configured to: specifically perform an addition operation on the first feature map and the second feature map to obtain the fused feature; or perform a concatenation operation on the first feature map and the second feature map to obtain the fused feature; or perform guided filtering on the first feature map to obtain a weight coefficient map corresponding to the first feature map, and generate the fused feature based on the weight coefficient map and the second feature map.

[0218] Exemplarily, when inputting the first initial component into a first neural network to obtain the first feature map, the determination module is specifically configured to extract M-dimensional (M is a positive integer) image features from the first initial component, concatenate the M-dimensional image features to obtain concatenated multidimensional features, and input the multidimensional features into a first neural network to obtain the first feature map.

[0219] Illustratively, the M-dimensional image features include at least one of image features after performing convolution on the first initial component with one convolution kernel, image features after performing cascade convolution on the first initial component with two convolution kernels, first-order spatial domain features after performing processing on the first initial component with a Sobel operator, second-order spatial domain features after performing processing on the first initial component with a Laplacian operator, and frequency domain features after performing a Fourier transform on the first initial component.

[0220] Exemplarily, the determination module is configured, when inputting the first initial component to a first neural network to obtain the first feature map, to specifically perform a wavelet transform on the first initial component to obtain multiple frequency bands after the wavelet transform, and input the multiple frequency bands or some of the multiple frequency bands into the first neural network to obtain the first feature map. When inputting the second initial component to a second neural network to obtain the second feature map, the determination module is configured, when inputting the second initial component to a second neural network to obtain the second feature map, to specifically perform a wavelet transform on the second initial component to obtain multiple frequency bands after the wavelet transform, and input the multiple frequency bands or some of the multiple frequency bands into the second neural network to obtain the second feature map.

[0221] Exemplarily, the determination module is configured to, before inputting the first initial component into a first neural network to obtain the first feature map, further perform preprocessing on the first initial component to obtain a preprocessed first initial component, which is used to input the preprocessed first initial component into the first neural network to obtain the first feature map. When preprocessing the first initial component to obtain the preprocessed first initial component, the determination module is specifically configured to perform edge enhancement on the first initial component to obtain edge-enhanced image features, perform multi-scale feature extraction on the edge-enhanced image features to obtain multi-scale features, and determine the preprocessed first initial component based on the multi-scale features.

[0222] Exemplarily, the determination module is configured, when performing multi-scale feature extraction on the edge-enhanced image features to obtain the multi-scale features, to specifically perform a convolution operation on the edge-enhanced image features to obtain convolved features, perform a downsampling operation on the convolved features to obtain down-sampled features, perform a channel transformation on the down-sampled features to obtain channel-transformed features, perform an upsampling operation on the channel-transformed features to obtain up-sampled features, and generate the multi-scale features based on the upsampling features and the channel-transformed features.

[0223] Illustratively, the processing module is further configured to, based on the first initial component and the second initial component, perform upsampling on the second initial component before generating an adjusted component corresponding to the second initial component to obtain an upsampled second initial component, where a resolution of the upsampled second initial component is equal to a resolution of the first initial component.

[0224] Illustratively, the processing module is configured to, before performing a feature enhancement process on the adjusted component and obtaining a restored target component corresponding to the second initial component, further perform upsampling on the adjusted component to obtain an upsampled adjusted component, where a resolution of the upsampled adjusted component is equal to a resolution of the first initial component.

[0225] Illustratively, the processing module is configured to perform a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component, and then perform upsampling on the target component to obtain an upsampled target component, where the resolution of the upsampled target component is equal to the resolution of the first initial component.

[0226] Illustratively, when performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component, the processing module is configured to perform the feature enhancement process on the adjusted component using at least one residual block network to obtain the target component, or to perform the feature enhancement process on the adjusted component using a U-Net network to obtain the target component.

[0227] Those skilled in the art will appreciate that embodiments of the present invention may be provided as a method, a system, or a computer program product. The present invention may employ an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Also, embodiments of the present invention may employ the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0228] The above is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art to make. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention. [Explanation of symbols]

[0229] Steps 201-203 811 processor 812 Machine-readable storage medium

Claims

1. 1. A decoding method comprising: decoding a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, the resolution of the first initial component being equal to or greater than the resolution of the second initial component; generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component; A decoding method comprising:

2. the reconstructed image block is a reconstructed image block in YUV format, the first initial component is a luminance component, and the second initial component is a chrominance U component and a chrominance V component; 2. The method of claim 1 .

3. generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, obtaining a first image feature corresponding to the first initial component; obtaining a second image feature corresponding to the second initial component; generating the adjusted component based on the first image feature and the second image feature; the first image features include image features that have not been processed by a neural network; the second image features include image features that have not been processed by a neural network; 2. The method of claim 1 .

4. the first image features include Haar wavelet frequency domain features; the second image features include Haar wavelet frequency domain features; 4. The method of claim 3.

5. generating an adjusted component corresponding to the second initial component based on the first initial component and the second initial component, performing at least one Haar wavelet transform on the first initial component, and acquiring a plurality of frequency bands after the transform as a first feature map; performing at least one Haar wavelet transform on the second initial component, and acquiring a plurality of frequency bands after the transform as a second feature map; performing a concatenation operation on the first feature map and the second feature map to obtain the adjusted components.

4. The method according to claim 1 or 3.

6. performing a concatenation operation on the first feature map and the second feature map, performing a concatenation operation on the first feature map and the second feature map along a channel dimension.

6. The method of claim 5.

7. performing at least one Haar wavelet transform on the first initial component to obtain a plurality of transformed frequency bands; performing two Haar wavelet transforms on the first initial component to obtain a plurality of frequency bands obtained by the two Haar wavelet transforms; performing at least one Haar wavelet transform on the second initial component to obtain a plurality of frequency bands after the transform; performing two Haar wavelet transforms on the second initial component to obtain a plurality of frequency bands obtained by the two Haar wavelet transforms; 6. The method of claim 5.

8. performing a Haar wavelet transform on the first initial component multiple times to obtain multiple frequency bands after the transform; performing two Haar wavelet transforms on the first initial component to obtain a plurality of frequency bands obtained by the two Haar wavelet transforms; performing a Haar wavelet transform on the second initial component multiple times to obtain multiple frequency bands after the transform; performing two Haar wavelet transforms on the second initial component to obtain a plurality of frequency bands obtained by the two Haar wavelet transforms; 6. The method of claim 5.

9. performing two Haar wavelet transforms on the first initial component and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms; performing a first Haar wavelet transform on the first initial component to obtain a plurality of frequency bands obtained by the first Haar wavelet transform; performing a second Haar wavelet transform on the plurality of frequency bands obtained by the first Haar wavelet transform, and obtaining the plurality of frequency bands obtained by the second Haar wavelet transform; performing two Haar wavelet transforms on the second initial component and obtaining a plurality of frequency bands obtained by the two Haar wavelet transforms; performing a first Haar wavelet transform on the second initial component to obtain a plurality of frequency bands obtained by the first Haar wavelet transform; performing a second Haar wavelet transform on the plurality of frequency bands obtained by the first Haar wavelet transform, and obtaining the plurality of frequency bands obtained by the second Haar wavelet transform; 9. The method according to claim 7 or 8.

10. The step of performing a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component includes: performing a feature enhancement process on the adjusted components using a plurality of residual block networks to obtain the target components; The method according to any one of claims 1 to 9.

11. 1. An encoding method comprising: encoding a first initial component and a second initial component of a current image block to obtain a bitstream corresponding to the current image block; a resolution of the first initial component is equal to or greater than a resolution of the second initial component; 10. A coding method comprising:

12. the current image block is in YUV format, the first initial component is a luma component, and the second initial component is a chroma U component and a chroma V component; 2. The method of claim 1 .

13. A decoding device, comprising: a decoding module configured to decode a bitstream corresponding to a current image block to obtain a reconstructed image block, the reconstructed image block including a first initial component and a second initial component, the resolution of the first initial component being equal to or greater than the resolution of the second initial component; a determination module configured to generate an adjusted component corresponding to the second initial component based on the first initial component and the second initial component; a processing module configured to perform a feature enhancement process on the adjusted component to obtain a restored target component corresponding to the second initial component. A decoding device characterized by:

14. 1. An encoding device, comprising: an encoding module configured to encode a first initial component and a second initial component of a current image block to obtain a bitstream corresponding to the current image block; a resolution of the first initial component is equal to or greater than a resolution of the second initial component; 1. An encoding device comprising:

15. A decryption side device, a processor; a machine-readable storage medium having stored thereon machine-executable instructions executable by said processor; The processor is configured to execute the machine-executable instructions to implement the decoding method of any one of claims 1 to 10. A decryption device characterized by:

16. An encoding side device, a processor; a machine-readable storage medium having stored thereon machine-executable instructions executable by said processor; The processor is configured to execute the machine-executable instructions to perform the encoding method of claim 11 or 12. An encoding device characterized by:

17. 1. A machine-readable storage medium, comprising: The machine-readable storage medium has stored thereon a plurality of computer instructions; The computer instructions, when executed by a processor, Implementing the decoding method according to any one of claims 1 to 10, or Implementing the encoding method according to claim 11 or 12 A machine-readable storage medium comprising:

18. A computer program product comprising a computer program, When the computer program is executed by a processor, Implementing the decoding method according to any one of claims 1 to 10, or Implementing the encoding method according to claim 11 or 12 1. A computer program product comprising:

Images