Image reconstruction, encoding and decoding methods, and related devices

Abstract

The present invention provides an image reconstruction, encoding and decoding method, and related devices. The image reconstruction method is based on an image reconstruction model comprising a feature processing layer, a super-resolution processing layer, and an enhancement processing layer. The method comprises: extracting features from an input image using the feature processing layer to obtain a feature image; processing the feature image using the super-resolution processing layer to obtain a first output image; and processing the feature image using the enhancement processing layer to obtain a second output image. The resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image. Image super-resolution and enhancement share the feature processing layer, improving computational performance and reducing computational resource overhead.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and in particular to an image reconstruction, coding and decoding method, and related devices. Background Art

[0002] Video image data is relatively large, and pixel data (RGB, YUV, etc.) typically needs to be compressed. This compressed data is called a video stream, which is transmitted to the user via a wired or wireless network for decoding and viewing. The entire video encoding process includes block division, prediction, transformation, quantization, and encoding. To more effectively compress video data, high-resolution images are downsampled to low-resolution images during encoding and decoding. When high-resolution images are needed, they are upsampled for magnification or reconstructed using super-resolution technology.

[0003] Super-resolution reconstruction technology not only requires enlarging the low-resolution image, but also reconstructing the missing information through a model to restore the high-resolution image. The model of super-resolution reconstruction technology usually includes priors, neural networks, etc.

[0004] In the existing technology, quality enhancement and super-resolution reconstruction networks are trained and used independently. When the two are used in combination, the overall performance is far lower than the sum of their independent performances, and the potential performance of the two cannot be fully tapped. Summary of the Invention

[0005] The present invention provides an image reconstruction, encoding and decoding method, and related devices, which can improve computing performance and reduce computing overhead.

[0006] To solve the above technical problems, the first technical solution provided by the present invention is: to provide an image reconstruction method, which is based on an image reconstruction model. The image reconstruction model includes a feature processing layer, a super-resolution processing layer and an enhancement processing layer. The method includes: using the feature processing layer to extract features of the input image to obtain a feature image; using the super-resolution processing layer to process the feature image to obtain a first output image; and using the enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0007] In order to solve the above technical problems, the second technical solution provided by the present invention is: to provide an image reconstruction device, the image reconstruction device includes: a feature processing module, a super-resolution processing module and an enhancement processing module; wherein, the feature processing module includes a feature processing layer, the super-resolution processing module includes a super-resolution processing layer, and the enhancement processing module includes an enhancement processing layer; the feature processing module is used to use the feature processing layer to extract features of the input image to obtain a feature image; the super-resolution processing module is used to use the super-resolution processing layer to process the feature image to obtain a first output image; and the enhancement processing module is used to use the enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0008] To solve the above technical problems, the third technical solution provided by the present invention is: providing a decoding method, comprising: decoding the acquired code stream to obtain an input image; using a feature processing layer to extract features of the input image to obtain a feature image; using a super-resolution processing layer to process the feature image to obtain a first output image; and using an enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0009] In order to solve the above technical problems, the fourth technical solution provided by the present invention is: providing a decoding device, including: a decoding module, used to decode the acquired code stream to obtain an input image; a feature processing module, used to use a feature processing layer to extract features of the input image to obtain a feature image; a super-resolution processing module, used to use the super-resolution processing layer to process the feature image to obtain a first output image; an enhancement processing module, used to use the enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0010] To solve the above technical problems, the fifth technical solution provided by the present invention is: providing a coding method, including: obtaining a coded reconstructed image, the coded reconstructed image being used as an input image; using a feature processing layer to extract features of the input image to obtain a feature image; using a super-resolution processing layer to process the feature image to obtain a first output image; and using an enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0011] In order to solve the above technical problems, the sixth technical solution provided by the present invention is: providing a coding device, including: an acquisition module, used to encode and reconstruct an image, the encoded and reconstructed image is used as an input image; a feature processing module, used to use a feature processing layer to extract features of the input image to obtain a feature image; a super-resolution processing module, used to use the super-resolution processing layer to process the feature image to obtain a first output image; an enhancement processing module, used to use the enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0012] To solve the above technical problems, the seventh technical solution provided by the present invention is: to provide an electronic device, including a processor and a memory coupled to each other, wherein the memory is used to store program instructions for implementing any of the above methods; and the processor is used to execute the program instructions stored in the memory.

[0013] In order to solve the above technical problems, the eighth technical solution provided by the present invention is: providing a computer-readable storage medium storing a program file, which can be executed to implement any of the above methods.

[0014] The beneficial effects of the present invention differ from those of the prior art. The image reconstruction method of the present invention is based on an image reconstruction model comprising a feature processing layer, a super-resolution processing layer, and an enhancement processing layer. The method comprises: extracting features from an input image using the feature processing layer to obtain a feature image; processing the feature image using the super-resolution processing layer to obtain a first output image; and processing the feature image using the enhancement processing layer to obtain a second output image. The resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image. Image super-resolution and enhancement share the same feature processing layer, improving computational performance and reducing computational resource overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0016] Figure 1 1 is a flow chart of a first embodiment of an image reconstruction method according to the present invention;

[0017] Figure 2a and Figure 2b Schematic diagram of image segmentation;

[0018] Figure 2c Schematic diagram of image combination;

[0019] Figure 3 is a flow chart of an embodiment of step S11;

[0020] Figure 4 is a flowchart of the image reconstruction method;

[0021] Figure 5 Schematic diagram of the processing flow of the residual module;

[0022] Figure 6 Schematic diagram of the processing flow of the attention module;

[0023] Figure 7 is a structural schematic diagram of an embodiment of an image reconstruction device of the present invention;

[0024] Figure 8 A schematic flow chart of an embodiment of a decoding method of the present invention;

[0025] Figure 9 A schematic diagram of a decoding process in a prior art embodiment;

[0026] Figure 10 A schematic diagram of a first embodiment of decoding according to the present invention;

[0027] Figure 11 A schematic diagram of a second embodiment of decoding according to the present invention;

[0028] Figure 12 A schematic diagram of a third embodiment of decoding according to the present invention;

[0029] Figure 13 A schematic diagram of a fourth embodiment of decoding according to the present invention;

[0030] Figure 14 A schematic diagram of a fifth embodiment of decoding according to the present invention;

[0031] Figure 15 1 is a schematic structural diagram of an embodiment of a decoding device of the present invention;

[0032] Figure 16 A schematic flow chart of an embodiment of an encoding method of the present invention;

[0033] Figure 17 1 is a schematic structural diagram of an embodiment of an encoding device of the present invention;

[0034] Figure 18 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention;

[0035] Figure 19It is a structural schematic diagram of the computer-readable storage medium of the present invention. Specific implementation methods

[0036] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0037] In the prior art, image resolution improvement and quality enhancement are achieved through two independent models. When the two are used in combination, the overall performance is far lower than the sum of the performance of the two when used independently, and the potential of the models cannot be fully released. The present application provides an image reconstruction method that can achieve resolution improvement and quality enhancement. The method can be implemented in the same model, and the resolution improvement and quality enhancement processes share a feature processing layer, reducing computational overhead, and the performance is better than the sum of the performance of the two when used independently. Specifically, the image reconstruction model includes: a feature processing layer, a super-resolution processing layer, and an enhancement processing layer. For details, please refer to Figure 1 , Figure 1 This is a flow chart of an embodiment of the image reconstruction method of the present invention, which specifically includes:

[0038] Step S11: Use the feature processing layer to extract features from the input image to obtain a feature image.

[0039] In a specific embodiment, before the input image is input into the image reconstruction model, the input image needs to be preprocessed. Specifically, assuming that the input size required by the image reconstruction model is in YUV4:2:0 format, the size of the input block is W*H, that is, the sizes of the Y component, U component, and V component are W*H, W / 2*H / 2, and W / 2*H / 2, respectively. Before the input image is input into the image reconstruction model, the image is segmented to obtain multiple input images. It should be noted that the image to be reconstructed includes a Y component image, a U component image, and a V component image, and the input image includes a Y component input image, a U component input image, and a V component input image. Taking the Y component image as an example, Figure 2a As shown, the Y component image is segmented to obtain n 128*128 input images. In another embodiment, in order to ensure the continuity of the image boundary, the boundary of the input image can be filled. In one embodiment, the input image is obtained based on the image block obtained by segmenting the image to be reconstructed. Specifically, the image to be reconstructed is segmented to obtain multiple image blocks, and the image blocks are filled with pixels around each image block to obtain the input image corresponding to each image block. Taking the Y component image as an example, as shown Figure 2bAs shown, the Y component image is segmented to obtain n 128*128 image blocks, and the image blocks are filled with 16 pixels around the 128*128 image blocks to obtain the input image, that is, the final input image is 160*160.

[0040] It can be understood that after the image reconstruction model processes each input image, the processed images output are spliced according to the corresponding positions to obtain the final image. Figure 2c As shown in the figure, the 160*160 input image in the second column and the second row is processed by the image reconstruction model, and an output block of 320*320 is output. The padding pixels in the output block are removed, and the valid area (256*256) of the output block is placed at the corresponding position in the second column and the second row.

[0041] For details, please combine Figure 3 , step S11 further includes:

[0042] Step S31: Process the input image and adjust the size of the input image to a preset size to obtain an image to be processed. The preset size is the input size of the image reconstruction model.

[0043] In the prior art, before inputting the image into the neural network, it is necessary to upsample the image in YUV4:2:0 format to YUV4:4:4. After the neural network outputs the processed image, the processed image in YUV4:4:4 format is downsampled to convert it into an image in YUV4:2:0 format. This process requires additional upsampling and downsampling, which is not conducive to the combination of the neural network and the encoder and decoder. The image reconstruction method provided by the present application does not require additional upsampling and downsampling processing, and can be processed into a suitable size in the image reconstruction model. Specifically, the present application provides an image reconstruction model, which includes a network input layer and a network processing layer. The network input layer can process the input image, adjust the size of the input image to a preset size, and then process it in the network processing layer, avoiding additional upsampling and downsampling, which is conducive to the combination of the model with the encoder and decoder.

[0044] In addition, in the currently available image reconstruction methods, the input of the image reconstruction model lacks additional prior information, which is not conducive to the image reconstruction results. The present application adds additional information to the input information of the image reconstruction model. Specifically, the input image input to the image reconstruction model includes additional information, and the additional information includes at least one of the image components, the quantization parameter map corresponding to the image components, and the intra-frame and inter-frame prediction values corresponding to the image components. Specifically, when the input image does not include additional information. Input the YUV components of the low-resolution image and output the YUV components of the high-resolution image. When the input image includes additional information. Input the YUV components of the low-resolution image and the additional information, and output the YUV components of the high-resolution image. Among them, the additional information includes but is not limited to any component of the reconstructed image, QP, any component of the intra-frame prediction image, any component of the inter-frame prediction, and various combinations.

[0045] In a specific embodiment, when the input image contains additional information, before the additional information is input into the image reconstruction model, the additional information needs to be preprocessed, and the size of the additional information is adjusted to a preset size. Specifically, the additional information is padded, and the size of the additional information is adjusted to a preset size. Specifically, when the additional information is single point data, for example, the additional information is a quantization parameter QP, and the quantization parameter QP=32, the quantization parameter QP needs to be padded to an input block of W*H. When the additional information is a one-dimensional vector, such as an upper reference pixel of 1*2W, it is necessary to multiply the transposed vector with the vector to obtain a 2W*2W matrix, and adjust the size to the input block size W*H; for example, a left reference pixel of 2H*1, it is necessary to multiply the vector with the transposed vector to obtain a 2H*2H matrix, and adjust the size to the input block size W*H. When the additional information is a two-dimensional matrix, such as when the matrix size is W / 2*H / 2, it is necessary to adjust the matrix size to the input block size W*H.

[0046] After preprocessing the image and additional information to obtain the input image, the input image is processed using the network input layer to adjust the size of the input image to a preset size. Specifically, the network input layer includes a deconvolution layer and a splicing layer. The deconvolution layer is used to process the U component input image and the V component input image so that the size of the U component input image and the V component input image matches the size of the Y component input image; the U component input image and the V component input image processed by the deconvolution layer are spliced with the Y component input image and the additional information, thereby adjusting the size of the input image to the preset size.

[0047] Please combine the specific Figure 4 , Figure 4The image reconstruction model of the present invention processes the input image. The convolution layer parameters are described using [in, k, k, out], where in is the number of input channels, k is the convolution kernel size, and out is the number of output channels. In addition, S is the convolution step size, which defaults to 1. Specifically, in the network input layer, a deconvolution layer with a step size of [1, 3, 3, 1] is used to transform the U component into the image U. Rec LR and V component input image V Rec LR Resize to match the size of the input image's Y component Rec LR The same size, and then through the concat layer to splice each input on the channel layer. Specifically, in the splicing layer, the U component is input to the image U Rec LR , V component input image V Rec LR , Y component input image size Y Rec LR and additional information E. After processing by the network output layer, the size of the input image can be adjusted to the preset size. It is understandable that when the input image does not include additional information, it is only necessary to splice the U component input image, the V component input image, and the Y component input image after deconvolution layer processing to adjust the size of the input image to the preset size.

[0048] Step S32: extracting features from the image to be processed to obtain a feature image.

[0049] Specifically, the adjusted input image is processed using the first convolutional layer; the output of the first convolutional layer is processed using the residual module; the output of the residual module is processed using the second convolutional layer, and the output of the first convolutional layer is added to the output of the second convolutional layer to obtain a feature image.

[0050] Please combine Figure 4 In the low-level feature extraction module, the first convolutional layer is used to process the image to be processed. The parameters of the first convolutional layer conv1 are [m, 3, 3, 64], where m is the input channel of the first convolutional layer. The number of input channels of the first convolutional layer is determined by the convolution kernel of the first convolutional layer and the amount of additional information. In a specific embodiment, m = 3 + n, where 3 represents the three YUV components and n represents the amount of additional information. In a specific embodiment, the additional information includes the predicted values of the input YUV components. That is, the amount of additional information is 3, and then the number of input channels of the first convolutional layer is 6.

[0051] In the repeating main module, the output of the first convolutional layer is processed using a residual module. Specifically, the repeating main module includes N cascaded residual units. In a specific embodiment, the number of residual units N=16.

[0052] Specifically, the output of the first convolutional layer is processed in sequence using N cascaded residual units, where N is greater than or equal to 1. The input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit.

[0053] Please combine Figure 5 , Figure 5 : This is a flow chart of an embodiment of a method for processing a residual unit. Specifically, in the Mth residual unit, R sub-residual units A are used in sequence to process the input of the Mth residual unit. The sub-residual unit includes a convolution layer and an activation layer. Figure 5 As shown in the figure, the number of sub-residual units A is 2. Specifically, the input of the M-th residual unit is processed in sequence using the convolutional layer conv7[64×3×3×64], the activation layer ReLU1, the convolutional layer conv8[64×3×3×64], and the activation layer ReLU2. If M is 1, the output of the first convolutional layer is processed in sequence using the convolutional layer conv7[64×3×3×64], the activation layer ReLU1, the convolutional layer conv8[64×3×3×64], and the activation layer ReLU2. After the second sub-residual unit is processed, the output of the R-th sub-residual unit (the second sub-residual unit) is processed using the third convolutional layer conv9[64×3×3×64]. The output of the third convolutional layer conv9[64×3×3×64] is processed using the attention module AB. Multiply the output of the attention module AB by the output of the third convolutional layer conv9[64×3×3×64] to obtain the product; add the input of the Mth residual unit to the product to obtain the output of the Mth residual unit. If M is 1, add the output of the first convolutional layer to the product to obtain the output of the first residual unit; if M is 2, add the output of the first residual unit to the product to obtain the output of the second residual unit.

[0054] Please combine Figure 6 , Figure 6This is a flowchart of the processing method of the attention module AB. Specifically, in the attention module AB, the output of the third convolutional layer conv9 [64×3×3×64] is processed by the pooling layer GAP, the convolutional layer conv10 [64×1×1×64], the activation layer ReLU, the convolutional layer conv11 [64×1×1×64], and the normalization layer softmax, which are cascaded in sequence in the attention module AB. Specifically, the pooling layer GAP processes the output of the third convolutional layer conv9 [64×3×3×64], the convolutional layer conv10 [64×1×1×64] processes the output of the pooling layer GAP, the activation layer ReLU processes the output of the convolutional layer conv10 [64×1×1×64], the convolutional layer conv11 [64×1×1×64] processes the output of the activation layer ReLU, and the normalization layer softmax processes the output of the convolutional layer conv11 [64×1×1×64].

[0055] For details, please combine Figure 4 In the high-level feature extraction module, the output of the module is processed using the second convolutional layer conv2[64×3×3×64]. After processing by the second convolutional layer, the output of the first convolutional layer is added to the output of the second convolutional layer to obtain the feature image.

[0056] Step S12: Process the feature image using a super-resolution processing layer to obtain a first output image; and process the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0057] Specifically, using the super-resolution processing layer to process the feature image to obtain the first output image includes: performing upsampling processing on the feature image; and processing the up-sampled feature image to obtain the first output image. Figure 4, the feature image is upsampled in the upsampling module. In the upsampling module, the feature image is processed by the upsampling layer. The upsampling layer includes the convolution layer conv3[64×3×3×256] and the shuffle function. The feature image is processed by the convolution layer conv3[64×3×3×256], and then the output of the convolution layer conv3[64×3×3×256] is processed by the shuffle function. The upsampled feature image is processed in the super-resolution output layer to obtain the first output image. Specifically, the purpose of the super-resolution output layer is to adjust the output size to the output size in the network so that the first output image can be used directly, thereby avoiding the upsampling and downsampling post-processing of the feature image in the prior art. For example, the network feature image of the prior art is in YUV4:4:4 format, and additional post-processing is required to convert the image into YUV4:2:0 format. However, the network proposed in this proposal can directly output images in YUV4:2:0 format. Please combine Figure 4 , the super-resolution output layer is three parallel [64,3,3,1] convolution layers, where the convolution step size of two convolution layers is 2. In the super-resolution output layer, the feature image is upsampled, and then the upsampled feature image is convolved by multiple parallel convolution layers to obtain the first output image. It can be understood that two convolution layers with a convolution step size of 2 are used to process the U component image and the V component image respectively, and another convolution layer is used to process the Y component image. The first output image output by the super-resolution output layer is a high-resolution image. The first output image includes the Y component first output image Y Rec SR , U component first output image U Rec SR , V component first output image V Rec SR The resolution of the first output image is greater than the resolution of the input image.

[0058] For details, please combine Figure 4In the enhanced output layer, the feature image is processed using the enhanced processing layer to produce a second output image. Specifically, the enhanced output layer consists of three parallel convolutional layers with a convolution stride of [64, 3, 3, 1], two of which have a convolution stride of 2. The second output image produced by the enhanced output layer is a high-quality image, consisting of a Y-component second output image (YQE), a U-component second output image (UQE), and a V-component second output image (VQE). The objective quality of the second output image is greater than that of the input image. It should be noted that objective quality is determined based on the peak signal-to-noise ratio (PSNR). After image compression, the output image typically differs to some extent from the original image. To measure the quality of processed images, the PSNR value is often used to determine whether a processing procedure is satisfactory. Specifically, in image processing, objective image evaluation often requires calculating the PSNR. The PSNR is an objective measure of image distortion or noise level. The higher the PSNR value between two images, the more similar they are. The general benchmark is 30dB, and image degradation below 30dB is more obvious.

[0059] Specifically, in the above content, it is proposed to segment the image to obtain multiple input images. Then, after processing each input image through the image reconstruction model, a first output image and a second output image are obtained. It is necessary to further combine the first output image and the second output image. Specifically, each first output image and second output image is obtained after processing an input image. By splicing all the first output images, the image after super-resolution of the image to be reconstructed can be obtained; by splicing all the second output images, the image after enhancement of the image to be reconstructed can be obtained. Specifically, by combining multiple first output images, a complete high-resolution image is obtained, and by combining multiple second output images, a complete enhanced image is obtained.

[0060] In one embodiment of the present application, an image reconstruction model needs to be trained. The corresponding image reconstruction model is determined based on image parameter information; the parameter information includes at least one of the frame type of the input image and the quantization parameter of the input image. Specifically, different image reconstruction models can be selected based on whether the input image frame type is an I frame or a B frame. Alternatively, different image reconstruction models can be selected based on the range of the quantization parameter in the input image. It should be noted that the quantization parameter represents the degree of image distortion.

[0061] In one embodiment, the initial network model is trained using multiple sample images and quantization parameters to obtain multiple image reconstruction models. For example, one image reconstruction model may be trained for each QP (quantization parameter). For example, one image reconstruction model may be trained when the QP is 22, one image reconstruction model may be trained when the QP is 27, one image reconstruction model may be trained when the QP is 32, one image reconstruction model may be trained when the QP is 37, and one model may be trained when the QP is 42.

[0062] In another embodiment, a model can be trained for a quantization parameter QP within a range. For example, model a is trained for a QP range of 22-32, where QPs within the range of 22-32 can be 22, 27, and 32. Model b is trained for a QP range of 32-42, where QPs within the range of 32-42 can be 32, 37, and 42. During image reconstruction and encoding and decoding, the quantization parameter is compared with a preset value, and the corresponding image reconstruction model is determined based on the comparison result. For example, model a is selected when the QP is not greater than the preset value of 32, and model b is selected when the QP is greater than the preset value of 32. For example, if the QP combinations are {17, 22, 27}, {22, 27, 32}, {27, 32, 37}, {32, 37, 42}, and {37, 42, 47}, models 1, 2, 3, 4, and 5 are trained respectively. Model 1 is selected when the QP is closest to 22, model 2 is selected when the QP is closest to 27, model 3 is selected when the QP is closest to 32, model 4 is selected when the QP is closest to 37, and model 5 is selected when the QP is closest to 42. The degree of quantization distortion is determined by the quantization parameter QP. Generally, a larger QP results in greater quantization distortion, and vice versa. When encoding a video sequence, the QP of each frame varies within a certain range based on the encoding configuration, taking the QP of the sequence as a benchmark. In other words, the quantization parameter characterizes the degree of distortion of the input image.

[0063] In another embodiment of the present application, the frame types of multiple sample images can be determined, and the initial network model can be trained using sample images of different frame types to obtain multiple image reconstruction models. For example, model a can be trained using sample images of I-frame type, and model b can be trained using sample images of B-frame type. When selecting an image reconstruction model, if the input image is an I-frame, model a is selected, and if the input image is a B-frame, model b is selected.

[0064] In another embodiment of the present application, the frame types of multiple sample images can also be determined; the initial network model is trained using sample images of different frame types and quantization parameters to obtain multiple image reconstruction models. Sample images of the frame type I are used to train an image reconstruction model with a QP of 22, an image reconstruction model with a QP of 27, an image reconstruction model with a QP of 32, an image reconstruction model with a QP of 37, and a model with a QP of 42. Alternatively, sample images of the frame type B are used to train an image reconstruction model with a QP of 22, an image reconstruction model with a QP of 27, an image reconstruction model with a QP of 32, an image reconstruction model with a QP of 37, and a model with a QP of 42. For another example, model a with a QP range of 22-32 and model b with a QP range of 32-42 are trained using sample images of the frame type I. During image reconstruction and encoding and decoding, the quantization parameter is compared with a preset value, and the image reconstruction model corresponding to the component is determined based on the frame type and the comparison result.

[0065] In one embodiment, to meet different QP requirements, models with QPs of 22, 27, 32, 37, and 42 are trained, and the QP range is [0, QPmax]. When QPs of 22, 27, 32, 37, and 42 are trained independently, the corresponding QP ranges used by each model during encoding are [0, 24], [25, 29], [30, 34], [35, 39], and [40, QPmax]. When three QP combinations are trained, for example, a lowQP model is trained with QP = {22, 27, 32}, and a highQP model is trained with QP = {32, 37, 42}, the QP range used during encoding for the lowQP model is [0, 32], and the QP range used for the highQP model is [33, QPmax]. When five QP combinations are trained into a single model, the entire QP range is directly applied during encoding.

[0066] In another embodiment, when a model is trained independently for each I and B frame, the training frame type is selected to be consistent with the current encoded image frame type during encoding. When a model is trained for both I and B frames, the model is directly applied to the current encoded image as an I frame or B frame type during encoding.

[0067] In another embodiment, to meet the requirements of combining different QPs with different frame types, training QPs include 22, 27, 32, 37, and 42, with a QP range of [0, QPmax], and frame types of I and B frames. Frame type and QP can be considered independent of each other, so the two can be directly combined. For example, when I and B frames are trained independently and their QPs are trained independently, five QP models are trained for I frames and five for B frames, for a total of ten models.

[0068] When QP is used as additional information as network input, only one model needs to be trained, and this model can be directly used for the entire QP range during encoding.

[0069] In one embodiment, the sample images of the training model include an encoded image corresponding to a first encoding condition and a first resolution, a real image corresponding to the first resolution, and an encoded image corresponding to the first encoding condition and a second resolution. Alternatively, the sample images include an encoded image corresponding to the first encoding condition and a first resolution, a real image corresponding to the first resolution, and a real image corresponding to the second resolution. Alternatively, the sample images include an encoded image corresponding to the first encoding condition and a first resolution, a real image corresponding to the first resolution, an encoded image corresponding to the first encoding condition and a second resolution, and a real image; wherein the second resolution is greater than the first resolution. Using the encoded high resolution as a super-resolution training label can achieve the purpose of improving network performance. In addition, when used as an upsampling module, since the encoded data is used as the label, the performance of the filtering and upsampling modules is kept independent, so that the performance of the combined use of the two can be superimposed.

[0070] The image reconstruction method proposed in this application does not require additional upsampling and downsampling preprocessing processes, making this method easier to combine with the codec. And using the additional prior information as network input can effectively improve the effect of network training, thereby improving the quality of video images. The input and output of the image reconstruction model are both images, which are highly interpretable and can directly learn images, fundamentally reducing the computational difficulty of the network, thereby effectively reducing the difficulty of network training. In addition, in the method proposed in this application, super-resolution processing and enhancement processing share the same feature processing layer, which improves the performance of the model and reduces computational overhead. This application can jointly train super-resolution processing and enhancement processing to effectively balance the performance between enhancement and super-resolution.

[0071] See Figure 7 , is a schematic diagram of the structure of an embodiment of the image reconstruction device of the present invention, the image reconstruction device includes: a feature processing module 71, a super-resolution processing module 72, and an enhancement processing module 73. Among them, the feature processing module 71 includes a feature processing layer, the super-resolution processing module 72 includes a super-resolution processing layer, and the enhancement processing module 73 includes an enhancement processing layer;

[0072] The feature processing module 71 is used to extract features from the input image using the feature processing layer to obtain a feature image.

[0073] The super-resolution processing module 72 is configured to process the feature image using the super-resolution processing layer to obtain a first output image; and the enhancement processing module 73 is configured to process the feature image using the enhancement processing layer to obtain a second output image. The resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0074] See Figure 8 , which is a flowchart of a first embodiment of a decoding method of the present invention, specifically includes:

[0075] Step S81: Decode the acquired code stream to obtain an input image.

[0076] Step S82: Use the feature processing layer to extract features from the input image to obtain a feature image.

[0077] Step S83: Process the feature image using a super-resolution processing layer to obtain a first output image; and process the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0078] Specifically, the decoding process of the prior art is as follows Figure 9 As shown, the process includes bitstream decoding → filtering (after filtering, a quality-enhanced image is obtained, i.e., the second output image) → upsampling (after upsampling, a high-resolution image is obtained, i.e., the first output image). In this embodiment, the image reconstruction model can directly replace the filtering module and the upsampling module. That is, the decoding process of the present application includes bitstream decoding → image reconstruction model (outputting the first output image and the second output image).

[0079] In another embodiment of the present application, the decoding method further includes upsampling the first output image. Figure 10 In this embodiment, the filtering module is replaced by an image reconstruction model, that is, the decoding process of this embodiment includes code stream decoding → image reconstruction model (outputting the first output image and the second output image) → upsampling (upsampling the first output image).

[0080] In another embodiment of the present application, before the step of extracting features from the input image using the feature processing layer to obtain the feature image, the step further includes: filtering the input image. Figure 11 In this embodiment, the upsampling module is replaced by an image reconstruction model, that is, the decoding process of this embodiment includes code stream decoding → filtering → image reconstruction model (outputting a first output image and a second output image).

[0081] In another embodiment of the present application, before the step of extracting features from the input image using the feature processing layer to obtain the feature image, the method further includes: determining the image reconstruction model corresponding to the input image from the model set based on the signal-to-noise ratio information; the model reconstruction model includes a first image reconstruction model and a second image reconstruction model; the first image reconstruction model is used to implement the above-mentioned image reconstruction method; the step of extracting features from the input image using the feature processing layer to obtain the feature image includes: extracting features from the input image using the feature processing layer in the selected image reconstruction model to obtain the feature image. Figure 12 In this embodiment, the image reconstruction model of the present application and existing neural networks are used as candidates, and a suitable model is selected from them based on the signal-to-noise ratio. Specifically, the existing neural network (i.e., the second image reconstruction model) includes a filtering module and an upsampling module. In other words, the decoding process of this embodiment includes bitstream decoding → filtering and upsampling or image reconstruction model.

[0082] In another embodiment of the present application, the existing neural network (ie, the second image reconstruction model) includes a filtering module, and the decoding method further includes upsampling the first output image. Figure 13 The decoding process of this embodiment includes code stream decoding→filtering or image reconstruction model→upsampling.

[0083] In another embodiment of the present application, the existing neural network (i.e., the second image reconstruction model) includes an upsampling module, and uses the feature processing layer in the selected image reconstruction model to extract features from the input image. Before obtaining the feature image, the step further includes: filtering the input image. Figure 14 The decoding process of this embodiment includes code stream decoding → filtering → upsampling or image reconstruction model.

[0084] In one embodiment, the code stream also includes a filter tag, which characterizes the type of image reconstruction model. When selecting an image reconstruction model, a selection can also be made based on the filter tag, which is a syntactic element. During the encoding and decoding process, if the image reconstruction model of the present application is directly used for processing, no syntactic element needs to be transmitted; if the image reconstruction model of the present application is used as a candidate module, additional syntactic elements need to be transmitted to indicate which candidate method is selected. The syntactic element CNN_FLAG is defined, and its values are 0, 1, 2, and 3. When the value is 0, it indicates that the image reconstruction model of the present application is not used. When the value is 1, it indicates that the existing filtering module and upsampling module are used. When the value is 2, it indicates that the filtering module is replaced by the image reconstruction model of the present application. When the value is 3, it indicates that the image reconstruction model of the present application replaces the upsampling module.

[0085] See Figure 15, is a structural diagram of an embodiment of the image decoding device of the present invention, which specifically includes: a decoding module 151, a feature processing module 152, a super-resolution output module 153 and an enhancement processing module 154.

[0086] The decoding module 151 is used to decode the acquired code stream to obtain an input image; the feature processing module 152 is used to use the feature processing layer to extract features of the input image to obtain a feature image; the super-resolution processing module 153 is used to use the super-resolution processing layer to process the feature image to obtain a first output image; the enhancement processing module 154 is used to use the enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0087] See Figure 16 , is a flow chart of an embodiment of the encoding method of the present invention, specifically comprising:

[0088] Step S161: Acquire a coded reconstructed image, which is used as an input image.

[0089] Step S162: Use the feature processing layer to extract features from the input image to obtain a feature image.

[0090] Step S163: Process the feature image using a super-resolution processing layer to obtain a first output image; and process the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0091] It should be noted that the coded reconstructed image in this embodiment is a pixel reconstructed image, and Figure 1 The reconstruction described is a high-resolution reconstruction.

[0092] In another embodiment, the filtering module can also be replaced by an image reconstruction model, that is, the encoding process of this embodiment includes image reconstruction model (outputting the first output image and the second output image) → upsampling (upsampling the first output image) → encoding to obtain a code stream.

[0093] In another embodiment, before the step of extracting features from the input image using the feature processing layer to obtain the feature image, the step further includes filtering the input image. Specifically, in this embodiment, the upsampling module is replaced by an image reconstruction model. That is, the encoding process of this embodiment includes bitstream filtering → image reconstruction model (outputting the first output image and the second output image) → encoding to obtain the bitstream.

[0094] In another embodiment of the present application, before the step of extracting features from the input image using the feature processing layer to obtain the feature image, the step further includes: determining the image reconstruction model corresponding to the input image from the model set based on the signal-to-noise ratio information; the model reconstruction model includes a first image reconstruction model and a second image reconstruction model; the first image reconstruction model is used to implement the above-mentioned image reconstruction method; the step of extracting features from the input image using the feature processing layer to obtain the feature image includes: extracting features from the input image using the feature processing layer in the selected image reconstruction model to obtain the feature image. Specifically, in this embodiment, the image reconstruction model of the present application and the existing neural network are used as candidates, and a suitable model is selected from them based on the signal-to-noise ratio. Specifically, the existing neural network (that is, the second image reconstruction model) includes a filtering module and an upsampling module. That is, the encoding process of this embodiment includes: bitstream filtering and upsampling or image reconstruction model → encoding to obtain bitstream.

[0095] In another embodiment of the present application, the existing neural network (i.e., the second image reconstruction model) includes a filtering module, and the encoding method further includes upsampling the first output image. Specifically, the decoding process of this embodiment includes: filtering or image reconstruction model → upsampling → encoding to obtain a bitstream.

[0096] In another embodiment of the present application, the existing neural network (i.e., the second image reconstruction model) includes an upsampling module, and uses a feature processing layer in the selected image reconstruction model to extract features from the input image. Before obtaining the feature image, the step further includes filtering the input image. Specifically, the encoding process of this embodiment includes filtering → upsampling or image reconstruction model → encoding to obtain a bitstream.

[0097] Specifically, after selecting a suitable model, a syntax element for the image reconstruction model used is established and encoded into the bitstream. During decoding, the image reconstruction model is selected from the model set based on the filter flag. In one embodiment, a syntax element CNN_FLAG is defined, with values of 0, 1, 2, and 3. A value of 0 indicates that the image reconstruction model of the present application is not used, a value of 1 indicates that the existing filtering module and upsampling module are used, a value of 2 indicates that the filtering module is replaced by the image reconstruction model of the present application, and a value of 3 indicates that the upsampling module is replaced by the image reconstruction model of the present application.

[0098] See Figure 17 , is a structural diagram of an embodiment of the encoding device of the present invention. Specifically, the encoding device includes an acquisition module 171, a feature processing module 172, a super-resolution output module 173 and an enhancement processing module 174.

[0099] The acquisition module 171 is used to acquire a coded reconstructed image, which is used as an input image; the feature processing module 172 is used to use a feature processing layer to extract features from the input image to obtain a feature image; the super-resolution processing module 173 is used to use a super-resolution processing layer to process the feature image to obtain a first output image; the enhancement processing module 174 is used to use an enhancement processing layer to process the feature image to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

[0100] See Figure 18 , which is a schematic structural diagram of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 132 and a processor 131 connected to each other.

[0101] The memory 132 is used to store program instructions for implementing any one of the above methods.

[0102] The processor 131 is configured to execute program instructions stored in the memory 132 .

[0103] The processor 131 may also be referred to as a CPU (Central Processing Unit). The processor 131 may be an integrated circuit chip having signal processing capabilities. The processor 131 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The general-purpose processor may be a microprocessor or any conventional processor.

[0104] The memory 132 can be a memory stick, a TF card, etc., which can store all the information in the electronic device, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory. It stores and retrieves information according to the location specified by the controller. Only with the memory can the electronic device have a memory function and ensure normal operation. The memory of the electronic device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented by other methods. For example, the device implementation method described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0106] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment as needed.

[0107] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0108] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, system server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application.

[0109] See also Figure 19, which is a structural diagram of the computer-readable storage medium of the present invention. The storage medium of the present application stores a program file 141 that can implement all the above methods, wherein the program file 141 can be stored in the above storage medium in the form of a software product, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage device includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0110] The above is only an implementation method of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image reconstruction method, characterized in that: The image reconstruction method is performed based on an image reconstruction model, wherein the image reconstruction model includes a feature processing layer, a super-resolution processing layer, and an enhancement processing layer. The method includes: When extracting features from the input image using the feature processing layer to obtain a feature image, the process includes: The image to be reconstructed includes a Y component image, a U component image, and a V component image, and the input image includes a Y component input image, a U component input image, and a V component input image; Using a deconvolution layer to process a U component input image and a V component input image so that the sizes of the U component input image and the V component input image match the size of the Y component input image, splicing the U component input image, the V component input image, and the Y component input image processed by the deconvolution layer, and then adjusting the size of the input image to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; Processing the image to be processed using a first convolutional layer; The output of the first convolutional layer is processed in sequence using N cascaded residual units of the residual module, where N is greater than or equal to 1, wherein the input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit; and in the Mth residual unit, the input of the Mth residual unit is processed in sequence using R sub-residual units, wherein the sub-residual unit includes a convolutional layer and an activation layer, and the output of the convolutional layer is used as the input of the activation layer; the output of the Rth sub-residual unit is processed using a third convolutional layer; the output of the third convolutional layer is processed using an attention module; the output of the attention module is multiplied by the output of the third convolutional layer to obtain a product; the input of the Mth residual unit is added to the product to obtain the output of the Mth residual unit; Processing the output of the residual module using a second convolutional layer; Adding the output of the first convolutional layer and the output of the second convolutional layer to obtain the feature image; Processing the feature image using the super-resolution processing layer to obtain a first output image; and The feature image is processed using the enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image.

2. The image reconstruction method according to claim 1, wherein: The input image is obtained based on image blocks obtained by segmenting the image to be reconstructed.

3. The image reconstruction method according to claim 2, wherein: Before the step of extracting features from the input image using the feature processing layer to obtain a feature image, the method includes: Segment the image to be reconstructed to obtain multiple image blocks; The image blocks are filled with pixel points around each of the image blocks to obtain the input image corresponding to each of the image blocks.

4. The image reconstruction method according to claim 1, wherein: The input image includes additional information, and the additional information includes at least one of an image component, a quantization parameter map corresponding to the image component, and an intra-frame or inter-frame prediction value corresponding to the image component.

5. The image reconstruction method according to claim 4, characterized in that: Processing the input image, adjusting the size of the input image to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; performing feature extraction on the image to be processed to obtain the output image; Before the step of processing the input image and adjusting the size of the input image to a preset size, the method includes: The additional information is filled and the size of the additional information is adjusted to the preset size.

6. The image reconstruction method according to claim 1, wherein: In response to the input image including additional information; The step of splicing the U component input image, the V component input image, and the Y component input image processed by the deconvolution layer includes: The U component input image, the V component input image, the Y component input image processed by the deconvolution layer and the additional information are spliced.

7. The image reconstruction method according to claim 1, wherein: The number of input channels of the first convolutional layer is determined by the number of convolution kernels and additional information of the first convolutional layer.

8. The image reconstruction method according to claim 1, wherein: The step of processing the output of the third convolutional layer by using the attention module includes: The output of the third convolutional layer is processed using the pooling layer, convolutional layer, activation layer, convolutional layer, and normalization layer cascaded in sequence in the attention module.

9. The image reconstruction method according to claim 1, wherein: The step of processing the feature image using the super-resolution processing layer to obtain a first output image includes: performing upsampling processing on the feature image; The up-sampled feature image is convolved using a plurality of convolutional layers connected in parallel to obtain the first output image.

10. The image reconstruction method according to claim 2, wherein: said processing the output image by using the super-resolution processing layer to obtain a first output image; After the output image is processed by the enhancement processing layer to obtain a second output image, the method further includes: The first output images are combined, and the second output images are combined.

11. The image reconstruction method according to claim 1, wherein: The step of extracting features from the input image using the feature processing layer to obtain image features may also include: A corresponding image reconstruction model is determined based on parameter information of the input image; the parameter information includes at least one of a frame type of the input image and a quantization parameter of the input image.

12. The image reconstruction method according to claim 11, characterized in that: The parameter information includes a quantization parameter; and the method further includes: The initial network model is trained using multiple sample images and the quantization parameters to obtain multiple image reconstruction models.

13. The image reconstruction method according to claim 11, characterized in that: The parameter information includes a frame type of the image; the method further includes: The initial network model is trained using sample images of different frame types to obtain multiple image reconstruction models.

14. The image reconstruction method according to claim 11, wherein: The parameter information includes a frame type and a quantization parameter of the image; the method further includes: The initial network model is trained using sample images of different frame types and the quantization parameters to obtain multiple image reconstruction models.

15. The image reconstruction method according to claim 12, wherein: The sample images include an encoded image corresponding to a first encoding condition and a first resolution, a real image corresponding to the first resolution, and an encoded image corresponding to the first encoding condition and a second resolution; or The sample images include an encoded image corresponding to a first encoding condition and a first resolution, a real image corresponding to the first resolution, and a real image corresponding to the second resolution; or The sample images include an encoded image corresponding to a first encoding condition and a first resolution, a real image corresponding to the first resolution, an encoded image corresponding to the first encoding condition and a second resolution, and a real image; The second resolution is greater than the first resolution.

16. An image reconstruction device, characterized in that: The image reconstruction device includes: a feature processing module, a super-resolution processing module and an enhancement processing module; wherein the feature processing module includes a feature processing layer, the super-resolution processing module includes a super-resolution processing layer, and the enhancement processing module includes an enhancement processing layer; The feature processing module is used to extract features from the input image using the feature processing layer to obtain a feature image; The super-resolution processing module is used to process the feature image using the super-resolution processing layer to obtain a first output image; and the enhancement processing module is used to process the feature image using the enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image; The feature processing module is used to extract features from the input image using the feature processing layer to obtain a feature image, including: The image to be reconstructed includes a Y component image, a U component image, and a V component image, and the input image includes a Y component input image, a U component input image, and a V component input image; The U component input image and the V component input image are processed by the deconvolution layer so that the sizes of the U component input image and the V component input image match the size of the Y component input image, and the U component input image, the V component input image and the Y component input image processed by the deconvolution layer are spliced, and then the size of the input image is adjusted to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; the image to be processed is processed by the first convolution layer; the output of the first convolution layer is processed in sequence by N cascaded residual units of the residual module, where N is greater than or equal to 1, wherein the input of the first residual unit is the output of the first convolution layer, and the input of the Nth residual unit is the Nth -1 residual unit output; and in the Mth residual unit, using R sub-residual units in sequence to process the input of the Mth residual unit, the sub-residual unit includes a convolution layer and an activation layer, and the output of the convolution layer is used as the input of the activation layer; using the third convolution layer to process the output of the Rth sub-residual unit; using the attention module to process the output of the third convolution layer; multiplying the output of the attention module with the output of the third convolution layer to obtain a product; adding the input of the Mth residual unit to the product to obtain the output of the Mth residual unit; using the second convolution layer to process the output of the residual module; adding the output of the first convolution layer with the output of the second convolution layer to obtain the feature image.

17. A decoding method, characterized in that: include: Decode the acquired code stream to obtain the input image; Use the feature processing layer to extract features from the input image to obtain a feature image; Processing the feature image using a super-resolution processing layer to obtain a first output image; as well as Processing the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image; The image to be reconstructed includes a Y component image, a U component image, and a V component image, the input image includes a Y component input image, a U component input image, and a V component input image, and the feature processing layer is used to extract features from the input image to obtain a feature image, including: Using a deconvolution layer to process the U component input image and the V component input image so that the sizes of the U component input image and the V component input image match the size of the Y component input image, splicing the U component input image, the V component input image, and the Y component input image processed by the deconvolution layer, and then adjusting the size of the input image to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; Processing the image to be processed using a first convolutional layer; The output of the first convolutional layer is processed in sequence using N cascaded residual units of the residual module, where N is greater than or equal to 1, wherein the input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit; and in the Mth residual unit, the input of the Mth residual unit is processed in sequence using R sub-residual units, wherein the sub-residual unit includes a convolutional layer and an activation layer, and the output of the convolutional layer is used as the input of the activation layer; the output of the Rth sub-residual unit is processed using a third convolutional layer; the output of the third convolutional layer is processed using an attention module; the output of the attention module is multiplied by the output of the third convolutional layer to obtain a product; the input of the Mth residual unit is added to the product to obtain the output of the Mth residual unit; Processing the output of the residual module using a second convolutional layer; The output of the first convolutional layer is added to the output of the second convolutional layer to obtain the feature image.

18. The decoding method according to claim 17, characterized in that: The method further comprises: Perform upsampling processing on the first output image.

19. The decoding method according to claim 17, wherein: Before the step of extracting features from the input image using the feature processing layer to obtain a feature image, the method further includes: The input image is filtered.

20. The decoding method according to claim 17, wherein: Before the step of extracting features from the input image using the feature processing layer to obtain a feature image, the method further includes: Determining an image reconstruction model corresponding to the input image from a model set based on signal-to-noise ratio information; the model reconstruction model includes a first image reconstruction model and a second image reconstruction model; the first image reconstruction model is used to implement the image reconstruction method according to any one of claims 1 to 15 above; The step of extracting features from the input image using the feature processing layer to obtain a feature image includes: The feature processing layer in the selected image reconstruction model is used to extract features from the input image to obtain a feature image.

21. The decoding method according to claim 20, characterized in that The second image reconstruction model includes a filtering module and an upsampling module.

22. The decoding method according to claim 20, characterized in that The second image reconstruction model includes a filtering module; The method further includes: performing upsampling processing on the first output image.

23. The decoding method according to claim 20, characterized in that The second image reconstruction model includes an upsampling module; Before the step of extracting features from the input image using the feature processing layer in the selected image reconstruction model to obtain a feature image, the method further includes: Perform filtering processing on the input image.

24. The decoding method according to claim 20, characterized in that The code stream further includes a filter flag, which indicates the type of the image reconstruction model.

25. A decoding device, characterized in that: include: A decoding module is used to decode the acquired code stream to obtain an input image; A feature processing module is used to extract features from the input image using the feature processing layer to obtain a feature image; a super-resolution processing module, configured to process the feature image using a super-resolution processing layer to obtain a first output image; an enhancement processing module, configured to process the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image; The image to be reconstructed includes a Y component image, a U component image, and a V component image, and the input image includes a Y component input image, a U component input image, and a V component input image. The feature processing module is used to extract features from the input image using a feature processing layer to obtain a feature image, including: Using a deconvolution layer to process the U component input image and the V component input image so that the sizes of the U component input image and the V component input image match the size of the Y component input image, splicing the U component input image, the V component input image, and the Y component input image processed by the deconvolution layer, and then adjusting the size of the input image to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; Processing the image to be processed using a first convolutional layer; The output of the first convolutional layer is processed in sequence using N cascaded residual units of the residual module, where N is greater than or equal to 1, wherein the input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit; and in the Mth residual unit, the input of the Mth residual unit is processed in sequence using R sub-residual units, wherein the sub-residual unit includes a convolutional layer and an activation layer, and the output of the convolutional layer is used as the input of the activation layer; the output of the Rth sub-residual unit is processed using a third convolutional layer; the output of the third convolutional layer is processed using an attention module; the output of the attention module is multiplied by the output of the third convolutional layer to obtain a product; the input of the Mth residual unit is added to the product to obtain the output of the Mth residual unit; Processing the output of the residual module using a second convolutional layer; The output of the first convolutional layer is added to the output of the second convolutional layer to obtain the feature image.

26. A coding method, characterized in that include: Acquire a coded reconstructed image, wherein the coded reconstructed image serves as an input image; Extracting features from the input image using a feature processing layer to obtain a feature image; Processing the feature image using a super-resolution processing layer to obtain a first output image; as well as Processing the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image; The image to be reconstructed includes a Y component image, a U component image, and a V component image, and the input image includes a Y component input image, a U component input image, and a V component input image. When extracting features from the input image using the feature processing layer to obtain a feature image, the process includes: Using a deconvolution layer to process the U component input image and the V component input image so that the sizes of the U component input image and the V component input image match the size of the Y component input image, splicing the U component input image, the V component input image, and the Y component input image processed by the deconvolution layer, and then adjusting the size of the input image to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; Processing the image to be processed using a first convolutional layer; The output of the first convolutional layer is processed in sequence using N cascaded residual units of the residual module, where N is greater than or equal to 1, wherein the input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit; and in the Mth residual unit, the input of the Mth residual unit is processed in sequence using R sub-residual units, wherein the sub-residual unit includes a convolutional layer and an activation layer, and the output of the convolutional layer is used as the input of the activation layer; the output of the Rth sub-residual unit is processed using a third convolutional layer; the output of the third convolutional layer is processed using an attention module; the output of the attention module is multiplied by the output of the third convolutional layer to obtain a product; the input of the Mth residual unit is added to the product to obtain the output of the Mth residual unit; Processing the output of the residual module using a second convolutional layer; The output of the first convolutional layer is added to the output of the second convolutional layer to obtain the feature image.

27. An encoding device, characterized in that include: An acquisition module, configured to acquire a coded reconstructed image, wherein the coded reconstructed image serves as an input image; A feature processing module is used to extract features from the input image using the feature processing layer to obtain a feature image; a super-resolution processing module, configured to process the feature image using a super-resolution processing layer to obtain a first output image; an enhancement processing module, configured to process the feature image using an enhancement processing layer to obtain a second output image; wherein the resolution of the first output image is greater than the resolution of the input image; and the objective quality of the second output image is greater than the objective quality of the input image; The image to be reconstructed includes a Y component image, a U component image, and a V component image, and the input image includes a Y component input image, a U component input image, and a V component input image. The feature processing module uses the feature processing layer to extract features from the input image to obtain a feature image, including: Using a deconvolution layer to process the U component input image and the V component input image so that the sizes of the U component input image and the V component input image match the size of the Y component input image, splicing the U component input image, the V component input image, and the Y component input image processed by the deconvolution layer, and then adjusting the size of the input image to a preset size to obtain an image to be processed, where the preset size is the input size of the image reconstruction model; Processing the image to be processed using a first convolutional layer; The output of the first convolutional layer is processed in sequence using N cascaded residual units of the residual module, where N is greater than or equal to 1, wherein the input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit; and in the Mth residual unit, the input of the Mth residual unit is processed in sequence using R sub-residual units, wherein the sub-residual unit includes a convolutional layer and an activation layer, and the output of the convolutional layer is used as the input of the activation layer; the output of the Rth sub-residual unit is processed using a third convolutional layer; the output of the third convolutional layer is processed using an attention module; the output of the attention module is multiplied by the output of the third convolutional layer to obtain a product; the input of the Mth residual unit is added to the product to obtain the output of the Mth residual unit; Processing the output of the residual module using a second convolutional layer; The output of the first convolutional layer is added to the output of the second convolutional layer to obtain the feature image.

28. An electronic device, characterized in that: It includes a processor and a memory coupled to each other, wherein: The memory is used to store program instructions for implementing the method according to any one of claims 1-15, 17-24, and 26; The processor is configured to execute the program instructions stored in the memory.

29. A computer-readable storage medium, characterized in that A program file is stored, and the program file can be executed to implement the method according to any one of claims 1-15, 17-24, and 26.

Images