Image reconstruction, encoding and decoding methods, and related devices

Abstract

The present invention provides an image reconstruction, encoding and decoding method, reconstruction model training method, and related devices. The image reconstruction method is based on an image reconstruction model, and the image reconstruction model includes a network input layer and a network processing layer. The method includes: using the network input layer to process an input residual image, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed, and the preset size is the input size of the image reconstruction model; using the network processing layer to process the residual image to be processed, and obtaining an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image. Avoiding upsampling or downsampling of the image outside the image reconstruction model is conducive to the combination of the image reconstruction model with the encoder and the decoder.

Description

Technical Field

[0001] The present invention relates to the technical field of video coding and decoding, and in particular to an image reconstruction, coding and decoding method, and related devices. Background Art

[0002] The amount of video image data is relatively large, and it is usually necessary to compress the video pixel data (RGB, YUV, etc.). The compressed data is called a video stream, which is transmitted to the user end through a wired or wireless network and then decoded for viewing. The entire video encoding process includes block division, prediction, transformation, quantization, encoding and other processes. In order to more effectively compress video data, high-resolution images are downsampled to low-resolution images during encoding and decoding. When high-resolution images are needed, they are enlarged through upsampling or reconstructed through super-resolution technology.

[0003] Super-resolution reconstruction technology not only needs to enlarge the low-resolution image, but also reconstruct the missing information through the model to restore the high-resolution image. The model of super-resolution reconstruction technology usually includes priors, neural networks, etc.

[0004] In the prior art, the reconstruction model of the super-resolution reconstruction technology is difficult to be combined with a decoder and an encoder when performing image reconstruction. Summary of the invention

[0005] The present invention provides an image reconstruction, encoding and decoding method and related devices, which can improve the combination effect of an image reconstruction model with a decoder and an encoder.

[0006] To solve the above technical problems, the first technical solution provided by the present invention is: to provide an image reconstruction method, the image reconstruction method is based on an image reconstruction model, the image reconstruction model includes a network input layer and a network processing layer, the method includes: using the network input layer to process the input residual image, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed, the preset size is the input size of the image reconstruction model; using the network processing layer to process the residual image to be processed to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0007] To solve the above technical problems, the second technical solution provided by the present invention is: providing an image reconstruction device, the image reconstruction device includes a network input module and a network processing module, the network input module includes a network input layer, and the network processing module includes a network processing layer; the network input module uses the network input layer to process the input residual image, adjusts the size of the input residual image to a preset size, and obtains a residual image to be processed; the preset size is the input size of the image reconstruction model; the network processing module uses the network processing layer to process the residual image to be processed to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0008] To solve the above technical problems, the third technical solution provided by the present invention is: providing a decoding method, comprising: decoding the acquired code stream to obtain an input residual image; using the network input layer in the image reconstruction model to process the input residual image, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed; the preset size is the input size of the image reconstruction model; using the network processing layer in the image reconstruction model to process the input residual image to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0009] To solve the above technical problems, the fourth technical solution provided by the present invention is: to provide a decoding device, comprising: a decoding module, used to decode the acquired code stream to obtain an input residual image; a network input module, used to process the input residual image using the network input layer in the image reconstruction model, and adjust the size of the input residual image to a preset size to obtain a residual image to be processed; the preset size is the input size of the image reconstruction model; a network processing module, used to process the input residual image using the network processing layer in the image reconstruction model to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0010] To solve the above technical problems, the fifth technical solution provided by the present invention is: providing a coding method, comprising: obtaining a coded residual image, the coded residual image being used as an input residual image; using a network input layer in an image reconstruction model to process the input residual image, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed; the preset size is the input size of the image reconstruction model; using a network processing layer in the image reconstruction model to process the input residual image to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0011] To solve the above technical problems, the sixth technical solution provided by the present invention is: to provide a coding device, comprising: an acquisition module, used to encode a residual image, the encoded residual image being used as an input residual image; a network input module, used to process the input residual image using a network input layer in an image reconstruction model, and adjust the size of the input residual image to a preset size to obtain a residual image to be processed; the preset size is the input size of the image reconstruction model; a network processing module, used to process the input residual image using a network processing layer in the image reconstruction model to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0012] To solve the above technical problems, the seventh technical solution provided by the present invention is: to provide an electronic device, comprising a processor and a memory coupled to each other, wherein the memory is used to store program instructions for implementing any of the above methods; and the processor is used to execute the program instructions stored in the memory.

[0013] In order to solve the above technical problems, the eighth technical solution provided by the present invention is: providing a computer-readable storage medium storing a program file, and the program file can be executed to implement any of the above methods.

[0014] The beneficial effect of the present invention is different from the prior art. The present invention uses the network input layer in the image reconstruction model to process the input residual image, adjusts the size of the input residual image to a preset size, and obtains a residual image to be processed. The preset size is the input size of the image reconstruction model; the network processing layer in the image reconstruction model is used to process the residual image to be processed to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image. It avoids upsampling or downsampling of the image outside the image reconstruction model, which is conducive to the combination of the image reconstruction model with the encoder and the decoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0016] Figure 1 is a schematic flow chart of a first embodiment of an image reconstruction method of the present invention;

[0017] Figure 2a and Figure 2b Schematic diagram of image segmentation;

[0018] Figure 2c Schematic diagram of image combination;

[0019] Figure 3 A schematic diagram of a flow chart of an embodiment of an image reconstruction method for an image reconstruction model;

[0020] Figure 4 is a flowchart of a processing method of an image processing layer;

[0021] Figure 5 Schematic diagram of the processing flow of the residual module;

[0022] Figure 6 Schematic diagram of the processing flow of the attention module;

[0023] Figure 7 is a schematic flow chart of a second embodiment of an image reconstruction method of the present invention;

[0024] Figure 8 is a schematic structural diagram of an embodiment of an image reconstruction device of the present invention;

[0025] Fig. 9 A schematic diagram of a flow chart of an embodiment of a decoding method of the present invention;

[0026] Fig.10 A schematic diagram of the structure of a decoding device according to an embodiment of the present invention;

[0027] Fig.11 A schematic diagram of a flow chart of an embodiment of an encoding method of the present invention;

[0028] Fig.12 is a schematic structural diagram of an embodiment of an encoding device of the present invention;

[0029] Fig.13 It is a structural schematic diagram of an embodiment of an electronic device of the present invention;

[0030] Fig.14 It is a schematic diagram of the structure of the computer-readable storage medium of the present invention. Specific implementation methods

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0032] In the prior art, before inputting the image into the neural network, it is necessary to upsample the image in YUV4:2:0 format to YUV4:4:4. After the neural network outputs the processed image, the processed image in YUV4:4:4 format is downsampled to convert it into an image in YUV4:2:0 format. This process requires additional upsampling and downsampling, which is not conducive to the combination of the neural network and the encoder and decoder. The image reconstruction method provided in the present application does not require additional upsampling and downsampling processing, and can be processed into a suitable size in the image reconstruction model. Specifically, the present application provides an image reconstruction model, which includes a network input layer and a network processing layer. The network input layer can process the input image, adjust the size of the input image to a preset size, and then process it in the network processing layer, avoiding additional upsampling and downsampling, which is conducive to the combination of the model with the encoder and decoder. For details, please refer to Figure 1 , the image reconstruction methods include:

[0033] Step S11: using the network input layer to process the input residual image, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed; the preset size is the input size of the image reconstruction model.

[0034] Specifically, the input residual image is input into the image reconstruction model, the input residual image is processed using the network input layer of the image reconstruction model, and the size of the input residual image is adjusted to a preset size, which is the input size of the image reconstruction model.

[0035] In a specific embodiment, before the input residual image is input into the image reconstruction model, the input residual image needs to be preprocessed. Specifically, it is assumed that the input size required by the image reconstruction model is in YUV4:2:0 format, and the size of the input block is W*H, that is, the sizes of the Y component, U component, and V component are W*H, W / 2*H / 2, and W / 2*H / 2, respectively. Before the input residual image is input into the image reconstruction model, the residual image to be reconstructed is segmented to obtain a plurality of input residual images. Specifically, the input residual image is obtained based on the image blocks obtained by segmenting the residual image to be reconstructed. It should be noted that the residual image to be reconstructed includes a Y component residual image, a U component residual image, and a V component residual image, and the input residual image includes a Y component input residual image, a U component input residual image, and a V component input residual image. In one embodiment, the residual image to be reconstructed is segmented to obtain a plurality of image blocks; the image blocks are filled with the pixels around each image block to obtain the input residual image corresponding to each image block. Specifically, taking the Y component residual image as an example, Figure 2aAs shown, the Y component residual image is segmented to obtain n 128*128 input residual images. In another embodiment, in order to ensure the continuity of the image boundary, the boundary of the input residual image can be filled. Specifically, the residual image is segmented to obtain multiple segmented blocks, and the segmented blocks are filled with pixels around each segmented block to obtain the input residual image. Taking the Y component residual image as an example, as shown in Figure 2b As shown, the Y component residual image is segmented to obtain n 128*128 segment blocks, and the segment blocks are filled with 16 pixels around the 128*128 segment blocks to obtain the input residual image, that is, the final input residual image is 160*160.

[0036] It can be understood that after the image reconstruction model processes each input residual image, the processed images are spliced ​​according to the corresponding positions to obtain the final image. Figure 2c As shown in FIG. 1 , the 160*160 input residual image in the second column and the second row is processed by the image reconstruction model, and an output block of 320*320 is output. The padding pixels in the output block are removed, and the valid area (256*256) of the output block is placed at the corresponding position of the second column and the second row.

[0037] In addition, in the current existing image reconstruction methods, the input of the image reconstruction model lacks additional prior information, which is not conducive to the image reconstruction result. The present application adds additional information to the input information of the image reconstruction model. Specifically, the input residual image input to the image reconstruction model includes additional information, and the additional information includes at least one of the image component, the quantization parameter map corresponding to the image component, and the intra-frame and inter-frame prediction values ​​corresponding to the image component. Specifically, when the input residual image does not include additional information. Input the YUV component of the low-resolution residual image, and output the YUV component of the high-resolution residual image. When the input residual image includes additional information. Input the YUV component of the low-resolution residual image and the additional information, and output the YUV component of the high-resolution residual image. Among them, the additional information includes but is not limited to any component of the reconstructed image, QP, any component of the intra-frame prediction image, any component of the inter-frame prediction, and various combinations.

[0038] In a specific embodiment, when the input residual image contains additional information, before the additional information is input into the image reconstruction model, the additional information needs to be preprocessed, and the size of the additional information is adjusted to a preset size. Specifically, the additional information is padded, and the size of the additional information is adjusted to a preset size. Specifically, when the additional information is single point data, for example, the additional information is a quantization parameter QP, and the quantization parameter QP=32, the quantization parameter QP needs to be padded to an input block of W*H. When the additional information is a one-dimensional vector, such as an upper reference pixel of 1*2W, it is necessary to multiply the transposed vector with the vector to obtain a 2W*2W matrix, and adjust the size to the input block size W*H; for example, a left reference pixel of 2H*1, it is necessary to multiply the vector with the transposed vector to obtain a 2H*2H matrix, and adjust the size to the input block size W*H. When the additional information is a two-dimensional matrix, such as when the matrix size is W / 2*H / 2, it is necessary to adjust the matrix size to the input block size W*H.

[0039] After the residual image and the additional information are preprocessed to obtain the input residual image, the input residual image is processed by the network input layer to adjust the size of the input residual image to a preset size. Specifically, the network input layer includes a deconvolution layer and a splicing layer. The U component input residual image and the V component input residual image are processed by the deconvolution layer so that the size of the U component input residual image and the V component input residual image matches the size of the Y component input residual image; the U component input residual image and the V component input residual image processed by the deconvolution layer are spliced ​​with the Y component input residual image and the additional information, and then the size of the input residual image is adjusted to a preset size; when the additional information is not included, the U component input residual image and the V component input residual image processed by the deconvolution layer are spliced ​​with the Y component input residual image, and then the size of the input residual image is adjusted to a preset size.

[0040] Please combine the specific Figure 3 , Figure 3 The processing process of the image reconstruction model of the present invention on the input residual model. The convolution layer parameters are described using [in, k, k, out], where in is the number of input channels, k is the convolution kernel size, and out is the number of output channels. In addition, S is the convolution step size, which defaults to 1. Specifically, in the network input layer, a deconvolution (deConv) layer with a step size of [1, 3, 3, 1] is used to input the U component into the residual image U Rec LR and V component input residual image V Rec LR The size of the Y component of the input residual image is adjusted to the size Y Rec LRThe sizes are the same, and then the inputs are concatenated on the channel layer through the concat layer. Specifically, in the concatenation layer, the U component is input into the residual image U Rec LR , V component input residual image V Rec LR , the size of the Y component input residual image Y Rec LR And additional information E are spliced. After being processed by the network output layer, the size of the input residual image can be adjusted to a preset size.

[0041] Step S12: using the network processing layer to process the residual image to be processed to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0042] Specifically, after the input residual image is processed by the network input layer, a residual image to be processed is obtained, and the residual image to be processed is further transmitted to the network processing layer, and the residual image to be processed is processed by the network processing layer to obtain an output residual image. It should be noted that the resolution of the output residual image is greater than the resolution of the input residual image. That is, the input residual image is a low-resolution residual image, and the output residual image is a high-resolution residual image.

[0043] For details, please combine Figure 4 , the residual image to be processed is processed using the network processing layer, including:

[0044] Step S41: using the first convolutional layer to process the residual image to be processed.

[0045] For details, please combine Figure 3 In the low-level feature extraction module, the first convolution layer is used to process the residual image to be processed. The parameters of the first convolution layer conv1 are [m, 3, 3, 64], where m is the input channel of the first convolution layer, and the number of input channels of the first convolution layer is determined by the convolution kernel of the first convolution layer and the number of additional information. In a specific embodiment, m = 3 + n, 3 represents the three components of YUV, and n represents the number of additional information. In a specific embodiment, the additional information includes the predicted value of the input YUV component, that is, the number of additional information is 3, then the input channel of the first convolution layer is 6.

[0046] Step S42: Process the output of the first convolutional layer using a residual module.

[0047] Please combine Figure 3 In the repetitive main module, the output of the first convolutional layer is processed by the residual module. Specifically, the repetitive main module, i.e., the residual module, includes N cascaded residual units. In a specific embodiment, the number of residual units N=16.

[0048] Specifically, the output of the first convolutional layer is processed in sequence using N cascaded residual units, where N is greater than or equal to 1. The input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit.

[0049] Please combine Figure 5 , Figure 5 : is a flow chart of an embodiment of a method for processing a residual unit. Specifically, in the Mth residual unit, R sub-residual units A are used in sequence to process the input of the Mth residual unit, and the sub-residual unit includes a convolution layer and an activation layer. Figure 5 As shown, the number of sub-residual units A is 2. Specifically, the input of the Mth residual unit is processed in sequence using the convolutional layer conv7[64×3×3×64], the activation layer ReLU1, the convolutional layer conv8[64×3×3×64], and the activation layer ReLU2. If M is 1, the output of the first convolutional layer is processed in sequence using the convolutional layer conv7[64×3×3×64], the activation layer ReLU1, the convolutional layer conv8[64×3×3×64], and the activation layer ReLU2. After the second sub-residual unit is processed, the output of the Rth sub-residual unit (the second sub-sub-residual unit) is processed using the third convolutional layer conv9[64×3×3×64]. The output of the third convolutional layer conv9[64×3×3×64] is processed using the attention module AB. The output of the attention module AB is multiplied by the output of the third convolutional layer conv9[64×3×3×64] to obtain the product; the input of the Mth residual unit is added to the product to obtain the output of the Mth residual unit. If M is 1, the output of the first convolutional layer is added to the product to obtain the output of the first residual unit; if M is 2, the output of the first residual unit is added to the product to obtain the output of the second residual unit.

[0050] Please combine Figure 6 , Figure 6: is a flowchart of the processing method of the attention module AB. Specifically, in the attention module AB, the output of the third convolution layer conv9[64×3×3×64] is processed by the sequentially cascaded pooling layer GAP, convolution layer conv10[64×1×1×64], activation layer ReLU, convolution layer conv11[64×1×1×64], and normalization layer softmax. Specifically, the pooling layer GAP processes the output of the third convolution layer conv9[64×3×3×64], the convolution layer conv10[64×1×1×64] processes the output of the pooling layer GAP, the activation layer ReLU processes the output of the convolution layer conv10[64×1×1×64], the convolution layer conv11[64×1×1×64] processes the output of the activation layer ReLU, and the normalization layer softmax processes the output of the convolution layer conv11[64×1×1×64].

[0051] Step S43: Use the second convolutional layer to process the output of the residual module.

[0052] For details, please combine Figure 3 ,In the high-level feature extraction module, the output of the residual module is processed using the second convolutional layer conv2[64×3×3×64].

[0053] Step S44: Use the upsampling layer to process the output of the second convolutional layer.

[0054] For details, please combine Figure 3 In the upsampling module, the output of the second convolutional layer is processed by the upsampling layer. The upsampling layer includes the convolutional layer conv3[64×3×3×256] and the shuffle function. The output of the second convolutional layer conv2[64×3×3×64] is processed by the convolutional layer conv3[64×3×3×256], and then the output of the convolutional layer conv3[64×3×3×256] is processed by the shuffle function.

[0055] Step S45: Use the output layer to process the output of the upsampling layer to obtain an output residual image.

[0056] Specifically, the purpose of the output layer is to adjust the output size to the output size in the network so that the output image can be used directly, thereby avoiding the up-sampling and down-sampling post-processing of the output image in the prior art. For example, the output image of the prior art network is in YUV4:4:4 format, which requires additional post-processing to convert the image to YUV4:2:0 format, while the network proposed in this proposal can directly output images in YUV4:2:0 format. Please combine Figure 3The output layer is three parallel convolution layers [64,3,3,1] Conv4, Conv5, Conv6, where the convolution step size of Conv4 and Conv5 is 2. The output residual image output by the output layer is a high-resolution residual image. The output residual image includes the Y component output residual image Y Rec SR , U component output residual image U Rec SR , V component output residual image V Rec SR .

[0057] By using the method of this embodiment, a high-resolution output residual image can be obtained.

[0058] See also Figure 7 , Figure 7 is a flow chart of a second embodiment of the image reconstruction method of the present invention. In this embodiment, step S71 and step S72 are the same as those described above. Figure 1 The step S11 and the step S12 in the first embodiment shown are the same, except that, after the step S72, the present embodiment further includes:

[0059] Step S73: Combine the output residual images to obtain a super-resolution image.

[0060] Specifically, in the above content, it is proposed to segment the residual image to obtain multiple input residual images, then after processing each input residual image through the image reconstruction model, an output residual image is obtained, and it is necessary to further combine the output residual images to obtain a super-resolution image. The super-resolution image is a complete image with a high resolution.

[0061] Step S74: up-sample the predicted image to obtain a sampled image.

[0062] The residual image is defined as a difference image formed by subtracting the predicted image from the real image. In order to obtain a complete reconstructed image, the super-resolution image needs to be added to the predicted image. Specifically, before adding the super-resolution image to the predicted image, the predicted image is upsampled to obtain a sampled image.

[0063] Step S75: Add the super-resolution image and the sampled image to obtain a reconstructed image.

[0064] By adding the super-resolution image to the sampled image, we can get the reconstructed image of the real image.

[0065] In one embodiment of the present application, it is also necessary to train an image reconstruction model, and determine the corresponding image reconstruction model based on the parameter information of the residual image; the parameter information includes at least one of the frame type of the input residual image and the additional information of the input residual image. Specifically, different image reconstruction models can be selected based on whether the frame type of the input residual image is an I frame or a B frame. Alternatively, different image reconstruction models can be selected based on the range of quantization parameters in the input residual image.

[0066] In one embodiment, the initial network model is trained using multiple sample images and quantization parameters to obtain multiple image reconstruction models. For example, one image reconstruction model may be trained for each QP (quantization parameter), for example, one image reconstruction model may be trained when the QP is 22. One image reconstruction model may be trained when the QP is 27. One image reconstruction model may be trained when the QP is 32. One image reconstruction model may be trained when the QP is 37. One model may be trained when the QP is 42.

[0067] In another embodiment, a model may be trained with a quantization parameter QP within a range. For example, a model a is trained when the QP is in the range of 22-32, wherein the QP in the range of 22-32 may be 22, 27, and 32. A model b is trained when the QP is in the range of 32-42, wherein the QP in the range of 32-42 may be 32, 37, and 42. During image reconstruction and encoding and decoding, the quantization parameter is compared with a preset value, and the corresponding image reconstruction model is determined based on the comparison result. For example, when the QP is not greater than the preset value 32, model a is selected, and when it is greater than the preset value 32, model b is selected. For another example, the QP combinations are {17, 22, 27}, {22, 27, 32}, {27, 32, 37}, {32, 37, 42}, {37, 42, 47}, and models 1, 2, 3, 4, and 5 are trained respectively. Model 1 is selected when QP is closest to 22, model 2 is selected when QP is closest to the preset value 27, model 3 is selected when QP is closest to the preset value 32, model 4 is selected when QP is closest to the preset value 37, and model 5 is selected when QP is closest to the preset value 42. The degree of quantization distortion is determined by the quantization parameter QP. Generally, the larger the QP, the greater the distortion caused by quantization, and vice versa. When encoding a video sequence, the QP of each image frame is based on the QP of the sequence and varies within a certain range according to the encoding configuration. That is, the quantization parameter characterizes the degree of distortion of the input image.

[0068] In another embodiment of the present application, the frame types of multiple sample images can also be determined, and the initial network model can be trained using sample images of different frame types to obtain multiple image reconstruction models. For example, a model a is trained using sample images of frame type I, and a model b is trained using sample images of frame type B. When selecting an image reconstruction model, if the input residual image is an I frame, model a is selected, and if the input residual image is a B frame, model b is selected.

[0069] In another embodiment of the present application, the frame types of multiple sample images can also be determined; the initial network model is trained using sample images of different frame types and quantization parameters to obtain multiple image reconstruction models. An image reconstruction model with a QP of 22, an image reconstruction model with a QP of 27, an image reconstruction model with a QP of 32, an image reconstruction model with a QP of 37, and a model with a QP of 42 are trained using sample images of frame types of I frames. Alternatively, an image reconstruction model with a QP of 22, an image reconstruction model with a QP of 27, an image reconstruction model with a QP of 32, an image reconstruction model with a QP of 37, and a model with a QP of 42 are trained using sample images of frame types of B frames. For another example, a model a with a QP range of 22-32 and a model b with a QP range of 32-42 are trained using sample images of frame types of I frames. During image reconstruction and encoding and decoding, the quantization parameter is compared with the preset value, and the image reconstruction model corresponding to the component is determined based on the frame type and the comparison result.

[0070] In one embodiment, for different QP requirements, the training QP includes models of 22, 27, 32, 37, and 42, and the QP range is [0, QPmax]. When QP is 22, 27, 32, 37, and 42 and trained independently, the QP usage intervals corresponding to each model during encoding are [0, 24], [25, 29], [30, 34], [35, 39], and [40, QPmax]. When three QPs are combined for training, for example, QP = {22, 27, 32} is used to train a lowQP model, and QP = {32, 37, 42} is used to train a highQP model, the QP usage interval of the lowQP model during encoding is [0, 32], and the QP usage interval of the high model during encoding is [33, QPmax]. When five QPs are combined to train a model, they are directly applied to the entire QP range during encoding.

[0071] In another embodiment, when a model is trained independently for each of the I and B frames, the training frame type is selected to be consistent with the current encoded image frame type during encoding. When a model is trained for both I and B frames, it is directly applied to the images currently encoded as I and B frames during encoding.

[0072] In another embodiment, for the combination requirements of different QPs and different frame types, the training QP includes 22, 27, 32, 37, and 42, the QP range is [0, QPmax], and the frame type is I and B frames. The frame type and QP can be considered independent of each other, so the two can be directly combined. For example, when I and B frames are trained independently and QP is trained independently, 5 QP models for I frames and B frames are trained respectively, for a total of 10 models.

[0073] When QP is used as additional information as network input, only one model needs to be trained, and this model can be directly used for the entire QP range during encoding.

[0074] In one embodiment, the sample image of the training model includes an input residual image corresponding to the first encoding condition, and an output residual image corresponding to the first encoding condition. Or the sample image of the training model includes an input residual image corresponding to the first encoding condition, and an unencoded output residual image. Or the sample image of the training model includes an input residual image corresponding to the first encoding condition, and an output residual image corresponding to the first encoding condition and an unencoded output residual image. Wherein, the resolution of the output residual image is greater than the resolution of the input residual image.

[0075] The image reconstruction method proposed in this application does not require additional upsampling and downsampling preprocessing processes, making the method easier to combine with the codec. And using the additional prior information as network input can effectively improve the effect of network training, thereby improving the quality of video images. The input and output of the image reconstruction model are both residual images, which are highly interpretable and directly learn the image residual, fundamentally reducing the computational difficulty of the network, thereby effectively reducing the difficulty of network training.

[0076] See also Figure 8 , is a schematic diagram of the structure of an embodiment of the image reconstruction device of the present invention, which specifically includes: a network input module 81 and a network processing module 82, wherein the network input module 81 includes a network input layer and the network processing module 82 includes a network processing layer.

[0077] The network input module 81 processes the input residual image using the network input layer, adjusts the size of the input residual image to a preset size, and obtains a residual image to be processed; the preset size is the input size of the image reconstruction model. The network processing module 82 processes the residual image to be processed using the network processing layer to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0078] The image reconstruction device of the present application does not need to perform additional upsampling and downsampling processing on the image outside the image reconstruction model, and can be better integrated with the encoder and decoder.

[0079] See also Fig. 9 , is a flow chart of an embodiment of a decoding method provided by the present invention, specifically comprising:

[0080] Step S91: Decode the acquired code stream to obtain an input residual image.

[0081] Step S92: Process the input residual image using the network input layer in the image reconstruction model, adjust the size of the input residual image to a preset size, and obtain a residual image to be processed; the preset size is the input size of the image reconstruction model.

[0082] Step S93: Process the input residual image using the network processing layer in the image reconstruction model to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0083] In this embodiment, the above Figures 1 to 7 The image reconstruction model of the image reconstruction method shown processes the input residual image.

[0084] In another embodiment, the code stream further includes a filter tag, which indicates the type of the image reconstruction model. Figures 1 to 7 The image reconstruction model of the image reconstruction method shown and the existing neural network are taken as candidates. In practical applications, a suitable image reconstruction model can be selected from the image reconstruction model of the present application and the existing neural network for encoding based on coding parameters, costs, etc., and the syntactic elements of the image reconstruction model used are established, and the syntactic elements are encoded into the bitstream. During decoding, the image reconstruction model is selected from the model set based on the filtering tag, and the input residual image is processed using the network input layer in the selected image reconstruction model.

[0085] Specifically, the model set includes a first image reconstruction model and a second image reconstruction model. The first image reconstruction model is used to achieve the above Figures 1 to 7 The image reconstruction model of the image reconstruction method shown, and the second image reconstruction model is an existing neural network.

[0086] In one embodiment, a syntax element SR_CNN_FLAG is defined, and its value is 0 and 1. When the value is 0, it indicates that the existing neural network is used, and when the value is 1, it indicates that the above neural network is implemented. Figures 1 to 7 An image reconstruction model of the image reconstruction method shown.

[0087] See also Fig.10 , is a schematic diagram of the structure of an embodiment of a decoding device of the present invention, which specifically includes: a decoding module 101, a network input module 102 and a network processing module 103, wherein the network input module 102 includes a network input layer and the network processing module 103 includes a network processing layer.

[0088] The decoding module 101 is used to decode the acquired code stream to obtain an input residual image.

[0089] The network input module 102 processes the input residual image using the network input layer, adjusts the size of the input residual image to a preset size, and obtains a residual image to be processed; the preset size is the input size of the image reconstruction model. The network processing module 103 processes the residual image to be processed using the network processing layer to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0090] The decoding device of the present application does not need to perform additional upsampling and downsampling processing on the image outside the image reconstruction model, and can be better integrated with the encoder and decoder.

[0091] See also Fig.11 , is a flow chart of an embodiment of the encoding method of the present invention, which specifically includes:

[0092] Step S111: obtaining a coded residual image, wherein the coded residual image is used as an input residual image.

[0093] Step S112: using the network input layer in the image reconstruction model to process the input residual image, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed; the preset size is the input size of the image reconstruction model.

[0094] Step S113: using the network processing layer in the image reconstruction model to process the input residual image to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0095] In this embodiment, the above Figures 1 to 7 The image reconstruction model of the image reconstruction method shown processes the input residual image to obtain an output residual image.

[0096] In practical applications, a suitable image reconstruction model can be selected from the image reconstruction model of the present application and the existing neural network based on coding parameters, costs, etc. to process the input residual image, and the syntactic elements of the image reconstruction model used can be established, and the syntactic elements can be encoded into the bitstream. During decoding, the image reconstruction model is selected from the model set based on the filtering tag, and the input residual image is processed using the network input layer in the selected image reconstruction model.

[0097] Specifically, the model set includes a first image reconstruction model and a second image reconstruction model. The first image reconstruction model is used to achieve the above Figures 1 to 7 The image reconstruction model of the image reconstruction method shown, and the second image reconstruction model is an existing neural network.

[0098] See also Fig.12, is a schematic diagram of the structure of an embodiment of the encoding device of the present invention, which specifically includes: an acquisition module 121, a network input module 122, and a network processing module 123, wherein the network input module 122 includes a network input layer. The network processing module 123 includes a network processing layer.

[0099] The acquisition module 121 is used to encode the residual image, and the encoded residual image is used as the input residual image.

[0100] The network input module 122 processes the input residual image using the network input layer, adjusts the size of the input residual image to a preset size, and obtains a residual image to be processed; the preset size is the input size of the image reconstruction model. The network processing module 123 processes the residual image to be processed using the network processing layer to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image.

[0101] The decoding device of the present application does not need to perform additional upsampling and downsampling processing on the image outside the image reconstruction model, and can be better integrated with the encoder and decoder.

[0102] See also Fig.13 , is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. The electronic device includes a memory 132 and a processor 131 connected to each other.

[0103] The memory 132 is used to store program instructions for implementing any one of the above methods.

[0104] The processor 131 is configured to execute program instructions stored in the memory 132 .

[0105] The processor 131 may also be referred to as a CPU (Central Processing Unit). The processor 131 may be an integrated circuit chip having signal processing capabilities. The processor 131 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0106] The memory 132 can be a memory stick, a TF card, etc., which can store all the information in the electronic device, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory. It stores and retrieves information according to the location specified by the controller. Only with the memory can the electronic device have a memory function and ensure normal operation. The memory of the electronic device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its purpose. There is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but it is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0107] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented by other methods. For example, the device implementation method described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a system server, or a network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application.

[0111] See also Fig.14 , which is a schematic diagram of the structure of the computer-readable storage medium of the present invention. The storage medium of the present application stores a program file 141 that can implement all the above methods, wherein the program file 141 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage device includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, tablets, etc.

[0112] The above is only an implementation method of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image reconstruction method, characterized in that: The image reconstruction method is performed based on an image reconstruction model, wherein the image reconstruction model includes a network input layer and a network processing layer, and the method includes: Processing the input residual image by using the network input layer, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed, wherein the preset size is the input size of the image reconstruction model, and the input residual image includes additional information, wherein the additional information includes at least one of an image component of a reconstructed image, a quantization parameter map corresponding to the image component, and an intra-frame or inter-frame prediction value corresponding to the image component; Processing the residual image to be processed by using the network processing layer to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image; Combining the output residual images to obtain a super-resolution image; Perform upsampling processing on the predicted image to obtain a sampled image; The super-resolution image is added to the sampled image to obtain a reconstructed image.

2. The image reconstruction method according to claim 1, characterized in that: The input residual image is obtained based on image blocks obtained by segmenting the residual image to be reconstructed.

3. The image reconstruction method according to claim 2, characterized in that: The residual image to be reconstructed includes a Y component residual image, a U component residual image and a V component residual image, and the input residual image includes a Y component input residual image, a U component input residual image and a V component input residual image.

4. The image reconstruction method according to claim 2, characterized in that: Before the step of processing the input residual image using the network input layer, the method includes: Segmenting the residual image to be reconstructed to obtain a plurality of image blocks; The image blocks are filled with pixel points around each of the image blocks to obtain the input residual image corresponding to each of the image blocks.

5. The image reconstruction method according to any one of claims 1 to 4, characterized in that: Before the step of processing the input residual image by using the network input layer to adjust the size of the input residual image to a preset size, the method includes: The additional information is filled and the size of the additional information is adjusted to the preset size.

6. The image reconstruction method according to claim 4, characterized in that: The step of processing the input residual image by using the network input layer to adjust the size of the input residual image to a preset size includes: Processing the U component input residual image and the V component input residual image by using a deconvolution layer so that the sizes of the U component input residual image and the V component input residual image match the size of the Y component input residual image; The U component input residual image, the V component input residual image and the Y component input residual image processed by the deconvolution layer are spliced, and then the size of the input residual image is adjusted to the preset size.

7. The image reconstruction method according to claim 6, characterized in that: In response to the input residual image comprising additional information; The step of splicing the U component input residual image, the V component input residual image and the Y component input residual image after being processed by the deconvolution layer comprises: The U component input residual image, the V component input residual image, the Y component input residual image and the additional information after being processed by the deconvolution layer are spliced.

8. The image reconstruction method according to claim 1, characterized in that: The step of processing the residual image to be processed by using the network processing layer to obtain an output residual image includes: Processing the residual image to be processed by using the first convolutional layer; Processing the output of the first convolutional layer using a residual module; Processing the output of the residual module using a second convolutional layer; Processing the output of the second convolutional layer using an upsampling layer; The output of the upsampling layer is processed by the output layer to obtain the output residual image.

9. The image reconstruction method according to claim 8, characterized in that: The number of input channels of the first convolutional layer is determined by the number of convolution kernels of the first convolutional layer and the number of additional information.

10. The image reconstruction method according to claim 8, characterized in that: The residual module includes N cascaded residual units; The step of processing the output of the first convolutional layer by using the residual module includes: Processing the output of the first convolutional layer using N cascaded residual units in sequence, where N is greater than or equal to 1; Among them, the input of the first residual unit is the output of the first convolutional layer, and the input of the Nth residual unit is the output of the N-1th residual unit.

11. The image reconstruction method according to claim 10, characterized in that: In the Mth residual unit, Using R sub-residual units in sequence to process the input of the M-th residual unit, the sub-residual unit includes a convolution layer and an activation layer; Processing the output of the Rth residual sub-unit using the third convolutional layer; Processing the output of the third convolutional layer using an attention module; Multiplying the output of the attention module by the output of the third convolutional layer to obtain a product; The input of the Mth residual unit is added to the product to obtain an output of the Mth residual unit.

12. The image reconstruction method according to claim 11, characterized in that: The step of processing the output of the third convolutional layer by using the attention module comprises: The output of the third convolutional layer is processed using the pooling layer, convolutional layer, activation layer, convolutional layer, and normalization layer cascaded in sequence in the attention module.

13. The image reconstruction method according to claim 1, characterized in that: Before the step of processing the input residual image by using the network input layer, the method further includes: A corresponding image reconstruction model is determined based on parameter information of the input residual image; the parameter information includes at least one of a frame type of the input residual image and a quantization parameter of the input residual image.

14. The image reconstruction method according to claim 13, characterized in that: The parameter information includes a quantization parameter; the method further includes: The initial network model is trained using multiple sample images and the quantization parameters to obtain multiple image reconstruction models.

15. The image reconstruction method according to claim 13, characterized in that: The parameter information includes a frame type of the image; the method further includes: The initial network model is trained using sample images of different frame types to obtain multiple image reconstruction models.

16. The image reconstruction method according to claim 12, characterized in that: The parameter information of the input residual image includes a frame type and a quantization parameter of the image; the method further includes: The initial network model is trained using sample images of different frame types and the quantization parameters to obtain multiple image reconstruction models.

17. The image reconstruction method according to claim 14, characterized in that: The sample image includes an input residual image corresponding to a first encoding condition, and an output residual image corresponding to the first encoding condition; or The sample image includes an input residual image corresponding to a first encoding condition, and an unencoded output residual image; or The sample image includes an input residual image corresponding to a first encoding condition, an output residual image corresponding to the first encoding condition, and an unencoded output residual image.

18. An image reconstruction device, characterized in that: The image reconstruction device comprises a network input module and a network processing module, wherein the network input module comprises a network input layer, and the network processing module comprises a network processing layer; The network input module processes the input residual image using the network input layer, adjusts the size of the input residual image to a preset size, and obtains a residual image to be processed; the preset size is the input size of the image reconstruction model, and the input residual image includes additional information, and the additional information includes at least one of an image component of the reconstructed image, a quantization parameter map corresponding to the image component, and an intra-frame and inter-frame prediction value corresponding to the image component; The network processing module processes the residual image to be processed using the network processing layer to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image; The image reconstruction device is also used to combine the output residual images to obtain a super-resolution image; up-sample the predicted image to obtain a sampled image; and add the super-resolution image and the sampled image to obtain a reconstructed image.

19. A decoding method, characterized in that: include: Decode the acquired code stream to obtain an input residual image; Processing the input residual image using the network input layer in the image reconstruction model, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed; the preset size is the input size of the image reconstruction model, and the input residual image includes additional information, and the additional information includes at least one of an image component of the reconstructed image, a quantization parameter map corresponding to the image component, and an intra-frame or inter-frame prediction value corresponding to the image component; Processing the input residual image using a network processing layer in an image reconstruction model to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image; Combining the output residual images to obtain a super-resolution image; Perform upsampling processing on the predicted image to obtain a sampled image; The super-resolution image is added to the sampled image to obtain a reconstructed image.

20. The decoding method according to claim 19, characterized in that: The code stream also includes a filter tag; the filter tag represents the type of the image reconstruction model; The step of processing the input residual image using the network input layer in the image reconstruction model comprises: Selecting an image reconstruction model from a model set based on the filter tag; the model set includes a first image reconstruction model and a second image reconstruction model, the first image reconstruction model being used to implement the image reconstruction method according to any one of claims 1 to 15; The input residual image is processed using the selected network input layer in the image reconstruction model.

21. A decoding device, characterized in that: include: A decoding module, used for decoding the acquired code stream to obtain an input residual image; A network input module, used to process an input residual image using a network input layer in an image reconstruction model, adjust the size of the input residual image to a preset size, and obtain a residual image to be processed; the preset size is the input size of the image reconstruction model, and the input residual image includes additional information, and the additional information includes at least one of an image component of a reconstructed image, a quantization parameter map corresponding to the image component, and an intra-frame or inter-frame prediction value corresponding to the image component; A network processing module, used to process the input residual image using a network processing layer in an image reconstruction model to obtain an output residual image; the resolution of the output residual image is greater than the resolution of the input residual image; The decoding device is also used to combine the output residual images to obtain a super-resolution image; up-sample the predicted image to obtain a sampled image; and add the super-resolution image and the sampled image to obtain a reconstructed image.

22. A coding method, characterized in that: include: Acquire a coded residual image, the coded residual image being used as an input residual image; Processing the input residual image using the network input layer in the image reconstruction model, adjusting the size of the input residual image to a preset size, and obtaining a residual image to be processed; the preset size is the input size of the image reconstruction model, and the input residual image includes additional information, and the additional information includes at least one of an image component of the reconstructed image, a quantization parameter map corresponding to the image component, and an intra-frame or inter-frame prediction value corresponding to the image component; Processing the input residual image using a network processing layer in an image reconstruction model to obtain an output residual image; The resolution of the output residual image is greater than the resolution of the input residual image.

23. The encoding method according to claim 22, characterized in that: The step of processing the input residual image using the network input layer in the image reconstruction model comprises: Selecting an image reconstruction model from a model set; the model set includes a first image reconstruction model and a second image reconstruction model, the first image reconstruction model being used to implement the image reconstruction method according to any one of claims 1 to 17; The input residual image is processed using the selected network input layer in the image reconstruction model.

24. A coding device, characterized in that: include: An acquisition module, used for encoding a residual image, wherein the encoded residual image is used as an input residual image; A network input module, used to process an input residual image using a network input layer in an image reconstruction model, adjust the size of the input residual image to a preset size, and obtain a residual image to be processed; the preset size is the input size of the image reconstruction model, and the input residual image includes additional information, and the additional information includes at least one of an image component of a reconstructed image, a quantization parameter map corresponding to the image component, and an intra-frame or inter-frame prediction value corresponding to the image component; A network processing module, used to process the input residual image using a network processing layer in an image reconstruction model to obtain an output residual image; The resolution of the output residual image is greater than the resolution of the input residual image.

25. An electronic device, characterized in that: It includes a processor and a memory coupled to each other, wherein: The memory is used to store program instructions for implementing the method according to any one of claims 1-17, 19-20, 22-23; The processor is configured to execute the program instructions stored in the memory.

26. A computer-readable storage medium, characterized in that: A program file is stored, and the program file can be executed to implement the method according to any one of claims 1-17, 19-20, 22-23.

Images