Image reconstruction method, image encoding and decoding method and related equipment

Abstract

The present invention discloses an image reconstruction method, an image encoding and decoding method and related equipment. The image reconstruction method comprises: obtaining an input image; inputting the input image into a global residual connection branch and a residual neural network branch of a super-resolution reconstruction network respectively, so as to upsample the input image using the global residual connection branch to obtain basic information of the input image, and reconstruct the input image using the residual neural network branch to obtain residual information of the input image; fusing the basic information and the residual information to generate an output image, and the resolution of the output image is higher than the resolution of the input image. In the above manner, the present invention can improve the performance of the super-resolution reconstruction network, so as to facilitate the realization of a good super-resolution reconstruction effect.

Description

Technical Field

[0001] The present invention relates to the field of video coding and decoding technology, and in particular to a super-resolution reconstruction method, an image coding method, an image decoding method, an encoder, a decoder, and a computer-readable storage medium. Background Art

[0002] The amount of video image data is relatively large, and the video pixel data usually needs to be compressed to form a video stream, which is transmitted to the user end via a wired or wireless network, and then decoded for the user to watch. After the pixels of the image are reconstructed, the image resolution is low. Therefore, reconstructing the lower resolution image to a higher resolution is one of the important requirements in the field of image processing so that enough information can be extracted. At present, due to the unreasonable design of the existing super-resolution reconstruction technology, it is easy to cause low image reconstruction quality. Summary of the invention

[0003] In view of this, the main technical problem solved by the present invention is to provide a super-resolution reconstruction method, an image encoding method, an image decoding method, an encoder, a decoder, and a computer-readable storage medium, which can improve the performance of the super-resolution reconstruction network to facilitate achieving good super-resolution reconstruction effects.

[0004] To solve the above technical problems, a technical solution adopted by the present invention is: to provide an image reconstruction method, which includes: obtaining an input image; inputting the input image into a global residual connection branch and a residual neural network branch of a super-resolution reconstruction network respectively, so as to upsample the input image using the global residual connection branch to obtain basic information of the input image, and reconstruct the input image using the residual neural network branch to obtain residual information of the input image; fusing the basic information and the residual information to generate an output image, wherein the resolution of the output image is higher than the resolution of the input image.

[0005] In one embodiment of the present invention, the input image is upsampled using a global residual connection branch, and obtaining basic information of the input image includes: using at least one upsampling processing module to upsample the input image respectively to obtain sub-basic information corresponding to each upsampling processing module, wherein at least one upsampling processing module includes at least one of a convolutional neural network and a priori filter; based on the sub-basic information corresponding to each upsampling processing module, basic information is obtained.

[0006] In one embodiment of the present invention, the sub-basis information corresponding to each upsampling processing module includes first sub-basis information corresponding to the convolutional neural network and second sub-basis information corresponding to the prior filter; based on the sub-basis information corresponding to each upsampling processing module, basic information is obtained, including: weighted fusion of the first sub-basis information and the second sub-basis information to obtain the basic information.

[0007] In one embodiment of the present invention, at least one upsampling processing module includes a convolutional neural network, the convolutional neural network includes a feature extraction layer, a first convolutional layer and a merge and sort layer that are cascaded in sequence, and the convolutional neural network is used to upsample the input image to obtain the first sub-basic information, including: using the feature extraction layer to extract features of the input image; using the output of the feature extraction layer as the input of the first convolutional layer, performing convolution processing to obtain multiple channel feature maps; using the merge and sort layer to fuse the multiple channel feature maps to obtain the first sub-basic information.

[0008] In one embodiment of the present invention, at least one upsampling processing module includes a priori filter, the priori filter includes a resampling filter, and upsampling the input image using the priori filter includes: performing upsampling interpolation filtering on the input image using the resampling filter.

[0009] In one embodiment of the present invention, the input image includes a luminance component and a chrominance component, and the tap coefficient corresponding to the luminance component is greater than the tap coefficient corresponding to the chrominance component.

[0010] In one embodiment of the present invention, before using a resampling filter to perform up-sampling interpolation filtering on an input image, the method includes: assigning a filter coefficient in the row direction of the resampling filter in combination with the coordinate value of the current pixel in the output image, the width of the input image, and the width of the output image; and assigning a filter coefficient in the column direction of the resampling filter in combination with the coordinate value of the current pixel in the output image, the height of the input image, and the height of the output image.

[0011] In one embodiment of the present invention, the input image includes a luminance component and a chrominance component; the filter coefficient in the row direction is defined as: frac = ((x*orgWidth<<4) / scaledWidth)&numFracPositions; the filter coefficient in the column direction is defined as: frac = ((y*orgHeight<<4) / scaledHeight)&numFracPositions; wherein, (x, y) represents the coordinates of the current pixel in the output image; orgWidth represents the width of the input image; scaledWidth represents the width of the output image; orgHeight represents the height of the input image; scaledHeight represents the height of the output image; when calculating the filter coefficient of the luminance component, the value of numFracPositions is the first preset value, and when calculating the filter coefficient of the chrominance component, the value of numFracPositions is the second preset value.

[0012] In one embodiment of the present invention, performing up-sampling interpolation filtering on an input image using a resampling filter includes: performing up-sampling interpolation filtering on a luminance component of the input image using 16 groups of 8-tap filter coefficient arrays, and performing up-sampling interpolation filtering on a chrominance component of the input image using 32 groups of 4-tap filter coefficient arrays.

[0013] In one embodiment of the present invention, at least one upsampling processing module includes a priori filter, the priori filter includes a bilinear cubic interpolation filter, and upsampling the input image using the priori filter includes: performing upsampling interpolation filtering on the input image using a bilinear cubic interpolation filter.

[0014] In one embodiment of the present invention, upsampling interpolation filtering is performed on an input image using a bilinear cubic interpolation filter, including: obtaining 4*4 neighborhood points within a preset range of a pixel to be interpolated, and using the coordinate values ​​of the neighborhood points to obtain the pixel value of the pixel to be interpolated; and upsampling interpolation filtering is performed on the input image in combination with the pixel value of the pixel to be interpolated and a bilinear cubic interpolation function.

[0015] In one embodiment of the present invention, obtaining the pixel value of the pixel point to be interpolated by using the coordinate value of the neighborhood point includes: the pixel value of the pixel point to be interpolated is defined as: Among them, (x, y) represents the coordinates of the pixel to be interpolated; (xi, yi) represents the coordinates of the neighborhood point, i = 0, 1, 2, 3, j = 0, 1, 2, 3; f represents the pixel value of the pixel; W represents the bilinear cubic interpolation function.

[0016] In one embodiment of the present invention, the bilinear cubic interpolation function is defined as: Wherein, a is a third preset value.

[0017] In one embodiment of the present invention, the global residual connection branch includes a first attention layer, and the first attention layer includes a second convolutional layer, a global mean pooling layer, a third convolutional layer, a first activation layer, a fourth convolutional layer and a normalization layer that are cascaded in sequence; weighted fusion of the first sub-basic information and the second sub-basic information to obtain basic information includes: fusing the first sub-basic information and the second sub-basic information and inputting them into the second convolutional layer for feature extraction; performing global mean pooling processing on the output of the second convolutional layer using the global mean pooling layer; performing feature extraction on the output of the global mean pooling layer using the third convolutional layer; using the output of the third convolutional layer as the input of the first activation layer, performing activation processing using the first activation layer, and inputting the feature map after activation processing into the fourth convolutional layer for feature extraction; performing normalized activation processing on the output of the fourth convolutional layer using the normalization layer to obtain weights corresponding to the first sub-basic information and the second sub-basic information; and using the obtained weights to weightedly fuse the first sub-basic information and the second sub-basic information.

[0018] In one embodiment of the present invention, the residual neural network includes a low-level feature extraction layer, a mapping layer, a high-level feature extraction layer and an upsampling layer cascaded in sequence, the output of the low-level feature extraction layer is connected to the input of the upsampling to form a local residual connection branch, and the mapping layer has a plurality of residual layers connected in sequence; the input image is reconstructed using the residual neural network branch to obtain the residual information of the input image, including: mapping the input image to a feature domain using the low-level feature extraction layer to extract a low-level feature image of the input image; mapping the low-level feature image to a first high-level feature image using the mapping layer; extracting features from the first high-level feature image using the high-level feature extraction layer to obtain a second high-level feature image; fusing the low-level feature image and the second high-level feature image using the local residual connection branch to obtain a fused feature image; and expanding and upsampling the fused feature image using the upsampling layer to obtain residual information.

[0019] In one embodiment of the present invention, the residual network also includes an input layer and an output layer, the input layer is connected to the low-level feature extraction layer, and the output layer is connected to the upsampling layer; the method also includes: inputting the input image into the input layer, performing input transformation processing on the input image, and using the transformed input image as the input of the low-level feature extraction layer; wherein the form of the transformed input image matches the form allowed for input by the low-level feature extraction layer; and / or, inputting the residual information into the output layer, performing output transformation processing on the residual information, and using the transformed residual information as the output of the residual neural network branch.

[0020] In one embodiment of the present invention, the mapping layer includes a plurality of first residual layers cascaded in sequence; the first residual layer includes a fifth convolutional layer, a second activation layer, a sixth convolutional layer and a second attention layer; the method further includes: using the fifth convolutional layer to perform feature extraction on the feature map of the low-level feature image / the output of the previous residual layer; using the second activation layer to perform activation processing on the output of the fifth convolutional layer, and using the sixth convolutional layer to perform feature extraction on the output of the second activation layer to obtain multiple mapping feature images corresponding to multiple channels; using the second attention layer to obtain the weights assigned to each mapping feature image; performing weighted fusion on the mapping feature images, and fusing the weighted fused image with the low-level feature image.

[0021] In one embodiment of the present invention, the mapping layer also includes a plurality of second residual layers and a plurality of third residual layers, and the first residual layer, the second residual layer and the third residual layer are connected; the second residual layer includes a seventh convolutional layer with a span greater than one and the first residual layer, and the seventh convolutional layer is connected to the first residual layer to downsample the feature map of the input second residual layer; the third residual layer includes a deconvolution layer with a span greater than one and the first residual layer, and the deconvolution layer is connected to the first residual layer to upsample the feature map of the input third residual layer; wherein the first residual layer of the feature map whose output size matches is connected to fuse the feature maps whose sizes match.

[0022] In one embodiment of the present invention, the method includes: extracting features from the feature map output by the low-level feature image / residual layer, and dividing the extracted feature map to obtain multiple image groups; fusing the multiple image groups multiple times to obtain multiple fused image groups, extracting features from the multiple fused image groups to obtain multiple fused feature maps; weighted fusing the multiple fused feature maps, and fusing the weighted fused feature map with the low-level feature image.

[0023] In one embodiment of the present invention, the mapping layer includes a plurality of fourth residual layers cascaded in sequence, and the fourth residual layer includes a tenth convolutional layer, a segmentation layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a splicing layer, a fourteenth convolutional layer and a second attention layer; extracting features from the low-level feature image / feature map output by the previous residual layer, and dividing the extracted feature map to obtain a plurality of image groups includes: inputting the feature map output by the low-level feature image / residual layer into the tenth convolutional layer for feature extraction; using the segmentation layer to divide the feature map output by the tenth convolutional layer into four groups, namely, a first image group, a second image group, a third image group and a fourth image group; fusing the plurality of image groups multiple times to obtain a plurality of fused image groups, extracting features from the plurality of fused image groups to obtain a plurality of fused feature maps includes: fusing the first image group with the second image group to obtain a first fused image group, and extracting features from the first fused image group using the eleventh convolutional layer to obtain a first fused feature map; The output result of the eleventh convolutional layer is fused with the third image group to obtain a second fused image group, and the twelfth convolutional layer is used to extract features from the second fused image group to obtain a second fused feature map; the output result of the twelfth convolutional layer is fused with the fourth image group to obtain a third fused image group, and the thirteenth convolutional layer is used to extract features from the third fused image group to obtain a third fused feature map; weighted fusion of multiple fused feature maps, and fusing the weighted fused feature map with the low-level feature image includes: inputting the first fused feature map, the second fused feature map and the third fused feature map into the splicing layer for splicing; using the fourteenth convolutional layer to perform feature transformation on the output result of the splicing layer; using the second attention layer to assign weights to the first fused feature map, the second fused feature map and the third fused feature map; weighted fusion of the first fused feature map, the second fused feature map and the third fused feature map, and fusing the weighted fused feature map with the low-level feature image / the feature map output by the previous residual layer.

[0024] In one embodiment of the present invention, the second attention layer includes a global mean pooling layer, an eighth convolutional layer, a third activation layer, a ninth convolutional layer and a normalization layer cascaded in sequence; the method also includes: using the global mean pooling layer to perform global mean pooling processing on the mapped feature image; using the eighth convolutional layer to perform feature extraction on the feature map after the global mean pooling processing; using the third activation layer to activate the feature map output by the eighth convolutional layer, and input the activated feature map into the ninth convolutional layer for feature extraction; using the normalization layer to perform normalized activation processing on the output of the ninth convolutional layer to assign weights to each component of the input image.

[0025] In one embodiment of the present invention, the input image includes a reconstructed image and an image formed using side information.

[0026] To solve the above technical problems, another technical solution adopted by the present invention is: to provide an image coding method, which includes: obtaining an original image, downsampling the original image to generate a downsampled image; encoding the downsampled image to obtain a coded reconstructed image and coding data; in response to the need to use a high-resolution image as a reference image, reconstructing the coded reconstructed image using the image reconstruction method described in any of the above embodiments.

[0027] To solve the above technical problems, another technical solution adopted by the present invention is: to provide an image decoding method, which includes: obtaining encoded data generated by the image encoding method as described in any of the above embodiments; decoding the encoded data to obtain a reconstructed image; and reconstructing the reconstructed image using the image reconstruction method as described in any of the above embodiments.

[0028] To solve the above technical problems, another technical solution adopted by the present invention is: to provide an encoder, which includes: the encoder includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program to execute instructions to implement the image reconstruction method described in any of the above embodiments, or the image encoding method described in any of the above embodiments.

[0029] To solve the above technical problems, another technical solution adopted by the present invention is: to provide a decoder, which includes: the decoder includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program to implement the image reconstruction method described in any of the above embodiments, or the image decoding method described in any of the above embodiments.

[0030] To solve the above technical problems, another technical solution adopted by the present invention is: providing a computer-readable storage medium, which is used to store instructions / program data, and the instructions / program data can be executed to implement the image reconstruction method as described in any of the above embodiments, or the image encoding method as described in any of the above embodiments, or the image decoding method as described in any of the above embodiments.

[0031] The beneficial effects of the present invention are as follows: different from the prior art, the present invention can utilize a super-resolution reconstruction network to perform super-resolution reconstruction on an input image, and the super-resolution reconstruction network includes a global residual connection branch and a residual neural network branch, thereby improving the performance of the super-resolution reconstruction network; the basic information obtained using the global residual connection branch can contain the basic information of the input image, and the residual information obtained using the residual neural network branch can contain higher-frequency information, and the output image generated by fusing the two can have a good super-resolution reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present invention, and together with the specification are used to explain the principles of the present invention. In addition, these drawings and textual descriptions are not intended to limit the scope of the present invention in any way, but to illustrate the concept of the present invention for those skilled in the art by reference to specific embodiments.

[0033] Figure 1 is a flow chart of an embodiment of an image reconstruction method of the present invention;

[0034] Figure 2 1 is a schematic diagram of the structure of an embodiment of a super-resolution reconstruction network of the present invention;

[0035] Figure 3 It is a structural schematic diagram of an embodiment of a convolutional neural network of the present invention;

[0036] Figure 4 4 is a schematic diagram of the structure of an embodiment of a 4*4 neighborhood of a pixel to be interpolated according to the present invention;

[0037] Figure 5 is a schematic diagram of the structure of an embodiment of the first attention layer of the present invention;

[0038] Figure 6 It is a schematic diagram of the structure of the first embodiment of the residual neural network of the present invention;

[0039] Figure 7 is a schematic diagram of the structure of the second embodiment of the residual neural network of the present invention;

[0040] Figure 8 is a schematic structural diagram of a third embodiment of a residual neural network of the present invention;

[0041] Fig. 9 is a schematic structural diagram of an embodiment of the first residual layer of the present invention;

[0042] Fig.10 is a schematic diagram of the structure of an embodiment of the second attention layer of the present invention;

[0043] Fig.11 is a structural diagram of an embodiment of a mapping layer of the present invention;

[0044] Fig.12 is a schematic structural diagram of an embodiment of a second residual layer of the present invention;

[0045] Fig.13 is a schematic structural diagram of an embodiment of a third residual layer of the present invention;

[0046] Fig.14 is a schematic structural diagram of an embodiment of a fourth residual layer of the present invention;

[0047] Fig.15 It is a flowchart of an embodiment of an image encoding method of the present invention;

[0048] Fig.16 It is a schematic structural diagram of an embodiment of an encoder of the present invention;

[0049] Fig.17 is a flowchart of a first embodiment of an image decoding method of the present invention;

[0050] Fig.18 It is a flowchart of an embodiment of a prior art image decoding method;

[0051] Fig.19 is a flowchart of a second embodiment of an image decoding method of the present invention;

[0052] Fig. 20 is a flowchart of a third embodiment of an image decoding method of the present invention;

[0053] Fig.21 It is a structural schematic diagram of an embodiment of a decoder of the present invention;

[0054] Fig. 22 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0056] In order to solve the technical problem of poor super-resolution reconstruction effect in the prior art, the present invention provides an image reconstruction method, which includes: obtaining an input image; inputting the input image into a global residual connection branch and a residual neural network branch of a super-resolution reconstruction network respectively, so as to upsample the input image using the global residual connection branch to obtain basic information of the input image, and reconstruct the input image using the residual neural network branch to obtain residual information of the input image; fusing the basic information and the residual information to generate an output image, wherein the resolution of the output image is higher than that of the input image. The present invention is described in detail below.

[0057] See also Figure 1 and Figure 2 , Figure 1is a flow chart of an embodiment of an image reconstruction method of the present invention, Figure 2 It is a schematic diagram of the structure of an embodiment of a super-resolution reconstruction network of the present invention. It should be noted that the image reconstruction method described in this embodiment is not limited to the following steps:

[0058] S101: Obtain an input image.

[0059] In this embodiment, an input image is obtained, and the input image is an image that needs to be super-resolution reconstructed. The input image may be a reconstructed image that has been processed with reconstructed pixels, or the input image may be a reconstructed image and an image formed using side information, and the side information may include one or more of prediction information, quantization parameter information, and division information, which are not limited here. Taking quantization parameter information as an example, an image may be constructed using quantization parameter information, and the pixel value of each pixel in the image is the quantization parameter.

[0060] In the process of encoding the image, the reconstructed image is the encoded reconstructed image; in the process of decoding the image, the reconstructed image is the decoded reconstructed image.

[0061] S102: Input the input image into the global residual connection branch and the residual neural network branch of the super-resolution reconstruction network respectively, so as to upsample the input image using the global residual connection branch to obtain basic information of the input image, and reconstruct the input image using the residual neural network branch to obtain residual information of the input image.

[0062] In this embodiment, the input image is input into the super-resolution reconstruction network, and the super-resolution reconstruction network includes a global residual connection branch and a residual neural network branch. The input image is respectively input into the global residual connection branch and the residual neural network branch, and the input image is pre-upsampled by using the global residual connection branch to obtain the basic information of the input image, thereby providing a basic value for the input image; and the input image is reconstructed by using the residual neural network branch to obtain the residual information of the input image, so as to obtain more information of the input image. It is easy to understand that the residual information is more complex than the basic information, that is, the residual information has higher frequency information.

[0063] Among them, the basic information and / or residual information can be represented by a feature map or by other data information, which is not limited here.

[0064] S103: Fusing the basic information and the residual information to generate an output image.

[0065] In this embodiment, after obtaining the basic information and the residual information, the basic information and the residual information are fused to generate an output image. The output image can contain information contained in both the basic information and the residual information. In other words, the output image contains both the basic information of the input image and the high-frequency information processed by the residual neural network branch, so as to achieve a good super-resolution reconstruction effect.

[0066] It can be seen that in this embodiment, the super-resolution reconstruction network includes a global residual connection branch and a residual neural network branch, and the super-resolution reconstruction network has a reasonable network structure, thereby improving the performance of the super-resolution reconstruction network; the basic information obtained by using the global residual connection branch can include the basic information of the input image, and the residual information obtained by using the residual neural network branch can include higher-frequency information, and the output image generated by fusing the two includes both the basic information of the input image and the high-frequency information processed by the residual neural network branch, which is conducive to achieving good super-resolution reconstruction effects.

[0067] The following is an example of the global residual connection branch of the present invention. Figure 2 and Figure 3 , Figure 3 It is a structural diagram of an embodiment of a convolutional neural network of the present invention.

[0068] In one embodiment, the global residual connection branch may include at least one upsampling processing module, and the input image is upsampled by using at least one upsampling processing module to obtain sub-basic information corresponding to each upsampling processing module; wherein, at least one upsampling processing module includes at least one of a convolutional neural network and a priori filter.

[0069] Based on the sub-basic information corresponding to each upsampling processing module, basic information for fusion with the residual information is obtained, wherein the sub-basic information corresponding to the upsampling processing module includes first sub-basic information obtained by using a convolutional neural network and second sub-basic information obtained by using a priori filter.

[0070] That is to say, based on the sub-basic information corresponding to each upsampling processing module, the basic information can be obtained specifically as follows: when using a convolutional neural network to obtain the first sub-basic information, the first sub-basic information is used as the basic information for fusion with the residual information; when using a priori filter to obtain the second sub-basic information, the second sub-basic information is used as the basic information for fusion with the residual information; when using a convolutional neural network to obtain the first sub-basic information and using a priori filter to obtain the second sub-basic information at the same time, the first sub-basic information can be fused with the second sub-basic information to generate basic information for fusion with the residual information. The technical solution for fusing the first sub-basic information with the second sub-basic information to obtain the basic information will be described in detail later. The following is a detailed description of the implementation method of using a convolutional neural network to obtain the first sub-basic information and using a priori filter to obtain the second sub-basic information.

[0071] The convolutional neural network performs upsampling processing on the input image to obtain the first sub-basic information. Specifically, the convolutional neural network includes a feature extraction layer, a first convolutional layer, and a merge sorting layer that are cascaded in sequence, wherein the convolutional neural network may include several feature extraction layers that are connected in sequence. Figure 3 The example in the figure shows that the convolutional neural network includes an implementation method of a feature extraction layer. The feature extraction layer is used to extract the features of the input image, and the output of the feature extraction layer is used as the input of the first convolutional layer to perform convolution to obtain multiple channel feature maps, which are used to represent the input image after convolution; the merge and sort layer is used to fuse multiple channel feature maps to obtain the first sub-basic information.

[0072] For example, the input channel of the convolutional neural network is one, so as to allow the input image to be input into the feature extraction layer. The feature extraction layer includes a convolution layer and an activation layer connected in sequence. The convolution layer of the feature extraction layer has a 3*3 convolution kernel, and the number of extracted features is 64, so as to reduce the large amount of data required to be processed by the super-resolution reconstruction network due to the large number of extracted features, and also reduce the poor reconstruction effect of the super-resolution reconstruction network due to the large number of extracted features. In other words, if the number of extracted features is selected, the computational cost of the super-resolution reconstruction network can be reduced while ensuring the reconstruction effect; the activation layer activates the feature map using the ReLU function, and then uses the first convolution layer to convolve again for feature extraction to obtain feature maps of four channels, and uses the merge sorting layer to fuse the feature maps of the four channels to obtain the first sub-basic information. Among them, the merge sorting layer can be a process of processing the feature maps of the four channels using the Shuffle function, etc.

[0073] It should be noted that, for ease of understanding, the following example is explained with the feature extraction layer having an extraction feature number of 64. When the extraction feature number of the feature extraction layer changes, the numerical values ​​exemplified below should change accordingly.

[0074] And / or, the input image is upsampled using a priori filter, and the a priori filter is pre-upsampled by upsampling interpolation filtering to obtain the second sub-basic information. Optionally, the a priori filter can be a reference image resampling filter, a bilinear cubic interpolation filter, a bilinear interpolation filter, a nearest neighbor interpolation filter, etc.

[0075] Taking the prior filter as a resampling filter as an example, the input image includes a brightness component and a chrominance component. For example, when the input image is in a YUV format, the brightness component is Y, and the chrominance components are U and V.

[0076] The input image is upsampled and interpolated by using a resampling filter, in which the tap coefficient corresponding to the luminance component is greater than the tap coefficient corresponding to the chrominance component.

[0077] For example, the resampling filter includes 16 groups of 8-tap filter coefficient arrays and 32 groups of 4-tap filter coefficient arrays. The 16 groups of 8-tap filter coefficient arrays are used to upsample and interpolate the luminance component, and the 32 groups of 4-tap filter coefficient arrays are used to upsample and interpolate the chrominance component.

[0078] The filter coefficient array corresponding to the brightness component is as follows:

[0079] {0, 0, 0, 64, 0, 0, 0, 0}

[0080] {0, 1, -3, 63, 4, -2, 1, 0}

[0081] {-1, 2, -5, 62, 8, -3, 1, 0}

[0082] {-1, 3, -8, 60, 13, -4, 1, 0}

[0083] {-1, 4, -10, 58, 17, -5, 1, 0}

[0084] {-1, 4, -11, 52, 26, -8, 3, -1}

[0085] {-1, 3, -9, 47, 31, -10, 4, -1}

[0086] {-1, 4, -11, 45, 34, -10, 4, -1}

[0087] {-1, 4, -11, 40, 40, -11, 4, -1}

[0088] {-1, 4, -10, 34, 45, -11, 4, -1}

[0089] {-1, 4, -10, 31, 47, -9, 3, -1}

[0090] {-1, 3, -8, 26, 52, -11, 4, -1}

[0091] {0, 1, -5, 17, 58, -10, 4, -1}

[0092] {0, 1, -4, 13, 60, -8, 3, -1}

[0093] {0, 1, -3, 8, 62, -5, 2, -1}

[0094] {0, 1, -2, 4, 63, -3, 1, 0}

[0095] The filter coefficient array corresponding to the chrominance component is as follows:

[0096] {0, 64, 0, 0}

[0097] {-1, 63, 2, 0}

[0098] {-2, 62, 4, 0}

[0099] {-2, 60, 7, -1}

[0100] {-2, 58, 10, -2}

[0101] {-3, 57, 12, -2}

[0102] {-4, 56, 14, -2}

[0103] {-4, 55, 15, -2}

[0104] {-4, 54, 16, -2}

[0105] {-5, 53, 18, -2}

[0106] {-6, 52, 20, -2}

[0107] {-6, 49, 24, -3}

[0108] {-6, 46, 28, -4}

[0109] {-5, 44, 29, -4}

[0110] {-4, 42, 30, -4}

[0111] {-4, 39, 33, -4}

[0112] {-4, 36, 36, -4}

[0113] {-4, 33, 39, -4}

[0114] {-4, 30, 42, -4}

[0115] {-4, 29, 44, -5}

[0116] {-4, 28, 46, -6}

[0117] {-3, 24, 49, -6}

[0118] {-2, 20, 52, -6}

[0119] {-2, 18, 53, -5}

[0120] {-2, 16, 54, -4}

[0121] {-2, 15, 55, -4}

[0122] {-2, 14, 56, -4}

[0123] {-2, 12, 57, -3}

[0124] {-2, 10, 58, -2}

[0125] {-1, 7, 60, -2}

[0126] {0, 4, 62, -2}

[0127] {0, 2, 63, -1}

[0128] The filter coefficients may be selected by combining the coordinate value of the current pixel in the output image, the size of the input image, and the size of the output image.

[0129] Specifically, the filter coefficient in the row direction of the resampling filter can be assigned in combination with the coordinate value of the current pixel in the output image, the width of the input image and the width of the output image, and the filter coefficient in the column direction of the resampling filter can be assigned in combination with the coordinate value of the current pixel in the output image, the height of the input image and the height of the output image.

[0130] For example, the coordinates of the current pixel in the output image can be set to (x, y), and the filter coefficient in the row direction of the fracth row can be defined as:

[0131] frac=((x*orgWidth<<4) / scaledWidth)&numFracPositions(1-1)

[0132] The filter coefficients in the column direction of the fracth row are defined as:

[0133] frac=((y*orgHeight<<4) / scaledHeight)&numFracPositions(1-2)

[0134] Among them, the value of frac is [0,15]; orgWidth represents the width of the input image, scaledWidth represents the width of the output image, orgHeight represents the height of the input image, and scaledHeight represents the height of the output image; numFracPositions represents the filter coefficient calculation parameter. When calculating the filter coefficient of the luminance component, the value of numFracPositions is the first preset value, and when calculating the filter coefficient of the chrominance component, the value of numFracPositions is the second preset value.

[0135] For example, in the present embodiment, “up-sampling interpolation filtering is performed on the luminance component using 16 groups of 8-tap filter coefficient arrays, and up-sampling interpolation filtering is performed on the chrominance component using 32 groups of 4-tap filter coefficient arrays”, the first preset value may be 15, and the second preset value may be 31. Of course, when other resampling filters are used for up-sampling interpolation filtering, the first preset value and the second preset value may change accordingly, which will not be described in detail here.

[0136] See also Figure 4 , Figure 4 The present invention is a structural diagram of an embodiment of a 4*4 neighborhood of a pixel to be interpolated.

[0137] Taking the prior filter as a bilinear cubic interpolation filter as an example, the bilinear cubic interpolation filter can be used to perform upsampling interpolation filtering on the input image.

[0138] Obtain 4x4 neighborhood points within the preset range of the pixel to be interpolated, and use the coordinate values ​​of the neighborhood points to obtain the pixel value of the pixel to be interpolated. The preset range may be a range including the pixel to be interpolated, for example Figure 4 The 4*4 neighborhood points within the preset range are shown in FIG. 1 , and the shaded circles are used to represent the pixels to be interpolated.

[0139] Assume the coordinates of the pixel to be interpolated are (x, y), and the pixel value of the pixel to be interpolated is defined as:

[0140]

[0141] Where f(x, y) represents the pixel value of the pixel to be interpolated; (xi, yi) represents the coordinates of the neighborhood point, where i = 0, 1, 2, 3, j = 0, 1, 2, 3; f(x i ,y i ) represents the pixel value of the pixel to be interpolated, and W represents the bilinear interpolation function.

[0142] Then, the input image is upsampled and interpolated by combining the pixel value of the pixel to be interpolated and the bilinear cubic interpolation function to obtain the second sub-basis information.

[0143] For example, the bilinear cubic interpolation function is defined as:

[0144]

[0145] Wherein, a is a third preset value. For example, in this embodiment, the value of a may be -0.5.

[0146] The following specifically describes a specific implementation method of weighted fusion of the first sub-basic information and the second sub-basic information after acquiring the first sub-basic information and the second sub-basic information.

[0147] Specifically, weights may be assigned to the first sub-basis information and the second sub-basis information in advance, or weights may be assigned to the first sub-basis information and the second sub-basis information through the first attention layer.

[0148] When weights are assigned to the first sub-basic information and the second sub-basic information in advance, the expression of the basic information is as follows:

[0149]

[0150] Wherein, N is the total number of upsampling methods, and the upsampling method may be one or more, including but not limited to the use of a priori filters and convolutional neural networks described in the above embodiments; j The weights assigned to different upsampling methods, w j Can be manually set or network-learned, and I j are the output images of different upsampling methods; O is the basic information.

[0151] It should be noted that when several convolutional neural networks are used to generate the first sub-basic information respectively, and when several prior filters are used to generate the second sub-basic information respectively, weights are assigned to each first sub-basic information and each second sub-basic information respectively, and the sum of the weight of each first sub-basic information and the weight of each second sub-basic information is 1.

[0152] Taking N=2 as an example, I1 is the first sub-basis information obtained using the convolutional neural network, and I2 is the second sub-basis information obtained using the resampling filter. The same weight can be assigned to the two, that is, w1=w2=0.5, O=0.5I1+0.5I2.

[0153] When weights are assigned to the first sub-basis information and the second sub-basis information through the first attention layer, it means that the global residual connection branch includes the first attention layer, wherein the first attention layer can be an attention module, etc., which can perform adaptive weight assignment, which can effectively improve the performance of the global residual connection branch, and thus improve the performance of the super-resolution reconstruction network, see Figure 5 , Figure 5 It is a structural diagram of an embodiment of the first attention layer of the present invention.

[0154] The first attention layer includes the second convolution layer, the global mean pooling layer, the third convolution layer, the first activation layer, the fourth convolution layer and the normalization layer, which are cascaded in sequence.

[0155] The first sub-basic information and the second sub-basic information are fused, which can be to fuse the first sub-basic information and the second sub-basic information in the first attention layer input channel, and input the fused information into the second convolution layer for feature extraction. The convolution kernel of the second convolution layer can be 1*1, and the number of extracted features can be 64; GAP (Global Average The feature map output by the second convolutional layer is subjected to global mean pooling by a global mean pooling (global mean pooling) layer; the output of the global mean pooling layer is subjected to feature extraction by a third convolutional layer, the convolution kernel of the third convolutional layer may be 1*1, and the number of extracted features may be 64; the output of the third convolutional layer is used as the input of the first activation layer, the first activation layer is used for activation processing, and the feature map after activation processing is input into the fourth convolutional layer for feature extraction, the convolution kernel of the fourth convolutional layer may be 1*1, and the number of extracted features may be 64; the output of the fourth convolutional layer is subjected to normalized activation processing by a normalization layer, wherein the normalization layer may be a softmax (normalized activation) function that is used to obtain weights corresponding to the first sub-basic information and the second sub-basic information, so as to assign weights to the first sub-basic information and the second sub-basic information, and the weights obtained are used to perform weighted fusion on the first sub-basic information and the second sub-basic information to obtain basic information.

[0156] Similarly, when several convolutional neural networks are used to generate the first sub-basis information respectively, and when several prior filters are used to generate the second sub-basis information respectively, weights are assigned to each first sub-basis information and each second sub-basis information respectively, and the sum of the weight of each first sub-basis information and the weight of each second sub-basis information is 1.

[0157] The above is an example of the global residual connection branch of the super-resolution reconstruction network. The following is an example of the residual neural network branch of the super-resolution reconstruction network.

[0158] See also Figure 6 and Figure 7 , Figure 6is a schematic diagram of the structure of the first embodiment of the residual neural network of the present invention, Figure 7 It is a schematic diagram of the structure of the second embodiment of the residual neural network of the present invention.

[0159] In one embodiment, the residual neural network includes a low-level feature extraction layer, a mapping layer, a high-level feature extraction layer, and an upsampling layer cascaded in sequence, and the output of the low-level feature extraction layer is connected to the input of the upsampling to form a local residual connection branch. Further, the residual neural network also includes an input layer and an output layer, the input layer is connected to the low-level feature extraction layer, and the output layer is connected to the upsampling layer.

[0160] Input the input image to the input layer of the residual neural network, perform input transformation processing on the input image so that the form of the transformed input image matches the form allowed by the low-level feature extraction layer, and use the transformed input image as the input of the low-level feature extraction layer; and / or, input the residual information to the output layer, perform output transformation processing on the residual information, so that the upsampling result of the residual neural network is converted into the desired output form, and use the transformed residual information as the output of the residual neural network branch.

[0161] like Figure 8 As shown in Figure 8 This is a structural diagram of the third embodiment of the residual neural network of the present invention, taking the use of the input layer to perform input transformation processing on the input image as an example, the input image includes a low-resolution coded image, a low-resolution predicted image and a QP (quantization parameter) image, wherein the QP image is an image formed by QP padding, and the low-resolution coded image, the low-resolution predicted image and the QP (quantization parameter) image are fused using the concat function.

[0162] Please continue reading Figure 6 and Figure 7 The low-level feature extraction layer may include a convolution layer, the convolution kernel is 3*3, the number of extracted features is 64, and the low-level feature extraction layer can be used to map the input image to the feature domain and extract the low-level feature image of the input image; wherein the low-level feature image is nonlinear.

[0163] The mapping layer may include a plurality of residual layers, and the residual layers may be a first residual layer, a second residual layer, a third residual layer, and a fourth residual layer, and the mapping layer is used to map the nonlinear low-level feature image to a first high-level feature image. For example, the mapping layer includes a plurality of first residual layers connected in sequence, such as 16 residual layers to fully map the low-level feature image to a first high-level feature image with higher complexity. The structure of the mapping layer will be specifically described later.

[0164] The high-level feature extraction layer may include a convolution layer, the convolution kernel is 3*3, the number of extracted features is 64, and the high-level feature extraction layer is used to extract features from the first high-level feature image to obtain a second high-level feature image. The low-level feature image and the second high-level feature image are fused using a local residual connection branch to obtain a fused feature image. That is to say, the local residual connection branch is connected from the output of the low-level feature extraction layer to the output of the high-level feature extraction layer. It should be noted that in the low-level feature image, the first high-level feature image and the second high-level feature image, high-level and low-level are corresponding concepts used to describe the complexity of the feature image. Generally speaking, the complexity of the high-level feature image is higher than that of the low-level feature image. For example, the second high-level feature image is obtained after mapping the low-level feature image, and the information contained in the second high-level feature image is more complex than that of the low-level feature image, that is, the second high-level feature image has higher frequency information.

[0165] The upsampling layer may include a convolution layer and a Shuffle function, and the upsampling layer is used to expand and upsample the fused feature image to obtain residual information. For example, the fused feature image may be convolved with a convolution kernel of 3*3 to expand and upsample, generating a feature map of 256 channels, and the expanded fused feature image may be processed using the Shuffle function to form a feature map of 64 channels, which matches the number of input channels of the output layer, so that the fused feature map is transformed into a form that allows input to the output layer, and the output layer is used to convert and fuse the feature map of 64 channels into 1 channel as the output of the residual neural network, that is, as residual information for fusion with the base image.

[0166] The mapping layer of the present invention is described below by way of example. Fig. 9 and Fig.10 , Fig. 9 is a schematic structural diagram of an embodiment of the first residual layer of the present invention, Fig.10 It is a structural diagram of an embodiment of the second attention layer of the present invention.

[0167] In one embodiment, the first residual layer includes a fifth convolution layer, a second activation layer, a sixth convolution layer, and a second attention layer. Specifically, the low-level feature image / feature map output by the previous residual layer is input into the fifth convolution layer, and the fifth convolution layer is used to extract features of the low-level feature image / feature map output by the previous residual layer, the convolution kernel of the fifth convolution layer is 3*3, and the number of extracted features is 64; the output of the fifth convolution layer is activated by the second activation layer (such as ReLU function, etc.); and the output of the second activation layer is extracted by the sixth convolution layer, the convolution kernel of the sixth convolution layer is 3*3, and the number of extracted features is 64, so as to obtain multiple mapping feature images corresponding to multiple channels; multiple mapping feature images are input into the second attention layer, and the weights assigned to each mapping feature image are obtained by the second attention layer; each mapping feature image is weighted fused, and the weighted fused image is fused with the input of the current first residual layer (low-level feature image / feature map output by the previous residual layer). Among them, the second attention layer can be an attention module, etc., which can perform adaptive weight allocation, which can effectively improve the performance of the residual neural network branch, and then improve the performance of the super-resolution reconstruction network.

[0168] in, Fig. 9 The fifth convolutional layer and the second activation layer shown in can be used as a feature extraction layer, and the first residual layer can include one or more feature extraction layers, which are not limited here. Fig. 9 An example of an implementation in which the first residual layer includes a feature extraction layer is shown.

[0169] Furthermore, the second attention layer includes a global mean pooling layer, an eighth convolutional layer, a third activation layer, a ninth convolutional layer and a normalization layer that are cascaded in sequence. The global mean pooling layer performs global mean pooling on the mapped feature image; the eighth convolutional layer is used to extract features from the feature map after the global mean pooling process, the convolution kernel of the eighth convolutional layer is 1*1, and the number of extracted features is 64; the third activation layer (such as ReLU function, etc.) performs activation processing on the feature map output by the eighth convolutional layer, and inputs the activated feature map into the ninth convolutional layer for feature extraction, the convolution kernel of the ninth convolutional layer is 1*1, and the number of extracted features is 64; the normalization layer (such as softmax function, etc.) is used to perform normalized activation processing on the output of the ninth convolutional layer, so as to realize that after the low-level feature map is mapped to a more complex feature, weights are assigned to each component of the input image, automatic weighting is achieved, and the performance of the super-resolution reconstruction network is improved. For example, for Figure 8 The coded image, predicted image, and QP image of the input residual neural network branch are weightedly fused as shown in FIG.

[0170] Please refer to Fig. 9 , Fig.11 , Fig.12 as well as Fig.13 , Fig.11 is a schematic diagram of the structure of an embodiment of a mapping layer of the present invention, Fig.12 is a schematic diagram of the structure of the second residual layer of the present invention, Fig.13 It is a schematic structural diagram of the third residual layer of the present invention.

[0171] In one embodiment, the mapping layer also includes a plurality of second residual layers and a plurality of third residual layers, the first residual layer, the second residual layer and the third residual layer are connected, the second residual layer is used to upsample the feature map, the third residual layer is used to downsample the feature map, and the first residual layer, the second residual layer and the third residual layer are connected, so as to fuse more information of the input image by performing feature scale transformation on the feature map, so as to improve the super-resolution reconstruction effect.

[0172] Specifically, the second residual layer includes a seventh convolutional layer with a span greater than one and a first residual layer, and the seventh convolutional layer is connected to the first residual layer to downsample the feature map input to the second residual layer. That is to say, the second residual layer includes the seventh convolutional layer, the fifth convolutional layer, the second activation layer, the sixth convolutional layer, and the second attention layer connected in sequence, and the output of the sixth convolutional layer is weighted fused with the weights assigned to the second attention layer, and the feature map output by the seventh convolutional layer is fused with the weighted fused features through a local residual connection to enrich the information contained in the feature map. Among them, the second attention layer can be as follows Fig.10 As shown in , the specific method of allocating weights will not be repeated here.

[0173] The third residual layer includes a deconvolution layer with a span greater than one and a first residual layer. The deconvolution layer is connected to the first residual layer to upsample the feature map input to the third residual layer. That is to say, the third residual layer includes a deconvolution layer, a fifth convolution layer, a second activation layer, a sixth convolution layer, and a second attention layer connected in sequence, and the output of the sixth convolution layer is weighted fused with the weights assigned to the second attention layer. The feature map output by the deconvolution layer is fused with the weighted fused features through a local residual connection to enrich the information contained in the feature map. Among them, the second attention layer can be as follows Fig.10 As shown in , the specific method of allocating weights will not be repeated here.

[0174] in, Fig.12 as well as Fig.13 The fifth convolutional layer and the second activation layer shown in can be used as a feature extraction layer, and the first residual layer can include one or more feature extraction layers, which are not limited here. Fig.12 as well as Fig.13 , respectively, illustrate examples of implementations in which the first residual layer includes a feature extraction layer.

[0175] Among them, the first residual layer of the feature map whose output size matches is connected to fuse the matching feature maps. For example, the span of the seventh convolutional layer can be 2, and correspondingly, the span of the deconvolution layer is also 2. The structure of the mapping layer can be as follows Fig.11 As shown in , it is easy to understand that the size of the feature map output by the first residual layer on the left matches the size of the feature map output by the first residual layer on the right, so the outputs of the two first residual layers are connected through the local residual connection branch to fuse the feature maps output by the two first residual layers. It should be noted that when the span of the seventh convolutional layer is different from the span of the deconvolution layer, the connection method of the mapping layer should be different, and the output feature maps with matching sizes should be fused.

[0176] See also Fig.14 , Fig.14 It is a schematic structural diagram of an embodiment of the fourth residual layer of the present invention.

[0177] The mapping layer may also include several fourth residual layers cascaded in sequence, which can perform feature extraction on the feature map output by the low-level feature image / residual layer, and divide the extracted feature map to obtain multiple image groups; perform multiple fusions on the multiple image groups to obtain multiple fused image groups, perform feature extraction on the multiple fused image groups to obtain multiple fused feature maps; weightedly fuse multiple fused feature maps, and fuse the weighted fused feature maps with the low-level feature image.

[0178] Specifically, Fig.14 As shown in , the fourth residual layer includes the tenth convolution layer, the segmentation layer, the eleventh convolution layer, the twelfth convolution layer, the thirteenth convolution layer, the splicing layer, the fourteenth convolution layer and the second attention layer.

[0179] The feature map output by the low-level feature image / previous residual layer is input into the tenth convolutional layer for feature extraction. The convolution kernel of the tenth convolutional layer is 1*1, and the number of extracted features is 64. The feature map output by the tenth convolutional layer is divided into four groups using the segmentation layer, namely the first image group, the second image group, the third image group, and the fourth image group. For example, the feature maps can be grouped using the Split function. The feature maps can be evenly divided to form image groups containing an equal number of feature maps, or the number of feature maps contained in each image group can be different. The following example takes the equal number of feature maps contained in each image group as an example, that is, each image group includes 16 feature maps.

[0180] The first image group and the second image group are fused to obtain a first fused image group, the first fused image group is input into the eleventh convolutional layer, and the eleventh convolutional layer is used to extract features to obtain a first fused feature map, the convolution kernel of the eleventh convolutional layer is 3*3, and the number of extracted features is 16; the first fused feature map is fused with the third image group to obtain a second fused image group, the second fused image group is input into the twelfth convolutional layer, and the twelfth convolutional layer is used to extract features to obtain a second fused feature map, so as to fuse more feature map information, the convolution kernel of the twelfth convolutional layer is 3*3, and the number of extracted features is 16; the second fused feature map is fused with the fourth image group, and then input into the thirteenth convolutional layer, and the thirteenth convolutional layer is used to extract features to obtain a third fused feature map, so as to fuse more feature map information, the convolution kernel of the thirteenth convolutional layer is 3*3, and the number of extracted features is 16.

[0181] The first fused feature map, the second fused feature map and the third fused feature map are input into the splicing layer for splicing; the output result of the splicing layer is transformed by the fourteenth convolution layer, the convolution kernel of the fourteenth convolution layer is 1*1, and the number of extracted features is 64; the second attention layer is used to assign weights to the first fused feature map, the second fused feature map and the third fused feature map, and the first fused feature map, the second fused feature map and the third fused feature map are weighted and fused, and the weighted feature map is fused with the input of the current fourth residual layer (low-level feature image / feature map output by the previous residual layer). It can be seen that the fourth residual layer introduces multiple branches, thereby expanding the receptive field of the residual layer, further improving the performance of the super-resolution reconstruction network, so as to achieve good super-resolution reconstruction effects. Among them, the second attention layer can be as follows Fig.10 As shown in , the specific method of allocating weights will not be repeated here.

[0182] In an alternative embodiment (not shown), feature extraction is performed on the feature map input of the low-level feature image / current residual layer output; and the feature map is divided into four groups, namely the first image group, the second image group, the third image group and the fourth image group as examples, and the first image group and the second image group can also be fused to generate a fourth fused image group, and feature extraction is performed on the fourth fused image group to obtain a fourth fused feature map; the third image group and the fourth image group are fused to generate a fifth fused image group, and feature extraction is performed on the fifth fused image group to obtain a fifth fused feature map; feature extraction is performed on the fourth fused feature map and the fifth fused feature map to obtain a sixth fused feature map, the fourth fused feature map, the fifth fused feature map and the sixth fused feature map are weightedly fused, and the weighted fused feature map is fused with the feature map of the low-level feature image / current residual layer output.

[0183] It can be seen that the image reconstruction method of the present invention utilizes the attention layer system, which can effectively improve the performance of the super-resolution reconstruction network, and uses side information as input to further improve the performance of the super-resolution reconstruction network. Convolutional neural networks and / or prior filters can be combined in the global residual connection branch to effectively reduce the difficulty of network training and make the network parameters sparse. In addition, the residual layer introduces the seventh convolution layer and deconvolution layer for up and down sampling, so that the residual layer can extract features of different scales; and / or, a multi-branch residual layer is introduced, and the feature maps of different branches are combined to effectively expand the receptive field of the residual layer.

[0184] See also Fig.15 , Fig.15 It is a flow chart of an embodiment of the image coding method of the present invention. It should be noted that the image coding method described in this embodiment is not limited to the following steps:

[0185] S201: Acquire an original image, and downsample the original image to generate a downsampled image.

[0186] In this embodiment, the original image is an image contained in an input video sequence, and the input video sequence is a video sequence that needs to be encoded.

[0187] S202: Encode the downsampled image to obtain a coded reconstructed image and coded data.

[0188] In this embodiment, a coded reconstructed image and coded data may be generated based on the downsampled image, and the coded data is used to be transmitted to a decoder for decoding processing.

[0189] S203: In response to the need to use a high-resolution image as a reference image, reconstruct the encoded reconstructed image using an image reconstruction method.

[0190] In this embodiment, the specific implementation of the image reconstruction method for reconstruction processing has been described in detail above, and will not be repeated here. The output image obtained by the image reconstruction method for reconstruction processing can be used as a reference image for encoding and reconstructing an image with the same resolution during encoding.

[0191] Optionally, the coded reconstructed image may be used as an input image for super-resolution reconstruction, or the coded reconstructed image and the side information may be used as input images for super-resolution reconstruction.

[0192] See also Fig.16 , Fig.16 It is a schematic structural diagram of an encoder embodiment of the present invention.

[0193] In this embodiment, the encoder 10 includes a memory and a processor 11 , and the memory (not shown in the figure) is used to store computer programs required for the processor 11 to run.

[0194] The processor 11 may also be referred to as a CPU (Central Processing Unit). The processor 11 may be an integrated circuit chip having signal processing capabilities. The processor 11 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 11 may also be any conventional processor, etc.

[0195] The processor 11 is used to execute a computer program to implement the image reconstruction method described in any of the above embodiments, or the image encoding method described in the above embodiments.

[0196] See also Fig.17 , Fig.17 FIG. 1 is a flow chart of the first embodiment of the image decoding method of the present invention. It should be noted that the image decoding method described in this embodiment is not limited to the following steps:

[0197] S301: Obtain coded data.

[0198] In this embodiment, the acquired coded data may be coded data generated using the above-mentioned image coding method. For example, the coded data may be generated based on the down-sampled image in step S202 of the above-mentioned embodiment.

[0199] S302: Decode and reconstruct the encoded data to obtain a decoded and reconstructed image.

[0200] S303: Reconstruct the decoded reconstructed image using an image reconstruction method.

[0201] In this embodiment, the specific implementation of the reconstruction process of the image reconstruction method has been described in detail above and will not be repeated here.

[0202] Optionally, the decoded reconstructed image may be used as an input image for super-resolution reconstruction, or the decoded reconstructed image and the side information may be used as an input image for super-resolution reconstruction.

[0203] The super-resolution reconstruction network provided by the present invention can be used as an upsampling module, that is, it can be used to replace the upsampling module in the encoding and decoding process, or as a candidate for the upsampling module.

[0204] Take the image reconstruction method applied to the image decoding method as an example, see Figures 18 to 20 , Fig.18 is a flow chart of an embodiment of a prior art image decoding method, Fig.19 is a flow chart of a second embodiment of an image decoding method according to the present invention, Fig. 20 It is a flowchart diagram of the third embodiment of the image decoding method of the present invention.

[0205] In the prior art, upsampling is usually performed directly through an upsampling module to achieve super-resolution reconstruction. Fig.19 As shown in , the decoded reconstructed image obtained by decoding the encoded data can be input into the super-resolution reconstruction network of the present invention, and the output image can be obtained by reconstructing the image using the image reconstruction method described in the above embodiment.

[0206] Or, if Fig. 20 As shown in , you can choose to use an upsampling module to perform super-resolution reconstruction, or choose to use the image reconstruction method described in the above embodiment to perform reconstruction processing to obtain an output image.

[0207] Optionally, during the encoding and decoding process, if the upsampling module is replaced by a super-resolution reconstruction network, no syntactic elements need to be transmitted. If the proposed super-resolution reconstruction network is used as a candidate module for the upsampling module, additional syntactic elements need to be transmitted to indicate the method of selecting upsampling; for example, a syntactic element SR_CNN_FLAG can be defined, and its value can be 0 or 1. When SR_CNN_FLAG takes a value of 0, it indicates that the super-resolution reconstruction method proposed in the present invention is not used for reconstruction processing, and when it takes a value of 1, it indicates that the super-resolution reconstruction method proposed in the present invention is used for reconstruction processing.

[0208] See also Fig.21 , Fig.21 It is a schematic structural diagram of a decoder according to an embodiment of the present invention.

[0209] In one embodiment, the decoder 20 includes a memory and a processor 21 , wherein the memory (not shown) is used to store computer programs required for the processor 21 to run.

[0210] The processor 21 may also be referred to as a CPU (Central Processing Unit). The processor 21 may be an integrated circuit chip having signal processing capabilities. The processor 21 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 21 may also be any conventional processor, etc.

[0211] The processor 21 is used to execute a computer program to implement the image reconstruction method described in any of the above embodiments, or the image decoding method described in the above embodiments.

[0212] See also Fig. 22 , Fig. 22 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present invention.

[0213] In one embodiment, a computer-readable storage medium 30 is used to store instruction / program data 31, and the instruction / program data 31 can be executed to implement an image reconstruction method as described in any of the above embodiments, or an image encoding method as described in the above embodiments, or an image decoding method as described in the above embodiments, which will not be repeated here.

[0214] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is schematic, for example, the division of modules or units is a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0215] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0216] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0217] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a computer-readable storage medium 30, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned computer-readable storage medium 30 includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, server and other media that can store program codes.

[0218] In addition, in the present invention, unless otherwise clearly specified and limited, the terms "connected", "connected", "stacked" and the like should be understood in a broad sense, for example, it can be fixedly connected, detachably connected, or integrated; it can be directly connected, or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image reconstruction method, characterized in that: include: Get the input image; The input image is respectively input into a global residual connection branch and a residual neural network branch of a super-resolution reconstruction network, so as to upsample the input image by using the global residual connection branch to obtain basic information of the input image, and reconstruct the input image by using the residual neural network branch to obtain residual information of the input image; Fusing the basic information and the residual information to generate an output image, wherein the resolution of the output image is higher than the resolution of the input image; The upsampling the input image by using the global residual connection branch to obtain basic information of the input image includes: Perform upsampling processing on the input image by using at least one upsampling processing module to obtain sub-basic information corresponding to each upsampling processing module, and obtain the basic information based on the sub-basic information corresponding to each upsampling processing module; The at least one upsampling processing module includes a priori filter, the priori filter includes a resampling filter, and using the priori filter to upsample the input image includes: The input image is subjected to up-sampling interpolation filtering using the resampling filter.

2. The image reconstruction method according to claim 1, characterized in that: The at least one upsampling processing module includes a convolutional neural network, and the sub-basic information corresponding to each of the upsampling processing modules includes first sub-basic information corresponding to the convolutional neural network and second sub-basic information corresponding to the priori filter; and obtaining the basic information based on the sub-basic information corresponding to each of the upsampling processing modules includes: The first sub-basic information and the second sub-basic information are weightedly fused to obtain the basic information.

3. The image reconstruction method according to claim 2, characterized in that: The convolutional neural network includes a feature extraction layer, a first convolutional layer, and a merge sorting layer that are cascaded in sequence. The upsampling process of the input image using the convolutional neural network to obtain the first sub-basic information includes: Extracting features of the input image using the feature extraction layer; Using the output of the feature extraction layer as the input of the first convolutional layer, and performing convolution processing to obtain multiple channel feature maps; The plurality of channel feature maps are fused using the merge-sort layer to obtain the first sub-basic information.

4. The image reconstruction method according to claim 1, characterized in that: The input image includes a luminance component and a chrominance component, and the tap coefficient corresponding to the luminance component is greater than the tap coefficient corresponding to the chrominance component.

5. The image reconstruction method according to claim 1, characterized in that: Before performing up-sampling interpolation filtering on the input image using the resampling filter, the method comprises: Assigning a filter coefficient in the row direction of the resampling filter in combination with the coordinate value of the current pixel in the output image, the width of the input image, and the width of the output image; The filter coefficients in the column direction of the resampling filter are assigned values ​​in combination with the coordinate value of the current pixel in the output image, the height of the input image and the height of the output image.

6. The image reconstruction method according to claim 5, characterized in that: The input image includes a luminance component and a chrominance component; The filter coefficients in the row direction are defined as: frac=((x*orgWidth<<4) / scaledWidth)&numFracPositions; The filter coefficients in the column direction are defined as: frac=((y*orgHeight<<4) / scaledHeight)&numFracPositions; Among them, (x, y) represents the coordinates of the current pixel in the output image; orgWidth represents the width of the input image; scaledWidth represents the width of the output image; orgHeight is the height of the input image; scaledHeight is the height of the output image; when calculating the filter coefficient of the luminance component, the value of numFracPositions is the first preset value, and when calculating the filter coefficient of the chrominance component, the value of numFracPositions is the second preset value.

7. The image reconstruction method according to any one of claims 1 to 6, characterized in that: The up-sampling interpolation filtering of the input image by using a resampling filter comprises: The luminance component of the input image is subjected to up-sampling interpolation filtering using 16 groups of 8-tap filter coefficient arrays, and the chrominance component of the input image is subjected to up-sampling interpolation filtering using 32 groups of 4-tap filter coefficient arrays.

8. The image reconstruction method according to claim 2, characterized in that: The a priori filter includes a bilinear cubic interpolation filter, and using the a priori filter to perform upsampling processing on the input image includes: The input image is subjected to up-sampling interpolation filtering using a bilinear cubic interpolation filter.

9. The image reconstruction method according to claim 8, characterized in that: The up-sampling interpolation filtering of the input image using a bilinear cubic interpolation filter comprises: Obtaining 4*4 neighborhood points within a preset range of the pixel point to be interpolated, so as to obtain the pixel value of the pixel point to be interpolated by using the coordinate values ​​of the neighborhood points; The input image is subjected to up-sampling interpolation filtering in combination with the pixel value of the pixel to be interpolated and a bilinear cubic interpolation function.

10. The image reconstruction method according to claim 9, characterized in that: The obtaining the pixel value of the pixel point to be interpolated by using the coordinate value of the neighborhood point comprises: The pixel value of the pixel to be interpolated is defined as: Among them, (x, y) represents the coordinates of the pixel to be interpolated; (xi, yi) represents the coordinates of the neighborhood point, i = 0, 1, 2, 3, j = 0, 1, 2, 3; f represents the pixel value of the pixel point; W represents a bilinear cubic interpolation function.

11. The image reconstruction method according to claim 9 or 10, characterized in that: The bilinear cubic interpolation function is defined as: Wherein, a is a third preset value.

12. The image reconstruction method according to claim 2, characterized in that: The global residual connection branch includes a first attention layer, which includes a second convolutional layer, a global mean pooling layer, a third convolutional layer, a first activation layer, a fourth convolutional layer and a normalization layer cascaded in sequence; the weighted fusion of the first sub-basic information and the second sub-basic information to obtain the basic information includes: The first sub-basic information and the second sub-basic information are fused and input into the second convolutional layer for feature extraction; Using the global mean pooling layer to perform global mean pooling processing on the output of the second convolutional layer; Using the third convolutional layer to extract features from the output of the global mean pooling layer; Using the output of the third convolutional layer as the input of the first activation layer, performing activation processing using the first activation layer, and inputting the feature map after the activation processing into the fourth convolutional layer for feature extraction; Using the normalization layer to perform normalization activation processing on the output of the fourth convolutional layer to obtain weights corresponding to the first sub-basis information and the second sub-basis information; The first sub-basic information and the second sub-basic information are weightedly fused using the obtained weight.

13. The image reconstruction method according to claim 1, characterized in that: The residual neural network comprises a low-level feature extraction layer, a mapping layer, a high-level feature extraction layer and an upsampling layer which are cascaded in sequence, wherein the output of the low-level feature extraction layer is connected to the input of the upsampling layer to form a local residual connection branch, and the mapping layer has a plurality of residual layers which are connected in sequence; the step of reconstructing the input image using the residual neural network branch to obtain the residual information of the input image comprises: Mapping the input image to a feature domain using the low-level feature extraction layer to extract a low-level feature image of the input image; Mapping the low-level feature image into a first high-level feature image using the mapping layer; Using the high-level feature extraction layer to extract features from the first high-level feature image to obtain a second high-level feature image; fusing the low-level feature image and the second high-level feature image using the local residual connection branch to obtain a fused feature image; The upsampling layer is used to expand the dimension and upsample the fused feature image to obtain residual information.

14. The image reconstruction method according to claim 13, characterized in that: The residual network further includes an input layer and an output layer, the input layer is connected to the low-level feature extraction layer, and the output layer is connected to the upsampling layer; the method further includes: Inputting the input image into the input layer, performing input transformation processing on the input image, and using the transformed input image as the input of the low-level feature extraction layer; wherein the form of the transformed input image matches the form allowed by the low-level feature extraction layer for input; And / or, inputting the residual information into the output layer, performing output transformation processing on the residual information, and using the transformed residual information as the output of the residual neural network branch.

15. The image reconstruction method according to claim 13, characterized in that: The mapping layer includes a plurality of first residual layers cascaded in sequence; the first residual layer includes a fifth convolution layer, a second activation layer, a sixth convolution layer and a second attention layer; The method further comprises: Using the fifth convolutional layer to perform feature extraction on the feature map output by the low-level feature image / previous residual layer; Using the second activation layer to perform activation processing on the output of the fifth convolutional layer, and using the sixth convolutional layer to perform feature extraction on the output of the second activation layer, so as to obtain a plurality of mapping feature images corresponding to multiple channels; Using the second attention layer to obtain the weights assigned to each of the mapped feature images; The mapped feature image is weightedly fused, and the weighted fused image is fused with the feature map of the low-level feature image / previous residual layer output.

16. The image reconstruction method according to claim 15, characterized in that: The mapping layer further includes a plurality of second residual layers and a plurality of third residual layers, and the first residual layer, the second residual layer and the third residual layer are connected; the second residual layer includes a seventh convolution layer with a span greater than one and the first residual layer, and the seventh convolution layer is connected to the first residual layer to downsample the feature map input to the second residual layer; the third residual layer includes a deconvolution layer with a span greater than one and the first residual layer, and the deconvolution layer is connected to the first residual layer to upsample the feature map input to the third residual layer; The first residual layer whose output size matches the feature map is connected to fuse the feature maps whose sizes match.

17. The image reconstruction method according to claim 13, characterized in that: The method comprises: Extracting features from the low-level feature image / the feature map output by the previous residual layer, and dividing the extracted feature map to obtain a plurality of image groups; fusing the multiple image groups multiple times to obtain multiple fused image groups, and extracting features from the multiple fused image groups to obtain multiple fused feature maps; The multiple fused feature maps are weighted-fused, and the weighted-fused feature map is fused with the feature map output by the low-level feature image / previous residual layer.

18. The image reconstruction method according to claim 17, characterized in that: The mapping layer includes a plurality of fourth residual layers cascaded in sequence, wherein the fourth residual layer includes a tenth convolutional layer, a segmentation layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a splicing layer, a fourteenth convolutional layer and a second attention layer; The feature extraction of the feature map of the low-level feature image / the feature map output by the previous residual layer and the division of the extracted feature map to obtain a plurality of image groups include: Inputting the feature map output by the low-level feature image / residual layer into the tenth convolutional layer for feature extraction; Using a segmentation layer, the feature map output by the tenth convolutional layer is divided into four groups, namely a first image group, a second image group, a third image group and a fourth image group; The performing multiple fusions on the multiple image groups to obtain multiple fused image groups, and performing feature extraction on the multiple fused image groups to obtain multiple fused feature maps comprises: Fusing the first image group with the second image group to obtain a first fused image group, and performing feature extraction on the first fused image group using the eleventh convolutional layer to obtain a first fused feature map; Fusing the output result of the eleventh convolutional layer with the third image group to obtain a second fused image group, and performing feature extraction on the second fused image group using the twelfth convolutional layer to obtain a second fused feature map; Fusing the output result of the twelfth convolutional layer with the fourth image group to obtain a third fused image group, and performing feature extraction on the third fused image group using the thirteenth convolutional layer to obtain a third fused feature map; The weighted fusion of the multiple fused feature maps, and fusing the weighted fused feature map with the low-level feature image includes: Inputting the first fused feature map, the second fused feature map, and the third fused feature map into a splicing layer for splicing; Using the fourteenth convolutional layer to perform feature transformation on the output result of the concatenation layer; Using the second attention layer to assign weights to the first fused feature map, the second fused feature map, and the third fused feature map; The first fused feature map, the second fused feature map and the third fused feature map are weighted fused, and the weighted fused feature map is fused with the low-level feature image.

19. The image reconstruction method according to any one of claims 15, 16 and 18, characterized in that: The second attention layer includes a global mean pooling layer, an eighth convolutional layer, a third activation layer, a ninth convolutional layer and a normalization layer which are cascaded in sequence; the method further includes: Using the global mean pooling layer to perform global mean pooling processing on the mapped feature image; Using the eighth convolutional layer to perform feature extraction on the feature map after global mean pooling processing; Using the third activation layer to activate the feature map output by the eighth convolutional layer, and inputting the activated feature map into the ninth convolutional layer for feature extraction; The output of the ninth convolutional layer is subjected to normalization activation processing by using a normalization layer to assign weights to components of the input image.

20. The image reconstruction method according to claim 1, characterized in that: The input image includes a reconstructed image and an image formed using side information.

21. An image encoding method, characterized in that: include: Acquire an original image, and downsample the original image to generate a downsampled image; Performing encoding processing on the downsampled image to obtain an encoded reconstructed image and encoded data; In response to the need to use a high-resolution image as a reference image, the encoded reconstructed image is reconstructed using the image reconstruction method according to any one of claims 1 to 20.

22. An image decoding method, characterized in that: include: Acquire coded data generated by the image coding method according to claim 21; Decoding the encoded data to obtain a decoded and reconstructed image; The reconstructed image is reconstructed using the image reconstruction method according to any one of claims 1 to 20.

23. An encoder, characterized in that: The encoder includes a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program to execute instructions to implement the image reconstruction method according to any one of claims 1 to 20, or the image encoding method according to claim 21.

24. A decoder, characterized in that The decoder includes a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program to implement the image reconstruction method according to any one of claims 1 to 20, or the image decoding method according to claim 22.

25. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store instructions / program data, which can be executed to implement the image reconstruction method according to any one of claims 1 to 20, or the image encoding method according to claim 21, or the image decoding method according to claim 22.

Images