Image restoration method, device, electronic device and computer storage medium

Abstract

The present application relates to the fields of image processing and computer vision, and provides an image restoration method, device, electronic device, and computer storage medium. The method comprises: performing texture encoding on an input image through a texture encoding module, performing texture restoration on the encoded input image in combination with a texture branch network, and outputting a final texture restoration image; performing gradient encoding on the input image through a gradient encoding module, performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restoration image; performing super-resolution processing on the input image through a backbone network, and outputting a final reconstructed feature map; and fusing the final texture restoration image, the final gradient restoration image, and the final reconstructed feature map to obtain a restored image carrying texture information and gradient information. The image restoration method provided in the embodiment of the present application retains texture information and gradient information during image super-resolution restoration, thereby improving the restoration quality of low-resolution images.

Description

Technical Field

[0001] The present application relates to the fields of image processing and computer vision, and in particular to an image restoration method, device, electronic device, and computer storage medium. Background Art

[0002] Single image super-resolution aims to restore a high-resolution image from a low-resolution image. With the development of deep learning, many super-resolution methods have been proposed. Most of these super-resolution methods are optimized by measuring the mean squared error of the pixel distance between the super-resolution image and the high-resolution image. However, the above optimization methods usually produce blurry images with a high peak signal-to-noise ratio. Recently, methods such as SRGAN, EnhanceNet, ESRGAN and NatSR, which aim to restore photo-realistic images, have utilized generative adversarial networks. Although generative adversarial methods can generate high-fidelity super-resolution results, they also produce geometric distortions while generating sharp edges and fine textures. Summary of the Invention

[0003] The present application provides an image restoration method, device, electronic device and computer storage medium, aiming to improve the restoration quality of low-resolution images.

[0004] In a first aspect, the present application provides an image restoration method, comprising:

[0005] The texture encoding module encodes the input image, and combines the texture branch network to restore the texture of the encoded input image, and outputs the final texture restoration image;

[0006] Performing gradient encoding on the input image through a gradient encoding module, and performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restored image;

[0007] Performing super-resolution processing on the input image through the backbone network and outputting a final reconstructed feature map;

[0008] The final texture restoration image, the final gradient restoration image and the final reconstructed feature map are fused to obtain a restoration image carrying texture information and gradient information.

[0009] In one embodiment, fusing the final texture restoration image, the final gradient restoration image, and the final reconstructed feature map to obtain a restoration image carrying texture information and gradient information includes:

[0010] Merging the final texture restoration image and the final reconstructed feature map by element-wise accumulation, and then obtaining a first target image through an activation function;

[0011] Performing threshold probability calculation on the first target image to obtain a first proportional image of the final texture restored image in the output image;

[0012] Multiplying the final reconstructed feature map and the first scale map element-wise and then adding the final texture restoration image to obtain a final fused texture map;

[0013] Merging the final gradient restored image and the final reconstructed feature map by element-wise accumulation, and then obtaining a second target image through the activation function;

[0014] Performing threshold probability calculation on the second target image to obtain a second proportional image of the final gradient restored image in the output image;

[0015] Multiplying the final gradient restored image and the second scale map element-wise, and then adding the final gradient restored image to obtain a final fused gradient map;

[0016] The final reconstructed feature map, the final fused texture map and the final fused gradient map are fused according to the dimensions to obtain the restored image.

[0017] The super-resolution processing of the input image is performed by the backbone network to output a final reconstructed feature map, including:

[0018] Performing feature extraction on the input image through a feature extraction convolution layer to obtain a feature map to be processed;

[0019] Reconstructing the feature map to be processed in sequence through multiple residual dense blocks in the backbone network to obtain a target feature map, and adding the target feature map and the feature map to be processed to obtain a final feature map;

[0020] The final feature map is upsampled by the upsampling module to obtain an upsampled feature map, and the upsampled feature map is processed by the feature restoration module to obtain the final reconstructed feature map.

[0021] The texture encoding module is used to perform texture encoding on the input image, and the texture branch network is used to perform texture restoration on the encoded input image, and the final texture restoration image is output, including:

[0022] Performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed;

[0023] Performing bitwise element accumulation on the grayscale image of the input image and the texture image to be processed through a fusion module to obtain a final input texture image;

[0024] The final input texture image is input into the texture branch network, and the features of the final input texture image are reconstructed through multiple residual dense blocks in the texture branch network, and the final texture restoration image is output.

[0025] The step of performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed includes:

[0026] Convolving the input image and its multiple texture feature convolution kernels through the texture convolution kernel module to obtain multiple convolved texture features;

[0027] The convolved texture features are multiplied by their corresponding coefficients and then accumulated to obtain the texture image to be processed.

[0028] The step of performing gradient encoding on the input image through the gradient encoding module, performing gradient restoration on the encoded input image in combination with the gradient branch network, and outputting a final gradient restored image comprises:

[0029] Performing gradient encoding on the input image through a gradient convolution module to obtain a gradient image to be processed;

[0030] The gradient image to be processed is input into the gradient branch network, and features of the gradient image to be processed are reconstructed through multiple residual dense blocks in the gradient branch network, and the final gradient restoration image is output.

[0031] The step of performing gradient encoding on the input image through a gradient convolution module to obtain a gradient image to be processed includes:

[0032] Convolving the input image and its multiple gradient feature convolution kernels through the gradient convolution module to obtain multiple convolved gradient features;

[0033] The plurality of convolved gradient features are added together to obtain the gradient image to be processed.

[0034] In a second aspect, the present application provides an image restoration device comprising:

[0035] The texture encoding module is used to perform texture encoding on the input image through the texture encoding module, and restore the texture of the encoded input image in combination with the texture branch network, and output the final texture restored image;

[0036] A gradient encoding module is used to perform gradient encoding on the input image through the gradient encoding module, and perform gradient restoration on the encoded input image in combination with the gradient branch network, and output a final gradient restored image;

[0037] A super-resolution processing module, configured to perform super-resolution processing on the input image through a backbone network and output a final reconstructed feature map;

[0038] The image fusion module is used to fuse the final texture restoration image, the final gradient restoration image and the final reconstructed feature map to obtain a restored image carrying texture information and gradient information.

[0039] In a third aspect, the present application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image restoration method described in the first aspect when executing the program.

[0040] In a fourth aspect, the present application further provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes a computer program, and when the computer program is executed by the processor, the image restoration method described in the first aspect is implemented.

[0041] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by the processor, it implements the image restoration method described in the first aspect.

[0042] The image restoration method, device, electronic device and computer storage medium provided in the present application not only perform super-resolution processing on the resolution of a low-resolution image during the image restoration process, but also restore the texture details and gradient details in the low-resolution image, that is, retain the texture information and gradient information in the low-resolution image. Therefore, in the process of image super-resolution restoration, the texture information and gradient information are retained, and a realistic restoration of the low-resolution image is achieved, thereby improving the restoration quality of the low-resolution image. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, a brief introduction is given below to the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 It is a flowchart of the image restoration method provided by this application;

[0045] Figure 2 It is a structural diagram of the texture inference module provided by this application;

[0046] Figure 3 It is a schematic diagram of the structure of the gradient reasoning module provided by this application;

[0047] Figure 4 It is a schematic diagram of the structure of the super-resolution inference module provided by this application;

[0048] Figure 5 It is a structural diagram of the fusion module provided by this application;

[0049] Figure 6 It is a structural schematic diagram of the image restoration device provided by this application;

[0050] Figure 7 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0052] Combine Figures 1 to 7 The present invention describes an image restoration method, device, electronic device, and computer storage medium. Figure 1 It is a flowchart of the image restoration method provided by this application; Figure 2 It is a structural diagram of the texture inference module provided by this application; Figure 3 It is a schematic diagram of the structure of the gradient reasoning module provided by this application; Figure 4 It is a schematic diagram of the structure of the super-resolution inference module provided by this application; Figure 5 It is a structural diagram of the fusion module provided by this application; Figure 6 It is a structural schematic diagram of the image restoration device provided by this application; Figure 7 It is a structural diagram of the electronic device provided in this application.

[0053] The embodiments of the present application provide embodiments of the image restoration method. It should be noted that although a logical order is shown in the flowchart, under certain data, the steps shown or described may be completed in an order different from that shown here.

[0054] The embodiment of the present application takes an electronic device as an example of an execution subject, and the embodiment of the present application takes an image restoration device as one of the manifestations of the electronic device, but is not limited thereto.

[0055] Reference Figure 1 , Figure 1 : is a flow chart of the image restoration method provided by this application. The image restoration method provided by the embodiment of this application includes:

[0056] Step S10: texture encoding the input image through the texture encoding module, and restoring the texture of the encoded input image in combination with the texture branch network, and outputting a final texture restored image.

[0057] It should be noted that the image restoration device includes but is not limited to a texture reasoning module, a gradient reasoning module, a super-resolution reasoning module and a fusion module, wherein the texture reasoning module includes a texture convolution kernel module (texture coding module), a fusion module, a texture branch network and a result output module; the gradient reasoning module includes a gradient convolution kernel module (gradient coding module), a gradient branch network and a result output module; the super-resolution reasoning module includes a feature extraction layer, a backbone network, an upsampling module, a feature restoration module and a result output module; the backbone network, the texture branch network and the gradient branch network all have multiple residual dense blocks, and the number of residual dense blocks is set according to actual conditions.

[0058] Furthermore, the image restoration device calculates texture features of the input image and restores the texture features. If the resolution of the input image does not meet a preset resolution, which is set based on actual conditions, it can be understood that the input image is a low-resolution image. Specifically, the image restoration device performs texture encoding on the input image (low-resolution image) using a texture encoding module, and simultaneously performs texture restoration on the texture-encoded input image using a texture branch network. The result output module then outputs a final texture-restored image of the input image, as described in steps S101 to S103.

[0059] Furthermore, steps S101 to S103 are described as follows:

[0060] Step S101, performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed;

[0061] Step S102, performing bitwise element accumulation on the grayscale image of the input image and the texture image to be processed by a fusion module to obtain a final input texture image;

[0062] Step S103: input the final input texture image into the texture branch network, and perform feature reconstruction on the final input texture image through multiple residual dense blocks in the texture branch network to output the final texture restoration image.

[0063] Specific combination Figure 2 Conduct analysis, Figure 2 The image restoration device inputs the image (low-resolution image) I input Input to the texture convolution kernel module (texture encoding module) F encoder In the texture convolution kernel module Fencoder For the input image I input Encode and obtain the texture image I to be processed texture , input image I input The encoding process can be expressed as I texture =F encoder (I input ).

[0064] Furthermore, the image restoration device inputs the image I input Grayscale image I gray and texture image I to be processed texture Input to the fusion module, where the fusion module is used to perform bitwise element accumulation on the two image matrices. Therefore, it can be understood that the image restoration device takes the input image I input Grayscale image I gray and texture image I to be processed texture Input to the fusion module, through the fusion module the grayscale image I gray and texture image I to be processed texture Perform bitwise element accumulation to obtain the final input texture image I input_texture .

[0065] Furthermore, the image restoration device will finally input the texture image I input_texture Input to the texture branch network, and the final input texture image I is processed by multiple residual dense blocks in the texture branch network. input_texture Perform feature reconstruction, where each residual dense block includes but is not limited to 5 convolutional layers and 5 ReLU activation layers. The convolution kernel size in the 5 convolutional layers is 3*3 and the step size is 2. It should be noted that I is added to each residual dense block. feature5 , I feature10 , I feature15 and I feature20 , used to assist in the reconstruction of texture features. feature5 , I feature10 , I feature15 and I feature20 It is the reconstructed feature map generated by the backbone super-resolution network (backbone network). The output of the last residual dense block I output_texture This is the final texture restoration image, and the final texture restoration image I is output through the result output module output_texture .

[0066] The embodiment of the present application processes the input image through the texture inference module and outputs the final texture restoration image, thereby retaining the output final texture restoration image.

[0067] Furthermore, the description of steps S1011 to S1012 is as follows:

[0068] Step S1011, convolving the input image and its multiple texture feature convolution kernels through the texture convolution kernel module to obtain multiple convolved texture features;

[0069] Step S1012: multiply the plurality of convolved texture features by their corresponding coefficients and then accumulate them to obtain the texture image to be processed.

[0070] Specifically, through the texture convolution kernel module F encoder Convolve the input image and its multiple texture feature convolution kernels respectively to obtain multiple convolved texture features. Further, the multiple convolved texture features are multiplied by their corresponding coefficients and then accumulated to obtain the texture image to be processed I texture .

[0071] In one embodiment, the input image I input The number of texture feature convolution kernels is 8, so it can be understood as: texture convolution kernel module F encoder For the input image I input And its 8 different texture feature convolution kernels are convolved to obtain 8 convolved texture features, among which the 8 different texture feature convolution kernels are as follows:

[0072] The texture feature convolution kernel 1 is: The texture feature convolution kernel 2 is: The texture feature convolution kernel 3 is: The texture feature convolution kernel 4 is: The texture feature convolution kernel 5 is: The texture feature convolution kernel 6 is: The texture feature convolution kernel 7 is: The texture feature convolution kernel 8 is: Furthermore, the 8 convolved texture features are multiplied by their corresponding coefficients and then accumulated to obtain the texture image to be processed I texture .

[0073] The embodiment of the present application processes the input image through a texture convolution kernel module to provide a data basis for outputting a final texture restoration image, thereby retaining the texture details in the image.

[0074] Step S20: performing gradient encoding on the input image through a gradient encoding module, and performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restored image.

[0075] The image restoration device calculates the input image I input The image restoration device uses the gradient encoding module to restore the gradient features of the input image I input(low-resolution image) is gradient encoded, and the gradient branch network is combined with the gradient encoded input image I input Perform gradient restoration and then output the input image I through the result output module input The final gradient restored image is as described in steps S201 to S202.

[0076] Furthermore, the description of step S201 to step S202 is as follows:

[0077] Step S201, performing gradient encoding on the input image through a gradient convolution module to obtain a gradient image to be processed;

[0078] Step S202: input the to-be-processed gradient image into the gradient branch network, and perform feature reconstruction on the to-be-processed gradient image through a plurality of residual dense blocks in the gradient branch network, and output the final gradient restored image.

[0079] Specific combination Figure 3 Conduct analysis, Figure 3 The image restoration device inputs the image I input Input to the gradient convolution kernel module (gradient encoding module) G encoder In the gradient convolution kernel module G encoder For the input image I input Encode and obtain the gradient image I to be processed input_gradient , input image I input The encoding process can be expressed as I input_gradient =G encoder (I input ).

[0080] Furthermore, the image restoration device converts the gradient image to be processed I input_gradient Input to the gradient branch network, and the gradient image I is processed through multiple residual dense blocks in the gradient branch network input_gradient Perform feature reconstruction, where each residual dense block includes but is not limited to 5 convolutional layers and 5 ReLU activation layers. The size of the convolution kernel in the 5 convolutional layers is 3*3 and the step size is 2. It should be noted that I is added to each residual dense block. feature5 , I feature10 , I feature15 and I feature20 , used to assist in the reconstruction of gradient features. feature5 , I feature10 , I feature15 and I feature20 It is the reconstructed feature map generated by the backbone super-resolution network (backbone network). The output of the last residual dense block I output_gradientThis is the final gradient restored image, and the final gradient restored image I is output through the result output module output_gradient .

[0081] The embodiment of the present application processes the input image through the gradient inference module and outputs the final gradient restored image, thereby retaining the output final gradient restored image.

[0082] Furthermore, the description of steps S2011 to S2012 is as follows:

[0083] Step S2011, convolving the input image and its multiple gradient feature convolution kernels through the gradient convolution module to obtain multiple convolved gradient features;

[0084] Step S2012: Adding the multiple convolved gradient features to obtain the gradient image to be processed.

[0085] Specifically, through the gradient convolution kernel module G encoder Convolve the input image and its multiple gradient feature convolution kernels respectively to obtain multiple convolved gradient features. Further, add the multiple convolved gradient features to obtain the gradient image to be processed I input_gradient .

[0086] In one embodiment, the input image I input The number of gradient feature convolution kernels is 2, so it can be understood as: gradient convolution kernel module G encoder For the input image I input And its two different gradient feature convolution kernels are convolved to obtain two convolved gradient features, among which the two different gradient feature convolution kernels are as follows:

[0087] The gradient feature convolution kernel 1 is: The gradient feature convolution kernel 2 is: Furthermore, the gradient features after the two convolutions are added together to obtain the gradient image to be processed I input_gradient .

[0088] The embodiment of the present application processes the input image through the gradient convolution kernel module to provide a data basis for outputting the final gradient restoration image, thereby retaining the gradient details in the image.

[0089] Step S30: performing super-resolution processing on the input image through the backbone network, and outputting a final reconstructed feature map.

[0090] Furthermore, the backbone network is used to process the input image I input Perform super-resolution processing and output the final reconstructed feature map I feature_final , specifically combined Figure 4 Conduct analysis, Figure 4 This is a schematic diagram of the structure of the super-resolution inference module provided by this application. The input image is input to the feature extraction convolution layer, and the feature extraction convolution layer extracts the features of the input image to obtain a feature map to be processed; the feature map to be processed is reconstructed in sequence by multiple residual dense blocks in the backbone network to obtain a target feature map, and the target feature map and the feature map to be processed are added to obtain a final feature map; the final feature map is upsampled by the upsampling module to obtain an upsampled feature map, and the upsampled feature map is processed by the feature restoration module to obtain the final reconstructed feature map.

[0091] Specifically, in one embodiment, the backbone network includes 23 residual dense blocks, so it can be understood as: a feature extraction convolution layer with a convolution kernel size of 3*3 and a step size of 1 is used to extract the input image I input Extract image features and obtain feature map I to be processed feature1 .

[0092] Furthermore, the feature map to be processed I feature1 The feature reconstruction is performed in the first residual dense block of the input backbone network. Each residual dense block contains 5 convolution layers with a convolution kernel size of 3*3 and a step size of 2 and 5 ReLU activation layers to obtain the feature map I to be processed. feature2 . Further, the feature map to be processed I feature2 Input the second residual dense block of the backbone network to reconstruct the features and obtain the feature map I to be processed feature3 . Further, the feature map to be processed I feature3 Input the third residual dense block of the backbone network to reconstruct the features and obtain the feature map I to be processed feature4 . Continue in sequence until the feature map to be processed I feature23 Input the 23rd residual dense block of the backbone network for feature reconstruction to obtain the target feature map I feature24 .

[0093] Furthermore, the feature map to be processed I feature1 and target feature map I feature24 Add them together to get the final feature map I feature . Further, the final feature map I is processed by upsampling module feature Upsample to obtain the upsampled feature map I upsample , then the upsampled feature map I is restored by a feature recovery module with two convolution kernels of size 3*3 and a step size of 1 upsample Processing is performed to obtain the final reconstructed feature map I feature_final .

[0094] Step S40 , fusing the final texture restoration image, the final gradient restoration image, and the final reconstructed feature map to obtain a restoration image carrying texture information and gradient information.

[0095] Furthermore, the final reconstructed feature map I obtained by the super-resolution inference module is feature_final , the final texture restoration image I obtained by the texture inference module output_texture The final gradient restored image I obtained by the gradient inference module output_gradient Perform feature fusion to obtain a high-resolution restored image that carries texture information and gradient information. Figure 5 Conduct analysis, Figure 5 This is a schematic diagram of the structure of the fusion module provided by this application. The final texture restoration image I output_texture And the final reconstructed feature map I feature_final Input to the merging module, and the final texture is restored to image I through the merging module output_texture And the final reconstructed feature map I feature_final The elements are accumulated and merged bit by bit, and then the first target image I1 is obtained through the activation function; the threshold probability calculation is performed on the first target image I1 to obtain the final texture restoration image I output_texture First scale diagram I in the output diagram texture_percentage ; The final feature map I will be reconstructed feature_final With the first scale figure I texture_percentage After element-wise multiplication, add the final texture restored image I output_texture , get the final fusion texture map I texture_mix .

[0096] The final gradient is restored to image I output_gradient And the final reconstructed feature map I feature_final Input to the merging module, and the final gradient restoration image I is restored by the merging module output_gradient And the final reconstructed feature map I feature_final The elements are accumulated and merged, and then the second target image I2 is obtained through the activation function; the threshold probability calculation is performed on the second target image I2 to obtain the final gradient restoration image I output_gradient Second scale diagram in output diagram I texture_percentage ; Restore the final gradient image I feature_final With the second scale figure I gradient_percentage After element-wise multiplication, add the final gradient to restore the image I output_gradient , and get the final fusion gradient map I gradient_mix .

[0097] The final reconstructed feature map I feature_final , Final fusion texture map I texture_mix And the final fused gradient map I gradient_mixFusion is performed according to the dimension to obtain the restored image I SR .

[0098] Furthermore, for the final texture restoration image I output_texture :The final reconstructed feature map I feature_final And the final texture restoration image I output_texture Input into the merging module, and the final feature map I is reconstructed by the merging module feature_final And the final texture restoration image I output_texture The elements are accumulated and merged bit by bit, and then a layer of ReLU activation function is applied to obtain the first target image I1. Further, the threshold probability calculation is performed on the first target image I1. The specific operation is to pass the first target image I1 through a layer of Sigmoid function to limit the values ​​in the first target image I1 to between (0, 1) to obtain the final texture restoration image I output_texture The proportion of the output diagram I texture_percentage . Further, the final feature map I is reconstructed feature_final and scale drawing I texture_percentage After element-wise multiplication, add the final texture restoration image I output_texture , and finally obtain the final fusion texture map I for fusion texture_mix .

[0099] Furthermore, for the final gradient restored image I output_gradient :The final reconstructed feature map I feature_final And the final gradient restored image I output_gradient Input into the merging module, and the final feature map I is reconstructed by the merging module feature_final And the final gradient restored image I output_gradient The elements are accumulated and merged bit by bit, and then a layer of ReLU activation function is applied to obtain the second target image I2. Further, the threshold probability calculation is performed on the second target image I2. The specific operation is to pass the second target image I2 through a layer of Sigmoid function to limit the values ​​in the second target image I2 to between (0, 1) to obtain the final gradient restored image I output_gradient The proportion of the output diagram I gradient_percentage . Further, the final feature map I is reconstructed feature_final and scale drawing I gradient_percentage After element-wise multiplication, add the final gradient to restore the image I output_gradient Get the final fusion gradient map I for fusion gradient_mix .

[0100] Furthermore, the feature map I is finally reconstructed feature_final , Final fusion texture map I texture_mix And the final fused gradient map I gradient_mixFusion is performed according to the dimension to obtain the restored image I SR .

[0101] The image restoration method provided in the embodiments of the present application, during the image restoration process, not only performs super-resolution processing on the resolution of a low-resolution image, but also restores the texture details and gradient details in the low-resolution image, that is, retains the texture information and gradient information in the low-resolution image. Therefore, during the image super-resolution restoration process, the texture information and gradient information are retained, thereby achieving realistic restoration of the low-resolution image, thereby improving the restoration quality of the low-resolution image.

[0102] Furthermore, the image restoration device provided by the present application is described below, and the image restoration device and the image restoration method can refer to each other.

[0103] like Figure 6 As shown, Figure 6 : is a schematic diagram of the structure of the image restoration device provided by this application, the image restoration device includes:

[0104] The texture encoding module 601 is used to perform texture encoding on the input image through the texture encoding module, and restore the texture of the encoded input image in combination with the texture branch network, and output a final texture restored image;

[0105] A gradient encoding module 602 is configured to perform gradient encoding on the input image through the gradient encoding module, and perform gradient restoration on the encoded input image in combination with a gradient branch network, and output a final gradient restored image;

[0106] A super-resolution processing module 603 is configured to perform super-resolution processing on the input image through a backbone network and output a final reconstructed feature map;

[0107] The image fusion module 604 is configured to fuse the final texture restoration image, the final gradient restoration image, and the final reconstructed feature map to obtain a restoration image carrying texture information and gradient information.

[0108] Furthermore, the texture encoding module 601 is further configured to:

[0109] Performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed;

[0110] Performing bitwise element accumulation on the grayscale image of the input image and the texture image to be processed through a fusion module to obtain a final input texture image;

[0111] The final input texture image is input into the texture branch network, and the features of the final input texture image are reconstructed through multiple residual dense blocks in the texture branch network, and the final texture restoration image is output.

[0112] Furthermore, the texture encoding module 601 is further configured to:

[0113] Convolving the input image and its multiple texture feature convolution kernels through the texture convolution kernel module to obtain multiple convolved texture features;

[0114] The convolved texture features are multiplied by their corresponding coefficients and then accumulated to obtain the texture image to be processed.

[0115] Furthermore, the gradient encoding module 602 is further configured to:

[0116] Performing gradient encoding on the input image through a gradient convolution module to obtain a gradient image to be processed;

[0117] The gradient image to be processed is input into the gradient branch network, and features of the gradient image to be processed are reconstructed through multiple residual dense blocks in the gradient branch network, and the final gradient restoration image is output.

[0118] Furthermore, the gradient encoding module 602 is further configured to:

[0119] Convolving the input image and its multiple gradient feature convolution kernels through the gradient convolution module to obtain multiple convolved gradient features;

[0120] The plurality of convolved gradient features are added together to obtain the gradient image to be processed.

[0121] Furthermore, the super-resolution processing module 603 is further configured to:

[0122] Performing feature extraction on the input image through a feature extraction convolution layer to obtain a feature map to be processed;

[0123] Reconstructing the feature map to be processed in sequence through multiple residual dense blocks in the backbone network to obtain a target feature map, and adding the target feature map and the feature map to be processed to obtain a final feature map;

[0124] The final feature map is upsampled by the upsampling module to obtain an upsampled feature map, and the upsampled feature map is processed by the feature restoration module to obtain the final reconstructed feature map.

[0125] Furthermore, the image fusion module 604 is further configured to:

[0126] Merging the final texture restoration image and the final reconstructed feature map by element-wise accumulation, and then obtaining a first target image through an activation function;

[0127] Performing threshold probability calculation on the first target image to obtain a first proportional image of the final texture restored image in the output image;

[0128] Multiplying the final reconstructed feature map and the first scale map element-wise and then adding the final texture restoration image to obtain a final fused texture map;

[0129] Merging the final gradient restored image and the final reconstructed feature map by element-wise accumulation, and then obtaining a second target image through the activation function;

[0130] Performing threshold probability calculation on the second target image to obtain a second proportional image of the final gradient restored image in the output image;

[0131] Multiplying the final gradient restored image and the second scale map element-wise, and then adding the final gradient restored image to obtain a final fused gradient map;

[0132] The final reconstructed feature map, the final fused texture map and the final fused gradient map are fused according to the dimensions to obtain the restored image.

[0133] The specific embodiments of the image restoration device provided in this application are basically the same as the embodiments of the above-mentioned image restoration method, and will not be described in detail here.

[0134] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call logic instructions in the memory 730 to execute an image restoration method, which includes:

[0135] The texture encoding module encodes the input image, and combines the texture branch network to restore the texture of the encoded input image, and outputs the final texture restoration image;

[0136] Performing gradient encoding on the input image through a gradient encoding module, and performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restored image;

[0137] Performing super-resolution processing on the input image through the backbone network and outputting a final reconstructed feature map;

[0138] The final texture restoration image, the final gradient restoration image and the final reconstructed feature map are fused to obtain a restoration image carrying texture information and gradient information.

[0139] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0140] On the other hand, the present application further provides a computer program product, comprising a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions. When the program instructions are executed by a computer, the computer is capable of performing the image restoration method provided by each of the above methods, the method comprising:

[0141] The texture encoding module encodes the input image, and combines the texture branch network to restore the texture of the encoded input image, and outputs the final texture restoration image;

[0142] Performing gradient encoding on the input image through a gradient encoding module, and performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restored image;

[0143] Performing super-resolution processing on the input image through the backbone network and outputting a final reconstructed feature map;

[0144] The final texture restoration image, the final gradient restoration image and the final reconstructed feature map are fused to obtain a restoration image carrying texture information and gradient information.

[0145] In another aspect, the present application further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program is implemented to perform the above-mentioned image restoration methods, the methods comprising:

[0146] The texture encoding module encodes the input image, and combines the texture branch network to restore the texture of the encoded input image, and outputs the final texture restoration image;

[0147] Performing gradient encoding on the input image through a gradient encoding module, and performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restored image;

[0148] Performing super-resolution processing on the input image through the backbone network and outputting a final reconstructed feature map;

[0149] The final texture restoration image, the final gradient restoration image and the final reconstructed feature map are fused to obtain a restoration image carrying texture information and gradient information.

[0150] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0152] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image restoration method, characterized in that: include: The texture encoding module encodes the input image, and combines the texture branch network to restore the texture of the encoded input image, and outputs the final texture restoration image; Performing gradient encoding on the input image through a gradient encoding module, and performing gradient restoration on the encoded input image in combination with a gradient branch network, and outputting a final gradient restored image; Performing super-resolution processing on the input image through the backbone network and outputting a final reconstructed feature map; Fusing the final texture restoration image, the final gradient restoration image, and the final reconstructed feature map to obtain a restoration image carrying texture information and gradient information; The fusing of the final texture restoration image, the final gradient restoration image and the final reconstructed feature map to obtain a restoration image carrying texture information and gradient information includes: Merging the final texture restoration image and the final reconstructed feature map by element-wise accumulation, and then obtaining a first target image through an activation function; Performing threshold probability calculation on the first target image to obtain a first proportional image of the final texture restored image in the output image; Multiplying the final reconstructed feature map and the first scale map element-wise and then adding the final texture restoration image to obtain a final fused texture map; Merging the final gradient restored image and the final reconstructed feature map by element-wise accumulation, and then obtaining a second target image through the activation function; Performing threshold probability calculation on the second target image to obtain a second proportional image of the final gradient restored image in the output image; Multiplying the final gradient restored image and the second scale map element-wise, and then adding the final gradient restored image to obtain a final fused gradient map; Fusing the final reconstructed feature map, the final fused texture map, and the final fused gradient map according to dimensions to obtain the restored image; The texture encoding module is used to perform texture encoding on the input image, and the texture branch network is used to perform texture restoration on the encoded input image, and the final texture restoration image is output, including: Performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed; Performing bitwise element accumulation on the grayscale image of the input image and the texture image to be processed through a fusion module to obtain a final input texture image; The final input texture image is input into the texture branch network, and the features of the final input texture image are reconstructed through multiple residual dense blocks in the texture branch network, and the final texture restoration image is output.

2. The image restoration method according to claim 1, wherein: The super-resolution processing of the input image is performed by the backbone network to output a final reconstructed feature map, including: Performing feature extraction on the input image through a feature extraction convolution layer to obtain a feature map to be processed; Reconstructing the feature map to be processed in sequence through multiple residual dense blocks in the backbone network to obtain a target feature map, and adding the target feature map and the feature map to be processed to obtain a final feature map; The final feature map is upsampled by an upsampling module to obtain an upsampled feature map, and the upsampled feature map is processed by a feature restoration module to obtain the final reconstructed feature map.

3. The image restoration method according to claim 1, wherein: The step of performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed includes: Convolving the input image and its multiple texture feature convolution kernels through the texture convolution kernel module to obtain multiple convolved texture features; The convolved texture features are multiplied by their corresponding coefficients and then accumulated to obtain the texture image to be processed.

4. The image restoration method according to claim 1, wherein: The step of performing gradient encoding on the input image through the gradient encoding module, performing gradient restoration on the encoded input image in combination with the gradient branch network, and outputting a final gradient restored image comprises: Performing gradient encoding on the input image through a gradient convolution module to obtain a gradient image to be processed; The gradient image to be processed is input into the gradient branch network, and features of the gradient image to be processed are reconstructed through multiple residual dense blocks in the gradient branch network, and the final gradient restoration image is output.

5. The image restoration method according to claim 4, wherein: The step of performing gradient encoding on the input image through a gradient convolution module to obtain a gradient image to be processed includes: Convolving the input image and its multiple gradient feature convolution kernels through the gradient convolution module to obtain multiple convolved gradient features; The plurality of convolved gradient features are added together to obtain the gradient image to be processed.

6. An image restoration device, characterized in that: include: The texture encoding module is used to perform texture encoding on the input image through the texture encoding module, and restore the texture of the encoded input image in combination with the texture branch network, and output the final texture restored image; A gradient encoding module is used to perform gradient encoding on the input image through the gradient encoding module, and perform gradient restoration on the encoded input image in combination with the gradient branch network, and output a final gradient restored image; A super-resolution processing module, configured to perform super-resolution processing on the input image through a backbone network and output a final reconstructed feature map; An image fusion module, configured to fuse the final texture restoration image, the final gradient restoration image, and the final reconstructed feature map to obtain a restored image carrying texture information and gradient information; The fusing of the final texture restoration image, the final gradient restoration image and the final reconstructed feature map to obtain a restoration image carrying texture information and gradient information includes: Merging the final texture restoration image and the final reconstructed feature map by element-wise accumulation, and then obtaining a first target image through an activation function; Performing threshold probability calculation on the first target image to obtain a first proportional image of the final texture restored image in the output image; Multiplying the final reconstructed feature map and the first scale map element-wise and then adding the final texture restoration image to obtain a final fused texture map; Merging the final gradient restored image and the final reconstructed feature map by element-wise accumulation, and then obtaining a second target image through the activation function; Performing threshold probability calculation on the second target image to obtain a second proportional image of the final gradient restored image in the output image; Multiplying the final gradient restored image and the second scale map element-wise, and then adding the final gradient restored image to obtain a final fused gradient map; Fusing the final reconstructed feature map, the final fused texture map, and the final fused gradient map according to dimensions to obtain the restored image; The texture encoding module is used to perform texture encoding on the input image, and the texture branch network is used to perform texture restoration on the encoded input image, and the final texture restoration image is output, including: Performing texture encoding on the input image through a texture convolution kernel module to obtain a texture image to be processed; Performing bitwise element accumulation on the grayscale image of the input image and the texture image to be processed through a fusion module to obtain a final input texture image; The final input texture image is input into the texture branch network, and the features of the final input texture image are reconstructed through multiple residual dense blocks in the texture branch network, and the final texture restoration image is output.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the image restoration method according to any one of claims 1 to 5 is implemented.

8. A non-transitory computer-readable storage medium comprising a computer program, characterized in that: When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 5 is implemented.

Images