Image restoration method, device, computer equipment, storage medium and program product

Abstract

The present application relates to an image restoration method, device, computer equipment, storage medium and program product. It can be used in the field of financial technology or other related fields. The method comprises: obtaining a target defective image; inputting the target defective image into a pre-trained image restoration model, performing feature extraction processing on the target defective image through a downsampling module and a feature extraction module in the image restoration model respectively, fusing the feature maps output by the downsampling module and the feature extraction module and then performing upsampling processing to output the target restoration image; wherein the feature extraction module comprises a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used for spatial feature extraction, the slicing layer is used for slicing the feature map output by the first feature extraction layer, and the first connection layer is used for connecting and outputting multiple feature maps. The use of this method can improve the quality of image restoration.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image restoration method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] Image restoration is a processing technology that improves image quality and is widely used in scientific research and engineering. In traditional technology, image restoration is performed based on a network model with a CNN (Convolutional Neural Networks) structure. However, in traditional technology, when performing deep network calculations, the network model based on the CNN structure loses the features of the restored image, and the restoration quality of the restored image is poor. Summary of the invention

[0003] Based on this, it is necessary to provide an image restoration method, apparatus, computer equipment, storage medium and program product that can improve the quality of image restoration in response to the above technical problems.

[0004] In a first aspect, the present application provides an image restoration method. The method comprises:

[0005] Acquire a target defect image;

[0006] The target defective image is input into a pre-trained image restoration model, and feature extraction processing is performed on the target defective image through the downsampling module and the feature extraction module in the image restoration model respectively, the feature maps output by the downsampling module and the feature extraction module are fused, and the fused feature maps are upsampled by the upsampling module in the image restoration model to output the target restoration image; wherein the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used for spatial feature extraction, the slicing layer is used for slicing the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used for connection processing and outputting the multiple second feature maps.

[0007] In one embodiment, the feature extraction module includes a plurality of cascaded first feature extraction layers, each of which is composed of a convolution layer and an activation layer, and the convolution kernel size of the convolution layer in each first feature extraction layer is different.

[0008] In one of the embodiments, the processing of the first slicing layer includes: performing a first slicing process on the first feature map based on a size parameter of the first feature map to obtain the plurality of second feature maps.

[0009] In one of the embodiments, the processing process of the first connection layer includes: based on the channel parameters of each second feature map, connecting and outputting each second feature map.

[0010] In one embodiment, the downsampling module includes a plurality of cascaded second feature extraction layers, and each second feature extraction layer is composed of a convolution layer, an activation layer and a pooling layer cascaded in sequence; the processing process of the second feature extraction layer includes: performing convolution processing on the input feature map through the convolution layer; performing activation processing on the feature map obtained by the convolution processing based on the activation function through the activation layer; performing maximum pooling processing on the feature map obtained by the activation processing through the pooling layer to obtain an output feature map.

[0011] In one embodiment, the upsampling module includes a plurality of cascaded upsampling layers, each of which is composed of a convolution layer, an activation layer and a deconvolution layer cascaded in sequence; the processing process of the upsampling layer includes: performing convolution processing on the input feature map through the convolution layer; performing activation processing on the feature map obtained by the convolution processing based on the activation function through the activation layer; performing deconvolution processing on the feature map obtained by the activation processing through the deconvolution layer to obtain an output image.

[0012] In one embodiment, the image restoration model also includes a preprocessing module, which includes a cascaded second slicing layer, a second connection layer and a third feature extraction layer, and the third feature extraction layer is composed of a convolution layer and an activation layer; the preprocessing module is used to perform pre-feature extraction processing on the target defective image, so that the downsampling module and the feature extraction module respectively perform feature extraction processing on the feature map output by the preprocessing module.

[0013] In one of the embodiments, the training process of the image restoration model includes: obtaining a training image set, which includes a plurality of simulated defective images; based on the training image set, using a focal loss function and an Adam gradient descent algorithm to train the initial image restoration model to obtain the image restoration model.

[0014] In one of the embodiments, the process of acquiring the simulated defect image includes: acquiring a normal image;

[0015] A mask image is generated by using a random mask; and the normal image is multiplied by the mask image to obtain the simulated defective image.

[0016] In a second aspect, the present application also provides an image restoration device. The device comprises:

[0017] An acquisition module, used for acquiring a target defective image;

[0018] An extraction module is used to input the target defective image into a pre-trained image restoration model, perform feature extraction processing on the target defective image through the downsampling module and the feature extraction module in the image restoration model respectively, fuse the feature maps output by the downsampling module and the feature extraction module, and upsample the fused feature maps through the upsampling module in the image restoration model to output the target restoration image; wherein the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used for spatial feature extraction, the slicing layer is used for slicing the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used for connecting and outputting the multiple second feature maps.

[0019] In one embodiment, the feature extraction module includes a plurality of cascaded first feature extraction layers, each of which is composed of a convolution layer and an activation layer, and the convolution kernel size of the convolution layer in each first feature extraction layer is different.

[0020] In one of the embodiments, the extraction module includes a slicing processing submodule, and the slicing processing submodule is used to: perform a first slicing process on the first feature map based on a size parameter of the first feature map to obtain the multiple second feature maps.

[0021] In one of the embodiments, the extraction module includes a connection processing submodule, and the connection processing submodule is used to: based on the channel parameters of each second feature map, perform connection processing on each second feature map and output it.

[0022] In one embodiment, the downsampling module includes a plurality of cascaded second feature extraction layers, and each second feature extraction layer is composed of a convolution layer, an activation layer, and a pooling layer cascaded in sequence;

[0023] The extraction module includes a downsampling processing submodule, which is used to: perform convolution processing on the input feature map through a convolution layer; perform activation processing on the feature map obtained by the convolution processing based on an activation function through an activation layer; and perform maximum pooling processing on the feature map obtained by the activation processing through a pooling layer to obtain an output feature map.

[0024] In one embodiment, the upsampling module includes a plurality of cascaded upsampling layers, each of which is composed of a convolutional layer, an activation layer, and a deconvolutional layer cascaded in sequence;

[0025] The extraction module includes an upsampling processing submodule, which is used to: perform convolution processing on the input feature map through a convolution layer; perform activation processing on the feature map obtained by the convolution processing based on an activation function through an activation layer; and perform deconvolution processing on the feature map obtained by the activation processing through a deconvolution layer to obtain an output image.

[0026] In one embodiment, the image restoration model also includes a preprocessing module, which includes a cascaded second slicing layer, a second connection layer and a third feature extraction layer, and the third feature extraction layer is composed of a convolution layer and an activation layer; the preprocessing module is used to perform pre-feature extraction processing on the target defective image, so that the downsampling module and the feature extraction module respectively perform feature extraction processing on the feature map output by the preprocessing module.

[0027] In one of the embodiments, the device also includes a training module, which is used to: obtain a training image set, which includes a plurality of simulated defective images; based on the training image set, use a Focal loss function and an Adam gradient descent algorithm to train the initial image restoration model to obtain the image restoration model.

[0028] In one of the embodiments, the device further includes a generating module, the generating module being used to: acquire a normal image; generate a mask image using a random mask; and multiply the normal image by the mask image to obtain the simulated defective image.

[0029] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect above.

[0031] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0032] The above-mentioned image restoration method, apparatus, computer equipment, storage medium and program product obtain a target defective image and input the target defective image into a pre-trained image restoration model, thereby performing feature extraction processing on the target defective image through the downsampling module and the feature extraction module in the image restoration model respectively, and then fusing the feature map output by the downsampling module and the feature extraction module, and upsampling the fused feature map through the upsampling module in the image restoration model to obtain the target restoration image output by the image restoration model, thereby realizing the restoration processing of the target defective image. Among them, since the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used to perform spatial feature extraction, thereby increasing the nonlinearity of the feature, improving the network expression ability of the image restoration model, and obtaining the spatial feature information of the image; the slicing layer is used to slice the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used to connect and output the multiple second feature maps; the feature maps output by the downsampling module and the feature extraction module are fused, while improving the restoration processing efficiency, a feature map containing rich spatial feature information is obtained, and the feature map containing spatial feature information output by the feature extraction module is fused with the feature map output by the downsampling module, thereby improving the restoration effect on the target defective image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic flow chart of an image restoration method in one embodiment;

[0034] Figure 2 is a schematic diagram of the structure of a feature extraction module in one embodiment;

[0035] Figure 3 is a schematic diagram of the structure of a downsampling module in one embodiment;

[0036] Figure 4 is a schematic flow chart of a second feature extraction layer processing process in one embodiment;

[0037] Figure 5 is a schematic diagram of the structure of an upsampling module in one embodiment;

[0038] Figure 6 Schematic diagram of the process of upsampling layer processing in one embodiment;

[0039] Figure 7 A schematic diagram of the structure of an image restoration model in one embodiment;

[0040] Figure 8 A schematic diagram of a process of training an image restoration model in one embodiment;

[0041] Fig. 9 is a structural schematic diagram of an image restoration model in another embodiment;

[0042] Fig.10 is a structural block diagram of an image restoration device in one embodiment;

[0043] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] Image restoration is a processing technology to improve image quality. It is a hot topic in the field of image processing research and is widely used in scientific research and engineering. In the process of image restoration, the traditional CNN structure network will lose the feature information of the image when performing deep network, and the effect of image restoration will also deteriorate.

[0046] Based on this, an embodiment of the present application provides an image restoration method, based on which the quality of the restored image can be improved to obtain a restored image with a better restoration result.

[0047] In one embodiment, Figure 1 As shown, an image restoration method is provided. The embodiment of the present application takes the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be a personal computer, a laptop, a smart phone or a tablet computer, etc., and the server can be implemented as an independent server or a server cluster composed of multiple servers. In the embodiment of the present application, the image restoration method includes the following steps:

[0048] Step 101, obtaining a target defective image.

[0049] Step 102, input the target defective image into a pre-trained image restoration model, perform feature extraction on the target defective image through the downsampling module and the feature extraction module in the image restoration model, fuse the feature maps output by the downsampling module and the feature extraction module, and perform upsampling on the fused feature maps through the upsampling module in the image restoration model to output the target restoration image. The feature extraction module includes a cascaded first feature extraction layer, a first slicing layer, and a first connection layer, the first feature extraction layer is used for spatial feature extraction, the slicing layer is used for slicing the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used for connecting and outputting the multiple second feature maps.

[0050] Among them, the target defective image refers to an image that needs to be repaired to be clearer and more complete. For example, the target defective image can be an old photo or a faded image. By repairing the target defective image, a more complete and clear repaired image corresponding to the target defective image is obtained. For example, in the scene of carrying out nostalgic activities, many old photos are designed, which have some areas damaged due to years of storage, so the old photos need to be repaired. In the embodiment of the present application, the target defective image is input into a pre-trained image repair model, and the image repair model is used to extract features and repair the target defective image, and the target repaired image corresponding to the target defective image is output.

[0051] Specifically, the image restoration model includes a downsampling module and a feature extraction module. The image restoration model performs downsampling processing on the target defective image through the downsampling module to obtain a feature map output by the downsampling module, thereby reducing the dimension of features in the image and retaining effective information, thereby improving the model processing efficiency. At the same time, the image restoration model performs feature extraction processing on the target defective image through the feature extraction module, thereby obtaining the spatial features lost by the target defective image during the processing of the downsampling module, so that the spatial feature information in the target defective image can be retained, thereby ensuring the quality of the restoration processing.

[0052] After the target defective image is processed by the downsampling module and the feature extraction module respectively, the feature maps output by the downsampling module and the feature extraction module are fused to obtain a fused feature image. The fused feature image is then upsampled by the upsampling module to restore the size of the fused feature image, and a target repair image having the same size as the target defective image is obtained. Thus, while the image is downsampled, the spatial feature information of the image is also retained, thereby improving the image repair effect. Since the downsampling module performs pooling processing, it discards a large number of spatial features of the target defective image. Therefore, through the fusion processing, the feature map is weighted, so that the repair effect is obvious. Specifically, the structure of the feature extraction module can be determined based on the attention mechanism, so that the feature extraction module is mainly used to extract spatial feature information of the target defective image.

[0053] The feature extraction module includes a cascaded first feature extraction layer, a first slicing layer, and a first connection layer, wherein the first feature extraction layer mainly extracts the spatial features of the target defective image through convolution processing to obtain a first feature map containing spatial feature information, and increases the nonlinearity of the spatial features through activation processing, thereby improving the expression ability of the image restoration model. Then, the feature extraction module slices the first feature map from which the spatial features are extracted through the first slicing layer to obtain multiple second feature maps, and connects the multiple second feature maps through the first connection layer, thereby achieving the purpose of downsampling while retaining the rich spatial feature information in the image. Optionally, the spatial feature information refers to features such as color, hierarchy, or correlation of the image.

[0054] Optionally, after acquiring the target defective image, the image restoration model may perform preliminary feature processing on the target defective image, and input the feature map obtained by the preliminary feature processing into the downsampling module and the feature extraction module respectively, so that the downsampling module and the feature extraction module perform feature extraction processing on the feature map obtained by the preliminary feature processing.

[0055] In the above-mentioned image restoration method, a target defective image is obtained and input into a pre-trained image restoration model, so that feature extraction processing is performed on the target defective image through the downsampling module and the feature extraction module in the image restoration model respectively, and then the feature map output by the downsampling module and the feature extraction module is fused, and the fused feature map is upsampled through the upsampling module in the image restoration model to obtain the target restoration image output by the image restoration model, thereby realizing the restoration processing of the target defective image. Among them, since the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used to perform spatial feature extraction, thereby increasing the nonlinearity of the feature, improving the network expression ability of the image restoration model, and obtaining the spatial feature information of the image; the slicing layer is used to slice the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used to connect and output the multiple second feature maps; the feature maps output by the downsampling module and the feature extraction module are fused, and while improving the restoration processing efficiency, a feature image containing rich spatial feature information is obtained, thereby improving the restoration effect on the target defective image.

[0056] In one embodiment, Figure 2 As shown, it shows a schematic diagram of the structure of a feature extraction module provided in an embodiment of the present application. The feature extraction module includes a plurality of cascaded first feature extraction layers, each of which is composed of a convolution layer and an activation layer, and the convolution kernel size of the convolution layer in each first feature extraction layer is different.

[0057] Optionally, the feature extraction module may include multiple first feature extraction layers, and each first feature extraction layer is cascaded. Exemplarily, Figure 2 In the figure, two first feature extraction layers are used for illustration. Figure 2For example, the input image of the feature extraction module is used as the input image of the first first feature extraction layer, and the input image is convolved based on the first convolution kernel by the first first feature extraction layer and then activated to obtain the feature map output by the first first feature extraction layer; the feature map output by the first first feature extraction layer is used as the input of the second first feature extraction layer, and it is convolved based on the second convolution kernel and then activated to obtain the feature map output by the second first feature extraction layer, so as to further slice the feature map output by the second first feature extraction layer through the first slicing layer, and connect it through the first connection layer. Wherein, the convolution kernel refers to the weighted average of pixels in a small area of ​​the input image when the input image is given during image processing, and each corresponding pixel in the output image is obtained after the weighted average, wherein the weight is defined by a function, which is called the convolution kernel. The convolution kernel can be measured by size. In the embodiment of the present application, the size of the convolution kernel of the convolution layer in each first feature extraction layer is different. Optionally, the size of the first convolution kernel can be 1×1, and the size of the second convolution kernel can be 3×3. Optionally, the activation layer in each second feature extraction layer implements activation processing through an activation function. Of course, other methods that can implement activation processing on the image can also be used for activation processing.

[0058] In the embodiment of the present application, multiple first feature extraction layers for feature extraction are provided, and each first feature extraction layer includes a convolution layer and an activation layer. The convolution layer performs convolution processing, so that the spatial features of the image input to the first feature extraction layer can be extracted. Furthermore, the activation layer performs activation processing, which increases the nonlinearity of the spatial features in the image. While extracting the spatial features of the image, the expression ability of the image restoration model can also be improved.

[0059] In one embodiment, the processing process of the first slicing layer includes: performing a first slicing process on the first feature map based on a size parameter of the first feature map to obtain the plurality of second feature maps.

[0060] Among them, the feature extraction module includes a first feature extraction layer, a first slicing layer and a first connection layer. The first feature extraction layer extracts spatial features of the target defective image and outputs a first feature map containing spatial feature information. The first feature map is input to the first slicing layer, and the first slicing layer performs a first slicing process on the first feature map based on the size parameter of the first feature map to obtain multiple second feature maps. Among them, the size of the image can usually be represented by parameters such as the number of channels, length, and width. In the embodiment of the present application, the size parameter can be length and width. For example, the number of channels is recorded as C, the length is recorded as H, and the width is recorded as W. For example, if the size of the first feature map is C×H×W, the first feature map is sliced ​​based on the size parameter, that is, the first feature map is sliced ​​according to the length and width, and is divided into four second feature maps with the same number of channels, the same length and width, and each second feature map has the same number of channels as the first feature map, but the length and width are different from the first feature map. For example, the size of each second feature map obtained after the slicing process is C×H / 2×W / 2.

[0061] In the embodiment of the present application, by slicing the first feature map based on the size parameter through the first slicing layer, the feature image of the complete area in the first feature map can help repair the feature image of the defective area to be repaired, and the loss of spatial features is extremely small, thereby providing feature information for obtaining a high-quality target repair image.

[0062] In one embodiment, the processing process of the first connection layer includes: based on the channel parameters of each second feature map, connecting and outputting each second feature map.

[0063] Specifically, the feature extraction module also connects and outputs each second feature map based on the channel parameter through the first connection layer. The output of the first connection layer is used as the output of the feature extraction module. The channel parameter is the number of channels of the image. Taking the four second feature maps obtained in the above text and the size of each second feature map being C×H / 2×W / 2 as an example, each second feature map is connected, that is, the four second feature maps are superimposed in the dimension of the number of channels, so that the feature map obtained after superposition has four times the number of channels as the single second feature map, and the length and width remain unchanged, that is, the size of the feature map obtained by the connection process is 4C×H / 2×W / 2.

[0064] In the embodiment of the present application, since the feature extraction module uses convolution processing and matrix-like splicing and slicing, the loss of spatial feature information of the image is extremely small. The feature map output by the feature extraction module based on the attention mechanism is fused with the feature map output by the downsampling module, which effectively repairs the defect that the spatial features of the image are damaged during the downsampling process.

[0065] In one embodiment, Figure 3 As shown, it shows a schematic diagram of the structure of a downsampling module provided in an embodiment of the present application. The downsampling module includes a plurality of cascaded second feature extraction layers, and each second feature extraction layer is composed of a convolution layer, an activation layer, and a pooling layer cascaded in sequence. Exemplarily, Figure 3 In the example of three second feature extraction layers, in the embodiment of the present application, the specific number of the second feature extraction layers can be determined according to the actual situation and is not specifically limited here. Wherein, for each feature extraction layer, it includes a cascaded convolution layer, an activation layer, and a pooling layer.

[0066] Please refer to Figure 4 , which shows a schematic flow chart of a second feature extraction layer processing process provided by an embodiment of the present application. The processing process of the second feature extraction layer includes:

[0067] Step 401, performing convolution processing on the input feature map through a convolution layer.

[0068] Optionally, the size of the convolution kernel of the convolution layer in each second feature extraction layer may be 3×3. Optionally, the size of the convolution kernel of the convolution layer in each second feature extraction layer may also be different.

[0069] Step 402: Activate the feature map obtained by convolution processing through the activation layer based on the activation function.

[0070] The activation layer can activate the feature map obtained by convolution based on the activation function. Optionally, the activation function can be a linear rectification function (Linear rectification function, ReLU). Of course, the feature map obtained by convolution can also be activated based on other activation functions or other methods that can achieve activation processing.

[0071] Step 403, performing maximum pooling processing on the feature map obtained by the activation processing through the pooling layer to obtain an output feature map.

[0072] Among them, the pooling layer is mainly used to further perform maximum pooling processing on the feature map obtained by activation processing, so as to reduce the dimension of the processed image and remove redundant information, thereby simplifying the complexity of the model processing process and reducing the amount of calculation and device memory consumption. In other words, the pooling layer can be a maximum pooling layer to achieve maximum pooling processing.

[0073] The pooling kernel is an important parameter for the pooling layer to implement pooling processing. It is equivalent to a filter. Pooling processing based on a pooling kernel of a preset size can make the features of the image more concentrated. Optionally, the size of the pooling kernel of the pooling layer in each second feature extraction layer can be 2×2, or, optionally, the size of the pooling kernel of the pooling layer in each second feature extraction layer can also be different.

[0074] Taking the example that the downsampling module includes three cascaded second feature extraction layers, the convolution kernel size of the convolution layer of each second feature extraction layer is 3×3, the size of the pooling kernel of the pooling layer of each second feature extraction layer is 2×2, and each activation layer uses a linear rectification function for activation processing, the size of the feature map input to the downsampling module is recorded as 4C×H / 2×W / 2, and the processing process of the downsampling module includes:

[0075] The feature map obtained by the downsampling module is used as the input of the first second feature extraction layer. The first second feature extraction layer performs convolution, activation and maximum pooling on the feature map, and outputs a feature map of size 4C×H / 4×W / 4. The feature map output by the first second feature extraction layer is used as the input of the second second feature extraction layer. The second second feature extraction layer performs convolution, activation and maximum pooling on the obtained feature map, and outputs a feature map of size 8C×H / 8×W / 8. The feature map output by the second second feature extraction layer is used as the input of the third second feature extraction layer. The third second feature extraction layer performs convolution, activation and maximum pooling on the feature map, and outputs a feature map of size 16C×H / 16×W / 16. The output of the third second feature extraction layer is recorded as the output of the downsampling module. Thus, the downsampling module implements downsampling processing on the input feature map.

[0076] In an embodiment of the present application, the input feature map is downsampled through a downsampling module to reduce the dimension of the feature and retain the effective information in the image, which can avoid overfitting to a certain extent, simplify the complexity of the model calculation process, and improve the operating efficiency of the model.

[0077] In one embodiment, Figure 5 As shown, it shows a schematic diagram of the structure of an upsampling module provided in an embodiment of the present application. The upsampling module includes a plurality of cascaded upsampling layers, each of which is composed of a convolutional layer, an activation layer, and a deconvolutional layer cascaded in sequence. Exemplarily, Figure 5 Taking four upsampling layers as an example, in the embodiment of the present application, the specific number of upsampling layers in the upsampling module can be determined according to the actual situation and is not specifically limited here. Wherein, for each upsampling layer, it includes a cascaded convolution layer, an activation layer, and a deconvolution layer. The upsampling module is mainly used to perform upsampling processing on the feature map obtained by fusing the feature map output by the downsampling module and the feature extraction module to restore the size of the image.

[0078] Please refer to Figure 6 , which shows a schematic flow chart of an upsampling layer processing process provided by an embodiment of the present application. The upsampling layer processing process includes:

[0079] Step 601, performing convolution processing on the input feature map through a convolution layer.

[0080] The input of the upsampling layer is used as the input of the convolution layer, and the input feature map is convolved by the convolution layer. Optionally, the size of the convolution kernel of each upsampling layer may be 3×3, and optionally, the size of the convolution kernel of each upsampling layer may be different.

[0081] Step 602: Activate the feature map obtained by convolution processing through the activation layer based on the activation function.

[0082] Optionally, the activation function used by the activation layer for activation processing may be a linear rectification function. Of course, the feature map obtained by convolution processing may also be activated based on other activation functions or other methods that can achieve activation processing.

[0083] Step 603, performing deconvolution processing on the feature map obtained by the activation processing through a deconvolution layer to obtain an output image.

[0084] Among them, the feature map obtained after downsampling is smaller than the target defective image. In order to restore the image size, each upsampling layer in the upsampling module needs to upsample the feature map.

[0085] Deconvolution is the opposite of convolution. It is mainly used to map the low-dimensional features obtained by convolution into high-dimensional inputs, which can achieve upsampling of the input feature map. That is, the feature map obtained by downsampling is smaller than the target defective image. In order to obtain a target repaired image of the same size as the target defective image after image repair, it is necessary to implement upsampling through the upsampling layer in the upsampling module to restore the image size.

[0086] Optionally, the size of the deconvolution kernel of the deconvolution layer in each upsampling layer may be 2×2. Optionally, the size of the deconvolution kernel of each deconvolution layer may be different.

[0087] Taking the upsampling module including four cascaded upsampling layers, the convolution kernel size of the convolution layer in each upsampling layer is 3×3, the size of the deconvolution kernel of the deconvolution layer of each upsampling layer is 2×2, and the activation function of each activation layer is a linear rectification function as an example, the size of the feature map obtained by the fusion process, that is, the feature map input by the upsampling module is recorded as 32C×H / 16×W / 16, and the processing process of the upsampling module includes:

[0088] The feature map obtained by the upsampling module is used as the input of the first upsampling layer. After the first upsampling layer performs convolution, activation and deconvolution on it, it outputs a feature map with a size of 8C×H / 8×W / 8; the feature map output by the first upsampling layer is input to the second upsampling layer for convolution, activation and deconvolution, and the output is a feature map with a size of 4C×H / 4×W / 4; the feature map output by the second upsampling layer is input to the third upsampling layer for convolution, activation and deconvolution, and the output is a feature map with a size of 2C×H / 2×W / 2; the feature map output by the third upsampling layer is input to the fourth upsampling layer for convolution, activation and deconvolution, and the output is a feature map with a size of C×H×W. The feature map with a size of C×H×W is the target repaired image.

[0089] In an embodiment of the present application, the feature map obtained after fusion processing is upsampled through an upsampling module so that the image size can be restored, and the output of the last upsampling layer is the target repaired image, so that the target repaired image and the target defective image are the same size and are clearer and more complete.

[0090] In one embodiment, Figure 7 As shown, it shows a structural schematic diagram of an image restoration model provided by an embodiment of the present application. The image restoration model also includes a preprocessing module, which includes a cascaded second slicing layer, a second connection layer, and a third feature extraction layer, and the third feature extraction layer is composed of a convolution layer and an activation layer. The preprocessing module is used to perform pre-feature extraction processing on the target defective image, so that the downsampling module and the feature extraction module respectively perform feature extraction processing on the feature map output by the preprocessing module.

[0091] Among them, the image restoration model also includes a preprocessing module, which takes the target defective image as the input of the preprocessing module. The preprocessing module performs slice processing on the target defective image through the second slice layer to obtain multiple first intermediate feature maps, and then performs splicing processing on the multiple first intermediate feature maps through the second connection layer to obtain the second intermediate feature map, and then performs feature extraction processing on the second intermediate feature map through the third feature extraction layer. Therefore, the downsampling module and the feature extraction module mainly perform further feature extraction processing on the feature map output by the preprocessing module. That is, the preprocessing module can realize a downsampling process of the target defective image through the splicing processing of the slice processing, wherein the slice and splicing processing, compared with the traditional direct downsampling through convolution processing and pooling processing, does not cause loss of spatial features in the target defective image, and retains the spatial feature information of the image.

[0092] Optionally, the size of the convolution kernel of the convolution layer in the third feature extraction layer may be 3×3. Optionally, the activation layer in the third feature extraction layer may use an activation function to implement activation processing.

[0093] Taking the convolution kernel size of the convolution layer in the third feature extraction layer as 3×3 and the target defective image size as C×H×W as an example, the processing process of the preprocessing module includes:

[0094] The target defective image is sliced ​​through the second slicing layer to obtain four first intermediate feature maps of size C×H / 2×W / 2, and the four first intermediate feature maps are input into the second connection layer for splicing to obtain the second intermediate feature map of size 4C×H / 2×W / 2 output by the second connection layer, and the second intermediate feature map is input into the third feature extraction layer for convolution processing and then activation processing to obtain a feature map of size 4C×H / 2×W / 2, i.e., the output of the preprocessing module.

[0095] The preprocessing module can realize a downsampling process of the target defective image through the splicing processing of the slice processing. Compared with the traditional downsampling directly achieved through convolution processing and pooling processing, the slice and splicing processing does not cause loss of spatial features in the target defective image and fully retains the spatial feature information of the image.

[0096] In one embodiment, Figure 8 As shown, it shows a flow chart of an image restoration model training process provided by an embodiment of the present application. The image restoration model training process includes:

[0097] Step 801: Acquire a training image set, where the training image set includes a plurality of simulated defect images.

[0098] The training image set is mainly used to train the initial image restoration model, thereby obtaining the image restoration model. The acquisition process of the simulated defective image includes: first acquiring a normal image; and using a random mask to generate a mask image; and then multiplying the normal image with the mask image to obtain the simulated defective image.

[0099] The normal image refers to a clear and complete image that does not require image restoration. The mask image is generated based on a random mask, and the size of the mask image is the same as that of the normal image. By multiplying the mask image with the normal image, the random mask can simulate the defective area in the normal image to obtain a simulated defective image, so as to continuously train the initial image restoration model based on the simulated defective image and the normal image corresponding to the simulated defective image to obtain a trained image restoration model.

[0100] Step 802: Based on the training image set, the initial image restoration model is trained using the Focal loss function and the Adam gradient descent algorithm to obtain the image restoration model.

[0101] Among them, since the size of the defective area and the size of the complete area in the defective image to be repaired are different, the areas occupied by the two are unbalanced. The complete area in the defective image to be repaired is recorded as a positive sample, and the defective area in the defective image to be repaired is recorded as a negative sample. The Focal loss function can be used to solve the problem of imbalance between positive and negative samples, thereby improving the efficiency of model iteration and the quality of model repair results.

[0102] Specifically, the Focal loss function is as follows:

[0103]

[0104] Among them, FL is the loss function, and α and γ are adjustment parameters. Among them, the value range of α is between 0 and 1, and the value of α is optional. It is 0.9, which can increase the weight of negative samples and reduce the weight of positive samples. The value range of γ is greater than 0, and the value of γ is optional. p represents the probability of whether the final calculated area is a normal area. If p is larger, it means that the sample is easier to repair, and the corresponding loss function is smaller, which reduces the weight of easy-to-repair samples. If p is smaller, it means that the sample is more difficult to repair, and the corresponding loss function is larger, which increases the weight of defective samples to achieve a positive and negative balance.

[0105] In addition, Adam's gradient descent algorithm can be used during the initial image restoration model training process to find the direction of the loss function descent as quickly as possible, thereby accelerating the convergence of the model and reducing training time.

[0106] Among them, Adam refers to the Adam optimizer. The processing process of the Adam optimizer includes:

[0107] 1. Calculate the gradient of the objective function f with respect to the current parameters, where the objective function is the Focal loss function:

[0108] in, Indicates m t The partial derivative of the function is obtained to get the gradient g t ; After multiple trainings, multiple historical gradients are obtained, namely g1, g2, ..., g t .

[0109] 2. Calculate the first-order momentum and second-order momentum based on the historical gradient:

[0110] First-order momentum m t =Φ(g1, g2, ..., gt ); second-order momentum V t =Φ(g1, g2, ..., g t ), where Φ(g1, g2, ..., g t ) means including g1,g2,...,g t Function of parameter, g t Represents the gradient calculated when the model parameters are updated for the tth time.

[0111] 3. Calculate the descent gradient at the current moment:

[0112] Among them, h t represents the descent gradient at the current moment; α is the learning rate;

[0113] 4. Update parameters according to the descending gradient: w t+1 =w t -h t , where w t+1 represents the parameters of the model after gradient descent, w t are the parameters of the model before gradient descent.

[0114] Optionally, the values ​​of two hyperparameters β1 and β2 in the Adam optimization process may be 0.9 and 0.999, respectively.

[0115] In one embodiment, Fig. 9 As shown, it shows a structural schematic diagram of another image restoration model provided by an embodiment of the present application. Among them, block is a slicing process, concat is a connection process, Conv is a convolution process, relu is an activation function, maxpool is a maximum pooling process, and up-conv is a deconvolution process. Specifically, normal images are collected and missing images are created using random masks. A training set is formed based on multiple created missing images, and the image restoration model is trained using the training set. Fig. 9 As shown, the image restoration process includes:

[0116] Input a sample image X of size C×H×W, divide the sample image into four feature maps of size C×H / 2×W / 2 through the block layer, and then splice the slices through the concat layer to obtain a feature map of size 4C×H / 2×W / 2. Among them, the process of slicing and splicing realizes a process of downsampling the feature image. This process does not cause loss of the spatial features of the image through the slicing and splicing of the image, which has great advantages over downsampling directly through convolution and pooling. Then, the spliced ​​feature map is convoluted and activated to obtain a feature map of size 4C×H / 2×W / 2.

[0117] Next, it enters two branches, one branch performs normal downsampling, and the other branch is the attention mechanism branch. Finally, the feature maps output by the two branches are spliced, and then a series of upsampling processes are performed to obtain the final repaired image. Specifically, the downsampling branch includes three downsampling layers, each of which performs convolution, activation, and maximum pooling on the input image. After downsampling through three downsampling layers, a feature map of size 16C×H / 16×W / 16 is obtained. The attention mechanism branch is a weighted downsampling branch, thereby adding spatial feature information lost in the feature image to the downsampling branch. Specifically, in the attention mechanism branch, the input feature map is first subjected to two layers of convolution feature extraction to increase the nonlinearity of spatial features, thereby improving the network's expressive power. Then, the extracted feature map containing spatial feature information is sliced ​​(sliced ​​by width and height into four sliced ​​feature maps with the same number of channels, length and width) and spliced ​​(the four sliced ​​feature maps are superimposed in the dimension of the number of channels, so that the number of channels of the spliced ​​feature image becomes four times the original, and the length and width remain unchanged) to achieve the purpose of downsampling, and the feature map output by the attention mechanism branch is obtained with a size of 16C×H / 16×W / 16. Among them, the block layer can make the spatial features of the complete area in the image help repair the spatial features of the area to be repaired. The attention mechanism branch only uses convolution and matrix splicing and slicing, which has minimal loss of spatial features.

[0118] By splicing the feature map output by the attention mechanism branch with the feature map output by the downsampling branch, a feature map of size 32C×H / 16×W / 16 with rich spatial feature information is obtained. The splicing of the downsampling branch and the attention branch can well repair the defects of damaged spatial features in the process of image downsampling, thereby improving the effect of image restoration. The pooling layer in the downsampling branch will discard a large number of spatial features. Therefore, using the attention mechanism branch for feature weighting through splicing processing will have an obvious image restoration effect.

[0119] Finally, the model includes four upsampling layers, each of which includes a convolution layer, an activation layer and a deconvolution layer. The feature map obtained by splicing is upsampled through four cascaded upsampling layers to obtain a repaired image of size C×H×W.

[0120] In addition, the Focal loss function is used in the training process of the image restoration model to solve the problem that the defective area in the image to be repaired is too small, resulting in an imbalance between positive samples (complete areas in the image) and negative samples (defective areas in the image), thereby improving the accuracy of the model. At the same time, the Adam gradient descent algorithm is used in the training process to speed up the convergence of the production network and reduce the training time.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides an image restoration device for implementing the above-mentioned image restoration method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more of the image restoration device embodiments provided below can refer to the limitations on the image restoration method above, and will not be repeated here.

[0123] In one embodiment, Fig.10 As shown, an image restoration device is provided, the image restoration device 1000 comprises: an acquisition module 1001 and an extraction module 1002, wherein:

[0124] An acquisition module 1001 is used to acquire a target defective image;

[0125] The extraction module 1002 is used to input the target defective image into a pre-trained image restoration model, perform feature extraction processing on the target defective image through the downsampling module and the feature extraction module in the image restoration model respectively, fuse the feature maps output by the downsampling module and the feature extraction module, and upsample the fused feature maps through the upsampling module in the image restoration model to output the target restoration image; wherein the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used for spatial feature extraction, the slicing layer is used for slicing the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used for connecting and outputting the multiple second feature maps.

[0126] In one embodiment, the feature extraction module includes a plurality of cascaded first feature extraction layers, each of which is composed of a convolution layer and an activation layer, and the convolution kernel sizes of the convolution layers in each of the first feature extraction layers are different.

[0127] In one embodiment, the extraction module 1002 includes a slicing processing submodule, and the slicing processing submodule is used to: perform a first slicing process on the first feature map based on a size parameter of the first feature map to obtain the multiple second feature maps.

[0128] In one embodiment, the extraction module 1002 includes a connection processing submodule, and the connection processing submodule is used to: perform connection processing on each second feature map based on the channel parameters of each second feature map and output the result.

[0129] In one embodiment, the downsampling module includes a plurality of cascaded second feature extraction layers, and each second feature extraction layer is composed of a convolution layer, an activation layer and a pooling layer cascaded in sequence; the extraction module 1002 includes a downsampling processing submodule, and the downsampling processing submodule is used to: perform convolution processing on the input feature map through the convolution layer; perform activation processing on the feature map obtained by the convolution processing based on the activation function through the activation layer; perform maximum pooling processing on the feature map obtained by the activation processing through the pooling layer to obtain an output feature map.

[0130] In one embodiment, the upsampling module includes a plurality of cascaded upsampling layers, each of which is composed of a convolution layer, an activation layer and a deconvolution layer cascaded in sequence; the extraction module 1002 includes an upsampling processing submodule, and the upsampling processing submodule is used to: perform convolution processing on the input feature map through the convolution layer; perform activation processing on the feature map obtained by the convolution processing based on the activation function through the activation layer; perform deconvolution processing on the feature map obtained by the activation processing through the deconvolution layer to obtain an output image.

[0131] In one embodiment, the image restoration model also includes a preprocessing module, which includes a cascaded second slicing layer, a second connection layer and a third feature extraction layer, and the third feature extraction layer is composed of a convolution layer and an activation layer; the preprocessing module is used to perform pre-feature extraction processing on the target defective image, so that the downsampling module and the feature extraction module respectively perform feature extraction processing on the feature map output by the preprocessing module.

[0132] In one embodiment, the image restoration device also includes a training module, which is used to: obtain a training image set, which includes multiple simulated defective images; based on the training image set, use the Focal loss function and the Adam gradient descent algorithm to train the initial image restoration model to obtain the image restoration model.

[0133] In one embodiment, the image restoration device further includes a generation module, which is used to: obtain a normal image; generate a mask image using a random mask; and multiply the normal image by the mask image to obtain the simulated defective image.

[0134] Each module in the above-mentioned image restoration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0135] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.11 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image restoration data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image restoration method is implemented.

[0136] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0138] Acquire a target defective image; input the target defective image into a pre-trained image restoration model, perform feature extraction on the target defective image through a downsampling module and a feature extraction module in the image restoration model, fuse the feature maps output by the downsampling module and the feature extraction module, and upsample the fused feature maps through an upsampling module in the image restoration model to output a target restoration image; wherein the feature extraction module comprises a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used for performing spatial feature extraction, the slicing layer is used for performing slicing on the first feature map output by the first feature extraction layer to obtain a plurality of second feature maps, and the first connection layer is used for performing connection processing on the plurality of second feature maps and outputting them.

[0139] In one embodiment, the feature extraction module includes a plurality of cascaded first feature extraction layers, each of which is composed of a convolution layer and an activation layer, and the convolution kernel sizes of the convolution layers in each of the first feature extraction layers are different.

[0140] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0141] Based on the size parameter of the first feature map, a first slicing process is performed on the first feature map to obtain the plurality of second feature maps.

[0142] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0143] Based on the channel parameters of each second feature map, each second feature map is connected and output.

[0144] In one embodiment, the downsampling module includes a plurality of cascaded second feature extraction layers, and each second feature extraction layer is composed of a convolution layer, an activation layer, and a pooling layer cascaded in sequence; when the processor executes the computer program, the following steps are also implemented: performing convolution processing on the input feature map through the convolution layer; performing activation processing on the feature map obtained by the convolution processing based on the activation function through the activation layer; performing maximum pooling processing on the feature map obtained by the activation processing through the pooling layer to obtain an output feature map.

[0145] In one embodiment, the upsampling module includes a plurality of cascaded upsampling layers, each of which is composed of a convolution layer, an activation layer, and a deconvolution layer cascaded in sequence; in one embodiment, the processor further implements the following steps when executing the computer program: performing convolution processing on the input feature map through the convolution layer; performing activation processing on the feature map obtained by the convolution processing based on the activation function through the activation layer; performing deconvolution processing on the feature map obtained by the activation processing through the deconvolution layer to obtain an output image.

[0146] In one embodiment, the image restoration model also includes a preprocessing module, which includes a cascaded second slicing layer, a second connection layer and a third feature extraction layer, and the third feature extraction layer is composed of a convolution layer and an activation layer; the preprocessing module is used to perform pre-feature extraction processing on the target defective image, so that the downsampling module and the feature extraction module respectively perform feature extraction processing on the feature map output by the preprocessing module.

[0147] In one embodiment, in one embodiment, the processor further implements the following steps when executing the computer program: obtaining a training image set, the training image set comprising a plurality of simulated defective images; based on the training image set, training the initial image restoration model using the Focal loss function and the Adam gradient descent algorithm to obtain the image restoration model.

[0148] In one embodiment, in one embodiment, when the processor executes the computer program, the following steps are further implemented: acquiring a normal image; generating a mask image using a random mask; and multiplying the normal image by the mask image to obtain the simulated defective image.

[0149] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0150] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0151] It should be noted that the method and device determined in the embodiments of the present application can be used in the image restoration processing scenario in the financial field, and can also be used for image restoration processing in any field other than the financial field. The application field of the method and device determined in the embodiments of the present application is not limited.

[0152] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0153] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image restoration method, characterized in that: The method comprises: Acquire a target defect image; Input the target defective image into a pre-trained image restoration model, perform feature extraction processing on the target defective image through a downsampling module and a feature extraction module in the image restoration model, fuse the feature maps output by the downsampling module and the feature extraction module, and perform upsampling processing on the fused feature maps through an upsampling module in the image restoration model to output a target restoration image; Among them, the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used to perform spatial feature extraction, the first slicing layer is used to slice the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the first connection layer is used to connect and output the multiple second feature maps; the processing process of the first slicing layer includes: based on the size parameter of the first feature map, performing a first slicing process on the first feature map to obtain the multiple second feature maps; the size parameter includes the length and width of the first feature map; the slicing process of the first feature map output by the first feature extraction layer to obtain multiple second feature maps includes: slicing the first feature map according to the length and width of the first feature map to obtain multiple second feature maps, wherein the number of channels of each second feature map is the same as the number of channels of the first feature map, and the length and width of each second feature map are different from the length and width of the first feature map.

2. The method according to claim 1, characterized in that The feature extraction module includes a plurality of cascaded first feature extraction layers, each of which is composed of a convolution layer and an activation layer, and the convolution kernel sizes of the convolution layers in each of the first feature extraction layers are different.

3. The method according to claim 1, characterized in that: The processing process of the first connection layer includes: Based on the channel parameters of each of the second feature maps, each of the second feature maps is connected and output.

4. The method according to any one of claims 1 to 3, characterized in that: The downsampling module includes a plurality of cascaded second feature extraction layers, and each of the second feature extraction layers is composed of a convolution layer, an activation layer, and a pooling layer cascaded in sequence; the processing process of the second feature extraction layer includes: The input feature map is convolved through the convolution layer; The feature map obtained by convolution processing is activated by the activation layer based on the activation function; The feature map obtained by the activation processing is subjected to maximum pooling processing through the pooling layer to obtain the output feature map.

5. The method according to any one of claims 1 to 3, characterized in that: The upsampling module includes a plurality of cascaded upsampling layers, each of which is composed of a convolutional layer, an activation layer, and a deconvolutional layer cascaded in sequence; the processing process of the upsampling layer includes: The input feature map is convolved through the convolution layer; The feature map obtained by convolution processing is activated by the activation layer based on the activation function; The feature map obtained by the activation processing is deconvolved through the deconvolution layer to obtain the output image.

6. The method according to any one of claims 1 to 3, characterized in that: The image restoration model further includes a preprocessing module, which includes a cascaded second slicing layer, a second connection layer, and a third feature extraction layer, wherein the third feature extraction layer is composed of a convolution layer and an activation layer; The preprocessing module is used to perform pre-feature extraction processing on the target defective image, so that the downsampling module and the feature extraction module respectively perform feature extraction processing on the feature map output by the preprocessing module.

7. The method according to claim 1, characterized in that The training process of the image restoration model includes: Acquire a training image set, wherein the training image set includes a plurality of simulated defect images; Based on the training image set, the initial image restoration model is trained using the Focal loss function and the Adam gradient descent algorithm to obtain the image restoration model.

8. The method according to claim 7, characterized in that The process of acquiring the simulated defective image includes: Get a normal image; Generate a mask image using a random mask; The normal image is multiplied by the mask image to obtain the simulated defect image.

9. An image restoration device, characterized in that: The device comprises: An acquisition module, used for acquiring a target defective image; The extraction module is used to input the target defective image into a pre-trained image restoration model, perform feature extraction processing on the target defective image through the down-sampling module and the feature extraction module in the image restoration model respectively, fuse the feature maps output by the down-sampling module and the feature extraction module, and up-sample the fused feature maps through the up-sampling module in the image restoration model to output the target restoration image; wherein the feature extraction module includes a cascaded first feature extraction layer, a first slicing layer and a first connection layer, the first feature extraction layer is used to perform spatial feature extraction, the first slicing layer is used to slice the first feature map output by the first feature extraction layer to obtain multiple second feature maps, and the The first connection layer is used to connect and output the multiple second feature maps; the extraction module includes a slicing processing submodule, and the slicing processing submodule is used to: based on the size parameter of the first feature map, perform a first slicing process on the first feature map to obtain the multiple second feature maps; the size parameter includes the length and width of the first feature map; the slicing process on the first feature map output by the first feature extraction layer to obtain multiple second feature maps includes: slicing the first feature map according to the length and width of the first feature map to obtain multiple second feature maps, wherein the number of channels of each of the second feature maps is the same as the number of channels of the first feature map, and the length and width of each of the second feature maps are different from the length and width of the first feature map.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images