An image inpainting method, device and equipment

Abstract

The present application discloses an image restoration method, apparatus and device. The to-be-restored image is obtained and input into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the to-be-restored image to obtain a normalized feature map corresponding to the to-be-restored image. Furthermore, the normalized feature map corresponding to the to-be-restored image is input into an image restoration model, so that the image restoration model restores the normalized feature map corresponding to the to-be-restored image to obtain the restored image output by the image restoration model. Among them, both the spatial normalization model and the image restoration model are models after training. The spatial normalization model is only used to map the to-be-restored image into a normalized space, and the image restoration model is only used to perform image restoration on the normalized feature map in the normalized space. In this way, by means of the diversity decoupling of the spatial normalization model and the image restoration model, the difficulty of image restoration can be greatly reduced and the efficiency of image restoration can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular, to an image restoration method, apparatus, and device. Background Art

[0002] The main purpose of image restoration is to repair the damaged parts in an image. Currently, the existing image restoration method is to directly input a defective image into an image restoration model to obtain the restored image output by the image restoration model. This image restoration method has high difficulty and low efficiency.

[0003] Therefore, how to provide an image restoration method with lower difficulty and higher efficiency is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image restoration method, apparatus, and device, which can reduce the difficulty of image restoration and improve the efficiency of image restoration.

[0005] To solve the above problems, the technical solutions provided by the embodiments of this application are as follows:

[0006] In a first aspect, an embodiment of this application provides an image restoration method, and the method includes:

[0007] Obtain an image to be restored;

[0008] Input the image to be restored into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be restored, and obtain a normalized feature map corresponding to the image to be restored;

[0009] Input the normalized feature map corresponding to the image to be restored into an image restoration model, so that the image restoration model repairs the normalized feature map corresponding to the image to be restored, and obtain the restored image output by the image restoration model;

[0010] Wherein, both the spatial normalization model and the image restoration model are models after training.

[0011] In a second aspect, an embodiment of this application provides an image restoration apparatus, and the apparatus includes:

[0012] A first acquisition module, configured to obtain an image to be restored;

[0013] A second acquisition module, configured to input the image to be restored into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be restored, and obtain a normalized feature map corresponding to the image to be restored;

[0014] A third acquisition module, configured to input the normalized feature map corresponding to the image to be repaired into an image repair model, so that the image repair model repairs the normalized feature map corresponding to the image to be repaired, and obtains the repaired image output by the image repair model;

[0015] Wherein, both the spatial normalization model and the image repair model are models after training.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0017] One or more processors;

[0018] A storage device storing one or more programs thereon,

[0019] When the one or more programs are executed by the one or more processors, the one or more processors implement the image repair method as described in any of the above embodiments.

[0020] In a fourth aspect, an embodiment of the present application provides a computer-readable medium, having a computer program stored thereon, wherein when the program is executed by a processor, the image repair method as described in any of the above embodiments is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product includes computer programs / instructions, and when the computer programs / instructions are executed by a processor, the image repair method as described in any of the above embodiments is implemented.

[0022] Thus, the embodiments of the present application have the following beneficial effects:

[0023] The embodiments of the present application provide an image repair method, apparatus and device. First, an image to be repaired is obtained, and the image to be repaired is input into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired, and obtains a normalized feature map corresponding to the image to be repaired. Furthermore, the normalized feature map corresponding to the image to be repaired is input into an image repair model, so that the image repair model repairs the normalized feature map corresponding to the image to be repaired, and obtains the repaired image output by the image repair model. Wherein, both the spatial normalization model and the image repair model are models after training. In the embodiments of the present application, the spatial normalization model is only used to map the image to be repaired into a normalized space. The input of the image repair model is the normalized feature map in a constant normalized space, that is, the image repair model is only used to perform image repair on the normalized feature map in the normalized space. In this way, by means of the diversity decoupling of the spatial normalization model and the image repair model, the difficulty of image repair can be greatly reduced, and the efficiency and effect of image repair can be effectively improved. Description of the Drawings

[0024] Figure 1 A schematic framework diagram of an exemplary application scenario provided by an embodiment of the present application;

[0025] Figure 2 A flowchart of a method for training a spatial normalization model provided by an embodiment of the present application;

[0026] Figure 3 A schematic structural diagram of a spatial normalization model provided by an embodiment of the present application;

[0027] Figure 4 A flowchart of a method for training an image restoration model provided by an embodiment of the present application;

[0028] Figure 5 A schematic structural diagram of an image restoration model provided by an embodiment of the present application;

[0029] Figure 6 A flowchart of an image restoration method provided by an embodiment of the present application;

[0030] Figure 7 A schematic structural diagram of an image restoration device provided by an embodiment of the present application;

[0031] Figure 8 A schematic diagram of the basic structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0032] To make the above objects, features, and advantages of the present application more obvious and understandable, the following further details the embodiments of the present application in conjunction with the accompanying drawings and specific implementation manners.

[0033] To facilitate the understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described first below.

[0034] The main purpose of image restoration is to repair the damaged parts in the image. After studying the traditional image restoration methods, the inventors found that the existing image restoration methods directly input the defective image into the image restoration model to obtain the restored image output by the image restoration model. However, this image restoration method is of high difficulty and low efficiency. For example, when the restored image output by the image restoration model has distortion such as ripples and twists, it is difficult to determine the specific problem, resulting in high difficulty and low efficiency of image restoration.

[0035] Therefore, how to provide an image restoration method with lower difficulty and higher efficiency is an urgent problem to be solved.

[0036] Based on this, the embodiments of the present application provide an image restoration method, apparatus, and device. First, an image to be restored is obtained, and the image to be restored is input into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be restored, and a normalized feature map corresponding to the image to be restored is obtained. Furthermore, the normalized feature map corresponding to the image to be restored is input into an image restoration model, so that the image restoration model restores the normalized feature map corresponding to the image to be restored, and a restored image output by the image restoration model is obtained. Among them, both the spatial normalization model and the image restoration model are models after training is completed. In the embodiments of the present application, the spatial normalization model is only used to map the image to be restored into a normalized space. The input of the image restoration model is the normalized feature map in a constant normalized space, that is, the image restoration model is only used to perform image restoration on the normalized feature map in the normalized space. In this way, by means of the diversity decoupling of the spatial normalization model and the image restoration model, the difficulty of image restoration can be greatly reduced, and the efficiency and effect of image restoration can be effectively improved.

[0037] To facilitate understanding of the image restoration method provided by the embodiments of the present application, the following Figure 1 is described with reference to the Figure 1 shown scenario example. As shown in

[0038] In practical applications, first, an image to be restored is obtained, and the image to be restored is a defective image. The image to be restored is input into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be restored, maps the image to be restored into a normalized space, and obtains a normalized feature map corresponding to the image to be restored. Furthermore, the obtained normalized feature map corresponding to the image to be restored is input into an image restoration model, so that the image restoration model restores the normalized feature map corresponding to the image to be restored, and a restored image output by the image restoration model is obtained.

[0039] Among them, both the spatial normalization model and the image restoration model are models after pre-training is completed.

[0040] To facilitate understanding of the present application, the following first describes a method for training a spatial normalization model provided by the embodiments of the present application with reference to the accompanying drawings.

[0041] As shown in Figure 2 shown, this figure is a flowchart of a method for training a spatial normalization model provided by the embodiments of the present application. As shown in Figure 3 shown, this figure is a schematic structural diagram of a spatial normalization model provided by the embodiments of the present application. As shown in Figure 3 shown, the spatial normalization model may include a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhancement feature extraction modules. Among them,Figure 3 The "M" module in

[0042] Combined with Figure 3 the spatial normalization model structure shown in Figure 2 as shown in

[0043] S201: Input at least one defective image into the residual module to obtain the original feature map corresponding to each defective image respectively.

[0044] First, obtain at least one original image. After defect processing on each original image, a defective image is obtained. When training the spatial normalization model, the defective image is used as the input image. As an optional example, the defect processing can be masking processing.

[0045] In one or more embodiments, when there are multiple defective images, the types of the multiple defective images are at least two. The type of the defective image is determined by the type of the corresponding original image. For example, multiple face images of the same person are of the same type and have similarity. Face images of different people are of different types and do not have similarity.

[0046] After obtaining the defective image, input the defective image into the residual module of the spatial normalization model so that the residual module extracts features from the defective image to obtain the original feature map corresponding to each defective image respectively. Furthermore, further image processing will be performed based on the original feature map later.

[0047] As an optional example, the residual module is a Resnet18 network. It can be understood that the original feature map obtained based on the Resnet18 network is a 4-fold downsampled feature map.

[0048] S202: Input the (i - 1)th target fusion feature map into the ith significant feature extraction module to obtain the ith target significant feature map; the value of i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)th target fusion feature map is the target original feature map, the target original feature map is the original feature map corresponding to the target defective image, and the target defective images are each of the at least one defective image.

[0049] For the convenience of description, each defective image in the at least one defective image is represented as a target defective image. After obtaining the original feature map corresponding to each defective image respectively, the target defective image corresponds to a target original feature map. The following will be described by taking the target original feature map as an example.

[0050] The significant feature extraction module can form a kind of attention-like mechanism for extracting the significant features of the feature map. The number of significant feature extraction modules is N. When the value of i is 1, the first significant feature extraction module is used to extract the significant features in the (i - 1)-th target fusion feature map. Among them, the (i - 1)-th target fusion feature map is the target original feature map. That is, the target original feature map is input into the first significant feature extraction module to obtain the first target significant feature map. When the value of i is not 1, the (i - 1)-th target fusion feature map is obtained through S204.

[0051] As an optional example, the significant feature extraction module consists of a 1×1 convolutional neural network layer, two fully connected layers, and an activation function layer (such as the sigmoid function).

[0052] S203: Input the (i - 1)-th target fusion feature map and the i-th target significant feature map into the i-th enhanced feature extraction module to obtain the i-th target enhanced feature map, and input the (i - 1)-th target fusion feature map and the i-th target significant feature map into the i-th difference feature extraction module to obtain the i-th target difference feature map.

[0053] Considering the further enhancement of the significant features in the feature map, as an optional example, the enhanced feature extraction module is used to perform an addition operation on the (i - 1)-th target fusion feature map and the i-th target significant feature map (that is, add the pixel values at the corresponding positions in the two feature maps). In this way, the enhanced feature extraction module can be used to further enhance the significant features on the feature map and strengthen the significant features in the feature map.

[0054] For example, when the value of i is 1, when obtaining the first target significant feature map, input the target original feature map and the obtained first target significant feature map into the first enhanced feature extraction module (such as Figure 3 the “+” module shown), and obtain the first target enhanced feature map output by the first enhanced feature extraction module.

[0055] Considering that the significant features in the feature map are easily captured by the model, while the subtly differentiated features are easily masked by the significant features. Therefore, as an optional example, the difference feature extraction module is used to perform a subtraction operation between the (i - 1)-th target fusion feature map and the i-th target significant feature map (that is, subtract the pixel values at the corresponding positions in the two feature maps). In this way, the difference feature extraction module can be used to extract the difference features between the input feature maps, and then more discriminative features can be captured from the difference features.

[0056] For example, when the value of i is 1, when obtaining the first target significant feature map, input the target original feature map and the obtained first target significant feature map into the first difference feature extraction module (such asFigure 3 Obtain the first target difference feature map output by the first difference feature extraction module from the "-" module shown.

[0057] S204: Concatenate the i-th target enhanced feature map and the i-th target difference feature map to obtain the i-th target fusion feature map. Repeat the steps of inputting the (i - 1)-th target fusion feature map into the i-th salient feature extraction module to obtain the i-th target salient feature map and subsequent steps until the value of i is N.

[0058] After obtaining the i-th target enhanced feature map and the i-th target difference feature map, concatenate the i-th target enhanced feature map and the i-th target difference feature map to obtain the i-th target fusion feature map. Furthermore, repeat S202 - S204 until the value of i is N.

[0059] For example, when the value of i is 1, the first target fusion feature map can be obtained. Furthermore, i = i + 1, and the value of i is 2. After obtaining the first target fusion feature map, in S202, input the first target fusion feature map into the second salient feature extraction module to obtain the second target salient feature map. Further, input the first target fusion feature map and the second target salient feature map into the second enhanced feature extraction module to obtain the second target enhanced feature map, and input the first target fusion feature map and the second target salient feature map into the second difference feature extraction module to obtain the second target difference feature map. Concatenate the second target enhanced feature map and the second target difference feature map to obtain the second target fusion feature map. In this way, repeat S202 - S204 until the value of i is N, and N target enhanced feature maps, N target difference feature maps, and N target fusion feature maps can be obtained.

[0060] Each feature fusion module includes a salient feature extraction module, a difference feature extraction module, and an enhanced feature extraction module, which can form a feature fusion module. Each feature fusion module is used to perform further feature extraction on the feature map. Then, N salient feature extraction modules, N difference feature extraction modules, and N enhanced feature extraction modules can form N cascaded feature fusion modules.

[0061] S205: Input the N-th target fusion feature map into the (N + 1)-th salient feature extraction module to obtain the (N + 1)-th target salient feature map, and input the N-th target fusion feature map and the (N + 1)-th target salient feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th target difference feature map.

[0062] The spatial normalization model includes the (N + 1)-th salient feature extraction module and the (N + 1)-th difference feature extraction module. As Figure 3As shown, after obtaining the Nth target fusion feature map, the Nth target fusion feature map is input into the (N + 1)th significant feature extraction module to obtain the (N + 1)th target significant feature map. Furthermore, the Nth target fusion feature map and the (N + 1)th target significant feature map are input into the (N + 1)th difference feature extraction module to obtain the (N + 1)th target difference feature map.

[0063] Among them, the (N + 1)th target difference feature map can be regarded as the finally obtained feature map.

[0064] S206: Combine the N target enhanced feature maps and the (N + 1)th target difference feature map to form a target feature map set corresponding to the target defect image.

[0065] After obtaining the (N + 1)th target difference feature map, the N target enhanced feature maps obtained through N enhanced feature extraction modules and the (N + 1)th target difference feature map are determined as the target feature map set corresponding to the target defect image.

[0066] In a possible implementation manner, after S205, a predicted normalized feature map output by the spatial normalization model can also be obtained, specifically including:

[0067] The N target enhanced feature maps and the (N + 1)th target difference feature map are spliced to obtain a target predicted normalized feature map corresponding to the target defect image.

[0068] During specific implementation, as Figure 3 shown, the spatial normalization model further includes a connection module. The N target enhanced feature maps and the (N + 1)th target difference feature map are input into the connection module, so that the connection module splices the N target enhanced feature maps and the (N + 1)th target difference feature map to obtain a target predicted normalized feature map corresponding to the target defect image.

[0069] The target predicted normalized feature map is the predicted normalized feature map corresponding to the target defect image output by the spatial normalization model. In this way, when there is at least one defect image input into the spatial normalization model, the predicted normalized feature maps corresponding to each defect image can be obtained.

[0070] It can be understood that due to the differences in the sizes, etc. of different defect images, before repairing the defect images, it is necessary to use the spatial normalization model to map the defect images to the same normalized space. That is, the normalization in the embodiments of the present application refers to the normalization in the feature space. For example, the dimensions of different defect images are unified, so that the dimensions of the predicted normalized feature maps corresponding to each finally obtained defect image are the same. In this way, it is convenient to uniformly input the predicted normalized feature maps of the same dimension into the subsequent image repair model.

[0071] S207: Obtain a first loss value according to the feature map sets respectively corresponding to each defect image, and train the residual module, N + 1 significant feature extraction modules, N + 1 differential feature extraction modules, and N enhanced feature extraction modules based on the first loss value until a first preset condition is reached.

[0072] After obtaining the feature map sets corresponding to each defect image, that is, the N enhanced feature maps and the (N + 1)-th differential feature map corresponding to each defect image, obtain a first loss value according to the feature map sets respectively corresponding to each defect image. And train the residual module, N + 1 significant feature extraction modules, N + 1 differential feature extraction modules, and N enhanced feature extraction modules based on the first loss value until a first preset condition is reached. When the first preset condition is not reached, repeat the execution of S201 and subsequent steps until the first preset condition is reached.

[0073] As an optional example, the first preset condition may be reaching a first preset number of training times or the first loss value reaching a first preset value. It can be understood that the embodiments of the present application do not limit the first preset number of training times and the first preset value, and can be set according to actual situations.

[0074] In a possible implementation manner, the embodiments of the present application provide a specific implementation manner for obtaining the first loss value according to the feature map sets respectively corresponding to each defect image in S207. For details, please refer to A1 - A3 below.

[0075] Based on the content of S201 - S207, a suitable spatial normalization model can be designed and the spatial normalization model can be trained. The spatial normalization model is independent of the image inpainting model, and the trained spatial normalization model can provide normalized feature maps corresponding to different images for the image inpainting model. In this way, through the decoupling method of the spatial normalization model and the image inpainting model, the difficulty of image inpainting can be greatly reduced, and the efficiency and effect of image inpainting can be effectively improved.

[0076] As an optional example, as Figure 3 shown, the spatial normalization model further includes a loss module. In one or more embodiments, an adversarial network loss module and a distance metric loss module.

[0077] Based on this, in a possible implementation manner, the embodiments of the present application provide a specific implementation manner for obtaining the first loss value according to the feature map sets respectively corresponding to each defect image in S207, including:

[0078] A1: Input the feature map sets respectively corresponding to each defect image into the adversarial network loss module to obtain an adversarial network loss value.

[0079] Among them, the adversarial network loss value can be called the GAN loss value. In practical applications, the feature map sets corresponding to each defective image are input into the adversarial network loss module, and the adversarial network loss value output by the adversarial network loss module can be obtained.

[0080] A2: Input the feature map sets corresponding to each defective image into the distance metric loss module to obtain the distance metric loss value.

[0081] When there are at least two types of defective images, it is necessary to ensure that the distance between similar images (i.e., images of the same type) in the normalized space is much smaller than the distance between dissimilar images (i.e., images of different types), that is, it is necessary to make the distance between the feature vectors of the same-class samples small and the distance between the feature vectors of different-class samples large. Based on this, the loss module in the embodiments of the present application further includes a distance metric loss module. Inputting the feature map sets corresponding to each defective image into the distance metric loss module can obtain the distance metric loss value. Using the distance metric loss value to train the spatial normalization model can achieve that the distance between similar pictures in the normalized space is much smaller than the distance between dissimilar pictures.

[0082] A3: The adversarial network loss value and the distance metric loss value form the first loss value.

[0083] After obtaining the adversarial network loss value and the distance metric loss value, the adversarial network loss value and the distance metric loss value form the first loss value. Thus, the first loss value can be used to train the residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhancement feature extraction modules in the spatial normalization model, etc.

[0084] Based on the content of A1 - A3, using the first loss value composed of the adversarial network loss value and the distance metric loss value to train the spatial normalization model can make the trained spatial normalization model meet the requirement of extracting the normalized feature map, and can ensure that the distance between the input similar images in the normalized space is much smaller than the distance between dissimilar images.

[0085] When training the spatial normalization model, it is also necessary to train the image repair model. The spatial normalization model and the image repair model are independently trained and connected in series.

[0086] To facilitate the understanding of the present application, a method for training an image repair model provided in the embodiments of the present application will be further described below with reference to the accompanying drawings.

[0087] See Figure 4 As shown, this figure is a flowchart of a method for training an image repair model provided in the embodiments of the present application. See Figure 5 As shown, this figure is a schematic structural diagram of an image repair model provided in the embodiments of the present application. As Figure 5As shown, the image inpainting model includes a first branch model and a second branch model.

[0088] Combined with Figure 5 the image inpainting model structure shown, such as Figure 4 shown, the image inpainting model training method may include S401 - S404:

[0089] S401: Input the target predicted normalized feature map into the first branch model, so that the first branch model performs detail inpainting on the target predicted normalized feature map to obtain the target predicted detail feature map.

[0090] In the embodiment of the above - mentioned spatial normalization model training method, the target predicted normalized feature map corresponding to the target defect image can be obtained. After obtaining the target predicted normalized feature map, the image inpainting model is trained using the target predicted normalized feature map.

[0091] Specifically, when implemented, input the target predicted normalized feature map into the first branch model, so that the first branch model performs detail inpainting on the target predicted normalized feature map to obtain the target predicted detail feature map. It can be understood that the first branch model is used to perform detail inpainting on the target predicted normalized feature map, and the obtained target predicted detail feature map can reflect the detail features of the defect image and is a fine - grained feature map.

[0092] As an optional example, as Figure 5 shown, the first branch model includes two gated convolutional neural networks and a context attention layer. The first gated convolutional neural network and the context attention layer are connected in series, and then another gated convolutional neural network is connected in series. The target predicted normalized feature map is first input into the first gated convolutional neural network. After obtaining the features output by the first gated convolutional neural network, the features output by the first gated convolutional neural network are input into the context attention layer. Then, the features output by the context attention layer are input into the second gated convolutional neural network to obtain the target predicted detail feature map.

[0093] In one or more embodiments, the context attention layer is composed of a 2 - layer 1*1 convolutional neural network, two fully - connected layers, and an activation function layer (such as a sigmoid activation function).

[0094] S402: Input the target predicted normalized feature map into the second branch model, so that the second branch model performs rough inpainting on the target predicted normalized feature map to obtain the target predicted contour feature map.

[0095] In addition, after obtaining the target prediction normalized feature map, the target prediction normalized feature map is simultaneously input into the second branch model, so that the second branch model performs rough repair on the target prediction normalized feature map to obtain the target prediction contour feature map. It can be understood that the second branch model can learn a dynamic feature selection mechanism for rough repair of the target prediction normalized feature map, and the obtained target prediction contour feature map can reflect the contour features of the defective image and is a coarse-grained feature map.

[0096] As an optional example, as Figure 5 shown, the second branch model includes a gated convolutional neural network and a dilated convolutional neural network. The gated convolutional neural network and the dilated convolutional neural network are connected in series. The target prediction normalized feature map is first input into the gated convolutional neural network. After obtaining the features output by the gated convolutional neural network, the features output by the gated convolutional neural network are then input into the dilated convolutional neural network to obtain the target prediction contour feature map.

[0097] S403: Concatenate the target prediction detail feature map and the target prediction contour feature map to obtain the target prediction image corresponding to the target original image.

[0098] After obtaining the target prediction detail feature map and the target prediction contour feature map, the target prediction detail feature map and the target prediction contour feature map are concatenated to obtain the target prediction image corresponding to the target original image.

[0099] The target prediction image is the image obtained after repairing the target original image. Thus, the obtained target prediction image can reflect both detail features and contour features, and can achieve a relatively high repair accuracy for the target prediction normalized feature map.

[0100] As an optional example, the image repair model further includes a connection module. Based on this, in practical applications, the target prediction detail feature map and the target prediction contour feature map are input into the connection module, so that the connection module concatenates the target prediction detail feature map and the target prediction contour feature map to obtain the target prediction image corresponding to the target original image.

[0101] S404: Obtain a second loss value based on each original image and the prediction image corresponding to each original image, and train the first branch model and the second branch model based on the second loss value until the second preset condition is reached.

[0102] After obtaining the target prediction image corresponding to the target original image, when there are at least one original images, each original image and the prediction image corresponding to each original image can be obtained. Furthermore, a second loss value is obtained based on each original image and the prediction image corresponding to each original image, and the first branch model and the second branch model are trained based on the second loss value until a second preset condition is reached. When the second preset condition is not reached, steps S401 and subsequent steps are repeatedly executed until the second preset condition is reached.

[0103] As an alternative example, the second preset condition can be reaching the second preset number of training times or the second loss value reaching the second preset value. It can be understood that the embodiments of the present application do not limit the second preset number of training times and the second preset value, which can be set according to actual situations.

[0104] Based on the content of S401-404, an image restoration model can be designed and the image restoration model can be trained. The image restoration model is independent of the spatial normalization model, and the image restoration model is connected in series with the spatial normalization model. The trained spatial normalization model combined with the trained image restoration model can be used to implement the restoration of defective images to obtain the restored images. In this way, by means of the diversity decoupling of the spatial normalization model and the image restoration model, the difficulty of image restoration can be greatly reduced, and the efficiency and effect of image restoration can be effectively improved.

[0105] As an alternative example, as Figure 5 shown, the image restoration model further includes a smoothing processing module. In one or more embodiments, the smoothing processing module is a gated convolutional neural network, such as a 6-layer gated convolutional neural network.

[0106] In a possible implementation manner, the embodiments of the present application provide a specific implementation manner for splicing the target prediction detail feature map and the target prediction contour feature map in S403 to obtain the target prediction image corresponding to the target original image, including:

[0107] B1: Splice the target prediction detail feature map and the target prediction contour feature map to obtain a target spliced feature map.

[0108] As Figure 5 shown, in practical applications, the target prediction detail feature map and the target prediction contour feature map are input into a connection module, so that the connection module splices the target prediction detail feature map and the target prediction contour feature map to obtain a target spliced feature map.

[0109] B2: Input the target spliced feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the target spliced feature map to obtain the target prediction image corresponding to the target original image.

[0110] Input the target spliced feature map into the smoothing module to obtain the target prediction image corresponding to the target original image. The smoothing module is used to perform a smoothing operation on the target spliced feature map to eliminate some burrs in the target spliced feature map, so that the repair effect of the obtained target prediction image is better.

[0111] In a possible implementation manner, the embodiments of the present application provide a specific implementation manner of obtaining a second loss value based on each original image and the prediction image corresponding to each original image in S404, and training the first branch model and the second branch model based on the second loss value until a second preset condition is reached, including:

[0112] Obtain a second loss value based on each original image and the prediction image corresponding to each original image, and train the first branch model, the second branch model, and the smoothing module based on the second loss value until a second preset condition is reached.

[0113] It can be understood that when the image repair model further includes a smoothing module, when training the image repair model based on the second loss value, it is specifically to train the first branch model, the second branch model, and the smoothing module based on the second loss value.

[0114] Based on the spatial normalization model training method and the image repair model training method provided in the above embodiments, after obtaining the trained spatial normalization model and image repair model, the embodiments of the present application further provide an image repair method.

[0115] See Figure 6 As shown, this figure is a flowchart of an image repair method provided by the embodiments of the present application. As Figure 6 shown, the method may include S601 - S603:

[0116] S601: Obtain the image to be repaired.

[0117] The image to be repaired is a defective image, and it is necessary to use the trained spatial normalization model and image repair model provided by the embodiments of the present application to repair it.

[0118] S602: Input the image to be repaired into the spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired to obtain a normalized feature map corresponding to the image to be repaired.

[0119] First, input the image to be repaired into the spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired, maps the image to be repaired to the normalized space, and obtains a normalized feature map corresponding to the image to be repaired.

[0120] In a possible implementation manner, based on the trained spatial normalization model, an embodiment of the present application provides a specific implementation manner of inputting an image to be repaired into the spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired to obtain a normalized feature map corresponding to the image to be repaired. For details, please refer to C1-C6 below.

[0121] S603: Input the normalized feature map corresponding to the image to be repaired into an image repair model, so that the image repair model repairs the normalized feature map corresponding to the image to be repaired, and obtains the repaired image output by the image repair model.

[0122] The spatial normalization model and the image repair model are used in series. After obtaining the normalized feature map corresponding to the image to be repaired, the normalized feature map corresponding to the image to be repaired is input into the image repair model. The image repair model is used to repair the normalized feature map corresponding to the image to be repaired, and finally obtains the repaired image output by the image repair model.

[0123] In a possible implementation manner, an embodiment of the present application provides a specific implementation manner of inputting the normalized feature map corresponding to the image to be repaired into an image repair model, so that the image repair model repairs the normalized feature map corresponding to the image to be repaired, and obtains the repaired image output by the image repair model. For details, please refer to D1-D3 below.

[0124] Based on the relevant content of S601-S603 above, an embodiment of the present application provides an image repair method. First, obtain an image to be repaired, and input the image to be repaired into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired to obtain a normalized feature map corresponding to the image to be repaired. Furthermore, input the normalized feature map corresponding to the image to be repaired into an image repair model, so that the image repair model repairs the normalized feature map corresponding to the image to be repaired, and obtains the repaired image output by the image repair model. Among them, both the spatial normalization model and the image repair model are models after training. In the embodiment of the present application, the spatial normalization model is only used to map the image to be repaired into a normalized space. The input of the image repair model is the normalized feature map in a constant normalized space, that is, the image repair model is only used to perform image repair on the normalized feature map in the normalized space. In this way, through the way of decoupling the spatial normalization model and the image repair model, the difficulty of image repair can be greatly reduced, and the efficiency and effect of image repair can be effectively improved.

[0125] As described in the embodiment of the above spatial normalization model training method, the spatial normalization model includes a residual module, N+1 significant feature extraction modules, N+1 difference feature extraction modules, and N enhancement feature extraction modules.

[0126] In a possible implementation manner, based on the trained spatial normalization model, an embodiment of the present application provides a specific implementation manner in S602 of inputting the image to be repaired into the spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired to obtain a normalized feature map corresponding to the image to be repaired, including:

[0127] C1: Input the image to be repaired into the residual module to obtain the original feature map corresponding to the image to be repaired.

[0128] After obtaining the image to be repaired, input the defective image into the residual module of the spatial normalization model, so that the residual module performs feature extraction on the defective image to obtain the original feature map corresponding to the image to be repaired.

[0129] C2: Input the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map; the value of i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)-th fused feature map is the original feature map corresponding to the image to be repaired.

[0130] For example, when the value of i is 1, use the first significant feature extraction module to extract the significant features in the (i - 1)-th fused feature map. Among them, the (i - 1)-th fused feature map is the original feature map corresponding to the image to be repaired. That is, input the original feature map corresponding to the image to be repaired into the first significant feature extraction module to obtain the first significant feature map. When the value of i is not 1, the (i - 1)-th fused feature map is the fused feature map obtained through C4.

[0131] C3: Input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th enhanced feature extraction module to obtain the i-th enhanced feature map, and input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th difference feature extraction module to obtain the i-th difference feature map.

[0132] For example, when the value of i is 1, when obtaining the first significant feature map, input the original feature map corresponding to the image to be repaired and the obtained first significant feature map into the first enhanced feature extraction module to obtain the first enhanced feature map output by the first enhanced feature extraction module.

[0133] In addition, when obtaining the first significant feature map, input the original feature map corresponding to the image to be repaired and the obtained first significant feature map into the first difference feature extraction module to obtain the first difference feature map output by the first difference feature extraction module.

[0134] C4: Concatenate the i-th enhanced feature map and the i-th difference feature map to obtain the i-th fused feature map. Repeat the steps of inputting the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map and subsequent steps until the value of i is N.

[0135] After obtaining the i-th enhanced feature map and the i-th difference feature map, concatenate the i-th enhanced feature map and the i-th difference feature map to obtain the i-th fused feature map. Furthermore, repeat C1 - C4 until the value of i is N.

[0136] For example, when the value of i is 1, the first fused feature map can be obtained. Furthermore, i = i + 1, and the value of i is 2. After obtaining the first fused feature map, perform the step in C2 of inputting the first fused feature map into the second significant feature extraction module to obtain the second significant feature map. Further, input the first fused feature map and the second significant feature map into the second enhanced feature extraction module to obtain the second enhanced feature map, and input the first fused feature map and the second significant feature map into the second difference feature extraction module to obtain the second difference feature map. Concatenate the second enhanced feature map and the second difference feature map to obtain the second fused feature map. In this way, repeat C2 - C4 until the value of i is N, and N enhanced feature maps, N difference feature maps, and N fused feature maps can be obtained.

[0137] C5: Input the N-th fused feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th significant feature map. Input the N-th fused feature map and the (N + 1)-th significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th difference feature map.

[0138] After obtaining the N-th fused feature map, input the N-th fused feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th significant feature map. Furthermore, input the N-th fused feature map and the (N + 1)-th significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th difference feature map.

[0139] C6: Concatenate the N enhanced feature maps and the (N + 1)-th difference feature map to obtain the normalized feature map corresponding to the image to be repaired.

[0140] In a possible implementation, the spatial normalization model further includes a connection module. Input the N enhanced feature maps and the (N + 1)-th difference feature map into the connection module, so that the connection module concatenates the N enhanced feature maps and the (N + 1)-th difference feature map to obtain the normalized feature map corresponding to the image to be repaired.

[0141] As can be seen from C1 - C6, the trained spatial normalization model can be used to map the image to be repaired into the normalized space to obtain the normalized feature map corresponding to the image to be repaired.

[0142] As described in the above embodiments, the image inpainting model includes a first branch model and a second branch model.

[0143] Based on this, in a possible implementation manner, an embodiment of the present application provides a specific implementation manner of inputting the normalized feature map corresponding to the image to be repaired into the image inpainting model in S603, so that the image inpainting model repairs the normalized feature map corresponding to the image to be repaired and obtains the repaired image output by the image inpainting model, including:

[0144] D1: Input the normalized feature map corresponding to the image to be repaired into the first branch model, so that the first branch model repairs the details of the normalized feature map corresponding to the image to be repaired to obtain the repaired detailed feature map.

[0145] D2: Input the normalized feature map corresponding to the image to be repaired into the second branch model, so that the second branch model performs rough repair on the normalized feature map corresponding to the image to be repaired to obtain the repaired contour feature map.

[0146] Input the normalized feature map corresponding to the image to be repaired into the first branch model and the second branch model respectively to obtain the repaired detailed feature map and the repaired contour feature map.

[0147] D3: Stitch the repaired detailed feature map and the repaired contour feature map together to obtain the repaired image.

[0148] It should be noted that D1 - D3 is similar to S401 - S403 in the above embodiments. For the technical details of D1 - D3, reference can be made to the above embodiments and will not be elaborated here.

[0149] When the image inpainting model further includes a smoothing processing module, in a possible implementation manner, an embodiment of the present application provides a specific implementation manner of stitching the repaired detailed feature map and the repaired contour feature map together in D3 to obtain the repaired image, including:

[0150] E1: Stitch the repaired detailed feature map and the repaired contour feature map together to obtain the stitched feature map.

[0151] E2: Input the stitched feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing operation on the stitched feature map to obtain the repaired image.

[0152] It can be understood that by using the smoothing processing module, some burrs in the spliced feature map can be eliminated, making the repair effect of the obtained repaired image better.

[0153] Based on the image repair method provided by the above method embodiment, the embodiment of the present application also provides an image repair device, which will be described below with reference to the accompanying drawings.

[0154] See Figure 7 As shown, this figure is a schematic structural diagram of an image repair device provided by an embodiment of the present application. As Figure 7 shown, the image repair device includes:

[0155] The first acquisition unit 701 is used to acquire the image to be repaired;

[0156] The second acquisition unit 702 is used to input the image to be repaired into the spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be repaired, and obtains the normalized feature map corresponding to the image to be repaired;

[0157] The third acquisition unit 703 is used to input the normalized feature map corresponding to the image to be repaired into the image repair model, so that the image repair model repairs the normalized feature map corresponding to the image to be repaired, and obtains the repaired image output by the image repair model;

[0158] Wherein, both the spatial normalization model and the image repair model are models after training.

[0159] In a possible implementation manner, the spatial normalization model includes a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhanced feature extraction modules. The device further includes a spatial normalization model training unit, and the spatial normalization model training unit includes:

[0160] The first input subunit is used to input at least one defective image into the residual module to obtain the original feature map corresponding to each defective image;

[0161] The second input subunit is used to input the (i - 1)th target fusion feature map into the ith significant feature extraction module to obtain the ith target significant feature map; the value of i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)th target fusion feature map is the target original feature map, the target original feature map is the original feature map corresponding to the target defective image, and the target defective images are each of at least one defective image;

[0162] A third input subunit, configured to input the (i-1)th target fusion feature map and the ith target saliency feature map into the ith enhanced feature extraction module to obtain the ith target enhanced feature map, and input the (i-1)th target fusion feature map and the ith target saliency feature map into the ith difference feature extraction module to obtain the ith target difference feature map;

[0163] A first splicing subunit, configured to splice the ith target enhanced feature map and the ith target difference feature map to obtain the ith target fusion feature map, and repeatedly execute the step of inputting the (i-1)th target fusion feature map into the ith saliency feature extraction module to obtain the ith target saliency feature map and subsequent steps until the value of i is N;

[0164] A fourth input subunit, configured to input the Nth target fusion feature map into the (N+1)th saliency feature extraction module to obtain the (N+1)th target saliency feature map, and input the Nth target fusion feature map and the (N+1)th target saliency feature map into the (N+1)th difference feature extraction module to obtain the (N+1)th target difference feature map;

[0165] A composition subunit, configured to compose N target enhanced feature maps and the (N+1)th target difference feature map into a target feature map set corresponding to the target defect image;

[0166] A first training subunit, configured to obtain a first loss value according to the feature map sets respectively corresponding to the respective defect images, and train the residual module, the (N+1) saliency feature extraction modules, the (N+1) difference feature extraction modules, and the N enhanced feature extraction modules based on the first loss value until a first preset condition is reached.

[0167] In a possible implementation manner, when there are multiple defect images, the types of the multiple defect images are at least two; the spatial normalization model further includes an adversarial network loss module and a distance metric loss module; the training subunit includes:

[0168] A first obtaining subunit, configured to input the feature map sets respectively corresponding to the respective defect images into the adversarial network loss module to obtain an adversarial network loss value;

[0169] A second obtaining subunit, configured to input the feature map sets respectively corresponding to the respective defect images into the distance metric loss module to obtain a distance metric loss value;

[0170] The adversarial network loss value and the distance metric loss value form the first loss value.

[0171] In a possible implementation manner, the apparatus further includes:

[0172] A splicing unit, configured to splice N target enhanced feature maps and the (N + 1)-th target difference feature map to obtain a target predicted normalized feature map corresponding to the target defect image;

[0173] The target defect image is obtained by performing defect processing on a target original image; the image restoration model includes a first branch model and a second branch model, and the apparatus further includes an image restoration model training unit, and the image restoration model training unit includes:

[0174] A third obtaining subunit, configured to input the target predicted normalized feature map into the first branch model, so that the first branch model performs detail restoration on the target predicted normalized feature map to obtain a target predicted detail feature map;

[0175] A fourth obtaining subunit, configured to input the target predicted normalized feature map into the second branch model, so that the second branch model performs rough restoration on the target predicted normalized feature map to obtain a target predicted contour feature map;

[0176] A fifth obtaining subunit, configured to splice the target predicted detail feature map and the target predicted contour feature map to obtain a target predicted image corresponding to the target original image;

[0177] A second training subunit, configured to obtain a second loss value based on each original image and the predicted image corresponding to each original image, and train the first branch model and the second branch model based on the second loss value until a second preset condition is reached.

[0178] In a possible implementation manner, the image restoration model further includes a smoothing processing module, and the fifth obtaining subunit includes:

[0179] A second splicing subunit, configured to splice the target predicted detail feature map and the target predicted contour feature map to obtain a target spliced feature map;

[0180] A sixth obtaining subunit, configured to input the target spliced feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the target spliced feature map to obtain a target predicted image corresponding to the target original image;

[0181] The second training subunit is specifically configured to:

[0182] Obtain a second loss value based on each original image and the predicted image corresponding to each original image, and train the first branch model, the second branch model, and the smoothing processing module based on the second loss value until a second preset condition is reached.

[0183] In a possible implementation, the spatial normalization model includes a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhanced feature extraction modules; the second acquisition unit 702 includes:

[0184] A fifth input subunit, configured to input the image to be repaired into the residual module to obtain an original feature map corresponding to the image to be repaired;

[0185] A sixth input subunit, configured to input the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map; i takes positive integer values from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)-th fused feature map is the original feature map corresponding to the image to be repaired;

[0186] A seventh input subunit, configured to input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th enhanced feature extraction module to obtain the i-th enhanced feature map, and input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th difference feature extraction module to obtain the i-th difference feature map;

[0187] A third splicing subunit, configured to splice the i-th enhanced feature map and the i-th difference feature map to obtain the i-th fused feature map, and repeatedly execute the step of inputting the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map and subsequent steps until the value of i is N;

[0188] An eighth input subunit, configured to input the N-th fused feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th significant feature map, and input the N-th fused feature map and the (N + 1)-th significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th difference feature map;

[0189] A fourth splicing subunit, configured to splice N enhanced feature maps and the (N + 1)-th difference feature map to obtain a normalized feature map corresponding to the image to be repaired.

[0190] In a possible implementation, the image repair model includes a first branch model and a second branch model, and the third acquisition unit 703 includes:

[0191] A first repair subunit, configured to input the normalized feature map corresponding to the image to be repaired into the first branch model, so that the first branch model performs detail repair on the normalized feature map corresponding to the image to be repaired to obtain a repaired detail feature map;

[0192] A second repair subunit, configured to input the normalized feature map corresponding to the image to be repaired into the second branch model, so that the second branch model performs rough repair on the normalized feature map corresponding to the image to be repaired, and obtains a repaired contour feature map;

[0193] A fifth splicing subunit, configured to splice the repaired detail feature map and the repaired contour feature map to obtain a repaired image.

[0194] In a possible implementation manner, the image repair model further includes a smoothing processing module, and the fifth splicing subunit includes:

[0195] A sixth splicing subunit, configured to splice the repaired detail feature map and the repaired contour feature map to obtain a spliced feature map;

[0196] A ninth input subunit, configured to input the spliced feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the spliced feature map to obtain a repaired image.

[0197] Based on the image repair method provided in the foregoing method embodiments, the present application further provides an electronic device, including: one or more processors; a storage device storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, enabling the one or more processors to implement the image repair method described in any of the foregoing embodiments.

[0198] Refer to the following Figure 8 , which shows a schematic structural diagram of an electronic device 1300 suitable for implementing the embodiments of the present application. The terminal device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices, tablet computers), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 8 The electronic device shown is only an example and should not impose any limitation on the functions and usage scope of the embodiments of the present application.

[0199] As Figure 8As shown, the electronic device 1300 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 1301, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1306 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the electronic device 1300 are also stored. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0200] Generally, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1306 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 8 the electronic device 1300 with various devices is shown, it should be understood that it is not required to implement or include all the shown devices. Instead, more or fewer devices may be implemented or included.

[0201] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart may be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes program codes for performing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 1309, or installed from the storage device 1306, or installed from the ROM 1302. When the computer program is executed by the processing device 1301, the above functions defined in the method of the embodiment of the present application are executed.

[0202] The electronic device provided in the embodiment of the present application and the image restoration method provided in the above embodiment belong to the same inventive concept. Technical details not described in detail in this embodiment may be referred to in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0203] Based on the image restoration method provided in the above method embodiment, an embodiment of the present application provides a computer-readable medium, on which a computer program is stored, wherein the program, when executed by a processor, implements the image restoration method as described in any of the above embodiments.

[0204] It should be noted that the above-mentioned computer-readable medium in this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In this application, the computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable signal medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0205] In some embodiments, the client and the server can communicate using any currently known or future-developed network protocol such as HTTP (HyperText Transfer Protocol), and can be interconnected with digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (e.g., the Internet), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0206] The above-mentioned computer-readable medium can be included in the above-mentioned electronic device; it can also exist separately without being assembled into the electronic device.

[0207] The above-mentioned computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to execute the above image repair method.

[0208] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include, but are not limited to, object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0209] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0210] The units involved in the embodiments described in this application can be implemented in software or in hardware. Among them, the name of the unit / module does not, in some cases, constitute a limitation on the unit itself. For example, the voice data acquisition module can also be described as the "data acquisition module".

[0211] The functions described above herein can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that can be used include: field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), and so on.

[0212] In the context of the present application, a machine-readable medium may be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0213] According to one or more embodiments of the present application, [Example 1] provides an image restoration method, the method comprising:

[0214] Obtaining an image to be restored;

[0215] Inputting the image to be restored into a spatial normalization model to enable the spatial normalization model to perform a spatial normalization operation on the image to be restored, and obtaining a normalized feature map corresponding to the image to be restored;

[0216] Inputting the normalized feature map corresponding to the image to be restored into an image restoration model to enable the image restoration model to restore the normalized feature map corresponding to the image to be restored, and obtaining a restored image output by the image restoration model;

[0217] Wherein, both the spatial normalization model and the image restoration model are models after training is completed.

[0218] According to one or more embodiments of the present application, [Example 2] provides an image restoration method, the spatial normalization model comprising a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhancement feature extraction modules, the training process of the spatial normalization model comprising:

[0219] Inputting at least one defective image into the residual module, and obtaining an original feature map corresponding to each defective image respectively;

[0220] Input the (i-1)-th target fusion feature map into the i-th significant feature extraction module to obtain the i-th target significant feature map; i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i-1)-th target fusion feature map is the target original feature map, and the target original feature map is the original feature map corresponding to the target defect image, and the target defect images are each of at least one defect image;

[0221] Input the (i-1)-th target fusion feature map and the i-th target significant feature map into the i-th enhanced feature extraction module to obtain the i-th target enhanced feature map, and input the (i-1)-th target fusion feature map and the i-th target significant feature map into the i-th difference feature extraction module to obtain the i-th target difference feature map;

[0222] Concatenate the i-th target enhanced feature map and the i-th target difference feature map to obtain the i-th target fusion feature map, and repeat the step of inputting the (i-1)-th target fusion feature map into the i-th significant feature extraction module to obtain the i-th target significant feature map and subsequent steps until the value of i is N;

[0223] Input the N-th target fusion feature map into the (N+1)-th significant feature extraction module to obtain the (N+1)-th target significant feature map, and input the N-th target fusion feature map and the (N+1)-th target significant feature map into the (N+1)-th difference feature extraction module to obtain the (N+1)-th target difference feature map;

[0224] Combine N target enhanced feature maps and the (N+1)-th target difference feature map to form the target feature map set corresponding to the target defect image;

[0225] Obtain the first loss value according to the feature map sets respectively corresponding to the respective defect images, and train the residual module, the (N+1) significant feature extraction modules, the (N+1) difference feature extraction modules, and the N enhanced feature extraction modules based on the first loss value until a first preset condition is reached.

[0226] According to one or more embodiments of the present application, [Example 3] provides an image restoration method. When there are multiple defect images, the types of the multiple defect images are at least two; the spatial normalization model further includes an adversarial network loss module and a distance metric loss module; the obtaining the first loss value according to the feature map sets respectively corresponding to the respective defect images includes:

[0227] Input the feature map sets respectively corresponding to the respective defect images into the adversarial network loss module to obtain the adversarial network loss value;

[0228] Input the feature map sets corresponding to each defect image into the distance metric loss module to obtain the distance metric loss value;

[0229] The adversarial network loss value and the distance metric loss value form the first loss value.

[0230] According to one or more embodiments of the present application, [Example Four] provides an image restoration method, and the method further includes:

[0231] Concatenate the N target enhanced feature maps and the (N + 1)-th target difference feature map to obtain the target predicted normalized feature map corresponding to the target defect image;

[0232] The target defect image is obtained by performing defect processing on the target original image; the image restoration model includes a first branch model and a second branch model, and the training process of the image restoration model includes:

[0233] Input the target predicted normalized feature map into the first branch model so that the first branch model performs detail restoration on the target predicted normalized feature map to obtain the target predicted detail feature map;

[0234] Input the target predicted normalized feature map into the second branch model so that the second branch model performs rough restoration on the target predicted normalized feature map to obtain the target predicted contour feature map;

[0235] Concatenate the target predicted detail feature map and the target predicted contour feature map to obtain the target predicted image corresponding to the target original image;

[0236] Obtain a second loss value based on each original image and the predicted image corresponding to each original image, and train the first branch model and the second branch model based on the second loss value until a second preset condition is reached.

[0237] According to one or more embodiments of the present application, [Example Five] provides an image restoration method, and the image restoration model further includes a smoothing processing module. The step of concatenating the target predicted detail feature map and the target predicted contour feature map to obtain the target predicted image corresponding to the target original image includes:

[0238] Concatenate the target predicted detail feature map and the target predicted contour feature map to obtain a target concatenated feature map;

[0239] Input the target concatenated feature map into the smoothing processing module so that the smoothing processing module performs a smoothing processing operation on the target concatenated feature map to obtain the target predicted image corresponding to the target original image;

[0240] Obtaining a second loss value based on each original image and the predicted image corresponding to each original image, and training the first branch model and the second branch model based on the second loss value until a second preset condition is reached, including:

[0241] Obtaining a second loss value based on each original image and the predicted image corresponding to each original image, and training the first branch model, the second branch model, and the smoothing processing module based on the second loss value until a second preset condition is reached.

[0242] According to one or more embodiments of the present application, [Example Six] provides an image restoration method. The spatial normalization model includes a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhancement feature extraction modules. Inputting the image to be restored into the spatial normalization model to enable the spatial normalization model to perform a spatial normalization operation on the image to be restored and obtain a normalized feature map corresponding to the image to be restored includes:

[0243] Inputting the image to be restored into the residual module to obtain an original feature map corresponding to the image to be restored;

[0244] Inputting the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map. The value of i is a positive integer from 1 to N, and the initial value of i is 1. When the value of i is 1, the (i - 1)-th fused feature map is the original feature map corresponding to the image to be restored;

[0245] Inputting the (i - 1)-th fused feature map and the i-th significant feature map into the i-th enhancement feature extraction module to obtain the i-th enhancement feature map, and inputting the (i - 1)-th fused feature map and the i-th significant feature map into the i-th difference feature extraction module to obtain the i-th difference feature map;

[0246] Concatenating the i-th enhancement feature map and the i-th difference feature map to obtain the i-th fused feature map, and repeating the steps of inputting the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map and subsequent steps until the value of i is N;

[0247] Inputting the N-th fused feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th significant feature map, and inputting the N-th fused feature map and the (N + 1)-th significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th difference feature map;

[0248] Concatenating N enhancement feature maps and the (N + 1)-th difference feature map to obtain the normalized feature map corresponding to the image to be restored.

[0249] According to one or more embodiments of the present application, [Example Seven] provides an image restoration method. The image restoration model includes a first branch model and a second branch model. Inputting the normalized feature map corresponding to the image to be restored into the image restoration model, so that the image restoration model restores the normalized feature map corresponding to the image to be restored, and obtaining the restored image output by the image restoration model includes:

[0250] Inputting the normalized feature map corresponding to the image to be restored into the first branch model, so that the first branch model performs detail restoration on the normalized feature map corresponding to the image to be restored, and obtaining the restored detail feature map;

[0251] Inputting the normalized feature map corresponding to the image to be restored into the second branch model, so that the second branch model performs rough restoration on the normalized feature map corresponding to the image to be restored, and obtaining the restored contour feature map;

[0252] Stitching the restored detail feature map and the restored contour feature map to obtain the restored image.

[0253] According to one or more embodiments of the present application, [Example Eight] provides an image restoration method. The image restoration model further includes a smoothing processing module. Stitching the restored detail feature map and the restored contour feature map to obtain the restored image includes:

[0254] Stitching the restored detail feature map and the restored contour feature map to obtain a stitched feature map;

[0255] Inputting the stitched feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the stitched feature map to obtain the restored image.

[0256] According to one or more embodiments of the present application, [Example Nine] provides an image restoration device, and the device includes:

[0257] A first acquisition unit, configured to acquire an image to be restored;

[0258] A second acquisition unit, configured to input the image to be restored into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be restored to obtain the normalized feature map corresponding to the image to be restored;

[0259] A third acquisition unit, configured to input the normalized feature map corresponding to the image to be restored into an image restoration model, so that the image restoration model restores the normalized feature map corresponding to the image to be restored to obtain the restored image output by the image restoration model;

[0260] Among them, both the spatial normalization model and the image inpainting model are models after training is completed.

[0261] According to one or more embodiments of the present application, [Example Ten] provides an image inpainting device. The spatial normalization model includes a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhanced feature extraction modules. The device further includes a spatial normalization model training unit, and the spatial normalization model training unit includes:

[0262] A first input subunit, configured to input at least one defective image into the residual module to obtain an original feature map corresponding to each defective image;

[0263] A second input subunit, configured to input the (i - 1)-th target fusion feature map into the i-th significant feature extraction module to obtain the i-th target significant feature map; the value of i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)-th target fusion feature map is the target original feature map, the target original feature map is the original feature map corresponding to the target defective image, and the target defective images are each of at least one defective image;

[0264] A third input subunit, configured to input the (i - 1)-th target fusion feature map and the i-th target significant feature map into the i-th enhanced feature extraction module to obtain the i-th target enhanced feature map, and input the (i - 1)-th target fusion feature map and the i-th target significant feature map into the i-th difference feature extraction module to obtain the i-th target difference feature map;

[0265] A first splicing subunit, configured to splice the i-th target enhanced feature map and the i-th target difference feature map to obtain the i-th target fusion feature map, and repeat the step of inputting the (i - 1)-th target fusion feature map into the i-th significant feature extraction module to obtain the i-th target significant feature map and subsequent steps until the value of i is N;

[0266] A fourth input subunit, configured to input the N-th target fusion feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th target significant feature map, and input the N-th target fusion feature map and the (N + 1)-th target significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th target difference feature map;

[0267] A composition subunit, configured to compose N target enhanced feature maps and the (N + 1)-th target difference feature map into a target feature map set corresponding to the target defective image;

[0268] The first training subunit is configured to obtain a first loss value according to the feature map sets respectively corresponding to each defective image, and train the residual module, the N+1 significant feature extraction modules, the N+1 difference feature extraction modules, and the N enhancement feature extraction modules based on the first loss value until a first preset condition is met.

[0269] According to one or more embodiments of the present application, [Example XI] provides an image restoration device. When there are multiple defective images, the types of the multiple defective images are at least two; the spatial normalization model further includes an adversarial network loss module and a distance metric loss module; the training subunit includes:

[0270] The first acquisition subunit is configured to input the feature map sets respectively corresponding to each defective image into the adversarial network loss module to obtain an adversarial network loss value;

[0271] The second acquisition subunit is configured to input the feature map sets respectively corresponding to each defective image into the distance metric loss module to obtain a distance metric loss value;

[0272] The adversarial network loss value and the distance metric loss value constitute the first loss value.

[0273] According to one or more embodiments of the present application, [Example XII] provides an image restoration device. The device further includes:

[0274] The splicing unit is configured to splice the N target enhancement feature maps and the (N+1)th target difference feature map to obtain the target predicted normalized feature map corresponding to the target defective image;

[0275] The target defective image is obtained by defect processing of the target original image; the image restoration model includes a first branch model and a second branch model. The device further includes an image restoration model training unit, and the image restoration model training unit includes:

[0276] The third acquisition subunit is configured to input the target predicted normalized feature map into the first branch model so that the first branch model performs detail restoration on the target predicted normalized feature map to obtain a target predicted detail feature map;

[0277] The fourth acquisition subunit is configured to input the target predicted normalized feature map into the second branch model so that the second branch model performs rough restoration on the target predicted normalized feature map to obtain a target predicted contour feature map;

[0278] The fifth acquisition subunit is configured to splice the target predicted detail feature map and the target predicted contour feature map to obtain the target predicted image corresponding to the target original image;

[0279] A second training subunit, configured to obtain a second loss value based on each original image and the predicted image corresponding to each original image, and train the first branch model and the second branch model based on the second loss value until a second preset condition is met.

[0280] According to one or more embodiments of the present application, [Example XIII] provides an image restoration device. The image restoration model further includes a smoothing processing module. The fifth acquisition subunit includes:

[0281] A second splicing subunit, configured to splice the target predicted detail feature map and the target predicted contour feature map to obtain a target spliced feature map;

[0282] A sixth acquisition subunit, configured to input the target spliced feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the target spliced feature map to obtain the predicted image corresponding to the target original image;

[0283] The second training subunit is specifically configured to:

[0284] Obtain a second loss value based on each original image and the predicted image corresponding to each original image, and train the first branch model, the second branch model, and the smoothing processing module based on the second loss value until a second preset condition is met.

[0285] According to one or more embodiments of the present application, [Example XIV] provides an image restoration device. The spatial normalization model includes a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhancement feature extraction modules; the second acquisition unit includes:

[0286] A fifth input subunit, configured to input the image to be restored into the residual module to obtain the original feature map corresponding to the image to be restored;

[0287] A sixth input subunit, configured to input the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map; i takes positive integer values from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)-th fused feature map is the original feature map corresponding to the image to be restored;

[0288] A seventh input subunit, configured to input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th enhancement feature extraction module to obtain the i-th enhancement feature map, and input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th difference feature extraction module to obtain the i-th difference feature map;

[0289] The third splicing subunit is configured to splice the i-th enhanced feature map and the i-th difference feature map to obtain the i-th fused feature map, and repeatedly execute the step of inputting the (i - 1)-th fused feature map into the i-th saliency feature extraction module to obtain the i-th saliency feature map and subsequent steps until the value of i is N;

[0290] The eighth input subunit is configured to input the N-th fused feature map into the (N + 1)-th saliency feature extraction module to obtain the (N + 1)-th saliency feature map, and input the N-th fused feature map and the (N + 1)-th saliency feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th difference feature map;

[0291] The fourth splicing subunit is configured to splice N enhanced feature maps and the (N + 1)-th difference feature map to obtain the normalized feature map corresponding to the image to be restored.

[0292] According to one or more embodiments of the present application, [Example XV] provides an image restoration device, the image restoration model includes a first branch model and a second branch model, and the third acquisition unit includes:

[0293] The first restoration subunit is configured to input the normalized feature map corresponding to the image to be restored into the first branch model, so that the first branch model performs detail restoration on the normalized feature map corresponding to the image to be restored to obtain a restored detail feature map;

[0294] The second restoration subunit is configured to input the normalized feature map corresponding to the image to be restored into the second branch model, so that the second branch model performs rough restoration on the normalized feature map corresponding to the image to be restored to obtain a restored contour feature map;

[0295] The fifth splicing subunit is configured to splice the restored detail feature map and the restored contour feature map to obtain a restored image.

[0296] According to one or more embodiments of the present application, [Example XVI] provides an image restoration device, the image restoration model further includes a smoothing processing module, and the fifth splicing subunit includes:

[0297] The sixth splicing subunit is configured to splice the restored detail feature map and the restored contour feature map to obtain a spliced feature map;

[0298] The ninth input subunit is configured to input the spliced feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the spliced feature map to obtain a restored image.

[0299] According to one or more embodiments of the present application, [Example Seventeen] provides an electronic device, including:

[0300] One or more processors;

[0301] A storage device on which one or more programs are stored,

[0302] When the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration method as described in any of the above.

[0303] According to one or more embodiments of the present application, [Example Eighteen] provides a computer-readable medium on which a computer program is stored, wherein when the program is executed by a processor, the image restoration method as described in any of the above is implemented.

[0304] According to one or more embodiments of the present application, [Example Nineteen] provides a computer program product, the computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the image restoration method as described in any of the above is implemented.

[0305] It should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0306] It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: only A exists, only B exists, and both A and B exist at the same time. Among them, A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after. "At least one (one) of the following" or its similar expression refers to any combination of these items, including any combination of single item (one) or plural items (ones). For example, at least one (one) of a, b, or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0307] It should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0308] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination thereof. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0309] The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image inpainting method, characterized in that, the method includes: obtaining an image to be inpainted; inputting the image to be inpainted into a spatial normalization model to enable the spatial normalization model to perform a spatial normalization operation on the image to be inpainted, and obtaining a normalized feature map corresponding to the image to be inpainted; inputting the normalized feature map corresponding to the image to be inpainted into an image inpainting model to enable the image inpainting model to inpaint the normalized feature map corresponding to the image to be inpainted, and obtaining the inpainted image output by the image inpainting model; wherein, both the spatial normalization model and the image inpainting model are models after training; the spatial normalization model includes a residual module, N+1 significant feature extraction modules, N+1 difference feature extraction modules, and N enhancement feature extraction modules; the step of inputting the image to be inpainted into the spatial normalization model to enable the spatial normalization model to perform a spatial normalization operation on the image to be inpainted, and obtaining a normalized feature map corresponding to the image to be inpainted, includes: inputting the image to be inpainted into the residual module, and obtaining an original feature map corresponding to the image to be inpainted; inputting the (i-1)th fused feature map into the ith significant feature extraction module, and obtaining the ith significant feature map; the value of i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i-1)th fused feature map is the original feature map corresponding to the image to be inpainted; inputting the (i-1)th fused feature map and the ith significant feature map into the ith enhancement feature extraction module, and obtaining the ith enhancement feature map, inputting the (i-1)th fused feature map and the ith significant feature map into the ith difference feature extraction module, and obtaining the ith difference feature map; concatenating the ith enhancement feature map and the ith difference feature map to obtain the ith fused feature map, and repeating the steps of inputting the (i-1)th fused feature map into the ith significant feature extraction module to obtain the ith significant feature map and subsequent steps until the value of i is N; inputting the Nth fused feature map into the (N+1)th significant feature extraction module, and obtaining the (N+1)th significant feature map, inputting the Nth fused feature map and the (N+1)th significant feature map into the (N+1)th difference feature extraction module, and obtaining the (N+1)th difference feature map; concatenating N enhancement feature maps and the (N+1)th difference feature map to obtain the normalized feature map corresponding to the image to be inpainted.

2. The method according to claim 1, characterized in that, the spatial normalization model includes a residual module, N+1 significant feature extraction modules, N+1 difference feature extraction modules, and N enhancement feature extraction modules, and the training process of the spatial normalization model includes: inputting at least one defective image into the residual module, and obtaining an original feature map corresponding to each defective image respectively; Input the (i - 1)-th target fusion feature map into the i-th significant feature extraction module to obtain the i-th target significant feature map; i takes positive integer values from 1 to N, the initial value of i is 1, and when i = 1, the (i - 1)-th target fusion feature map is the target original feature map, and the target original feature map is the original feature map corresponding to the target defect image, and the target defect images are each of at least one defect image; Input the (i - 1)-th target fusion feature map and the i-th target significant feature map into the i-th enhanced feature extraction module to obtain the i-th target enhanced feature map, and input the (i - 1)-th target fusion feature map and the i-th target significant feature map into the i-th difference feature extraction module to obtain the i-th target difference feature map; Concatenate the i-th target enhanced feature map and the i-th target difference feature map to obtain the i-th target fusion feature map, and repeat the step of inputting the (i - 1)-th target fusion feature map into the i-th significant feature extraction module to obtain the i-th target significant feature map and subsequent steps until i = N; Input the N-th target fusion feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th target significant feature map, and input the N-th target fusion feature map and the (N + 1)-th target significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th target difference feature map; Compose the N target enhanced feature maps and the (N + 1)-th target difference feature map into the target feature map set corresponding to the target defect image; Obtain the first loss value according to the feature map sets respectively corresponding to each defect image, and train the residual module, the (N + 1) significant feature extraction modules, the (N + 1) difference feature extraction modules, and the N enhanced feature extraction modules based on the first loss value until a first preset condition is reached.

3. The method according to claim 2, wherein, when there are multiple defect images, the types of the multiple defect images are at least two; the spatial normalization model further includes an adversarial network loss module and a distance metric loss module; The obtaining the first loss value according to the feature map sets respectively corresponding to each defect image includes: Input the feature map sets respectively corresponding to each defect image into the adversarial network loss module to obtain the adversarial network loss value; Input the feature map sets respectively corresponding to each defect image into the distance metric loss module to obtain the distance metric loss value; The adversarial network loss value and the distance metric loss value constitute the first loss value.

4. The method according to claim 2, wherein, The method further includes: Concatenate the N target enhanced feature maps and the (N + 1)-th target difference feature map to obtain the target predicted normalized feature map corresponding to the target defect image; The target defect image is obtained by defect processing of the target original image; the image repair model includes a first branch model and a second branch model, and the training process of the image repair model includes: Input the target prediction normalized feature map into the first branch model so that the first branch model repairs the details of the target prediction normalized feature map to obtain a target prediction detailed feature map; Input the target prediction normalized feature map into the second branch model so that the second branch model repairs the roughness of the target prediction normalized feature map to obtain a target prediction contour feature map; Stitch the target prediction detailed feature map and the target prediction contour feature map to obtain a target prediction image corresponding to the target original image; Obtain a second loss value based on each original image and the prediction image corresponding to each original image, and train the first branch model and the second branch model based on the second loss value until a second preset condition is reached.

5. The method according to claim 4, wherein, the image repair model further includes a smoothing processing module, and the step of stitching the target prediction detailed feature map and the target prediction contour feature map to obtain a target prediction image corresponding to the target original image includes: Stitch the target prediction detailed feature map and the target prediction contour feature map to obtain a target stitched feature map; Input the target stitched feature map into the smoothing processing module so that the smoothing processing module performs a smoothing processing operation on the target stitched feature map to obtain a target prediction image corresponding to the target original image; The step of obtaining a second loss value based on each original image and the prediction image corresponding to each original image, and training the first branch model and the second branch model based on the second loss value until a second preset condition is reached includes: Obtain a second loss value based on each original image and the prediction image corresponding to each original image, and train the first branch model, the second branch model, and the smoothing processing module based on the second loss value until a second preset condition is reached.

6. The method according to claim 1, wherein, the image repair model includes a first branch model and a second branch model, and the step of inputting the normalized feature map corresponding to the image to be repaired into the image repair model so that the image repair model repairs the normalized feature map corresponding to the image to be repaired to obtain a repaired image output by the image repair model includes: Input the normalized feature map corresponding to the image to be repaired into the first branch model so that the first branch model repairs the details of the normalized feature map corresponding to the image to be repaired to obtain a repaired detailed feature map; Input the normalized feature map corresponding to the image to be repaired into the second branch model so that the second branch model repairs the roughness of the normalized feature map corresponding to the image to be repaired to obtain a repaired contour feature map; Stitch the repaired detailed feature map and the repaired contour feature map to obtain a repaired image.

7. The method according to claim 6, wherein, The image inpainting model further includes a smoothing processing module. The step of splicing the inpainted detailed feature map and the inpainted contour feature map to obtain an inpainted image includes: Splicing the inpainted detailed feature map and the inpainted contour feature map to obtain a spliced feature map; Inputting the spliced feature map into the smoothing processing module, so that the smoothing processing module performs a smoothing processing operation on the spliced feature map to obtain an inpainted image.

8. An image inpainting device Characterized in that The device includes: A first acquisition unit, configured to acquire an image to be inpainted; A second acquisition unit, configured to input the image to be inpainted into a spatial normalization model, so that the spatial normalization model performs a spatial normalization operation on the image to be inpainted to obtain a normalized feature map corresponding to the image to be inpainted; A third acquisition unit, configured to input the normalized feature map corresponding to the image to be inpainted into an image inpainting model, so that the image inpainting model inpaints the normalized feature map corresponding to the image to be inpainted to obtain an inpainted image output by the image inpainting model; Wherein, both the spatial normalization model and the image inpainting model are models after training; The spatial normalization model includes a residual module, N + 1 significant feature extraction modules, N + 1 difference feature extraction modules, and N enhancement feature extraction modules; the second acquisition unit is specifically configured to: Input the image to be inpainted into the residual module to obtain an original feature map corresponding to the image to be inpainted; Input the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map; i is a positive integer from 1 to N, the initial value of i is 1, and when the value of i is 1, the (i - 1)-th fused feature map is the original feature map corresponding to the image to be inpainted; Input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th enhancement feature extraction module to obtain the i-th enhancement feature map, and input the (i - 1)-th fused feature map and the i-th significant feature map into the i-th difference feature extraction module to obtain the i-th difference feature map; Splice the i-th enhancement feature map and the i-th difference feature map to obtain the i-th fused feature map, and repeat the steps of inputting the (i - 1)-th fused feature map into the i-th significant feature extraction module to obtain the i-th significant feature map and subsequent steps until the value of i is N; Input the N-th fused feature map into the (N + 1)-th significant feature extraction module to obtain the (N + 1)-th significant feature map, and input the N-th fused feature map and the (N + 1)-th significant feature map into the (N + 1)-th difference feature extraction module to obtain the (N + 1)-th difference feature map; Splice N enhancement feature maps and the (N + 1)-th difference feature map to obtain a normalized feature map corresponding to the image to be inpainted.

9. An electronic device Characterized in that It includes: One or more processors; A storage device on which one or more programs are stored When the one or more programs are executed by the one or more processors, the one or more processors implement the image restoration method according to any one of claims 1-7.

10. A computer-readable medium, characterized in that, a computer program is stored thereon, wherein when the program is executed by a processor, the image restoration method according to any one of claims 1-7 is implemented.

11. A computer program product, characterized in that, the computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, the image restoration method according to any one of claims 1-7 is implemented.

Images