Image inpainting model training method, image inpainting method, device and storage medium

Abstract

The application discloses an image repairing model training method, an image repairing method, an image repairing device and a storage medium. The image repairing model training method comprises the following steps: inputting a first image to be repaired and an i-th random noise in N random noises into a generating network of an image repairing model to generate an i-th second image; determining a first loss value according to a relative entropy of a first parameter and a second parameter; the first parameter is determined according to normalized results of the i-th random noise and the N random noises; the second parameter is determined according to normalized results of the i-th second image and N second images corresponding to the N random noises; the N is an integer greater than 1; and the weight parameters of the image repairing model are updated according to the determined first loss value.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image restoration model training method, image restoration method, device, electronic device and storage medium. Background Technology

[0002] Image restoration refers to the process of reconstructing lost or damaged parts of an image. In some applications, an image to be restored may correspond to multiple reasonable restoration results. For example, in an image of a face with a damaged nose area, a reasonable restored image could be either a high nose bridge or a low nose bridge.

[0003] In related technologies, in image restoration methods based on random noise, only a single restored image can be obtained when restoring an image to be restored, and it is not possible to provide multiple restored images for a single image to be restored. Summary of the Invention

[0004] In view of this, embodiments of this application provide an image restoration model training method, an image restoration method, an apparatus, an electronic device, and a storage medium, so as to at least solve the problem that related technologies cannot provide a variety of restoration images for a single image to be restored when performing image restoration.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides an image restoration model training method, including:

[0007] The first image to be repaired and the i-th random noise from N random noises are input into the generative network of the image repair model to generate the i-th second image;

[0008] The first loss value is determined based on the relative entropy of the first parameter and the second parameter; the first parameter is determined based on the normalized result of the i-th random noise and the N random noises; the second parameter is determined based on the normalized result of the i-th second image and the N second images generated corresponding to the N random noises; where N is an integer greater than 1.

[0009] The weight parameters of the image restoration model are updated based on the determined first loss value.

[0010] In the above scheme, before updating the weight parameters of the image inpainting model based on the determined first loss value, the method further includes:

[0011] The discriminant network of the image restoration model is used to classify the i-th second image. Based on the classification result of the i-th second image and the classification result of the third image, a second loss value is determined. The third image represents the calibration result corresponding to the first image.

[0012] Based on the i-th second image and the third image, a third loss value is determined;

[0013] The step of updating the weight parameters of the image restoration model based on the determined first loss value includes:

[0014] The weight parameters of the image restoration model are updated based on the determined first loss value, second loss value, and third loss value.

[0015] In the above scheme, determining the third loss value based on the i-th second image and the third image includes:

[0016] The fourth loss value is determined based on the norm of the i-th second image and the third image;

[0017] A first feature vector is determined based on the i-th second image, a second feature vector is determined based on the third image, and a fifth loss value is determined based on the norm of the vector difference between the first feature vector and the second feature vector.

[0018] The third loss value is determined based on the fourth loss value and the fifth loss value.

[0019] In the above scheme, before inputting the first image to be repaired and the i-th random noise from N random noises into the generator network of the image repair model, the method further includes:

[0020] Determine the fourth image from the established sample image library;

[0021] The determined third image is masked to obtain the first image.

[0022] In the above scheme, determining the fourth image from the set sample image library includes:

[0023] The fifth image in the set sample image library is subjected to target detection, and the fifth image is cropped based on the target rectangle located during the target detection process to determine the fourth image.

[0024] This application also provides an image restoration method, including:

[0025] The sixth image to be repaired and N random noises are input into the generator network of the image repair model, which outputs N seventh images; where,

[0026] The image restoration model is trained using any of the image restoration model training methods described above.

[0027] This application also provides an image restoration model training device, including:

[0028] The generation unit is used to input the first image to be repaired and the i-th random noise from N random noises into the generation network of the image repair model to generate the i-th second image;

[0029] A first processing unit is configured to determine a first loss value based on the relative entropy of a first parameter and a second parameter; the first parameter is determined based on the normalization result of the i-th random noise and the N random noises; the second parameter is determined based on the normalization result of the i-th second image and the N second images generated corresponding to the N random noises; where N is an integer greater than 1.

[0030] The training unit is used to update the weight parameters of the image restoration model based on the determined first loss value.

[0031] This application also provides an image restoration apparatus, including:

[0032] The inpainting unit is used to input the sixth image to be inpainted and N random noises into the generator network of the image inpainting model, and output N seventh images; where,

[0033] The image restoration model is trained using any of the image restoration model training methods described above.

[0034] This application also provides a first electronic device, including: a first processor and a first communication interface; wherein,

[0035] The first processor is configured to input the first image to be repaired and the i-th random noise from N random noises into the generative network of the image repair model to generate the i-th second image;

[0036] The first loss value is determined based on the relative entropy of the first parameter and the second parameter; the first parameter is determined based on the normalized result of the i-th random noise and the N random noises; the second parameter is determined based on the normalized result of the i-th second image and the N second images generated corresponding to the N random noises; where N is an integer greater than 1.

[0037] The weight parameters of the image restoration model are updated based on the determined first loss value.

[0038] This application also provides a second electronic device, including: a second processor and a second communication interface; wherein,

[0039] The second processor is used to input the sixth image to be repaired and N random noises into the generator network of the image repair model, and output N seventh images; wherein,

[0040] The image restoration model is trained using any of the image restoration model training methods described above.

[0041] This application also provides a first electronic device, including: a first processor and a first memory for storing a computer program capable of running on the processor.

[0042] Wherein, when the first processor is used to run the computer program, it executes the steps of any of the above-described image restoration model training methods.

[0043] This application also provides a second electronic device, including: a second processor and a second memory for storing a computer program capable of running on the processor.

[0044] The second processor is used to execute the steps of the above-described image restoration method when running the computer program.

[0045] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the image restoration model training method described above, or implements the steps of the image restoration method described above.

[0046] The image inpainting model training method, image inpainting method, apparatus, electronic device, and storage medium provided in this application embodiment input the first image to be repaired and the i-th random noise from N random noises into the generator network of the image inpainting model to generate the i-th second image. The normalized result of the i-th random noise and the N random noises is used as a first parameter, and the normalized result of the N second images generated corresponding to the i-th second image and the N random noises is used as a second parameter. A first loss value is determined based on the relative entropy of the first and second parameters, and the weight parameters of the generator network of the image inpainting model are updated according to the first loss value. During image inpainting model training, updating the weight parameters of the generator network of the image inpainting model through the first loss value enables the generated N second images to overcome the pattern collapse problem present in generative adversarial networks. Thus, based on multiple random noises, diverse image inpainting results corresponding to the same image to be repaired can be generated, thereby providing diverse image inpainting results for the same image to be repaired. Attached Figure Description

[0047] Figure 1 A flowchart illustrating an image restoration model training method provided in this application embodiment;

[0048] Figure 2A flowchart illustrating an image restoration method provided in an embodiment of this application;

[0049] Figure 3 A schematic flowchart of an image restoration method provided for an application embodiment of this application;

[0050] Figure 4 A schematic diagram of an image restoration model provided for an application embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the structure of an image restoration model training device provided in an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of this application;

[0053] Figure 7 This is a schematic diagram of the structure of a first electronic device provided in an embodiment of this application;

[0054] Figure 8 This is a schematic diagram of the structure of a second electronic device provided in an embodiment of this application. Detailed Implementation

[0055] Image restoration refers to the process of reconstructing lost or damaged parts of an image. In some applications, an image to be restored may correspond to multiple reasonable restoration results. For example, in an image of a face with a damaged nose area, a reasonable restored image could be either a high nose bridge or a low nose bridge.

[0056] In related technologies, in image restoration methods based on random noise, only a single restored image can be obtained when restoring an image to be restored, and it is not possible to provide multiple restored images for a single image to be restored.

[0057] Based on this, the image inpainting model training method, image inpainting method, apparatus, electronic device, and storage medium provided in this application embodiment input the first image to be inpainted and the i-th random noise from N random noises into the generative network of the image inpainting model to generate the i-th second image. The normalized result of the i-th random noise and the N random noises is used as the first parameter, and the normalized result of the N second images generated corresponding to the i-th second image and the N random noises is used as the second parameter. A first loss value is determined based on the relative entropy of the first and second parameters, and the weight parameters of the generative network of the image inpainting model are updated according to the first loss value. When training the image inpainting model, updating the weight parameters of the generative network of the image inpainting model using the first loss value ensures that among the N generated second images, those generated based on noises that are close in the random noise space are close in the image space, while those generated based on noises that are far apart in the random noise space are far apart in the image space, thereby overcoming the pattern collapse problem existing in generative adversarial networks. In this way, based on multiple random noises, diverse image restoration results corresponding to the same image to be restored can be generated, thus providing diverse image restoration results for the same image to be restored.

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0059] A flowchart illustrating the image restoration model training method provided in this application embodiment is shown below. Figure 1 As shown, the image restoration model training methods include:

[0060] Step 101: Input the first image to be repaired and the i-th random noise from N random noises into the generator network of the image repair model to generate the i-th second image.

[0061] Wherein, N is an integer greater than 1.

[0062] The first image to be repaired and the i-th random noise from N random noises are input into the generator network of the image inpainting model to generate the i-th second image. N is an integer greater than 1, and i is any positive integer from 1 to N. Here, the first image represents the image to be repaired that is input into the image inpainting model, and the second image represents the repaired image corresponding to the first image output by the image inpainting model. The image inpainting model can be a type of Generative Adversarial Network (GAN), such as a Conditional Generative Adversarial Network (CGAN), which inputs additional information as conditions to both the generator and discriminator networks.

[0063] Random noise can be input into the generative network of the image inpainting model in the form of feature vectors. The random noise can be obtained by sampling random samples from a prior distribution, such as a uniform distribution or a multidimensional Gaussian distribution with a covariance matrix of I. The value of N can be a power of 2, such as 8, 16, 32, etc., and is determined according to the training device, such as the video memory of a GPU.

[0064] Step 102: Determine the first loss value based on the relative entropy of the first parameter and the second parameter.

[0065] The first parameter is determined based on the normalization result of the i-th random noise and the N random noises; the second parameter is determined based on the normalization result of the i-th second image and the N second images generated corresponding to the N random noises.

[0066] The first parameter is determined based on the normalization results of the i-th random noise and N random noises. The second parameter is determined based on the normalization results of the i-th second image and the N second images generated corresponding to the N random noises. The first loss value is determined based on the relative entropy results of the first parameter and the second parameter.

[0067] Step 103: Update the weight parameters of the image restoration model based on the determined first loss value.

[0068] The weight parameters of the image restoration model are updated to improve the discrimination results of the discriminant network for the restored image output by the generative network of the image restoration model.

[0069] During training, the generative and discriminative networks of the image inpainting model are trained iteratively, with weight parameters updated, until they reach a dynamic equilibrium, i.e., Nash equilibrium. The optimal state is when the discriminative network cannot distinguish between the generated inpainted image and the real image; that is, with the original GAN ​​loss, the discrimination probability of both in the discriminator is 0.5.

[0070] Specifically, the first loss value is backpropagated in the image restoration model. During the backpropagation of the loss value to each layer of the image restoration model, the gradient of the loss function is calculated based on the loss value, and the weight parameters backpropagated to the current layer are updated along the descent direction of the gradient.

[0071] The updated weight parameters will be used as the weight parameters for the trained image restoration model.

[0072] Here, an update stopping condition can be set. When the update stopping condition is met, the weight parameters obtained from the last update are determined as the weight parameters used by the trained image restoration model. The update stopping condition is such as the set number of training iterations. One training iteration is the process of training the image restoration model once based on the first image. Of course, the update stopping condition is not limited to this; for example, it can also be a threshold value of the loss function.

[0073] It's important to note that the loss function measures the degree of inconsistency between the model's predictions and the true values ​​(calibrated values). In practical applications, model training is achieved by minimizing the loss function.

[0074] Backpropagation is the opposite of forward propagation. Forward propagation refers to the feedforward processing of the model, while backpropagation proceeds in the opposite direction. Backpropagation involves updating the weight parameters of each layer of the model based on the model's output.

[0075] In this way, when using the generative network of the trained image restoration model to restore images, the pattern collapse problem existing in generative adversarial networks can be overcome. Based on multiple random noises, diverse image restoration results corresponding to the same image to be restored can be generated, thus providing users with a variety of image restoration results for the same image to be restored.

[0076] In practical applications, the first loss value can be determined in the following ways.

[0077] The i-th first parameter P i It can be determined by the following formula (1):

[0078]

[0079] in,

[0080] z i Let i be the i-th random noise in N random noises.

[0081] z j Let j be the j-th random noise in N random noises.

[0082] The i-th second parameter Q i It can be determined by the following formula (2):

[0083]

[0084] in,

[0085] The generative network for the image inpainting model is based on the first image to be inpainted. The second image generated from the i-th random noise,

[0086] The generative network for the image inpainting model is based on the first image to be inpainted. The j-th second image is generated from the j-th random noise.

[0087] First loss value L multimodal It can be determined by the following formula (3):

[0088] L multimodal =λ1KL(P i ||Q i (3)

[0089] in,

[0090] λ1 represents the weight of the first loss value.

[0091] KL divergence calculation probability P i With Q i The expected value of the logarithmic difference.

[0092] In one embodiment, before updating the weight parameters of the image inpainting model based on the determined first loss value, the method further includes:

[0093] The discriminant network of the image restoration model is used to classify the i-th second image. Based on the classification result of the i-th second image and the classification result of the third image, a second loss value is determined. The third image represents the calibration result corresponding to the first image.

[0094] Based on the i-th second image and the third image, a third loss value is determined;

[0095] The step of updating the weight parameters of the image restoration model based on the determined first loss value includes:

[0096] The weight parameters of the image restoration model are updated based on the determined first loss value, second loss value, and third loss value.

[0097] Before updating the weight parameters of the image inpainting model based on the determined first loss value, the discriminative network of the image inpainting model classifies the i-th second image and the third image respectively. A second loss value is determined based on the classification results of the i-th second image and the third image, and a third loss value is determined based on the i-th second image and the third image. The weight parameters of the image inpainting model are then updated based on the determined first, second, and third loss values. Here, the third image represents the calibration result corresponding to the first image; it can be the fourth image corresponding to the first image, i.e., the third image is the original image of the first image; or it can be an image determined from the image database based on the feature information of the first image.

[0098] In this way, the weight parameters are updated based on the determined first, second, and third loss values. The introduction of the second loss value makes the generated restored image visually more realistic. Simultaneously, the pixel-level losses of the generated and calibrated images are determined to make the restored image as similar as possible to the corresponding real image in pixel space. Furthermore, by comparing high-level perceptual and semantic differences between images, it is ensured that the image restoration result is semantically consistent with the corresponding real image.

[0099] Here, the second loss value L adv It can be determined by the following formula (4):

[0100]

[0101] Where D and G are the outputs of the discriminator network and the generator network, respectively.

[0102] The generative network for the image inpainting model is based on the first image to be inpainted;

[0103] x represents the calibration result corresponding to the first image, i.e., the third image;

[0104] z i Let i be the i-th random noise in N random noises.

[0105] In one embodiment, determining the third loss value based on the i-th second image and the third image includes:

[0106] The fourth loss value is determined based on the norm of the i-th second image and the third image;

[0107] A first feature vector is determined based on the i-th second image, a second feature vector is determined based on the third image, and a fifth loss value is determined based on the norm of the vector difference between the first feature vector and the second feature vector.

[0108] The third loss value is determined based on the fourth loss value and the fifth loss value.

[0109] The fourth loss value is determined based on the L1 norm of the i-th second image and the third image. A first feature vector is determined based on the i-th second image, and a second feature vector is determined based on the third image. The fifth loss value is determined based on the L1 norm of the vector difference between the first and second feature vectors. The third loss value is then calculated using the determined fourth and fifth loss values. Here, the feature vectors can be determined by inputting the corresponding images into a neural network model pre-trained using ImageNet, such as VGGNet.

[0110] Fourth loss value L L1 It can be determined by the following formula (5):

[0111]

[0112] in,

[0113] λ2 represents the weight of the fourth loss value.

[0114] Here, the fifth loss value L perceptual It can be determined by the following formula (6):

[0115]

[0116] in,

[0117] λ3 represents the weight of the fifth loss value.

[0118] F(·) represents the feature vector corresponding to the image.

[0119] The third loss value L3 can be determined by the following formula (7):

[0120] L3 = L L1 +L perceptual (7)

[0121] Thus, the weight parameters are updated based on the determined first, second, and third loss values. The introduction of the second loss value makes the generated restored image visually more realistic. Simultaneously, the fourth loss value determines the pixel-level loss between the generated and calibrated images, ensuring the restored image is as similar as possible to the corresponding real image in pixel space. Furthermore, by introducing a fifth loss value to compare high-level perceptual and semantic differences between images, the semantic consistency between the image restoration result and the corresponding real image is ensured.

[0122] In one embodiment, before inputting the first image to be repaired and the i-th random noise from N random noises into the generative network of the image inpainting model, the method further includes:

[0123] Determine the fourth image from the established sample image library;

[0124] The determined fourth image is masked to obtain the first image.

[0125] A fourth image is selected from a predefined sample image library. A designated region of this fourth image is then masked to obtain the first image to be repaired for training. The predefined sample image library can be a commonly used database such as ImageNet or SUN, or it can be a sample image library created as needed. Masking can be implemented using OpenCV or similar methods.

[0126] When training an image restoration model, depending on the model's purpose of restoring face images, landscape images, etc., the shape of the set area of ​​the fourth image can be a defined shape such as a rectangle or an ellipse, or it can be the recognition result of a specific object in the image.

[0127] Depending on the intended use of the model, masks are randomly added at appropriate locations. For example, masks are added to the face region when restoring face images, and to the regions corresponding to specific objects (such as human bodies, trees, rocks, buildings, etc.) when restoring landscape images.

[0128] In one embodiment, determining the fourth image from a set sample image library includes:

[0129] The fifth image in the set sample image library is subjected to target detection, and the fifth image is cropped based on the target rectangle located during the target detection process to determine the fourth image.

[0130] Object detection is performed on the fifth image in the designated sample image library. Based on the target bounding boxes located during the detection process, the target in the image is selected, and the coordinate boxes of the target locations are obtained. The detection boxes containing the target are then cropped according to these coordinate boxes to obtain the fourth image. Here, the fifth image in the sample image library is the image dataset used for training. During cropping, image alignment and correction can also be performed to obtain images that meet the set criteria.

[0131] In this way, by performing object detection and cropping on the original dataset images, the neural network model can focus on learning the target information of the image during training while retaining the recognized target, thus reducing the training complexity of the model. The resulting image restoration model has good application performance.

[0132] A flowchart illustrating the image restoration method provided in this application embodiment is shown below. Figure 2 As shown, image restoration methods include:

[0133] Step 201: Input the sixth image to be repaired and N random noises into the generator network of the image inpainting model, and output N seventh images; where,

[0134] The image restoration model is trained using any of the image restoration model training methods described above.

[0135] The sixth image to be repaired and N random noises are input into the generator network of the image inpainting model, which outputs N repaired seventh images. Here, the generator network used in the image inpainting model is trained using any of the image inpainting model training methods described above. The generator network of the image inpainting model groups the sixth image and each of the N random noises as a pair; each pair of images and random noises can generate a corresponding seventh image. Thus, N random noises and the sixth image can generate N seventh images.

[0136] In this way, when using the generative network of the trained image restoration model to restore images, the pattern collapse problem existing in generative adversarial networks can be overcome. Based on multiple random noises, diverse image restoration results corresponding to the same image to be restored can be generated, thus providing diverse image restoration results for the same image to be restored.

[0137] The following are application examples of the embodiments of this application:

[0138] In the field of unsupervised learning, GANs consist of a generator network and a discriminator network, which can learn to fit the image distribution and generate visually realistic images that conform to the image distribution after training.

[0139] like Figure 3 The diagram illustrates a process flow for an image restoration method, including the following steps:

[0140] Step 301: Obtain the training dataset for the image to be repaired.

[0141] Here, the training dataset that does not meet the training requirements is preprocessed, including but not limited to normalizing the images in the training dataset and adding masks to the images in the training set according to specific needs. The preprocessed images are then used as the images to be repaired in the training set for training.

[0142] Depending on the purpose of the model, masks are randomly added at appropriate locations. For example, masks are added to the face area when restoring face images, and to the area corresponding to specific objects (such as human bodies, trees, rocks, buildings, etc.) when restoring landscape images.

[0143] Step 302: Construct an unsupervised image inpainting model.

[0144] like Figure 4The diagram shows a schematic of an image inpainting model, which includes a random noise-driven module, an image input module to be inpainted, a generative network inpainting module, a discriminative network module, an adversarial loss module, a pixel-level loss module, a perceptual loss module, and a multimodal constraint module.

[0145] The image to be repaired input module processes the image to be repaired into an input format that meets the requirements of the generative repair module, including but not limited to alignment, correction, and cropping. The image processed by the image to be repaired input module, along with random noise samples, is then input into the generative network repair module.

[0146] The random noise driving module samples random noise samples from the prior distribution and inputs them, along with the image to be repaired determined by the image to be repaired input module, into the generative network repair module.

[0147] The generative network inpainting module is a generative network structure, typically consisting of an encoder-decoder network architecture. It can contain multiple skip connection layers, or it can consist of nearest neighbor upsampling layers, nearest neighbor downsampling layers, and multiple residual blocks. Its main function is to accept sample inputs from the random noise driving module and the image input module to be repaired, and generate diverse repaired images.

[0148] Once trained, the image restoration model can generate diverse image restoration results based on different noise samples.

[0149] The discriminative network module is used to determine the realism of the restored image, stimulating the generative network inpainting module to generate visually realistic and semantically consistent image restoration results. The discriminative network module is a discriminative network structure, typically a deep convolutional neural network.

[0150] Step 303: Train the constructed image restoration model.

[0151] An image inpainting model is constructed by training an adversarial loss module, a pixel-level loss module, a perceptual loss module, and a multimodal constraint module, wherein:

[0152] The loss-adversarial module can calculate the loss value L using formula (4). adv .

[0153] Where D and G are the outputs of the discriminant network module and the generator network repair module, respectively, and x and These are the actual complete image and the masked image to be repaired in the training set, respectively. i Let i be the i-th noise sample among N random noise samples sampled from a prior distribution. The prior distribution is usually a multidimensional standard Gaussian distribution or a uniform distribution.

[0154] The adversarial loss module can improve the realism of the restored image.

[0155] The pixel-level loss module can calculate the loss value L using formula (5). L1 .

[0156] Where ||·||1 represents the L1 norm, and λ2 represents the weight of the pixel-level loss, which is used to adjust the weight of this loss in the overall loss function.

[0157] The pixel-level loss module is used to calculate the pixel-level loss between the generated and calibrated images, so that the restored image is as similar as possible to the corresponding real image in pixel space.

[0158] The loss perception module can calculate the loss value L using formula (6). perceptual .

[0159] Where F(·) represents the feature vector obtained by inputting the corresponding image into a neural network model pre-trained on ImageNet, where the neural network model can be VGGNet, and λ3 represents the weight of the perceptual loss, which is used to adjust the weight of this loss in the overall loss function.

[0160] The perceptual loss module ensures that the image restoration result is semantically consistent with the corresponding real image by comparing high-level perceptual and semantic differences between images.

[0161] The multimodal constraint module can calculate the loss value L using formula (3). multimodal .

[0162] For each image to be repaired After normalizing the Euclidean distances between a set number (N) of random noise samples z using the softmax function, the probability distribution is denoted as P. i The P corresponding to the i-th random noise sample among N random noise samples. i It can be determined by the following formula (1).

[0163] For images generated during training Similarly, the softmax function is used for probability normalization, and the probability distribution is denoted as Q. i Q is the value corresponding to the i-th generated image out of N generated images. i It can be determined by the following formula (2).

[0164] Based on P using formula (3) i and Q i Determine L multimodal .

[0165] Where λ1 represents the weights of the multimodal loss function, and the KL divergence is used to calculate the probability P. i With Qi The expected value of the logarithmic difference.

[0166] The multimodal constraint module excites the generative adversarial network to generate diverse image inpainting results based on the input random noise, thereby solving the mode collapse problem in image inpainting.

[0167] Thus, the loss function of the image restoration model can be calculated using the following formula (8).

[0168] L total =L adv +L L1 +L percetual +L multimodal (8)

[0169] Step 304: Input the image to be repaired into the trained image repair model.

[0170] After the image restoration model is trained, the image to be restored is input into the network restoration module, and multiple random noise z are sampled from the prior distribution to generate diverse image restoration results.

[0171] Since the image inpainting dataset used for training is constructed by applying a random mask to the complete images in the training set, for CGAN-based image inpainting methods, each image to be inpainted in the training set corresponds to only one calibration result (real image). CGAN cannot learn the conditional distribution under the conditions of the image to be inpainted. The trained model can only obtain one inpainting result for each image to be inpainted, and cannot obtain various semantically meaningful and reasonable inpainting results.

[0172] Furthermore, GANs suffer from pattern collapse, resulting in similar restoration outcomes that fail to produce diverse results. However, in some application scenarios, it is possible for a single image to be restored to correspond to multiple reasonable restoration results. For example, in an image of a damaged nose region on a face, the image that meets the user's needs could be either a high nose bridge or a low nose bridge. In this case, the inability to produce diverse restoration results fails to meet the requirements of image restoration.

[0173] In this application embodiment, an image inpainting model driven by prior noise is designed. By perturbing the input noise, visually realistic, semantically consistent, and diverse image inpainting results are obtained for the same image to be inpainted. Furthermore, the network framework and loss function based on the image inpainting model solve the pattern collapse problem in the field of image inpainting applications, thereby achieving diversity in image inpainting results.

[0174] To implement the method of this application embodiment, this application embodiment also provides an image restoration model training device, which is disposed on a first electronic device, such as... Figure 5As shown, the device includes:

[0175] The generation unit 501 is used to input the first image to be repaired and the i-th random noise from N random noises into the generation network of the image repair model to generate the i-th second image;

[0176] The first processing unit 502 is configured to determine a first loss value based on the relative entropy of a first parameter and a second parameter; the first parameter is determined based on the normalization result of the i-th random noise and the N random noises; the second parameter is determined based on the normalization result of the i-th second image and the N second images generated corresponding to the N random noises; where N is an integer greater than 1.

[0177] Training unit 503 is used to update the weight parameters of the image restoration model based on the determined first loss value.

[0178] In one embodiment, the device further includes:

[0179] The second processing unit is used to classify the i-th second image through the discriminative network of the image restoration model, and determine a second loss value based on the classification result of the i-th second image and the classification result of the third image; the third image represents the calibration result corresponding to the first image;

[0180] The third processing unit is used to determine a third loss value based on the i-th second image and the third image;

[0181] The training unit 503 is used for:

[0182] The weight parameters of the image restoration model are updated based on the determined first loss value, second loss value, and third loss value.

[0183] In one embodiment, the third processing unit is configured to:

[0184] The fourth loss value is determined based on the norm of the i-th second image and the third image;

[0185] A first feature vector is determined based on the i-th second image, a second feature vector is determined based on the third image, and a fifth loss value is determined based on the norm of the vector difference between the first feature vector and the second feature vector.

[0186] The third loss value is determined based on the fourth loss value and the fifth loss value.

[0187] In one embodiment, the device further includes:

[0188] The fourth processing unit is used to determine the fourth image from the set sample image library;

[0189] The fifth processing unit is used to perform masking processing on the determined set area of ​​the fourth image to obtain the first image.

[0190] In one embodiment, the fourth processing unit is configured to:

[0191] The fifth image in the set sample image library is subjected to target detection, and the fifth image is cropped based on the target rectangle located during the target detection process to determine the fourth image.

[0192] In practical applications, the generation unit 501, the first processing unit 502, the training unit 503, the second processing unit, the third processing unit, the fourth processing unit, and the fifth processing unit can be implemented by processors in the image restoration model training device, such as central processing units (CPUs), digital signal processors (DSPs), microcontroller units (MCUs), or field-programmable gate arrays (FPGAs).

[0193] It should be noted that the image restoration model training device provided in the above embodiments is only illustrated by the division of the above-described program modules when training the image restoration model. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the image restoration model training device and the image restoration model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0194] To implement the method of the embodiments of this application, the embodiments of this application also provide an image restoration device, which is disposed on a second electronic device, such as... Figure 6 As shown, the device includes:

[0195] Repair unit 601 is used to input the sixth image to be repaired and N random noises into the generator network of the image repair model, and output N seventh images; wherein,

[0196] The image restoration model is trained using any of the image restoration model training methods described above.

[0197] In practical applications, the repair unit 601 can be implemented by a processor in the image repair device, such as a CPU, DSP, MCU or FPGA.

[0198] It should be noted that the image restoration device provided in the above embodiments is only illustrated by the division of the above-described program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. In addition, the image restoration device and the image restoration method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0199] Based on the hardware implementation of the above program modules, and in order to implement the image restoration model training method of this application embodiment, this application embodiment also provides a first electronic device, such as... Figure 7 As shown, the first electronic device 700 includes:

[0200] The first communication interface 701 is capable of exchanging information with other network nodes;

[0201] The first processor 702 is connected to the first communication interface 701 to enable information interaction with other network nodes. When running a computer program, it executes the methods provided by one or more technical solutions on the first electronic device side. The computer program is stored in the first memory 703.

[0202] Specifically, the first processor 702 is used to input the first image to be repaired and the i-th random noise from N random noises into the generative network of the image repair model to generate the i-th second image;

[0203] The first loss value is determined based on the relative entropy of the first parameter and the second parameter; the first parameter is determined based on the normalized result of the i-th random noise and the N random noises; the second parameter is determined based on the normalized result of the i-th second image and the N second images generated corresponding to the N random noises; where N is an integer greater than 1.

[0204] The weight parameters of the image restoration model are updated based on the determined first loss value.

[0205] In one embodiment, the first processor 702 is configured to classify the i-th second image using the discriminative network of the image restoration model, and determine a second loss value based on the classification result of the i-th second image and the classification result of the third image; the third image represents the calibration result corresponding to the first image.

[0206] Based on the i-th second image and the third image, a third loss value is determined;

[0207] The step of updating the weight parameters of the image restoration model based on the determined first loss value includes:

[0208] The weight parameters of the image restoration model are updated based on the determined first loss value, second loss value, and third loss value.

[0209] In one embodiment, the first processor 702 is configured to determine a fourth loss value based on the norm of the i-th second image and the third image;

[0210] A first feature vector is determined based on the i-th second image, a second feature vector is determined based on the third image, and a fifth loss value is determined based on the norm of the vector difference between the first feature vector and the second feature vector.

[0211] The third loss value is determined based on the fourth loss value and the fifth loss value.

[0212] In one embodiment, the first processor 702 is configured to determine a fourth image from a set sample image library;

[0213] The determined fourth image is masked to obtain the first image.

[0214] In one embodiment, the first processor 702 is configured to perform target detection on the fifth image in the set sample image library, and crop the fifth image based on the target bounding box located during the target detection process to determine the fourth image.

[0215] It should be noted that the specific processing procedures of the first processor 702 and the first communication interface 701 can be understood by referring to the above method.

[0216] Of course, in practical applications, the various components in the first electronic device 700 are coupled together via a bus system 704. It can be understood that the bus system 704 is used to realize communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 7 The general designated all buses as Bus System 704.

[0217] The first memory 703 in this embodiment is used to store various types of data to support the operation of the first electronic device 700. Examples of such data include any computer program used to operate on the first electronic device 700.

[0218] The methods disclosed in the above embodiments of this application can be applied to the first processor 702, or implemented by the first processor 702. The first processor 702 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the first processor 702. The first processor 702 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The first processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the first memory 703. The first processor 702 reads the information in the first memory 703 and completes the steps of the aforementioned method in combination with its hardware.

[0219] Optionally, when the first processor 702 executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0220] In an exemplary embodiment, the first electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0221] Based on the hardware implementation of the above program modules, and in order to implement the image restoration method of this application embodiment, this application embodiment also provides a second electronic device, such as... Figure 8 As shown, the second electronic device 800 includes:

[0222] The second communication interface 801 is capable of exchanging information with other network nodes;

[0223] The second processor 802 is connected to the second communication interface 801 to enable information interaction with other network nodes and to execute the image restoration method provided by the above-mentioned technical solution when running a computer program. The computer program is stored in the second memory 803.

[0224] Specifically, the second processor 802 is used to input the sixth image to be repaired and N random noises into the generator network of the image repair model, and output N seventh images; wherein,

[0225] The image restoration model is trained using any of the image restoration model training methods described above.

[0226] It should be noted that the specific processing procedures of the second processor 802 and the second communication interface 801 can be understood by referring to the above method.

[0227] Of course, in practical applications, the various components in the second electronic device 800 are coupled together through the bus system 804. It can be understood that the bus system 804 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, in... Figure 8 The general labeled all buses as Bus System 804.

[0228] The second memory 803 in this embodiment is used to store various types of data to support the operation of the second electronic device 800. Examples of such data include any computer program used to operate on the second electronic device 800.

[0229] The methods disclosed in the embodiments of this application can be applied to the second processor 802, or implemented by the second processor 802. The second processor 802 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in the second processor 802. The second processor 802 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The second processor 802 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in the second memory 803. The second processor 802 reads the information in the second memory 803 and completes the steps of the aforementioned method in combination with its hardware.

[0230] Optionally, when the second processor 802 executes the program, it implements the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0231] In an exemplary embodiment, the second electronic device 800 may be implemented by one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0232] It is understood that the memories (first memory 703, second memory 803) in the embodiments of this application can be volatile memories or non-volatile memories, or both. Non-volatile memories can be read-only memories (ROM), programmable read-only memories (PROM), erasable programmable read-only memories (EPROM), electrically erasable programmable read-only memories (EEPROM), magnetic random access memories (FRAM), flash memories, magnetic surface memories, optical discs, or compact disc read-only memories (CD-ROM); magnetic surface memories can be disk storage or magnetic tape storage. Volatile memories can be random access memories (RAM), which are used as external caches. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0233] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a first memory 703 storing a computer program, which can be executed by a first processor 702 of a first electronic device 700 to complete the steps described in the aforementioned method on the first electronic device side. Another example is a second memory 803 storing a computer program, which can be executed by a second processor 802 of a second electronic device 800 to complete the steps described in the aforementioned method on the second electronic device side. The computer-readable storage medium can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0234] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0235] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0236] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A method for training an image restoration model, characterized in that, include: The first image to be repaired and the i-th random noise from N random noises are input into the generative network of the image repair model to generate the i-th second image; The first loss value is determined based on the relative entropy of the first and second parameters; The first parameter is determined based on the normalization result of the i-th random noise and the N random noises; The second parameter is determined based on the normalization result of the N second images generated corresponding to the i-th second image and the N random noises; where N is an integer greater than 1. Based on the determined first loss value, the weight parameters of the image restoration model are updated; Among the N random noises, the i-th first parameter corresponds to the i-th random noise. Through formula It is confirmed that, among them, Let i be the i-th random noise in N random noises. The j-th random noise in N random noises; the i-th second parameter corresponding to the i-th generated image in N generated images. Through formula It is confirmed that, among them, The generative network for the image inpainting model is based on the first image to be inpainted. The second image generated from the i-th random noise, The generative network for the image inpainting model is based on the first image to be inpainted. The j-th second image is generated from the j-th random noise.

2. The image restoration model training method according to claim 1, before updating the weight parameters of the image restoration model based on the determined first loss value, the method further includes: The discriminant network of the image restoration model is used to classify the i-th second image. Based on the classification result of the i-th second image and the classification result of the third image, a second loss value is determined. The third image represents the calibration result corresponding to the first image. Based on the i-th second image and the third image, a third loss value is determined; The step of updating the weight parameters of the image restoration model based on the determined first loss value includes: The weight parameters of the image restoration model are updated based on the determined first loss value, second loss value, and third loss value.

3. The image restoration model training method according to claim 2, wherein determining the third loss value based on the i-th second image and the third image includes: The fourth loss value is determined based on the norm of the i-th second image and the third image; A first feature vector is determined based on the i-th second image, a second feature vector is determined based on the third image, and a fifth loss value is determined based on the norm of the vector difference between the first feature vector and the second feature vector. The third loss value is determined based on the fourth loss value and the fifth loss value.

4. The image inpainting model training method according to claim 1, further comprising, before inputting the first image to be inpainted and the i-th random noise from N random noises into the generator network of the image inpainting model: Determine the fourth image from the established sample image library; The determined fourth image is masked to obtain the first image.

5. The image restoration model training method according to claim 4, wherein determining the fourth image from the set sample image library includes: The fifth image in the set sample image library is subjected to target detection, and the fifth image is cropped based on the target rectangle located during the target detection process to determine the fourth image.

6. An image restoration method, characterized in that, include: The sixth image to be repaired and N random noises are input into the generator network of the image repair model, which outputs N seventh images; where, The image restoration model is trained using the image restoration model training method described in any one of claims 1 to 5.

7. An image restoration model training device, characterized in that, include: The generation unit is used to input the first image to be repaired and the i-th random noise from N random noises into the generation network of the image repair model to generate the i-th second image; The first processing unit is used to determine the first loss value based on the relative entropy of the first parameter and the second parameter; The first parameter is determined based on the normalization result of the i-th random noise and the N random noises; The second parameter is determined based on the normalization result of the N second images generated corresponding to the i-th second image and the N random noises; where N is an integer greater than 1. The training unit is used to update the weight parameters of the image restoration model based on the determined first loss value; Among the N random noises, the i-th first parameter corresponds to the i-th random noise. Through formula It is confirmed that, among them, Let i be the i-th random noise in N random noises. The j-th random noise in N random noises; the i-th second parameter corresponding to the i-th generated image in N generated images. Through formula It is confirmed that, among them, The generative network for the image inpainting model is based on the first image to be inpainted. The second image generated from the i-th random noise, The generative network for the image inpainting model is based on the first image to be inpainted. The j-th second image is generated from the j-th random noise.

8. An image restoration device, characterized in that, include: The inpainting unit is used to input the sixth image to be inpainted and N random noises into the generator network of the image inpainting model, and output N seventh images; where, The image restoration model is trained using the image restoration model training method described in any one of claims 1 to 5.

9. A first electronic device, characterized in that, include: A first processor and a first communication interface; wherein... The first processor is configured to input the first image to be repaired and the i-th random noise from N random noises into the generative network of the image repair model to generate the i-th second image; The first loss value is determined based on the relative entropy of the first parameter and the second parameter; the first parameter is determined based on the normalized result of the i-th random noise and the N random noises; the second parameter is determined based on the normalized result of the i-th second image and the N second images generated corresponding to the N random noises; where N is an integer greater than 1. Based on the determined first loss value, update the weight parameters of the image restoration model; Among the N random noises, the i-th first parameter corresponds to the i-th random noise. Through formula It is confirmed that, among them, Let i be the i-th random noise in N random noises. The j-th random noise in N random noises; the i-th second parameter corresponding to the i-th generated image in N generated images. Through formula It is confirmed that, among them, The generative network for the image inpainting model is based on the first image to be inpainted. The second image generated from the i-th random noise, The generative network for the image inpainting model is based on the first image to be inpainted. The j-th second image is generated from the j-th random noise.

10. A second electronic device, characterized in that, include: A second processor and a second communication interface; wherein... The second processor is used to input the sixth image to be repaired and N random noises into the generator network of the image repair model, and output N seventh images; wherein, The image restoration model is trained using the image restoration model training method described in any one of claims 1 to 5.

11. A first electronic device, characterized in that, include: A first processor and a first memory for storing computer programs capable of running on the processor. Wherein, when the first processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 5.

12. A second electronic device, characterized in that, include: A second processor and a second memory for storing computer programs that can run on the processor. Wherein, when the second processor is used to run the computer program, it executes the steps of the method of claim 6.

13. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5, or the steps of the method according to claim 6.

Images