Image inpainting model training method, image inpainting method, device and electronic equipment

Abstract

This invention discloses an image inpainting model training method, image inpainting method, apparatus, and electronic device, relating to the field of image processing technology, to solve the problem of poor image inpainting effects in related image inpainting schemes. The method includes: acquiring a lossless image sample set; determining a first grayscale image and a first contour image of a lost sample image corresponding to the lossless sample image based on the lossless sample image and a first mask in the lossless image sample set; inputting the first grayscale image, the first contour image, the first mask, and the lost sample image into an image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network; determining a loss function value based on the inpainted image and the lossless sample image; and training the parameters of the image inpainting network based on the loss function value. This invention ensures that the trained image inpainting network can accurately locate the lost region in the image, achieving better image inpainting results.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image restoration model training method, image restoration method, apparatus, and electronic device. Background Technology

[0002] With the advancement of computer technology and the development of machine learning, various methods for repairing image defects now exist. Currently, deep learning-based image inpainting technology has become the mainstream trend. However, the image inpainting solutions in these technologies often suffer from poor image restoration results due to inaccurate localization of the lost image areas. Summary of the Invention

[0003] This invention provides an image restoration model training method, an image restoration method, an apparatus, and an electronic device to address the problem of poor image restoration performance in related technologies.

[0004] In a first aspect, embodiments of the present invention provide an image restoration model training method, comprising:

[0005] Obtain a lossless image sample set;

[0006] Based on the lossless sample images and the first mask in the lossless image sample set, determine the first grayscale image and the first contour image of the loss sample image corresponding to the lossless sample image;

[0007] The first grayscale image, the first contour image, the first mask, and the lost sample image are input into an image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network.

[0008] Based on the repaired image and the lossless sample image, determine the loss function value;

[0009] The parameters of the image inpainting network are trained based on the loss function value.

[0010] Optionally, the first grayscale image is equal to the Hadamard product of the matrix corresponding to the grayscale image of the lossless sample image and the first matrix, and the first contour image is equal to the Hadamard product of the matrix corresponding to the contour image of the lossless sample image and the first matrix. The first matrix is ​​equal to the difference between the matrix with all elements set to 1 and the matrix corresponding to the first mask. The matrix corresponding to the first mask uses element value 1 to represent the loss region and element value 0 to represent the non-loss region.

[0011] Optionally, the image restoration network includes a contour completion network and a color filling network;

[0012] The step of inputting the first grayscale image, the first contour image, the first mask, and the lost sample image into an image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network includes:

[0013] The first grayscale image, the first contour image, and the first mask are input into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network.

[0014] The lost sample image and the third contour map corresponding to the lost region in the lost sample image are input into the color filling network to obtain the repaired image of the lost sample image output by the color filling network.

[0015] The step of determining the loss function value based on the repaired image and the lossless sample image includes:

[0016] Based on the second contour map and the first contour map, determine the contour loss function value;

[0017] Based on the repaired image and the lossless sample image, determine the image loss function value;

[0018] The step of training the parameters of the image inpainting network based on the loss function value includes:

[0019] Based on the contour loss function value, the parameters of the contour completion network are trained;

[0020] The parameters of the color filling network are trained based on the image loss function value.

[0021] Optionally, determining the image loss function value based on the repaired image and the lossless sample image includes:

[0022] The L1 norm loss function value is determined based on the difference between each pixel in the repaired image and the lossless sample image, the loss region in the lossless sample image, and the number of pixels in the lossless sample image.

[0023] Optionally, the L1 norm loss function value includes the L1 loss value of the loss region of the loss sample image and the L1 loss value of the non-loss region of the loss sample image;

[0024] Wherein, the L1 loss value of the non-loss region is equal to the L1 norm of the Hadamard product of the matrix corresponding to the second mask and the second matrix divided by the number of pixels, the second matrix is ​​equal to the difference between the matrix corresponding to the repaired image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss region in the loss sample image, and the matrix corresponding to the second mask uses element value 0 to represent the loss region and element value 1 to represent the non-loss region.

[0025] The L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of the third matrix and the second matrix divided by the number of pixels, wherein the third matrix is ​​equal to the difference between the matrix with all elements equal to 1 and the matrix corresponding to the second mask.

[0026] Optionally, determining the image loss function value based on the repaired image and the lossless sample image includes:

[0027] Based on the restored image and the lossless sample image, the L1 norm loss function value, the perceptual loss function value, and the style loss function value are calculated respectively.

[0028] The image loss function value is obtained by calculating the weighted sum of the L1 norm loss function value, the perceptual loss function value, and the style loss function value.

[0029] Optionally, the image restoration network is a U-shaped network based on partial convolution.

[0030] Optionally, the image inpainting network is a generative adversarial network, and the generator of the generative adversarial network is a U-shaped network based on partial convolution.

[0031] Optionally, both the generator and the discriminator in the generative adversarial network are trained using spectral normalization.

[0032] Secondly, embodiments of the present invention also provide an image restoration method, comprising:

[0033] Obtain the target image to be repaired;

[0034] The target image is input into an image restoration network to obtain the image restored by the image restoration network.

[0035] The image restoration network is a network trained using the image restoration model training method described in the first aspect above.

[0036] Thirdly, embodiments of the present invention also provide an image restoration model training apparatus, comprising:

[0037] The first acquisition module is used to acquire a lossless image sample set;

[0038] The first determining module is used to determine, based on the lossless sample images and the first mask in the lossless image sample set, a first grayscale image and a first contour image of the lossless sample image corresponding to the lossless sample image.

[0039] The processing module is used to input the first grayscale image, the first contour image, the first mask and the lost sample image into the image inpainting network, and obtain the inpainted image of the lost sample image output by the image inpainting network;

[0040] The second determining module is used to determine the loss function value based on the repaired image and the lossless sample image;

[0041] The training module is used to train the parameters of the image inpainting network based on the loss function value.

[0042] Optionally, the first grayscale image is equal to the Hadamard product of the matrix corresponding to the grayscale image of the lossless sample image and the first matrix, and the first contour image is equal to the Hadamard product of the matrix corresponding to the contour image of the lossless sample image and the first matrix. The first matrix is ​​equal to the difference between the matrix with all elements set to 1 and the matrix corresponding to the first mask. The matrix corresponding to the first mask uses element value 1 to represent the loss region and element value 0 to represent the non-loss region.

[0043] Optionally, the image restoration network includes a contour completion network and a color filling network;

[0044] The processing module includes:

[0045] The first processing unit is configured to input the first grayscale image, the first contour image, and the first mask into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network.

[0046] The second processing unit is used to input the lost sample image and the third contour map corresponding to the lost region in the lost sample image into the color filling network to obtain the repaired image of the lost sample image output by the color filling network.

[0047] The second determining module includes:

[0048] The first determining unit is used to determine the contour loss function value based on the second contour map and the first contour map;

[0049] The second determining unit is used to determine the image loss function value based on the repaired image and the lossless sample image;

[0050] The training module includes:

[0051] The first training unit is used to train the parameters of the contour completion network based on the contour loss function value;

[0052] The second training unit is used to train the parameters of the color filling network based on the image loss function value.

[0053] Optionally, the second determining unit is used to determine the L1 norm loss function value based on the difference between each pixel in the repaired image and the lossless sample image, the loss region in the lossless sample image, and the number of pixels in the lossless sample image.

[0054] Optionally, the L1 norm loss function value includes the L1 loss value of the loss region of the loss sample image and the L1 loss value of the non-loss region of the loss sample image;

[0055] Wherein, the L1 loss value of the non-loss region is equal to the L1 norm of the Hadamard product of the matrix corresponding to the second mask and the second matrix divided by the number of pixels, the second matrix is ​​equal to the difference between the matrix corresponding to the repaired image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss region in the loss sample image, and the matrix corresponding to the second mask uses element value 0 to represent the loss region and element value 1 to represent the non-loss region.

[0056] The L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of the third matrix and the second matrix divided by the number of pixels, wherein the third matrix is ​​equal to the difference between the matrix with all elements equal to 1 and the matrix corresponding to the second mask.

[0057] Optionally, the second determining unit includes:

[0058] The first calculation subunit is used to calculate the L1 norm loss function value, the perceptual loss function value, and the style loss function value based on the restored image and the lossless sample image, respectively.

[0059] The second calculation subunit is used to calculate the weighted sum of the L1 norm loss function value, the perceptual loss function value, and the style loss function value to obtain the image loss function value.

[0060] Optionally, the image restoration network is a U-shaped network based on partial convolution.

[0061] Optionally, the image inpainting network is a generative adversarial network, and the generator of the generative adversarial network is a U-shaped network based on partial convolution.

[0062] Optionally, both the generator and the discriminator in the generative adversarial network are trained using spectral normalization.

[0063] Fourthly, embodiments of the present invention also provide an image restoration apparatus, comprising:

[0064] The second acquisition module is used to acquire the target image to be repaired;

[0065] The repair module is used to input the target image into the image repair network and obtain the repaired image output by the image repair network.

[0066] The image restoration network is a network trained using the image restoration model training method described in the first aspect above.

[0067] Fifthly, embodiments of the present invention also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the image restoration model training method described above; or to implement the steps in the image restoration method described above.

[0068] In a sixth aspect, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the image restoration model training method described above; or implements the steps in the image restoration method described above.

[0069] In this embodiment of the invention, a lossless image sample set is obtained; based on the lossless sample images and a first mask in the lossless image sample set, a first grayscale image and a first contour image of the loss sample image corresponding to the lossless sample image are determined; the first grayscale image, the first contour image, the first mask, and the loss sample image are input into an image inpainting network to obtain the inpainted image of the loss sample image output by the image inpainting network; based on the inpainted image and the lossless sample image, a loss function value is determined; based on the loss function value, the parameters of the image inpainting network are trained. Thus, by training the image inpainting network using the grayscale image of the lossless original image, the loss contour image, and the mask, it can be ensured that the trained image inpainting network can accurately locate the loss region in the image and has high accuracy in recognizing image edges, thereby achieving a better image inpainting effect. Attached Figure Description

[0070] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart of the image restoration model training method provided in this embodiment of the invention;

[0072] Figure 2 This is a schematic diagram of the U-shaped network structure provided in an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the model training process provided in an embodiment of the present invention;

[0074] Figure 4 This is a flowchart of the image restoration method provided in the embodiments of the present invention;

[0075] Figure 5 This is a structural diagram of the image restoration model training device provided in an embodiment of the present invention;

[0076] Figure 6 This is a structural diagram of the image restoration device provided in an embodiment of the present invention;

[0077] Figure 7 This is a structural diagram of the electronic device provided in an embodiment of the present invention;

[0078] Figure 8 This is a structural diagram of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0079] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0080] See Figure 1 , Figure 1 This is a flowchart of the image restoration model training method provided in the embodiments of the present invention, such as... Figure 1 As shown, it includes the following steps:

[0081] Step 101: Obtain a lossless image sample set.

[0082] In this embodiment of the invention, to train an image inpainting model capable of accurately identifying and effectively repairing lost regions in an image, it is necessary to train the model based on a lossless image sample set and a corresponding lost image sample set. Therefore, during pre-training, a lossless image sample set can be obtained first. This lost image sample set can be obtained by loading masks onto the lossless sample images in the lossless image sample set to simulate lost regions. In one implementation, masks of different sizes and shapes can be used to generate lost sample images of randomly sized and shaped lost regions from each lossless sample image, serving as the lost image sample set to ensure the robustness of the trained image inpainting model.

[0083] Step 102: Based on the lossless sample images and the first mask in the lossless image sample set, determine the first grayscale image and the first contour image of the loss sample image corresponding to the lossless sample image.

[0084] Before training begins, the lossless sample images in the lossless image sample set can be preprocessed. Specifically, based on the lossless sample images and the first mask, the corresponding loss sample images can be determined, and the loss sample images can be converted into grayscale images to obtain a first grayscale image. Additionally, the contour maps of the loss regions in the loss sample images can be determined to obtain a first contour map.

[0085] The aforementioned first mask can refer to a mask used to simulate the loss region. This mask can be fixed or randomly generated. Specifically, the loss sample image can be obtained by loading the first mask onto the lossless sample image, and the loss sample image can be converted into a grayscale image to obtain the first grayscale image. Alternatively, the lossless sample image can be converted into a grayscale image first, and then the first mask can be loaded onto the grayscale image of the lossless sample image to obtain the first grayscale image.

[0086] Step 103: Input the first grayscale image, the first contour image, the first mask, and the lost sample image into the image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network.

[0087] After determining the first grayscale image and the first contour image of the loss sample image corresponding to the lossless sample image, the first grayscale image, the first contour image, the first mask, and the loss sample image can be used as input data into the image inpainting network. The image inpainting network processes the input data, that is, it identifies and repairs the loss region in the loss sample image. After processing, the predicted image output by the image inpainting network is obtained, that is, the repaired image after repairing the loss sample image is output. The first contour image and the first mask can be used as limiting conditions to train the network to generate a loss region contour image that is as close as possible to the first contour image (the true loss region contour) when identifying the loss region.

[0088] The initial training model of the image restoration network can be selected according to actual needs. For example, it can be a neural network model such as a convolutional network, a partially convolutional network, a U-shaped network, or a generative adversarial network, or it can be a combination of multiple neural networks.

[0089] Optionally, the first grayscale image is equal to the Hadamard product of the matrix corresponding to the grayscale image of the lossless sample image and the first matrix, and the first contour image is equal to the Hadamard product of the matrix corresponding to the contour image of the lossless sample image and the first matrix. The first matrix is ​​equal to the difference between the matrix with all elements set to 1 and the matrix corresponding to the first mask. The matrix corresponding to the first mask uses element value 1 to represent the loss region and element value 0 to represent the non-loss region.

[0090] In one embodiment, the lossless sample image can be converted into a grayscale image, and then a Hadamard product operation (i.e., multiplying corresponding elements in the two matrices) can be performed on the grayscale image of the lossless sample image and the first mask in matrix form to obtain the grayscale image matrix of the lossless sample image. Alternatively, the contour map matrix of the lossless sample image can be obtained by calculating the contour map of the lossless sample image and performing a Hadamard product operation on the contour map of the lossless sample image and the first mask in matrix form. The contour map calculation of the lossless sample image can be performed using the Canny operator.

[0091] Specifically, matrix I can be used. gt The lossless sample image represents the grayscale image and the contour image of the lossless sample image, respectively represented by matrix C. gt Sum matrix I gray In other words, matrix M is the first mask, and the first grayscale image of the loss sample image corresponding to the lossless sample image is... The first contour map of the lost sample image In matrix M, an element with a value of 1 represents the loss region, and an element with a value of 0 represents the non-loss region.

[0092] The image inpainting network can be based on the first grayscale image. First contour map And the first mask M, generate a contour map C of the loss region in the loss sample image. pred The specific relation can be, The image inpainting network can also be based on the generated contour map C of the lost region. pred The image inpainting network uses the image features in the lost sample image to repair the lost region, and can use the corresponding lossless sample image as a limiting condition to encourage the image inpainting network to generate a repaired image that is as similar as possible to the lossless sample image (lossless original image).

[0093] In this way, since the method of using the Hadamard product of matrices to calculate the grayscale and contour maps of the lost sample images is not a simple mapping relationship, it can ensure the accuracy of image edge recognition and the accurate localization of the image loss region.

[0094] Step 104: Determine the loss function value based on the repaired image and the lossless sample image.

[0095] After obtaining the restored image output by the image restoration network, the restored image can be compared with the lossless sample image, and the loss function value can be calculated using a loss function. The loss function can be selected according to actual needs. For example, the L1 norm loss function, style loss function, perceptual loss function, etc. can be selected to calculate the loss function value, or a combination of multiple loss functions can be used for calculation.

[0096] Step 105: Train the parameters of the image inpainting network based on the loss function value.

[0097] After calculating the loss function value, the parameters of the image restoration network can be trained based on this loss function value. If the loss function value does not meet the requirements, such as when the loss function value is too large, the parameters of the image restoration network can be adjusted, and the image restoration network can be iterated and trained again as in steps 102 to 105 until the loss function value meets the requirements and is minimized, thus obtaining the trained image restoration network.

[0098] Optionally, the image restoration network includes a contour completion network and a color filling network;

[0099] Step 103 includes:

[0100] The first grayscale image, the first contour image, and the first mask are input into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network.

[0101] The lost sample image and the third contour map corresponding to the lost region in the lost sample image are input into the color filling network to obtain the repaired image of the lost sample image output by the color filling network.

[0102] Step 104 includes:

[0103] Based on the second contour map and the first contour map, determine the contour loss function value;

[0104] Based on the repaired image and the lossless sample image, determine the image loss function value;

[0105] Step 105 includes:

[0106] Based on the contour loss function value, the parameters of the contour completion network are trained;

[0107] The parameters of the color filling network are trained based on the image loss function value.

[0108] In one embodiment, the image restoration network can be composed of two networks: a contour completion network and a color filling network. The contour completion network is used to identify the contours of the lost regions in the input lost image, and the color filling network is used to repair the lost regions output by the contour completion network, ultimately outputting a restored image.

[0109] Thus, after obtaining the first grayscale image and the first contour image of the loss sample image, the first grayscale image, the first contour image, and the first mask can be input into the contour completion network to generate a second contour image of the loss region in the loss sample image. For example, the contour completion network can be based on the input first grayscale image, the first contour image, and the first mask, and the relational expression... Output the contour map C predicted for the loss region in the lost sample image. pred .

[0110] Then, by comparing the second contour image output by the contour completion network with the true original image contour (i.e., the first contour image), the contour loss function value can be calculated using a loss function. Based on this contour loss function value, the parameters of the contour completion network can be trained. If the contour loss function value does not meet the requirements, such as if the contour loss function value is too large, the parameters of the contour completion network can be adjusted, and the contour completion network can be trained again until the contour loss function value meets the requirements. This ensures that the contour generated by the contour completion network is as close as possible to the true original image contour, ultimately resulting in a well-trained contour completion network.

[0111] The color filling network can be trained together with the contour completion network or separately. When trained together, the second contour map output by the contour completion network and the loss sample image with the loss region can be input into the color filling network. When trained separately, the loss sample image with the outline of the loss region can be input into the color filling network. That is, the third contour map can be either the second contour map output by the contour completion network or the contour map marked in the loss sample image. The color filling network can generate a repaired image of the loss region in the loss sample image based on the loss sample image and the third contour map.

[0112] Next, by comparing the repaired image output by the color fill network with the lossless original image (i.e., the lossless sample image), the image loss function value can be calculated using a loss function. Based on this image loss function value, the parameters of the color fill network can be trained. If the image loss function value does not meet the requirements, such as if the image loss function value is too large, the parameters of the color fill network can be adjusted, and the color fill network can be trained again until the image loss function value meets the requirements. This ensures that the color distribution of the repaired image generated by the color fill network is as close as possible to the lossless original image, ultimately resulting in a well-trained color fill network.

[0113] It should be noted that when training the contour completion network and the color filling network together, the parameters of the contour completion network and the color filling network can be adjusted based on the contour loss function value and the image loss function value, respectively, and then they can be entered into the next iteration training process together.

[0114] In this way, by using a joint network with two different functions, the image loss region can be identified more accurately, resulting in better image restoration. Furthermore, the use of a contour loss function not only aims to improve the reconstruction accuracy of individual pixels but also considers factors related to the overall structure, namely whether the restored image pixels can be well integrated into the surrounding environment pixels.

[0115] Optionally, determining the image loss function value based on the repaired image and the lossless sample image includes:

[0116] The L1 norm loss function value is determined based on the difference between each pixel in the repaired image and the lossless sample image, the loss region in the lossless sample image, and the number of pixels in the lossless sample image.

[0117] In one embodiment, for the color fill network, the L1 norm loss function can be used to calculate the output loss value of the network. The L1 norm loss is also called the Least Absolute Deviation (LAD) or Least Absolute Error (LAE). The Mean Absolute Error (MAE) is obtained by dividing the L1 norm loss by the number of pixel values ​​in a certain area. In this embodiment, MAE can be used to calculate the L1 norm loss function value of the color fill network.

[0118] Specifically, the difference between each pixel in the repaired image and the lossless sample image can be calculated. Then, by combining the loss region in the loss sample image and the number of pixels in the lossless sample image, the L1 norm loss function value, i.e., the mean absolute error, can be calculated.

[0119] In this way, by calculating the loss value of the restored image output by the color fill network at the image pixel level, it is possible to ensure that the color fill network trained based on the loss value has higher accuracy, thereby obtaining better image restoration results and avoiding a large number of problems such as blurring, distortion, and noise in the restoration.

[0120] Optionally, the L1 norm loss function value includes the L1 loss value of the loss region of the loss sample image and the L1 loss value of the non-loss region of the loss sample image;

[0121] Wherein, the L1 loss value of the non-loss region is equal to the L1 norm of the Hadamard product of the matrix corresponding to the second mask and the second matrix divided by the number of pixels, the second matrix is ​​equal to the difference between the matrix corresponding to the repaired image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss region in the loss sample image, and the matrix corresponding to the second mask uses element value 0 to represent the loss region and element value 1 to represent the non-loss region.

[0122] The L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of the third matrix and the second matrix divided by the number of pixels, wherein the third matrix is ​​equal to the difference between the matrix with all elements equal to 1 and the matrix corresponding to the second mask.

[0123] In one implementation, the image loss value of the color filling network can be calculated from both the loss of the lost region and the loss of the non-loss region.

[0124] Specifically, individual pixel reconstruction loss functions can be defined for the loss region and the non-loss region, respectively. The L1 loss function for the non-loss region of the lost sample image can be... The L1 loss function for the loss region of the lost sample image can be... Among them, L valid L1 is the loss value of the non-loss region. hole Let I be the L1 loss value of the loss region. gt This represents the original image without loss, i.e., the lossless sample image, I. out The image represents the repaired image predicted by the network, i.e., the output of the color-filling network. M is a second mask representing the loss region of the lost sample image, where the pixel value in the loss region is 0 and the pixel value in the non-loss region is 1. The lossless sample image I represents gt The number of pixels in the image is represented by C×H×W, where C, H, and W are the number of channels, image height, and image width, respectively.

[0125] In this way, the L1 loss value of the network output can be calculated quickly and accurately. By calculating the L1 loss value of the loss region and the non-loss region separately, and training the color filling network based on the calculated loss value, the global repair effect of the model can be guaranteed, and structural disorder of the repaired image can be avoided.

[0126] Optionally, determining the image loss function value based on the repaired image and the lossless sample image includes:

[0127] Based on the restored image and the lossless sample image, the L1 norm loss function value, the perceptual loss function value, and the style loss function value are calculated respectively.

[0128] The image loss function value is obtained by calculating the weighted sum of the L1 norm loss function value, the perceptual loss function value, and the style loss function value.

[0129] In another embodiment, multiple loss functions can be combined to calculate the image loss function value. Specifically, based on the restored image and the lossless sample image, the L1 norm loss function value, the perceptual loss function value, and the style loss function value can be calculated respectively, and the image loss function value can be determined by weighted summation of the L1 norm loss function value, the perceptual loss function value, and the style loss function value.

[0130] The perceptual loss function can be calculated based on the first pooling layer, the second pooling layer, and the third pooling layer (i.e., pool1 layer, pool2 layer, and pool3 layer) in the color filling network.

[0131] Style loss function refers to a loss function used in style transfer. Style loss is similar to perceptual loss, but before calculating the L1 loss value, each feature map generated by the color-filling network calculates its autocorrelation function value (i.e., the gram matrix). The gram matrix contains the relationships and connections between different features in the feature map, i.e., the style of that feature map. Based on this, the style loss between the feature map and the restored image output by the color-filling network can be calculated.

[0132] In addition, when weighting the L1 norm loss function value, the perceptual loss function value, and the style loss function value, the weight coefficient of each loss function value can be determined through multiple image restoration tests.

[0133] In this way, by combining loss functions that represent different image parameters to train the color filling network, the restoration effect of the network can be further improved, ensuring that the restored image has a better structure and is more in line with the user's visual perception.

[0134] Optionally, the image restoration network is a U-shaped network based on partial convolution.

[0135] In one implementation, a U-shaped network based on partial convolution can be used as the image inpainting network to ensure that the model learns the structural features of the image better, and to reduce the amount of computation and improve the model's processing efficiency.

[0136] The U-shaped network based on partial convolution performs image inpainting by using stacked partial convolution operations and mask update steps. This U-shaped network, based on partial convolution layers, replaces the convolutional layers in a traditional U-shaped network with partial convolutional layers. The U-shaped network structure is a variation of a fully convolutional neural network, containing only convolutional and pooling layers. In this embodiment, all traditional convolutional layers in the U-shaped network structure are replaced with partial convolutional layers to construct a variant of the U-shaped network.

[0137] Partial convolution and the mask update function together are called a partially convolutional layer. Partial convolution means that convolution is performed only in the effective region of the image (i.e., the part where the mask is 0), and the image mask iterates and shrinks as the network depth increases. In other words, both the image with the mask and the mask participate in training. The formula for partial convolution is:

[0138]

[0139] Where W is the weight of the convolution kernel, X is the feature (or pixel) value corresponding to the current convolution window, M is the binary mask corresponding to X, b is the corresponding offset value, x′ represents the output of the input image after convolution, and ⊙ represents pixel-wise multiplication. The scaling factor sum(1) / sum(M) is used to adjust the effective input variation by appropriate scaling. Matrix 1 and M have the same shape, such as 3×3 matrices, but all values ​​in matrix 1 are 1. After each partial convolution operation, the mask needs to be updated, and the mask update function formula is:

[0140]

[0141] Here, m′ represents the output of the input Mask after convolution. After partial convolution, the Mask is updated. The update rule can be: if at least one of the Mask values ​​corresponding to each pixel in the convolution window is 1, then update the Mask at the corresponding position after convolution to 1. As long as the network depth is sufficient, the Mask region size can be shrunk to 0.

[0142] In the U-shaped network based on partial convolution, the input image size and the Mask size are the same, as are the convolution kernel sizes. The difference is that the input image's convolution kernel is continuously updated, while the Mask's convolution kernel is always 1 and has no offset. This reduces the number of training parameters required for convolution, decreases resource consumption, and speeds up the training process. During Mask updates, each pixel in Mask takes either 1 or 0, without decimal values.

[0143] Compared to traditional convolutional layers, the partial convolutional layers in the U-shaped network based on partial convolution have masks. In traditional convolutional layers, every pixel participates in the convolution operation, while in partial convolutional layers, only pixels in the non-loss region participate in the convolution operation, thereby improving processing efficiency.

[0144] The structure of the U-shaped network can be as follows: Figure 2 As shown, the entire network structure is U-shaped. The structure from the leftmost point to the middle of the bottom of this U-shaped network can be considered an encoder, and the structure from the middle of the bottom to the rightmost point can be considered a decoder. Figure 2As shown, each layer from the encoder on the left to the decoder on the right has a concatenation connection. This concatenation connection refers to cropping the encoder's output and using it as input parameters to concatenate with the decoder's deconvolution result. Batch normalization can be introduced into this network structure to unify the scattered data, allowing the network to learn in its optimal state, which is especially necessary for neural networks with many layers in this model. Figure 2 The number of network layers shown is for illustrative purposes only; the actual number of U-shaped network layers can be designed according to requirements.

[0145] In the U-shaped network based on partial convolution, the Leaky ReLU activation function can be used in the decoder stage so that the network can still learn when the data input is negative and the neurons will not die.

[0146] In the last partially convolutional layer of the partially convolutional U-shaped network, the layer input values ​​include the original image with the loss region input at the beginning and the mask, so that the partially convolutional U-shaped network outputs the pixels of the unlossed part. Figure 2 Each jumper in the process connects the left and right feature maps and the mask respectively. After they are synthesized, they are spliced ​​into a feature map and a mask. Then, a new feature map is obtained through deconvolution. This new feature map and the updated mask are used as two new input parameters to provide to the next part of the convolutional layer. This ensures that the effective pixels in the initial input image can provide reference information for image restoration during the decoding stage.

[0147] In this way, by using a U-shaped network based on partial convolution as the image inpainting network, not only can the model processing efficiency be improved, but also good model inpainting performance can be guaranteed.

[0148] Optionally, the image inpainting network is a generative adversarial network, and the generator of the generative adversarial network is a U-shaped network based on partial convolution.

[0149] In one embodiment, a generative adversarial network (GAN) can be used as the image inpainting network, and a U-shaped network based on partial convolution can be used as the generator of the GAN to identify the image loss region and generate the inpainted image. The output of the U-shaped network based on partial convolution is used as the input of the discriminator of the GAN to determine whether the generated loss region contour map is real and whether the generated inpainted image is sufficiently similar to the original image without loss.

[0150] The discriminator network can be trained using objective evaluation metrics such as adversarial loss and feature matching loss. The feature matching loss compares the feature maps in the intermediate layers of the discriminator and forces the generator to produce results with features similar to real images by limiting the data generated by the intermediate layers, thereby stabilizing the training process and allowing the model to converge.

[0151] It should be noted that when the image restoration network comprises a contour completion network and a color filling network, both the contour completion network and the color filling network can employ generative adversarial networks (GANs), and both can use a partially convolutional U-shaped network as the generator. Thus, the training process for the contour completion network and the color filling network can be as follows: Figure 3 As shown.

[0152] In this way, by incorporating a U-shaped network with some convolutional layers into the network structure of a generative adversarial network, this implementation allows the model to learn the structural features of the image better and avoids the abstraction of invalid or even harmful regions, thereby further improving the model's repair effect.

[0153] Optionally, both the generator and the discriminator in the generative adversarial network are trained using spectral normalization.

[0154] In one implementation, both the generator and discriminator in the generative adversarial network can be trained using spectral normalization. This reduces the maximum values ​​of the adversarial loss, feature matching loss, etc., proportionally decreasing the weight matrix, thereby making the training process more stable and effectively limiting the network's Lipshitz constant to 1.

[0155] In this way, by using spectral normalization in the generator and discriminator, the amount of parameter and gradient change in a short period of time can be effectively limited, so that the adversarial loss can play its maximum role in the variance calculation of the discriminator, and the feature matching loss can effectively constrain the feature map trained by the generator.

[0156] The image restoration model training method of this invention involves: acquiring a lossless image sample set; determining a first grayscale image and a first contour image of a loss sample image corresponding to the lossless sample image based on the lossless sample image and a first mask; inputting the first grayscale image, the first contour image, the first mask, and the loss sample image into an image restoration network to obtain a restored image of the loss sample image output by the image restoration network; determining a loss function value based on the restored image and the lossless sample image; and training the parameters of the image restoration network based on the loss function value. In this way, by training the image restoration network using the grayscale image of the lossless original image, the loss contour image, and the mask, it is ensured that the trained image restoration network can accurately locate the loss region in the image and has high accuracy in identifying image edges, thereby achieving a better image restoration effect.

[0157] See Figure 4 , Figure 4 This is a flowchart of the image restoration method provided in the embodiments of the present invention, such as... Figure 4 As shown, it includes the following steps:

[0158] Step 401: Obtain the target image to be repaired.

[0159] When it is necessary to repair a lost image, the target image to be repaired can be obtained, that is, the target image contains a lost region.

[0160] Step 402: Input the target image into the image inpainting network to obtain the image output by the image inpainting network after inpainting the target image;

[0161] The image inpainting network is a network that... Figure 1 The network trained by the image restoration model training method in the illustrated embodiment.

[0162] In this step, the target image to be repaired can be input into an image inpainting network to identify and repair the lost regions of the target image through the image inpainting network, thereby obtaining the image after repair of the target image output by the image inpainting network.

[0163] The image restoration network is... Figure 1 The network trained by the image restoration model training method in the illustrated embodiment can be found in the relevant descriptions in the foregoing embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0164] The image restoration method of this invention involves: acquiring a target image to be restored; inputting the target image into an image restoration network; and acquiring an image restored by the image restoration network. The image restoration network is a network that... Figure 1 The network trained using the image inpainting model training method in the illustrated embodiment is thus obtained. Because the network inpainting model is trained using a lossless grayscale image of the original image, a loss contour map, and a mask, a good image inpainting effect can be guaranteed.

[0165] This invention also provides an image restoration model training device. See [link to related document]. Figure 5 , Figure 5 This is a structural diagram of the image restoration model training device provided in an embodiment of the present invention. Since the principle by which the image restoration model training device solves the problem is similar to the image restoration model training method in this embodiment of the present invention, the implementation of this image restoration model training device can refer to the implementation of the method, and repeated details will not be described again.

[0166] like Figure 5 As shown, the image restoration model training device 500 includes:

[0167] The first acquisition module 501 is used to acquire a lossless image sample set;

[0168] The first determining module 502 is used to determine, based on the lossless sample image and the first mask in the lossless image sample set, the first grayscale image and the first contour image of the lossless sample image corresponding to the lossless sample image.

[0169] Processing module 503 is used to input the first grayscale image, the first contour image, the first mask and the lost sample image into an image inpainting network, and obtain the inpainted image of the lost sample image output by the image inpainting network;

[0170] The second determining module 504 is used to determine the loss function value based on the repaired image and the lossless sample image;

[0171] Training module 505 is used to train the parameters of the image inpainting network based on the loss function value.

[0172] Optionally, the first grayscale image is equal to the Hadamard product of the matrix corresponding to the grayscale image of the lossless sample image and the first matrix, and the first contour image is equal to the Hadamard product of the matrix corresponding to the contour image of the lossless sample image and the first matrix. The first matrix is ​​equal to the difference between the matrix with all elements set to 1 and the matrix corresponding to the first mask. The matrix corresponding to the first mask uses element value 1 to represent the loss region and element value 0 to represent the non-loss region.

[0173] Optionally, the image restoration network includes a contour completion network and a color filling network;

[0174] Processing module 503 includes:

[0175] The first processing unit is configured to input the first grayscale image, the first contour image, and the first mask into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network.

[0176] The second processing unit is used to input the lost sample image and the third contour map corresponding to the lost region in the lost sample image into the color filling network to obtain the repaired image of the lost sample image output by the color filling network.

[0177] The second determining module 504 includes:

[0178] The first determining unit is used to determine the contour loss function value based on the second contour map and the first contour map;

[0179] The second determining unit is used to determine the image loss function value based on the repaired image and the lossless sample image;

[0180] Training module 505 includes:

[0181] The first training unit is used to train the parameters of the contour completion network based on the contour loss function value;

[0182] The second training unit is used to train the parameters of the color filling network based on the image loss function value.

[0183] Optionally, the second determining unit is used to determine the L1 norm loss function value based on the difference between each pixel in the repaired image and the lossless sample image, the loss region in the lossless sample image, and the number of pixels in the lossless sample image.

[0184] Optionally, the L1 norm loss function value includes the L1 loss value of the loss region of the loss sample image and the L1 loss value of the non-loss region of the loss sample image;

[0185] Wherein, the L1 loss value of the non-loss region is equal to the L1 norm of the Hadamard product of the matrix corresponding to the second mask and the second matrix divided by the number of pixels, the second matrix is ​​equal to the difference between the matrix corresponding to the repaired image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss region in the loss sample image, and the matrix corresponding to the second mask uses element value 0 to represent the loss region and element value 1 to represent the non-loss region.

[0186] The L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of the third matrix and the second matrix divided by the number of pixels, wherein the third matrix is ​​equal to the difference between the matrix with all elements equal to 1 and the matrix corresponding to the second mask.

[0187] Optionally, the second determining unit includes:

[0188] The first calculation subunit is used to calculate the L1 norm loss function value, the perceptual loss function value, and the style loss function value based on the restored image and the lossless sample image, respectively.

[0189] The second calculation subunit is used to calculate the weighted sum of the L1 norm loss function value, the perceptual loss function value, and the style loss function value to obtain the image loss function value.

[0190] Optionally, the image restoration network is a U-shaped network based on partial convolution.

[0191] Optionally, the image inpainting network is a generative adversarial network, and the generator of the generative adversarial network is a U-shaped network based on partial convolution.

[0192] Optionally, both the generator and the discriminator in the generative adversarial network are trained using spectral normalization.

[0193] The image restoration model training device provided in this embodiment of the invention can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0194] The image restoration model training device 500 of this embodiment acquires a lossless image sample set; based on the lossless sample images and a first mask in the lossless image sample set, it determines a first grayscale image and a first contour image of the loss sample image corresponding to the lossless sample image; it inputs the first grayscale image, the first contour image, the first mask, and the loss sample image into an image restoration network to obtain a restored image of the loss sample image output by the image restoration network; it determines a loss function value based on the restored image and the lossless sample image; and it trains the parameters of the image restoration network based on the loss function value. Thus, by training the image restoration network using the grayscale image of the lossless original image, the loss contour image, and the mask, it can ensure that the trained image restoration network can accurately locate the loss region in the image and has high accuracy in recognizing image edges, thereby achieving a better image restoration effect.

[0195] This invention also provides an image restoration apparatus. See [link to related document]. Figure 6 , Figure 6This is a structural diagram of the image restoration device provided in an embodiment of the present invention. Since the principle by which the image restoration device solves the problem is similar to the image restoration method in this embodiment, the implementation of this image restoration device can be referred to the implementation of the method, and repeated details will not be described again.

[0196] like Figure 6 As shown, the image restoration apparatus 600 includes:

[0197] The second acquisition module 601 is used to acquire the target image to be repaired;

[0198] Repair module 602 is used to input the target image into an image repair network and obtain the repaired image of the target image output by the image repair network;

[0199] The image inpainting network is a network that... Figure 1 The network trained by the image restoration model training method in the embodiment.

[0200] The image restoration device provided in this embodiment of the invention can perform the above-described method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0201] The image restoration apparatus 600 of this embodiment acquires a target image to be restored; inputs the target image into an image restoration network, and acquires an image restored by the image restoration network; wherein, the image restoration network is a network that... Figure 1 The network trained using the image inpainting model training method in the illustrated embodiment is thus obtained. Because the network inpainting model is trained using a lossless grayscale image of the original image, a loss contour map, and a mask, a good image inpainting effect can be guaranteed.

[0202] This invention also provides an electronic device. Since the principle by which this electronic device solves the problem is similar to the image inpainting model training method in this invention, the implementation of this electronic device can be found in the method implementation; repeated details will not be repeated. Figure 7 As shown, the electronic device of this embodiment includes:

[0203] Processor 700 is used to read the program from memory 720 and execute the following procedures:

[0204] Obtain a lossless image sample set;

[0205] Based on the lossless sample images and the first mask in the lossless image sample set, determine the first grayscale image and the first contour image of the loss sample image corresponding to the lossless sample image;

[0206] The first grayscale image, the first contour image, the first mask, and the lost sample image are input into an image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network.

[0207] Based on the repaired image and the lossless sample image, determine the loss function value;

[0208] The parameters of the image inpainting network are trained based on the loss function value.

[0209] Among them, Figure 7 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 700) and memory (memory 720). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 700 is responsible for managing the bus architecture and general processing, and memory 720 can store data used by processor 700 during operation.

[0210] Optionally, the processor 700 is also used to read the program from the memory 720 and perform the following steps:

[0211] The first grayscale image is equal to the Hadamard product of the matrix corresponding to the grayscale image of the lossless sample image and the first matrix. The first contour image is equal to the Hadamard product of the matrix corresponding to the contour image of the lossless sample image and the first matrix. The first matrix is ​​equal to the difference between the matrix with all elements set to 1 and the matrix corresponding to the first mask. The matrix corresponding to the first mask uses an element value of 1 to represent the loss region and an element value of 0 to represent the non-loss region.

[0212] Optionally, the image restoration network includes a contour completion network and a color filling network;

[0213] The processor 700 is also used to read the program in the memory 720 and perform the following steps:

[0214] The first grayscale image, the first contour image, and the first mask are input into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network.

[0215] The lost sample image and the third contour map corresponding to the lost region in the lost sample image are input into the color filling network to obtain the repaired image of the lost sample image output by the color filling network.

[0216] Based on the second contour map and the first contour map, determine the contour loss function value;

[0217] Based on the repaired image and the lossless sample image, determine the image loss function value;

[0218] Based on the contour loss function value, the parameters of the contour completion network are trained;

[0219] The parameters of the color filling network are trained based on the image loss function value.

[0220] Optionally, the processor 700 is also used to read the program from the memory 720 and perform the following steps:

[0221] The L1 norm loss function value is determined based on the difference between each pixel in the repaired image and the lossless sample image, the loss region in the lossless sample image, and the number of pixels in the lossless sample image.

[0222] Optionally, the L1 norm loss function value includes the L1 loss value of the loss region of the loss sample image and the L1 loss value of the non-loss region of the loss sample image;

[0223] Wherein, the L1 loss value of the non-loss region is equal to the L1 norm of the Hadamard product of the matrix corresponding to the second mask and the second matrix divided by the number of pixels, the second matrix is ​​equal to the difference between the matrix corresponding to the repaired image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss region in the loss sample image, and the matrix corresponding to the second mask uses element value 0 to represent the loss region and element value 1 to represent the non-loss region.

[0224] The L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of the third matrix and the second matrix divided by the number of pixels, wherein the third matrix is ​​equal to the difference between the matrix with all elements equal to 1 and the matrix corresponding to the second mask.

[0225] Optionally, the processor 700 is further configured to read the program from the memory 720 and perform the following steps:

[0226] Based on the restored image and the lossless sample image, the L1 norm loss function value, the perceptual loss function value, and the style loss function value are calculated respectively.

[0227] The image loss function value is obtained by calculating the weighted sum of the L1 norm loss function value, the perceptual loss function value, and the style loss function value.

[0228] Optionally, the image restoration network is a U-shaped network based on partial convolution.

[0229] Optionally, the image inpainting network is a generative adversarial network, and the generator of the generative adversarial network is a U-shaped network based on partial convolution.

[0230] Optionally, both the generator and the discriminator in the generative adversarial network are trained using spectral normalization.

[0231] The electronic device provided in this embodiment of the invention can execute the above-described image restoration model training method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0232] This invention also provides another electronic device. Since the principle by which this electronic device solves the problem is similar to the image restoration method in this invention, the implementation of this electronic device can be found in the implementation of the method, and repeated details will not be described again. Figure 8 As shown, the electronic device of this embodiment includes:

[0233] Processor 800 is used to read the program from memory 820 and execute the following procedures:

[0234] Obtain the target image to be repaired;

[0235] The target image is input into an image restoration network to obtain the image restored by the image restoration network.

[0236] The image inpainting network is the one described above. Figure 1 The network trained by the image restoration model training method in the illustrated embodiment.

[0237] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 820). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 800 is responsible for managing the bus architecture and general processing, and memory 820 can store data used by processor 800 during operation.

[0238] The electronic device provided in this embodiment of the invention can execute the above-described image restoration method embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0239] Furthermore, the computer-readable storage medium of this embodiment of the invention is used to store a computer program, which can be executed by a processor to implement as follows: Figure 1 The various steps in the method embodiments shown, or implementations such as Figure 4The steps in the method embodiment shown.

[0240] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0241] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit described above can be implemented in hardware or in the form of hardware plus software functional units.

[0242] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0243] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for training an image restoration model, characterized in that, include: Obtain a lossless image sample set; Based on the lossless sample images and the first mask in the lossless image sample set, determine the first grayscale image and the first contour image of the loss sample image corresponding to the lossless sample image; The first grayscale image, the first contour image, the first mask, and the lost sample image are input into an image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network. Based on the repaired image and the lossless sample image, determine the loss function value; Based on the loss function value, the parameters of the image inpainting network are trained; The image restoration network includes a contour completion network and a color filling network; The step of inputting the first grayscale image, the first contour image, the first mask, and the lost sample image into an image inpainting network to obtain the inpainted image of the lost sample image output by the image inpainting network includes: The first grayscale image, the first contour image, and the first mask are input into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network. The lost sample image and the third contour map corresponding to the lost region in the lost sample image are input into the color filling network to obtain the repaired image of the lost sample image output by the color filling network. The step of determining the loss function value based on the repaired image and the lossless sample image includes: Based on the second contour map and the first contour map, determine the contour loss function value; Based on the repaired image and the lossless sample image, determine the image loss function value; The step of training the parameters of the image inpainting network based on the loss function value includes: Based on the contour loss function value, the parameters of the contour completion network are trained; The parameters of the color filling network are trained based on the image loss function value.

2. The method according to claim 1, characterized in that, The first grayscale image is equal to the Hadamard product of the matrix corresponding to the grayscale image of the lossless sample image and the first matrix. The first contour image is equal to the Hadamard product of the matrix corresponding to the contour image of the lossless sample image and the first matrix. The first matrix is ​​equal to the difference between the matrix with all elements set to 1 and the matrix corresponding to the first mask. The matrix corresponding to the first mask uses an element value of 1 to represent the loss region and an element value of 0 to represent the non-loss region.

3. The method according to claim 1, characterized in that, The step of determining the image loss function value based on the repaired image and the lossless sample image includes: The L1 norm loss function value is determined based on the difference between each pixel in the repaired image and the lossless sample image, the loss region in the lossless sample image, and the number of pixels in the lossless sample image.

4. The method according to claim 3, characterized in that, The L1 norm loss function value includes the L1 loss value of the lost region of the lost sample image and the L1 loss value of the non-loss region of the lost sample image; Wherein, the L1 loss value of the non-loss region is equal to the L1 norm of the Hadamard product of the matrix corresponding to the second mask and the second matrix divided by the number of pixels, the second matrix is ​​equal to the difference between the matrix corresponding to the repaired image and the matrix corresponding to the lossless sample image, the second mask is determined based on the loss region in the loss sample image, and the matrix corresponding to the second mask uses element value 0 to represent the loss region and element value 1 to represent the non-loss region. The L1 loss value of the loss region is equal to the L1 norm of the Hadamard product of the third matrix and the second matrix divided by the number of pixels, wherein the third matrix is ​​equal to the difference between the matrix with all elements equal to 1 and the matrix corresponding to the second mask.

5. The method according to claim 3, characterized in that, The step of determining the image loss function value based on the repaired image and the lossless sample image includes: Based on the restored image and the lossless sample image, the L1 norm loss function value, the perceptual loss function value, and the style loss function value are calculated respectively. The image loss function value is obtained by calculating the weighted sum of the L1 norm loss function value, the perceptual loss function value, and the style loss function value.

6. An image restoration method, characterized in that, include: Obtain the target image to be repaired; The target image is input into an image restoration network to obtain the image restored by the image restoration network. The image restoration network is a network trained using the image restoration model training method described in any one of claims 1 to 5.

7. An image restoration model training device, characterized in that, include: The first acquisition module is used to acquire a lossless image sample set; The first determining module is used to determine, based on the lossless sample images and the first mask in the lossless image sample set, a first grayscale image and a first contour image of the lossless sample image corresponding to the lossless sample image. The processing module is used to input the first grayscale image, the first contour image, the first mask and the lost sample image into the image inpainting network, and obtain the inpainted image of the lost sample image output by the image inpainting network; The second determining module is used to determine the loss function value based on the repaired image and the lossless sample image; The training module is used to train the parameters of the image inpainting network based on the loss function value; The image restoration network includes a contour completion network and a color filling network; The processing module includes: The first processing unit is configured to input the first grayscale image, the first contour image, and the first mask into the contour completion network to obtain the second contour image of the loss region in the loss sample image output by the contour completion network. The second processing unit is used to input the lost sample image and the third contour map corresponding to the lost region in the lost sample image into the color filling network to obtain the repaired image of the lost sample image output by the color filling network. The second determining module includes: The first determining unit is used to determine the contour loss function value based on the second contour map and the first contour map; The second determining unit is used to determine the image loss function value based on the repaired image and the lossless sample image; The training module includes: The first training unit is used to train the parameters of the contour completion network based on the contour loss function value; The second training unit is used to train the parameters of the color filling network based on the image loss function value.

8. An image restoration device, characterized in that, include: The second acquisition module is used to acquire the target image to be repaired; The repair module is used to input the target image into the image repair network and obtain the repaired image output by the image repair network. The image restoration network is a network trained using the image restoration model training method described in any one of claims 1 to 5.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program in the memory to implement the steps in the image restoration model training method as described in any one of claims 1 to 5; or to implement the steps in the image restoration method as described in claim 6.

10. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps in the image restoration model training method as described in any one of claims 1 to 5; or implements the steps in the image restoration method as described in claim 6.

Images