An image inpainting method and system based on a generative adversarial network
By using gated convolutional layers and mean squared error correction techniques in generative adversarial networks (GANs), the problem of poor restoration results in GAN image inpainting was solved, and higher quality image inpainting results were achieved.
Patent Information
- Application Number
- CN202410927940.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2044-07-11
AI Technical Summary
Generative adversarial networks (GANs) tend to generate images with no missing parts that differ significantly from the original image during image inpainting, resulting in poor inpainting performance. This is especially true when generating missing parts, where existing techniques struggle to effectively improve inpainting quality.
An image inpainting method based on generative adversarial networks is adopted. The features of the image to be inpainted and the mask image are extracted by gated convolutional layers. The coarse inpainting image is corrected by calculating the mean square error and further optimized in the fine inpainting sub-network. The generator and discriminator are optimized in combination with the training process to improve the inpainting effect.
It improves the fidelity of non-repaired areas during image restoration, reduces information loss, and enhances the quality and efficiency of image restoration.
Smart Images
Figure CN118918041B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image restoration, specifically to an image restoration method and system based on generative adversarial networks. Background Technology
[0002] Image restoration can remove blemishes from images, such as scratches, stains, and fading. It can also perform reasoned repair on missing parts, improving the overall quality of the image and making old or damaged photographs clearer and more aesthetically pleasing. It is of great significance in the restoration of cultural relics, paintings, murals, and old films. The development of image restoration has progressed from manual restoration to digital restoration and then to artificial intelligence-based restoration. Early image restoration relied on the manual skills of restorers, which was time-consuming and costly. With the development of computer technology, digital image processing tools such as Photoshop have become increasingly widespread, improving restoration efficiency. In recent years, the application of deep learning and generative adversarial networks (GANs) has ushered in an era of intelligent image restoration, enabling the automatic and efficient repair of complex image damage, significantly improving restoration quality and speed.
[0003] In recent years, many scholars and research institutions have begun to study the use of generative adversarial networks (GANs) for image restoration, proposing many good models that have achieved excellent results in face restoration and building restoration. However, GANs not only generate images of missing parts but also regenerate the complete parts. If the complete parts differ significantly from the original during the GAN process, the regenerated missing parts will also be of poor quality. How to improve the performance of image restoration is a problem that urgently needs to be solved in this field. Summary of the Invention
[0004] To address the above problems, this invention provides an image inpainting method based on generative adversarial networks, the method comprising the following steps:
[0005] The image to be repaired and its mask image are obtained. The image to be repaired and the mask image are input into the generator in the generative adversarial network. The coarse repair sub-network of the generator includes multiple gated convolutional layers. The gated convolutional layers extract features from the input feature maps of the image to be repaired and the mask image through gated convolution to obtain the output feature map of the gated convolutional layer.
[0006] Obtain the coarse repair image from the coarse repair sub-network, divide the non-repaired portion of the coarse repair image into multiple regions, calculate the mean square error between each region and the corresponding region in the image to be repaired, correct the coarse repair image based on the mean square error, input the corrected coarse repair image into the fine repair sub-network, and use the output of the fine repair sub-network as the image repair result.
[0007] Preferably, the gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer, specifically:
[0008] The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation. An activation function is used to restrict the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and then multiplies it element by element with the second channel map to obtain the feature map of the second channel.
[0009] The feature map of the mask image is converted into a mask image feature map with the same size as the feature maps of the two channels using a convolution operation. An activation function is used to restrict the element values of the mask image feature map to between 0 and 1, and then multiply it element by element with the first channel image to obtain the feature map of the first channel.
[0010] The output feature map of the gated convolutional layer is obtained by averaging the feature maps of the first and second channels.
[0011] Preferably, the step of correcting the coarsely repaired image based on the mean square error specifically involves:
[0012] When the mean square error is greater than a preset value, the corresponding region in the coarse repair image is replaced with the image of the region in the image to be repaired.
[0013] The replaced coarsely repaired image is then smoothed.
[0014] Preferably, the method further includes a training process, which specifically includes:
[0015] Training samples are input into the generative adversarial network (GAN) to obtain coarsely repaired images from the coarse repair subnetwork. The mean square error (MSE) between the non-repaired regions and the corresponding regions in the ground truth image in the coarse repaired image is calculated, as is the MSE between the repaired regions and the ground truth image in the coarse repaired image. The loss of the coarse repair subnetwork is calculated based on these two average errors, and the coarse repair subnetwork is trained based on the loss. After the coarse repair subnetwork is trained, the entire generator is trained. After the generator is trained, the entire GAN is trained.
[0016] Preferably, the step of calculating the loss of the coarse repair subnet based on the above two averaged errors specifically involves:
[0017] Set the weights for the mean squared error of the non-repaired region and the mean squared error of the repaired region.
[0018] Calculate the product of the mean square error of the non-repaired region and the corresponding weight, and the product of the mean square error of the repaired region and the corresponding weight. Use the sum of the two products as the loss of the coarse repaired subnet.
[0019] In another aspect, the present invention also provides an image inpainting system based on generative adversarial networks, the system comprising the following modules:
[0020] The feature extraction module is used to acquire the image to be repaired and the mask image of the image to be repaired. The image to be repaired and the mask image are input into the generator in the generative adversarial network. The coarse repair sub-network of the generator includes multiple gated convolutional layers. The gated convolutional layers extract features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer.
[0021] The image inpainting module is used to acquire the coarse inpainting image of the coarse inpainting sub-network, divide the non-inpainting part of the coarse inpainting image into multiple regions, calculate the mean square error between each region and the corresponding region in the image to be inpainted, correct the coarse inpainting image according to the mean square error, input the corrected coarse inpainting image into the fine inpainting sub-network, and use the output of the fine inpainting sub-network as the image inpainting result.
[0022] Preferably, the gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer, specifically:
[0023] The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation. An activation function is used to restrict the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and then multiplies it element by element with the second channel map to obtain the feature map of the second channel.
[0024] The feature map of the mask image is converted into a mask image feature map with the same size as the feature maps of the two channels using a convolution operation. An activation function is used to restrict the element values of the mask image feature map to between 0 and 1, and then multiply it element by element with the first channel image to obtain the feature map of the first channel.
[0025] The output feature map of the gated convolutional layer is obtained by averaging the feature maps of the first and second channels.
[0026] Preferably, the step of correcting the coarsely repaired image based on the mean square error specifically involves:
[0027] When the mean square error is greater than a preset value, the corresponding region in the coarse repair image is replaced with the image of the region in the image to be repaired.
[0028] The replaced coarsely repaired image is then smoothed.
[0029] Preferably, the method further includes a training process, which specifically includes:
[0030] Training samples are input into the generative adversarial network (GAN) to obtain coarsely repaired images from the coarse repair subnetwork. The mean square error (MSE) between the non-repaired regions and the corresponding regions in the ground truth image in the coarse repaired image is calculated, as is the MSE between the repaired regions and the ground truth image in the coarse repaired image. The loss of the coarse repair subnetwork is calculated based on these two average errors, and the coarse repair subnetwork is trained based on the loss. After the coarse repair subnetwork is trained, the entire generator is trained. After the generator is trained, the entire GAN is trained.
[0031] Preferably, the step of calculating the loss of the coarse repair subnet based on the above two averaged errors specifically involves:
[0032] Set the weights for the mean squared error of the non-repaired region and the mean squared error of the repaired region.
[0033] Calculate the product of the mean square error of the non-repaired region and the corresponding weight, and the product of the mean square error of the repaired region and the corresponding weight. Use the sum of the two products as the loss of the coarse repaired subnet.
[0034] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.
[0035] To address the problem of content loss in non-repaired areas of the image to be repaired during image restoration, this invention corrects the coarsely generated image. Specifically, it divides the non-repaired areas of the coarse restoration image into multiple regions, calculates the mean square error (MSE) between each region and the corresponding region in the image to be repaired, and corrects the coarse restoration image based on the MSE. This compensates for the impact and information loss on non-repaired areas during the coarse restoration stage, improving the restoration effect. Furthermore, this invention fuses features from the mask image into the image to be repaired using gated convolution. Through gated convolution, information from the mask image is learned, especially information about the region of interest (ROI) in the mask image, i.e., the region to be repaired, further improving the effect of the coarse restoration stage. Attached Figure Description
[0036] Figure 1 This is a flowchart of Example 1;
[0037] Figure 2 A schematic diagram of the generator structure in a generative adversarial network;
[0038] Figure 3 This is a comparison diagram of the effects of the present invention and the traditional DeepFillv2;
[0039] Figure 4 This is a structural diagram of Example 2. Detailed Implementation
[0040] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] Example 1: This invention provides an image inpainting method based on generative adversarial networks, such as... Figure 1 As shown, the method includes the following steps:
[0043] S1, obtain the image to be repaired and the mask image of the image to be repaired, and input the image to be repaired and the mask image into the generator in the generative adversarial network. The coarse repair sub-network of the generator includes multiple gated convolutional layers. The gated convolutional layers extract features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer.
[0044] When using a generative adversarial network (GAN) for image inpainting, the image to be inpainted and its mask image are first obtained. The mask image is an image of the same size as the image to be inpainted and is a binary image, or simply an image of both. Regions of no interest have pixel values of 0, while regions of interest have pixel values of 1 or 255. In this invention, the region of interest in the mask image is the missing portion or the area to be repaired in the image to be inpainted. The image to be inpainted and the mask image are input into the GAN, which includes a generator and a discriminator. In a more specific embodiment, the GAN uses a DeepFill network structure. The generator of the GAN includes a coarse inpainting subnet and a fine inpainting subnet. A preliminary inpainting result is generated in the coarse inpainting stage, and then the fine inpainting subnet performs further inpainting. Figure 2 As shown.
[0045] The generator in a generative adversarial network includes multiple gated convolutions. These gated convolutions adaptively select and emphasize useful contextual information, improving the restoration effect. In one embodiment, the gated convolutional layer extracts features from the input feature map of the image to be restored and the feature map of the mask image using gated convolution to obtain the output feature map of the gated convolutional layer. Specifically, the output feature map of the gated convolutional layer is obtained using a coarse restoration subnet of DeepFillv2, i.e., feature extraction is performed using a gated convolutional layer of DeepFillv2. To fully learn the mask image information, in another specific embodiment, the gated convolutional layer extracts features from the input feature map of the image to be restored and the feature map of the mask image using gated convolution to obtain the output feature map of the gated convolutional layer. Specifically:
[0046] The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation. An activation function is used to restrict the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and then multiplies it element by element with the second channel map to obtain the feature map of the second channel.
[0047] The feature map of the mask image is converted into a mask image feature map with the same size as the feature maps of the two channels using a convolution operation. An activation function is used to restrict the element values of the mask image feature map to between 0 and 1, and then multiply it element by element with the first channel image to obtain the feature map of the first channel.
[0048] The output feature map of the gated convolutional layer is obtained by averaging the feature maps of the first and second channels.
[0049] For each gated convolutional layer, the input consists of the feature map of the image to be repaired and the feature map of the mask image. The feature map of the image to be repaired is convolved with two kernels to obtain two channel maps. These channel maps are obtained by convolving the feature map of the image to be repaired. Activation functions such as ReLU or ELU are used to activate the first channel map, restricting the element value of each pixel in the first channel feature map to the range [0,1]. Then, the first and second channel maps are multiplied according to pixel position. For example, if the pixel value at position [5,10] in the first channel map is 0.5, and the pixel value at position [5,10] in the second channel map is 86, then after multiplication, the pixel value at position [5,10] in the second channel feature map is 43.
[0050] The mask image can distinguish the regions to be repaired in the image to be repaired. The processing of the regions to be repaired differs from that of other regions. This information is obtained by learning the feature map corresponding to the mask image. Specifically, a convolution operation is used to convert the feature map of the mask image into a mask image feature map with the same size as the feature maps of the two channels. For example, if the size of the first channel image and the second channel image is 64×64, then the size of the mask image feature map is also 64×64. Similarly, the elements of the mask image feature map are restricted to the range [0,1] using an activation function, and then multiplied element-wise with the first channel image to obtain the feature map of the first channel.
[0051] Finally, the feature maps of the first and second channels are averaged to obtain the output feature map of the gated convolutional layer. For example, if the feature value at position [8,6] in the first channel is 10 and the feature value at position [8,6] in the second channel is 12, then the average value is 11.
[0052] S2, obtain the coarse repair image of the coarse repair sub-network, divide the non-repaired part of the coarse repair image into multiple regions, calculate the mean square error between each region and the corresponding region in the image to be repaired, correct the coarse repair image according to the mean square error, input the corrected coarse repair image into the fine repair sub-network, and use the output of the fine repair sub-network as the image repair result.
[0053] After passing through the coarse repair sub-network, a coarse repair image is output. This image includes repair of the region of interest (ROI), i.e., the region to be repaired, as well as images generated for the non-ROI regions. The primary goal of the repair is to repair the ROI, but the non-ROI regions are also regenerated. These regenerated images will lose some content from the non-ROI regions in the original image. To prevent this, in one embodiment, the coarse repair image is corrected based on the mean square error, specifically as follows:
[0054] When the mean square error is greater than a preset value, the corresponding region in the coarse repair image is replaced with the image of the region in the image to be repaired.
[0055] The replaced coarsely repaired image is then smoothed.
[0056] The non-repairable portion of the image to be repaired is divided into multiple regions. The mean square error of each region in the image to be repaired and in the coarse repair image is calculated. If the mean square error is greater than a preset value, the corresponding region in the original image to be repaired is used to replace the corresponding region in the coarse repair image. This avoids the impact of coarse repair on the subsequent repair of the region to be repaired. Specifically, the image to be repaired is divided into multiple image blocks to form an image block set, for example, into image blocks of size 8×8. If an image block contains a region to be repaired, it is removed from the image block set. The remaining image blocks are the multiple regions divided from the non-repairable portion. Since replacement may lead to obvious boundaries between different regions, in a more specific embodiment, the coarse repair image after replacement is smoothed.
[0057] In a more specific embodiment, the fine inpainting subnetwork adopts the fine inpainting subnetwork of DeepFillv2, and the discriminator also adopts the discriminator structure of DeepFillv2; the coarse inpainting subnetwork further includes dilated convolution and deconvolution. Before using the generative adversarial network for image inpainting, the generative adversarial network needs to be trained. During training, a discriminator is used to continuously optimize the discriminator, enabling it to more accurately distinguish between real data and generated data. The generator continuously improves its generation effect, and the discriminator continuously improves its discrimination effect, thereby achieving the goal of improving the generator's generation effect. In a more specific embodiment, the method further includes a training process, which specifically includes:
[0058] Training samples are input into the generative adversarial network (GAN) to obtain coarsely repaired images from the coarse repair subnetwork. The mean square error (MSE) between the non-repaired regions and the corresponding regions in the ground truth image in the coarse repaired image is calculated, as is the MSE between the repaired regions and the ground truth image in the coarse repaired image. The loss of the coarse repair subnetwork is calculated based on these two average errors, and the coarse repair subnetwork is trained based on the loss. After the coarse repair subnetwork is trained, the entire generator is trained. After the generator is trained, the entire GAN is trained.
[0059] Training a Generative Adversarial Network (GAN) involves several steps. The first step is to train only the coarse-grained restoration sub-network. Training the coarse-grained sub-network allows it to quickly achieve good results and prevents bias in the fine-grained sub-network and the discriminator. Specifically, the mean square error between the non-restored regions in the coarse-restored image and their corresponding regions in the ground truth image is calculated, as well as the mean square error between the restored regions in the coarse-restored image and their ground truth images. The loss of the coarse-grained sub-network is calculated based on these two errors, and this loss is used to train the coarse-grained sub-network. The ground truth image is the image in the training samples without the missing regions added; the image to be restored is the one with the missing regions added to it.
[0060] The loss of the coarse restoration subnet is calculated based on the mean square error between the non-restored region and the corresponding region of the real image, as well as the average error between the restored region and the real image. There are several methods for this. One method is to calculate the loss using the two largest average errors. Another method is to pre-set the weights of the two average errors. In a more specific embodiment, the calculation of the loss of the coarse restoration subnet based on the aforementioned two average errors is as follows:
[0061] Set the weights for the mean squared error of the non-repaired region and the mean squared error of the repaired region.
[0062] Calculate the product of the mean square error of the non-repaired region and the corresponding weight, and the product of the mean square error of the repaired region and the corresponding weight. Use the sum of the two products as the loss of the coarse repaired subnet.
[0063] After training the coarse repair subnet, the entire generator is trained. After the generator is trained, the entire generative adversarial network is trained.
[0064] In a more specific embodiment, the training samples are divided into three parts: the first part is used to train the coarse repair subnet, the second part is used to train the entire generator, and the third part is used to train the entire generative adversarial network. There are various division ratios, such as 1:1:8 or 2:2:6, etc., and this invention does not impose a specific limitation on the division ratio.
[0065] Figure 3 The image shows a comparison of the results of the present invention and DeepFillv2 in repairing the same image. Figure 3 (A) is the image to be repaired, 3(B) is the image after repair by DeepFillv2, and 3(C) is the image after repair by the present invention.
[0066] Example 2: This invention also provides an image inpainting system based on generative adversarial networks, such as... Figure 4 As shown, the system includes the following modules:
[0067] The feature extraction module is used to acquire the image to be repaired and the mask image of the image to be repaired. The image to be repaired and the mask image are input into the generator in the generative adversarial network. The coarse repair sub-network of the generator includes multiple gated convolutional layers. The gated convolutional layers extract features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer.
[0068] The image inpainting module is used to acquire the coarse inpainting image of the coarse inpainting sub-network, divide the non-inpainting part of the coarse inpainting image into multiple regions, calculate the mean square error between each region and the corresponding region in the image to be inpainted, correct the coarse inpainting image according to the mean square error, input the corrected coarse inpainting image into the fine inpainting sub-network, and use the output of the fine inpainting sub-network as the image inpainting result.
[0069] Preferably, the gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer, specifically:
[0070] The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation. An activation function is used to restrict the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and then multiplies it element by element with the second channel map to obtain the feature map of the second channel.
[0071] The feature map of the mask image is converted into a mask image feature map with the same size as the feature maps of the two channels using a convolution operation. An activation function is used to restrict the element values of the mask image feature map to between 0 and 1, and then multiply it element by element with the first channel image to obtain the feature map of the first channel.
[0072] The output feature map of the gated convolutional layer is obtained by averaging the feature maps of the first and second channels.
[0073] Preferably, the step of correcting the coarsely repaired image based on the mean square error specifically involves:
[0074] When the mean square error is greater than a preset value, the corresponding region in the coarse repair image is replaced with the image of the region in the image to be repaired.
[0075] The replaced coarsely repaired image is then smoothed.
[0076] Preferably, the method further includes a training process, which specifically includes:
[0077] Training samples are input into the generative adversarial network (GAN) to obtain coarsely repaired images from the coarse repair subnetwork. The mean square error (MSE) between the non-repaired regions and the corresponding regions in the ground truth image in the coarse repaired image is calculated, as is the MSE between the repaired regions and the ground truth image in the coarse repaired image. The loss of the coarse repair subnetwork is calculated based on these two average errors, and the coarse repair subnetwork is trained based on the loss. After the coarse repair subnetwork is trained, the entire generator is trained. After the generator is trained, the entire GAN is trained.
[0078] Preferably, the step of calculating the loss of the coarse repair subnet based on the above two averaged errors specifically involves:
[0079] Set the weights for the mean squared error of the non-repaired region and the mean squared error of the repaired region.
[0080] Calculate the product of the mean square error of the non-repaired region and the corresponding weight, and the product of the mean square error of the repaired region and the corresponding weight. Use the sum of the two products as the loss of the coarse repaired subnet.
[0081] In embodiment three, the present invention also provides a computer-readable storage medium on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method described in embodiment one.
[0082] In Embodiment 4, the present invention also provides a computer device, the computer device including at least a memory and a processor, wherein a computer program is stored in the memory, and the computer program, when executed by the processor, implements the method described in Embodiment 1.
[0083] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Other embodiments may also be used. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image inpainting method based on generative adversarial networks, characterized in that, The method includes the following steps: The image to be repaired and its mask image are obtained. The image to be repaired and the mask image are input into the generator in the generative adversarial network. The coarse repair sub-network of the generator includes multiple gated convolutional layers. The gated convolutional layers extract features from the input feature maps of the image to be repaired and the mask image through gated convolution to obtain the output feature map of the gated convolutional layer. Obtain the coarse repair image of the coarse repair sub-network, divide the non-repaired part of the coarse repair image into multiple regions, calculate the mean square error between each region and the corresponding region in the image to be repaired, correct the coarse repair image according to the mean square error, input the corrected coarse repair image into the fine repair sub-network, and use the output of the fine repair sub-network as the image repair result. The gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer. Specifically: The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation. An activation function is used to restrict the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and then multiplies it element by element with the second channel map to obtain the feature map of the second channel. The feature map of the mask image is converted into a mask image feature map with the same size as the feature maps of the two channels above using a convolution operation. An activation function is used to restrict the element values of the mask image feature map to between 0 and 1, and then multiply it element by element with the first channel image to obtain the feature map of the first channel. The output feature map of the gated convolutional layer is obtained by averaging the feature maps of the first and second channels.
2. The method as described in claim 1, characterized in that, The step of correcting the coarsely repaired image based on the mean square error specifically involves: When the mean square error is greater than a preset value, the corresponding region in the coarse repair image is replaced with the image of the region in the image to be repaired. The replaced coarsely repaired image is then smoothed.
3. The method as described in claim 1, characterized in that, The method further includes a training process, which specifically includes: The training samples are input into the generative adversarial network to obtain the coarsely repaired image of the coarse repair subnetwork. The mean square error between the non-repaired region and the corresponding region of the real image in the coarsely repaired image is calculated, and the mean square error between the repaired region and the real image in the coarsely repaired image is also calculated. The loss of the coarse repair subnetwork is calculated based on the above two mean square errors, and the coarse repair subnetwork is trained based on the loss. After the coarse repair subnetwork is trained, the entire generator is trained. After the generator is trained, the entire generative adversarial network is trained.
4. The method as described in claim 1, characterized in that, The calculation of the loss of the coarse repair subnet based on the above two average errors is as follows: Set the weights for the mean square error of the non-repaired region and the mean square error of the repaired region; Calculate the product of the mean square error of the non-repaired region and the corresponding weight, and the product of the mean square error of the repaired region and the corresponding weight. Use the sum of the two products as the loss of the coarse repaired subnet.
5. An image inpainting system based on generative adversarial networks, characterized in that, The system includes the following modules: The feature extraction module is used to obtain the image to be repaired and the mask image of the image to be repaired. The image to be repaired and the mask image are input into the generator in the generative adversarial network. The coarse repair sub-network of the generator includes multiple gated convolutional layers. The gated convolutional layers extract features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer. The image inpainting module is used to acquire the coarse inpainting image of the coarse inpainting sub-network, divide the non-inpainting part of the coarse inpainting image into multiple regions, calculate the mean square error between each region and the corresponding region in the image to be inpainted, correct the coarse inpainting image according to the mean square error, input the corrected coarse inpainting image into the fine inpainting sub-network, and use the output of the fine inpainting sub-network as the image inpainting result. The gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain the output feature map of the gated convolutional layer. Specifically: The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation. An activation function is used to restrict the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and then multiply it element by element with the second channel map to obtain the feature map of the second channel. The feature map of the mask image is converted into a mask image feature map with the same size as the feature maps of the two channels above using a convolution operation. An activation function is used to restrict the element values of the mask image feature map to between 0 and 1, and then multiply it element by element with the first channel image to obtain the feature map of the first channel. The output feature map of the gated convolutional layer is obtained by averaging the feature maps of the first and second channels.
6. The system as described in claim 5, characterized in that, The step of correcting the coarsely repaired image based on the mean square error specifically involves: When the mean square error is greater than a preset value, the corresponding region in the coarse repair image is replaced with the image of the region in the image to be repaired. The replaced coarsely repaired image is then smoothed.
7. The system as described in claim 5, characterized in that, The system also includes a training module, which specifically comprises: The training samples are input into the generative adversarial network to obtain the coarsely repaired image of the coarse repair subnetwork. The mean square error between the non-repaired region and the corresponding region of the real image in the coarsely repaired image is calculated, and the mean square error between the repaired region and the real image in the coarsely repaired image is also calculated. The loss of the coarse repair subnetwork is calculated based on the above two mean square errors, and the coarse repair subnetwork is trained based on the loss. After the coarse repair subnetwork is trained, the entire generator is trained. After the generator is trained, the entire generative adversarial network is trained.
8. A computer storage device, wherein a computer program is stored on the storage device, characterized in that, The computer program, when executed by a processor, implements the method as described in any one of claims 1-4.
Citation Information
Patent Citations
Filling pipeline crack detection method
CN111784645A
Image restoration method based on wavelet self-attention generative adversarial network
CN115861108A