Image restoration method and system based on generative adversarial network
By using gated convolutional layer and mean square error correction technology in the generative adversarial network, the problem of poor generation effect of generative adversarial network in image repair is solved, and the repair effect and content retention are improved.
Patent Information
- Application Number
- CN202410927940.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2044-07-11
AI Technical Summary
During the image repair process of generative adversarial network, when the generated missing parts are far from the original image, the repair effect is poor and it is easy to lead to the loss of content in non-repaired areas.
Using an image repair method based on a generative adversarial network, the features of the image to be repaired and the masked image are extracted through the gated convolution layer, the mean square error of each area is calculated, the coarse repaired image is corrected, and the corrected image is input. Repair subnets carefully to improve repair results.
Improves the effect during image repair, reduces the loss of content in non-repaired areas, and enhances the accuracy and quality of repair results.
Smart Images

Figure CN118918041A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image restoration, and in particular to an image restoration method and system based on a generative adversarial network. Background Art
[0002] Image restoration can remove defects in images, such as scratches, stains, and fading, and can infer missing parts to improve the overall quality of the image, making old or damaged photos clearer and more beautiful. It is of great significance in the restoration of cultural relics, paintings, murals, old films, etc. The development process of image restoration has gone through manual restoration, digital restoration, and artificial intelligence-based restoration. Early image restoration relied on the manual skills of restorers, which was time-consuming and costly. With the development of computer technology, digital image processing tools such as Photoshop have gradually become popular, improving the efficiency of restoration. In recent years, the application of deep learning and generative adversarial networks (GANs) has brought image restoration into the intelligent era, which can automatically and efficiently repair complex image damage, significantly improving the quality and speed of restoration.
[0003] In recent years, many scholars and research institutions have begun to study the use of generative adversarial networks to repair images, and have proposed many good models, which have achieved good results in face repair and building repair. However, the generative adversarial network not only generates images of missing parts, but also regenerates the non-missing parts. If the non-missing parts are far from the original parts in the middle of the generative adversarial network process, the effect of the regenerated missing parts will be relatively poor. How to improve the effect of the image repair process is an urgent problem to be solved in this field. Summary of the invention
[0004] In order to solve the above problems, the present invention provides an image restoration method based on a generative adversarial network, the method comprising the following steps: An image to be repaired and a mask image of the image to be repaired are obtained, and the image to be repaired and the mask image are input into a generator in a generative adversarial network. The coarse repair subnetwork of the generator includes a plurality of gated convolutional layers, and the gated convolutional layer performs feature extraction on an input feature map of the image to be repaired and a feature map of the mask image through gated convolution to obtain an output feature map of the gated convolutional layer.
[0005] Obtain a coarse inpainted image of the coarse inpainting subnetwork, divide the non-inpainted part of the coarse inpainting image into multiple regions, calculate the mean square error between each region and the corresponding position region in the image to be inpainted, correct the coarse inpainted image according to the mean square error, input the corrected coarse inpainted image into the fine inpainting subnetwork, and use the output of the fine inpainting subnetwork as the image inpainting result.
[0006] Preferably, the gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain an output feature map of the gated convolution layer, specifically: The convolution operation is used to convert the feature map of the image to be repaired into a first channel map and a second channel map. The activation function is used to limit the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and the first channel map is multiplied element by element with the second channel map to obtain the feature map of the second channel.
[0007] The feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps by using a convolution operation, and the element values of the mask image feature map are limited to between 0 and 1 by using an activation function, and the feature map of the first channel is obtained by multiplying the first channel map element by element.
[0008] The feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolutional layer.
[0009] Preferably, the correcting the rough repaired image according to the mean square error is specifically: When the mean square error is greater than a preset value, the image of the area in the image to be repaired is used to replace the corresponding area in the rough repair image.
[0010] The replaced rough repaired image is smoothed.
[0011] Preferably, the method further comprises a training process, and the training process is specifically: The training samples are input into the generative adversarial network to obtain a coarse repaired image of the coarse repaired subnet, the mean square error between the non-repaired area in the coarse repaired image and the corresponding area of the real image is calculated, and the mean square error between the repaired area in the coarse repaired image and the real image is calculated, the loss of the coarse repaired subnet is calculated according to the above two average errors, and the coarse repaired subnet is trained based on the loss; after the coarse repaired subnet is trained, the entire generator is trained; after the generator training is completed, the entire generative adversarial network is trained.
[0012] Preferably, the loss of the coarse repair subnet is calculated based on the two average errors, specifically: Set the weight of the mean square error of the non-repaired area and the weight of the mean square error of the repaired area.
[0013] The product of the mean square error of the non-repaired area and the corresponding weight, as well as the product of the mean square error of the repaired area and the corresponding weight are calculated, and the sum of the two products is taken as the loss of the coarse repair subnet.
[0014] On the other hand, the present invention also provides an image restoration system based on a generative adversarial network, the system comprising the following modules: A feature extraction module is used to obtain an image to be repaired and a mask image of the image to be repaired, and input the image to be repaired and the mask image into a generator in a generative adversarial network. The coarse repair subnetwork of the generator includes multiple gated convolutional layers. The gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain an output feature map of the gated convolutional layer.
[0015] The image restoration module is used to obtain the coarse restoration image of the coarse restoration subnetwork, divide the non-restoration part in the coarse restoration image into multiple regions, calculate the mean square error between each region and the corresponding position region in the image to be restored, correct the coarse restoration image according to the mean square error, input the corrected coarse restoration image into the fine restoration subnetwork, and use the output of the fine restoration subnetwork as the image restoration result.
[0016] Preferably, the gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain an output feature map of the gated convolution layer, specifically: The convolution operation is used to convert the feature map of the image to be repaired into a first channel map and a second channel map. The activation function is used to limit the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and the first channel map is multiplied element by element with the second channel map to obtain the feature map of the second channel.
[0017] The feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps by using a convolution operation, and the element values of the mask image feature map are limited to between 0 and 1 by using an activation function, and the feature map of the first channel is obtained by multiplying the first channel map element by element.
[0018] The feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolutional layer.
[0019] Preferably, the correcting the rough repaired image according to the mean square error is specifically: When the mean square error is greater than a preset value, the image of the area in the image to be repaired is used to replace the corresponding area in the rough repair image.
[0020] The replaced rough repaired image is smoothed.
[0021] Preferably, the method further comprises a training process, and the training process is specifically: The training samples are input into the generative adversarial network to obtain a coarse repaired image of the coarse repaired subnet, the mean square error between the non-repaired area in the coarse repaired image and the corresponding area of the real image is calculated, and the mean square error between the repaired area in the coarse repaired image and the real image is calculated, the loss of the coarse repaired subnet is calculated according to the above two average errors, and the coarse repaired subnet is trained based on the loss; after the coarse repaired subnet is trained, the entire generator is trained; after the generator training is completed, the entire generative adversarial network is trained.
[0022] Preferably, the loss of the coarse repair subnet is calculated based on the two average errors, specifically: Set the weight of the mean square error of the non-repaired area and the weight of the mean square error of the repaired area.
[0023] The product of the mean square error of the non-repaired area and the corresponding weight, as well as the product of the mean square error of the repaired area and the corresponding weight are calculated, and the sum of the two products is taken as the loss of the coarse repair subnet.
[0024] In addition, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.
[0025] In view of the problem that the content of the non-repaired area in the image to be repaired is easily lost during the image repair process, the present invention corrects the coarse generated image, specifically by dividing the non-repaired area in the coarse repair image into multiple areas, calculating the mean square error between each area and the image of the corresponding position area in the image to be repaired, and correcting the coarse repair image according to the mean square error, thereby compensating for the impact and information loss on the non-repaired area in the coarse repair stage, and improving the repair effect. Here, the present invention fuses the features in the mask image into the image to be repaired by means of gated convolution, and learns the information in the mask image through gated convolution, especially the information of the area of interest in the mask image, that is, the area to be repaired, to further improve the effect of the coarse repair stage. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a flow chart of Embodiment 1; Figure 2 This is a schematic diagram of the structure of the generator in the generative adversarial network; Figure 3 This is a comparison diagram between the present invention and the traditional DeepFillv2; Figure 4 This is a structural diagram of the second embodiment. DETAILED DESCRIPTION
[0027] In this article, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0028] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] Embodiment 1, the present invention provides an image restoration method based on a generative adversarial network, such as Figure 1 As shown, the method comprises the following steps: S1, obtaining an image to be repaired and a mask image of the image to be repaired, inputting the image to be repaired and the mask image into a generator in a generative adversarial network, wherein a coarse repair subnetwork of the generator includes a plurality of gated convolutional layers, and the gated convolutional layer extracts features of an input feature map of the image to be repaired and a feature map of the mask image through gated convolution to obtain an output feature map of the gated convolutional layer.
[0030] When using a generative adversarial network for image restoration, first obtain an image to be restored and a mask image of the image to be restored, wherein the mask image is an image of the same size as the image to be restored, and the mask image is a binary image, or a two-image, and the pixel values of the area of no interest are 0, and the pixel values of the area of interest are 1 or 255. In the present invention, the area of interest in the mask image is the missing part of the image to be restored or the area to be repaired. The image to be restored and the mask image are input into a generative adversarial network. The generative adversarial network includes a generator and a discriminator. In a more specific embodiment, the generative adversarial network adopts a DeepFill network structure. The generator of the generative adversarial network includes a coarse repair subnet and a fine repair subnet. A preliminary repair result will be generated in the coarse repair stage, and then the repair will be performed again through the fine repair subnet, such as Figure 2 shown.
[0031] The generator of the generative adversarial network includes multiple gated convolutions, which can adaptively select and emphasize useful contextual information to improve the repair effect. In one embodiment, the gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain the output feature map of the gated convolution layer, specifically: the coarse repair subnet of DeepFillv2 is used to obtain the output feature map of the gated convolution layer, that is, the gated convolution layer of DeepFillv2 is used for feature extraction. In order to fully learn the mask image information, in another specific embodiment, the gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain the output feature map of the gated convolution layer, specifically: The convolution operation is used to convert the feature map of the image to be repaired into a first channel map and a second channel map. The activation function is used to limit the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and the first channel map is multiplied element by element with the second channel map to obtain the feature map of the second channel.
[0032] The feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps by using a convolution operation, and the element values of the mask image feature map are limited to between 0 and 1 by using an activation function, and the feature map of the first channel is obtained by multiplying the first channel map element by element.
[0033] The feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolutional layer.
[0034] For each gated convolution layer, the input is the feature map of the image to be repaired and the feature map of the mask image. The feature map of the image to be repaired is convolved with two convolution kernels to obtain two channel maps. The channel map is obtained by convolving the feature map of the image to be repaired. The first channel map is activated by using activation functions such as ReLU and ELU, that is, the element value of each pixel in the first channel feature map is limited to [0,1]. Then the first channel map and the second channel map are multiplied according to the pixel position. For example, the pixel value at position [5,10] in the first channel map is 0.5, and the pixel value at position [5,10] in the second channel map is 86. After the two are multiplied, the pixel value at position [5,10] in the feature map of the second channel is 43.
[0035] The mask image can be used to distinguish the area to be repaired in the image to be repaired. The treatment of the area to be repaired is different from that of other areas. This part of information is learned by learning the feature map corresponding to the mask image. Specifically, the feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps using a convolution operation. For example, if the size of the first channel map and the second channel map is 64×64, the size of the mask image feature map is also 64×64. Similarly, the elements of the mask image feature map are restricted to [0,1] using an activation function, and are multiplied element by element with the first channel map to obtain the feature map of the first channel.
[0036] Finally, the feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolution layer. For example, the feature value at position [8,6] of the feature map of the first channel is 10, and the feature value at position [8,6] of the feature map of the second channel is 12, so the average value is 11.
[0037] S2, obtaining a coarse repaired image of the coarse repair subnetwork, dividing the non-repaired part of the coarse repaired image into multiple regions, calculating the mean square error between each region and the corresponding position region in the image to be repaired, correcting the coarse repaired image according to the mean square error, inputting the corrected coarse repaired image into the fine repair subnetwork, and taking the output of the fine repair subnetwork as the image repair result.
[0038] After passing through the coarse repair sub-network, a coarse repair image will be output. The coarse repair image includes the repair of the region of interest, that is, the region to be repaired, and also includes the image generated for the non-interest region. In the repair, the main purpose is to repair the region of interest, but the non-interest region will also be regenerated, and the image generated by the non-interest region will lose part of the content of the non-interest region in the image to be repaired. In order to prevent this situation, in one embodiment, the coarse repair image is corrected according to the mean square error, specifically: When the mean square error is greater than a preset value, the image of the area in the image to be repaired is used to replace the corresponding area in the rough repair image.
[0039] The replaced rough repaired image is smoothed.
[0040] The non-repaired part of the image to be repaired is divided into multiple regions, and the mean square error of each region in the image to be repaired and the coarse repaired image is calculated. If the mean square error is greater than a preset value, the image of the corresponding area in the original image to be repaired is used to replace the image of the corresponding area in the coarse repaired image. This can avoid the impact of the coarse repair on the subsequent repair of the area to be repaired. The division method is specifically to divide the image to be repaired into multiple image blocks to form an image block set, for example, divided into image blocks of size 8×8. If the image block contains an area to be repaired, it is removed from the image block set, and the remaining image blocks are the multiple areas divided by the non-repaired part. Since the replacement may cause obvious boundaries between different areas, in a more specific embodiment, the replaced coarse repaired image is smoothed.
[0041] In a more specific embodiment, the fine repair subnet adopts the fine repair subnet of DeepFillv2, and the discriminator also adopts the discriminator structure of DeepFillv2; wherein, the coarse repair subnet also includes hole convolution and deconvolution. Before using the generative adversarial network for image restoration, the generative adversarial network needs to be trained. The discriminator needs to be used during training. By continuously optimizing the discriminator, it can more accurately distinguish between real data and generated data, the generator continuously improves the generation effect, and the discriminator continuously improves the discrimination effect, thereby achieving the purpose of improving the generation effect of the generator. In a more specific embodiment, the method also includes a training process, and the training process is specifically as follows: The training samples are input into the generative adversarial network to obtain a coarse repaired image of the coarse repaired subnet, the mean square error between the non-repaired area in the coarse repaired image and the corresponding area of the real image is calculated, and the mean square error between the repaired area in the coarse repaired image and the real image is calculated, the loss of the coarse repaired subnet is calculated according to the above two average errors, and the coarse repaired subnet is trained based on the loss; after the coarse repaired subnet is trained, the entire generator is trained; after the generator training is completed, the entire generative adversarial network is trained.
[0042] When training the generative adversarial network, it is divided into multiple steps. The first step is to train only the coarse repair subnet. By training the coarse repair subnet, the coarse repair subnet can achieve good results as soon as possible, and prevent the fine repair subnet and the discriminator from deviating. Specifically, the mean square error between the non-repaired area in the coarse repair image and the corresponding area of the real image is calculated, and the mean square error between the repaired area in the coarse repair image and the real image is calculated. The loss of the coarse repair subnet is calculated based on these two errors, and this loss is used to train the coarse repair subnet. Among them, the real image is the image without the missing area added in the training sample, and the image of the missing area is added to the real image to form the image to be repaired.
[0043] There are multiple ways to calculate the loss of the coarse repair subnet based on the mean square error between the non-repaired area and the corresponding area of the real image, and the average error between the repaired area and the real image. One way is to calculate the loss by taking the two average errors with the largest average errors, and the other is to pre-set the weights of the two average errors. In a more specific embodiment, the loss of the coarse repair subnet is calculated based on the above two average errors, specifically: Set the weight of the mean square error of the non-repaired area and the weight of the mean square error of the repaired area.
[0044] The product of the mean square error of the non-repaired area and the corresponding weight, as well as the product of the mean square error of the repaired area and the corresponding weight are calculated, and the sum of the two products is taken as the loss of the coarse repair subnet.
[0045] After training the coarse repair subnet, the entire generator is trained, and after the generator is trained, the entire generative adversarial network is trained.
[0046] In a more specific embodiment, the training samples are divided into three parts, the first part is used to train the coarse repair subnet, the second part is used to train the entire generator, and the third part is used to train the entire generative adversarial network. There are many division ratios, such as 1:1:8 or 2:2:6, etc., and the present invention does not specifically limit the division ratio.
[0047] Figure 3 The figure shows the comparison of the effects of the solution of the present invention and DeepFillv2 on the same image, where: Figure 3 (A) is the image to be repaired, 3 (B) is the image repaired by DeepFillv2, and 3 (C) is the image repaired by the present invention.
[0048] Embodiment 2, the present invention also provides an image restoration system based on a generative adversarial network, such as Figure 4 As shown, the system includes the following modules: A feature extraction module is used to obtain an image to be repaired and a mask image of the image to be repaired, and input the image to be repaired and the mask image into a generator in a generative adversarial network. The coarse repair subnetwork of the generator includes multiple gated convolutional layers. The gated convolutional layer extracts features from the input feature map of the image to be repaired and the feature map of the mask image through gated convolution to obtain an output feature map of the gated convolutional layer.
[0049] The image restoration module is used to obtain the coarse restoration image of the coarse restoration subnetwork, divide the non-restoration part in the coarse restoration image into multiple regions, calculate the mean square error between each region and the corresponding position region in the image to be restored, correct the coarse restoration image according to the mean square error, input the corrected coarse restoration image into the fine restoration subnetwork, and use the output of the fine restoration subnetwork as the image restoration result.
[0050] Preferably, the gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain an output feature map of the gated convolution layer, specifically: The convolution operation is used to convert the feature map of the image to be repaired into a first channel map and a second channel map. The activation function is used to limit the element values of the first channel map of the feature map of the image to be repaired to between 0 and 1, and the first channel map is multiplied element by element with the second channel map to obtain the feature map of the second channel.
[0051] The feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps by using a convolution operation, and the element values of the mask image feature map are limited to between 0 and 1 by using an activation function, and the feature map of the first channel is obtained by multiplying the first channel map element by element.
[0052] The feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolutional layer.
[0053] Preferably, the correcting the rough repaired image according to the mean square error is specifically: When the mean square error is greater than a preset value, the image of the area in the image to be repaired is used to replace the corresponding area in the rough repair image.
[0054] The replaced rough repaired image is smoothed.
[0055] Preferably, the method further comprises a training process, and the training process specifically comprises: The training samples are input into the generative adversarial network to obtain a coarse repaired image of the coarse repaired subnet, the mean square error between the non-repaired area in the coarse repaired image and the corresponding area of the real image is calculated, and the mean square error between the repaired area in the coarse repaired image and the real image is calculated, the loss of the coarse repaired subnet is calculated according to the above two average errors, and the coarse repaired subnet is trained based on the loss; after the coarse repaired subnet is trained, the entire generator is trained; after the generator training is completed, the entire generative adversarial network is trained.
[0056] Preferably, the loss of the coarse repair subnet is calculated based on the two average errors, specifically: Set the weight of the mean square error of the non-repaired area and the weight of the mean square error of the repaired area.
[0057] The product of the mean square error of the non-repaired area and the corresponding weight, as well as the product of the mean square error of the repaired area and the corresponding weight are calculated, and the sum of the two products is taken as the loss of the coarse repair subnet.
[0058] Embodiment 3, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in embodiment 1 is implemented.
[0059] Embodiment 4: The present invention further provides a computer device, which includes at least a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the method described in Embodiment 1 is implemented.
[0060] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. An image restoration method based on a generative adversarial network, characterized in that: The method comprises the following steps: Acquire an image to be repaired and a mask image of the image to be repaired, input the image to be repaired and the mask image into a generator in a generative adversarial network, wherein a coarse repair subnetwork of the generator includes a plurality of gated convolutional layers, and the gated convolutional layer extracts features of an input feature map of the image to be repaired and a feature map of the mask image through gated convolution to obtain an output feature map of the gated convolutional layer; Obtain a coarse inpainted image of the coarse inpainting subnetwork, divide the non-inpainted part of the coarse inpainting image into multiple regions, calculate the mean square error between each region and the corresponding position region in the image to be inpainted, correct the coarse inpainted image according to the mean square error, input the corrected coarse inpainted image into the fine inpainting subnetwork, and use the output of the fine inpainting subnetwork as the image inpainting result.
2. The method according to claim 1, characterized in that The gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain the output feature map of the gated convolution layer, specifically: The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation, and the element value of the first channel map of the feature map of the image to be repaired is limited to between 0 and 1 by using an activation function, and the first channel map is multiplied element by element with the second channel map to obtain the feature map of the second channel; The feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps by using a convolution operation, the element values of the mask image feature map are limited to between 0 and 1 by using an activation function, and the mask image feature map is element-wise multiplied with the first channel map to obtain a feature map of the first channel; The feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolutional layer.
3. The method according to claim 1, characterized in that The correcting the rough repaired image according to the mean square error is specifically: When the mean square error is greater than a preset value, the image of the area in the image to be repaired is used to replace the corresponding area in the rough repair image; The replaced rough repaired image is smoothed.
4. The method according to claim 1, characterized in that The method further includes a training process, which specifically includes: Input the training sample into the generative adversarial network to obtain a coarse repair image of the coarse repair subnet, calculate the mean square error between the non-repaired area in the coarse repair image and the corresponding area of the real image, and calculate the mean square error between the repaired area in the coarse repair image and the real image, calculate the loss of the coarse repair subnet based on the above two average errors, and train the coarse repair subnet based on the loss; after the coarse repair subnet is trained, train the entire generator; After the generator training is completed, the entire generative adversarial network is trained.
5. The method according to claim 1, characterized in that The loss of the coarse repair subnet is calculated based on the above two average errors, specifically: Set the weight of the mean square error of the non-repaired area and the weight of the mean square error of the repaired area; The product of the mean square error of the non-repaired area and the corresponding weight, as well as the product of the mean square error of the repaired area and the corresponding weight are calculated, and the sum of the two products is taken as the loss of the coarse repair subnet.
6. An image restoration system based on a generative adversarial network, characterized in that: The system includes the following modules: A feature extraction module, used to obtain an image to be repaired and a mask image of the image to be repaired, and input the image to be repaired and the mask image into a generator in a generative adversarial network, wherein a coarse repair subnetwork of the generator includes a plurality of gated convolutional layers, and the gated convolutional layer extracts features from an input feature map of the image to be repaired and a feature map of the mask image through gated convolution to obtain an output feature map of the gated convolutional layer; The image restoration module is used to obtain the coarse restoration image of the coarse restoration subnetwork, divide the non-restoration part in the coarse restoration image into multiple regions, calculate the mean square error between each region and the corresponding position region in the image to be restored, correct the coarse restoration image according to the mean square error, input the corrected coarse restoration image into the fine restoration subnetwork, and use the output of the fine restoration subnetwork as the image restoration result.
7. The system according to claim 6, characterized in that The gated convolution layer extracts features of the input image feature map to be repaired and the mask image feature map through gated convolution to obtain the output feature map of the gated convolution layer, specifically: The feature map of the image to be repaired is converted into a first channel map and a second channel map by using a convolution operation, and the element value of the first channel map of the feature map of the image to be repaired is limited to between 0 and 1 by using an activation function, and the first channel map is multiplied element by element with the second channel map to obtain the feature map of the second channel; The feature map of the mask image is converted into a mask image feature map of the same size as the two channel feature maps by using a convolution operation, the element values of the mask image feature map are limited to between 0 and 1 by using an activation function, and the mask image feature map is element-wise multiplied with the first channel map to obtain a feature map of the first channel; The feature map of the first channel and the feature map of the second channel are averaged to obtain the output feature map of the gated convolutional layer.
8. The system according to claim 6, characterized in that The correcting the rough repaired image according to the mean square error is specifically: When the mean square error is greater than a preset value, the image of the area in the image to be repaired is used to replace the corresponding area in the rough repair image; The replaced rough repaired image is smoothed.
9. The system according to claim 6, characterized in that The system also includes a training module, which is specifically: Input the training sample into the generative adversarial network to obtain a coarse repair image of the coarse repair subnet, calculate the mean square error between the non-repaired area in the coarse repair image and the corresponding area of the real image, and calculate the mean square error between the repaired area in the coarse repair image and the real image, calculate the loss of the coarse repair subnet based on the above two average errors, and train the coarse repair subnet based on the loss; after the coarse repair subnet is trained, train the entire generator; After the generator training is completed, the entire generative adversarial network is trained.
10. A computer storage device, wherein a computer program is stored on the storage device, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Filling pipeline crack detection method
CN111784645A
Generative adversarial network image restoration method based on multi-scale texture feature branches
CN113902630A
Image restoration method based on wavelet self-attention generative adversarial network
CN115861108A
Very high-resolution image in-painting with neural networks
US20210150678A1
Cited By
Temporomandibular joint disease automatic diagnosis method and system based on artificial intelligence
CN119314661A
