An image inpainting system combining attention mechanism and generative adversarial network

Abstract

The application relates to the technical field of image restoration, in particular to an image restoration system combining an attention mechanism and a generative adversarial network. An input unit receives an input image and a mask image and performs preprocessing; a generation unit generates a restored image through an encoding layer, a double-flow parallel attention bottleneck layer and a decoding layer, wherein the double-flow parallel attention bottleneck layer generates global structure features by using a global context attention module, a mask-guided sparse attention module extracts local texture features, and the two are fused through an adaptive gating fusion module; a discrimination unit adopts a multi-scale structure, performs true-false judgment on images with different resolutions through parallel sub-discriminators, and introduces a gradient map as an additional channel; a training unit optimizes the generative adversarial network by using a composite loss function, and the composite loss function comprises an adversarial loss, a reconstruction loss and a multi-level frequency loss; and an output unit outputs a final restored image; the system effectively improves the structure consistency and texture fidelity of image restoration.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, specifically to an image restoration system that combines an attention mechanism with a generative adversarial network. Background Technology

[0002] In the field of digital image processing, image inpainting is a key technology that aims to automatically fill in or repair missing or damaged parts of an image based on information from known areas in the image. This technology has wide application value in scenarios such as cultural relic restoration, old photo restoration, film and television special effects, and target removal.

[0003] Early image inpainting methods were mainly based on pixel diffusion or texture synthesis. For example, methods based on partial differential equations filled the missing areas by gradually diffusing pixel information from the boundaries of the region to be repaired into the interior. However, this method was only suitable for small-area defects and did not perform well in restoring large-area textures and structures. Methods based on sample block matching filled the missing areas by searching for the most similar texture blocks in the known regions of the image. Although they could generate good textures, they were difficult to understand and restore complex semantic structures, and were prone to structural distortion and boundary artifacts.

[0004] In recent years, with the development of deep learning, generative adversarial networks (GANs) have been able to generate highly realistic and semantically consistent repair results through adversarial training of the generator and discriminator. However, traditional GAN ​​models still face challenges in handling large-area irregular missing parts.

[0005] (1) Standard convolutional neural networks have a limited receptive field when extracting features, making it difficult to capture long-distance dependencies in images, resulting in distorted or discontinuous structures.

[0006] (2) The model has difficulty simultaneously focusing on global semantic consistency and local texture details.

[0007] (3) Traditional repair networks do not make full use of the contextual information around the missing area and cannot understand the scene content, thus generating content that does not match the surrounding environment.

[0008] To address this, an image inpainting system combining attention mechanisms and generative adversarial networks is proposed. Summary of the Invention

[0009] The purpose of this invention is to provide an image restoration system that combines attention mechanisms with generative adversarial networks.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] An image inpainting system combining attention mechanisms and generative adversarial networks, comprising:

[0012] The input unit is used to receive the input image and the corresponding mask image, and preprocess them to obtain a preprocessed image;

[0013] The generation unit includes an encoding layer, a dual-stream parallel attention bottleneck layer, and a decoding layer, used to receive a preprocessed image and generate a repaired image. The dual-stream parallel attention bottleneck layer generates structural features through a global context attention module, extracts texture features from known regions through a mask-guided sparse attention module, and fuses structural features and texture features through an adaptive gated fusion module to generate a repaired image.

[0014] The discrimination unit adopts a multi-scale structure design based on gradient correlation. It is used to receive the restored image and the real image, and to judge the authenticity of the images at different resolutions through parallel sub-discriminators. The gradient maps of the restored image and the real image are used as additional channels of the discriminator.

[0015] The training unit is used to train the generative adversarial network using a composite loss function, which includes: adversarial loss, reconstruction loss, and multi-level frequency loss. The adversarial loss is the sum of the losses of all parallel sub-discriminators, and the multi-level frequency loss is calculated using Laplacian pyramid decomposition.

[0016] The output unit is used to output the final repaired image after training and optimization.

[0017] Furthermore, the preprocessing of the input image and the corresponding mask image includes:

[0018] The input unit receives the input image and the corresponding mask image;

[0019] Adjust the size of the input image and the mask image to the resolution required by the generation unit;

[0020] The input image and the mask image are normalized to scale the pixel values ​​to the range of [0, 1] to meet the input requirements of the generation unit.

[0021] The normalized input image and mask image are used as preprocessed images and passed to the generation unit for further processing.

[0022] Furthermore, the generation unit receives the preprocessed image, and the process of generating the repaired image includes:

[0023] The coding layer downsamples the preprocessed image through multiple convolutional operations to extract low-dimensional feature representations to capture the semantic information of the image;

[0024] The dual-stream parallel attention bottleneck layer processes the output features of the coding layer, generates global structural features through the global context attention module, generates local texture features through the mask-guided sparse attention module, and fuses the two features through the adaptive gating fusion module.

[0025] The decoding layer upsamples the fused features through multiple deconvolution operations to generate a repaired image with the same resolution as the input image.

[0026] Furthermore, the process by which the dual-stream parallel attention bottleneck layer processes the output features of the coding layer includes:

[0027] The global context attention module calculates the global correlation of the output features of the encoding layer through a self-attention mechanism, generating global structural features;

[0028] The mask-guided sparse attention module uses the region to be repaired as the query object based on the mask image, obtains key objects and value objects from the known regions, performs local attention calculations, and extracts local texture features.

[0029] The adaptive gating fusion module learns adaptive weights to perform weighted fusion of global structural features and local texture features to generate fused features.

[0030] Furthermore, the process by which the discrimination unit judges the authenticity of the restored image and the real image includes:

[0031] Calculate the gradient maps of the restored image and the real image, and pair them with the corresponding restored image and the real image to form image pairs;

[0032] Image pairs are downsampled and then real or fake image pairs at different resolutions are distinguished by parallel sub-discriminators. Each sub-discriminator processes image features at the corresponding scale.

[0033] The outputs of all parallel sub-discriminators are aggregated to generate the final true / false result to guide the optimization of the generation unit.

[0034] Furthermore, the training process of the training unit includes:

[0035] Generative adversarial networks are optimized using a composite loss function, which includes adversarial loss, reconstruction loss, and multi-level frequency loss.

[0036] Adversarial loss measures the consistency of the distribution between the restored image and the real image by calculating the sum of the losses of all parallel sub-discriminators;

[0037] Reconstruction loss measures the pixel accuracy of the restored image by comparing the pixel-level differences between the restored image and the original image.

[0038] Multi-level frequency loss is decomposed using the Laplacian pyramid to calculate the differences between the restored image and the real image at different frequency components, measuring the consistency of the image in terms of structure and detail.

[0039] The parameters of the generating and discriminating units are iteratively updated through backpropagation and optimization algorithms until the loss converges, generating the final restored image.

[0040] Furthermore, before outputting the final repaired image after training and optimization, the output unit applies super-resolution technology to the repaired image to meet the resolution requirements.

[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0042] 1. This invention proposes a dual-stream parallel attention processing procedure. It captures long-distance dependencies by utilizing the self-attention mechanism of the global context attention module to generate global structural features. The mask-guided sparse attention module fully utilizes contextual information to extract texture features from known regions for the area to be repaired, avoiding the generation of content that does not match the environment due to insufficient utilization of context in traditional methods. Finally, the adaptive gating fusion module dynamically fuses global structural and local texture features to ensure a balance between global consistency and local details in the repaired image, significantly improving the realism and naturalness of the repair effect.

[0043] 2. This invention proposes a gradient-correlation-based structural consistency discriminator. By using parallel sub-discriminators to determine the authenticity of image pairs at different resolutions, and combining this with gradient maps as an additional channel, the discriminator's ability to perceive image structure and details is enhanced, overcoming the shortcomings of traditional GAN ​​models in judging large-area missing images for restoration. The aggregation of multi-scale discrimination results guides the optimization of the generation unit, ensuring that the restored image closely approximates the real image at different resolutions, reducing boundary artifacts and structural discontinuities in traditional methods, and improving the overall restoration quality.

[0044] 3. This invention proposes a composite loss function for optimizing image inpainting networks. The composite loss function includes adversarial loss, reconstruction loss, and multi-level frequency loss, which comprehensively optimizes the quality of the inpainted image and solves the shortcomings of traditional GAN ​​models in balancing global semantics and local details. By calculating multi-level frequency loss through Laplacian pyramid decomposition, it ensures that the inpainted image is consistent with the real image in different frequency components, which significantly improves the problem of poor texture and structure restoration in traditional methods when inpainting large areas. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the structure of an image restoration system combining an attention mechanism and a generative adversarial network according to the present invention;

[0046] Figure 2This is a schematic diagram of the structure of the dual-stream parallel attention bottleneck layer of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the process by which the discrimination unit of the present invention judges the authenticity of the repaired image and the real image. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Please see Figures 1 to 3 This invention provides an image inpainting system that combines an attention mechanism with a generative adversarial network. The technical solution is as follows:

[0050] Example 1:

[0051] To improve image restoration quality, especially in large-area scenarios, a company used an image restoration system proposed in this invention that combines an attention mechanism with a generative adversarial network. The system structure is as follows: Figure 1 As shown, it includes:

[0052] The input unit is used to receive the input image and the corresponding mask image, and preprocess them to obtain a preprocessed image;

[0053] Furthermore, the preprocessing of the input image and the corresponding mask image includes:

[0054] The input unit receives the input image and the corresponding mask image;

[0055] Adjust the size of the input image and the mask image to the resolution required by the generation unit;

[0056] The input image and the mask image are normalized to scale the pixel values ​​to the range of [0, 1] to meet the input requirements of the generation unit.

[0057] The normalized input image and mask image are used as preprocessed images and passed to the generation unit for subsequent processing.

[0058] Furthermore, the process of obtaining the mask image corresponding to the input image is as follows: the user draws the area to be repaired in the input image on the touch screen or mouse; the background system generates a binary mask image in real time based on the area to be repaired, which is used to mark the area drawn by the user; where 0 represents the area to be repaired; 255 represents the known area, i.e., the non-repaired area;

[0059] Furthermore, bilinear interpolation is used to adjust the dimensions of the input image and the mask image;

[0060] Furthermore, the input image and the mask image are normalized by dividing the pixel values ​​of the image by 255. In this case, 0 in the normalized mask image represents the area to be repaired, and 1 represents the known area.

[0061] By preprocessing the input image and the mask image, the standardization of the input data of the generation unit is ensured, avoiding deviations in the generation results caused by inconsistent inputs, and providing a data foundation for the subsequent image restoration process.

[0062] The generation unit includes an encoding layer, a dual-stream parallel attention bottleneck layer, and a decoding layer, which are used to receive the preprocessed image and generate the repaired image. The dual-stream parallel attention bottleneck layer generates structural features through a global context attention module, extracts texture from known regions through a mask-guided sparse attention module to generate texture features, and fuses the structural features and texture features through an adaptive gated fusion module to generate the repaired image.

[0063] Furthermore, the generation unit receives the preprocessed image, and the process of generating the repaired image includes:

[0064] The encoding layer downsamples the preprocessed image through multiple convolutional operations to extract low-dimensional feature representations to capture the semantic information of the image;

[0065] The dual-stream parallel attention bottleneck layer processes the output features of the encoding layer, generates global structural features through the global context attention module, generates local texture features through the mask-guided sparse attention module, and fuses the two features through the adaptive gating fusion module.

[0066] The decoding layer upsamples the fused features through multiple deconvolution operations to generate a repaired image with the same resolution as the input image.

[0067] Furthermore, both the encoding and decoding layers use convolutional neural networks. The encoding layer reduces spatial resolution through a 3×3 convolution operation with a stride of 2 and performs three downsampling operations. The decoding layer gradually increases spatial resolution through a 3×3 transposed convolution operation with a stride of 2. The last layer uses the sigmoid function to ensure that the output pixel value is within the range of [0, 1]. Skip connections can be used to combine the features of the encoding layer during the decoding process to enhance detail recovery.

[0068] Through multi-stage processing of the generation unit, the encoding layer extracts low-dimensional features and captures semantic information of the image through multi-layer convolutional downsampling; the dual-stream parallel attention bottleneck layer generates global structural features and local texture features respectively and performs dynamic fusion; the decoding layer generates the repaired image through deconvolutional upsampling; the system can generate semantically consistent and textured repaired images, overcoming the defects of traditional methods in structural distortion or discontinuity when dealing with large-area missing areas.

[0069] Furthermore, the structure of the dual-stream parallel attention bottleneck layer is as follows: Figure 2 As shown, the process of processing the output features of the coding layer includes:

[0070] The global context attention module calculates the global correlation of the output features of the encoding layer through a self-attention mechanism, generating global structural features;

[0071] The mask-guided sparse attention module uses the region to be repaired as the query object based on the mask image, obtains key objects and value objects from the known regions, performs local attention calculations, and extracts local texture features.

[0072] The adaptive gating fusion module learns adaptive weights to perform weighted fusion of global structural features and local texture features to generate fused features;

[0073] Furthermore, the process of the global context attention module generating global structural features is as follows: receiving the output features of the encoding layer and converting them into a query matrix, a key matrix, and a value matrix through a linear transformation; dividing the result of the dot product of the query matrix and the key transpose matrix by the square root of the key vector dimension to obtain the attention score; applying the softmax function to the attention score to generate attention weights, and using these weights to perform a weighted summation of the value matrix to generate global context features;

[0074] Furthermore, the process of extracting local texture features by the mask-guided sparse attention module is as follows: the output features of the coding layer are segmented into features of the region to be repaired and features of the known region using the downsampled mask image; the features of the region to be repaired are converted into a query matrix, and the features of the known region are converted into a key matrix and a value matrix; the process of calculating attention scores and weights is consistent with that in the global context attention module, and the value matrix of the known region is weighted and summed using attention weights to extract local texture features;

[0075] Furthermore, the global context features and local texture features are concatenated along the channel dimension to obtain concatenated features. A weight generation network is used to process the concatenated features and output a weight matrix, including global context feature weights and local texture feature weights. The weight generation network is a fully connected network and is trained together with the generation and discriminator units, and optimized through a composite loss function. The generated global context feature weights and local texture feature weights are used to perform a weighted summation of the global context features and local texture features, respectively, to generate fused features.

[0076] Furthermore, the mask-guided sparse attention module only focuses on the repair area and the known area during the extraction of local texture features, neglecting the boundary changes between regions. To address this, an edge detection algorithm is used to process the mask image, generating a mask boundary map. Then, the downsampled mask boundary map is used to segment the output features of the encoding layer to extract boundary features. These features are then merged with the features of the area to be repaired, serving as a common query object. This reduces unnatural transitions between the repair area and the known area, improving the fidelity of edge details.

[0077] By leveraging the self-attention mechanism of the global context attention module to capture long-distance dependencies, global structural features are generated. The mask-guided sparse attention module fully utilizes contextual information to extract texture features from known regions for the area to be repaired, avoiding the generation of content that does not match the environment due to insufficient use of context in traditional methods. Finally, the adaptive gating fusion module dynamically fuses global structural and local texture features to ensure a balance between global consistency and local details in the repaired image, significantly improving the realism and naturalness of the repair effect.

[0078] The discrimination unit adopts a multi-scale structure design based on gradient correlation. It is used to receive the restored image and the real image, and to judge the authenticity of the images at different resolutions through parallel sub-discriminators. The gradient maps of the restored image and the real image are used as additional channels of the discriminator.

[0079] Furthermore, the discrimination unit performs a process of determining the authenticity of the restored image and the original image as follows: Figure 3 As shown, it includes:

[0080] Calculate the gradient maps of the restored image and the real image, and pair them with the corresponding restored image and the real image to form image pairs;

[0081] Image pairs are downsampled and then real or fake image pairs at different resolutions are distinguished by parallel sub-discriminators. Each sub-discriminator processes image features at the corresponding scale.

[0082] The outputs of all parallel sub-discriminators are aggregated to generate the final true / false discrimination result to guide the optimization of the generation unit;

[0083] Furthermore, the Sobel operator is applied to the restored image and the real image, and the gradients in the corresponding directions are calculated using Sobel kernels in the horizontal and vertical directions. Then, the gradient map is paired with the original image to form an image pair, which includes a five-channel restored image pair and a five-channel real image pair. For example, the five-channel restored image pair consists of the RGB three channels of the restored image and the gradient channels in the horizontal and vertical directions.

[0084] Furthermore, stride convolution is used to downsample the image pairs multiple times to generate image pairs with different resolutions, in order to meet the input requirements of different sub-discriminators;

[0085] Furthermore, the sub-discriminator is composed of multiple convolutional layers and LeakyReLU activation functions, outputting a true / false probability map at the corresponding scale; the true / false probability map output by each sub-discriminator is averaged to obtain a scalar probability, which is used as the discrimination result; the multi-scale discrimination results are integrated by weighted averaging to evaluate the authenticity of the restored image;

[0086] Furthermore, a pre-trained semantic segmentation model is used to extract corresponding semantic features from the unsampling repaired image and the real image, respectively. Then, the semantic features of the repaired image and the real image are used as additional input channels for the sub-discriminators at the corresponding scale, while the inputs of the other sub-discriminators remain unchanged. The pre-trained semantic segmentation model can adopt the DeepLab model, which can improve the discriminator's ability to evaluate the semantic authenticity of the repaired image and enhance the consistency between the repair results and the global semantics.

[0087] By using a parallel sub-discriminator to determine the authenticity of image pairs at different resolutions and combining gradient maps as an additional channel, the discriminator's ability to perceive image structure and details is enhanced, making up for the shortcomings of traditional GAN ​​models in judging large-area missing images for restoration. The aggregation of multi-scale discrimination results guides the optimization of the generation unit, making the restored image close to the real image at different resolutions, reducing boundary artifacts and structural discontinuities in traditional methods, and improving the overall restoration quality.

[0088] The training unit is used to train the generative adversarial network using a composite loss function, which includes: adversarial loss, reconstruction loss, and multi-level frequency loss. The adversarial loss is the sum of the losses of all parallel sub-discriminators, and the multi-level frequency loss is calculated using Laplacian pyramid decomposition.

[0089] Furthermore, the training process of the training unit includes:

[0090] Generative adversarial networks are optimized using composite loss functions, which include adversarial loss, reconstruction loss, and multi-level frequency loss.

[0091] Adversarial loss measures the consistency of the distribution between the restored image and the real image by calculating the sum of the losses of all parallel sub-discriminators;

[0092] Reconstruction loss measures the pixel accuracy of the restored image by comparing the pixel-level differences between the restored image and the original image.

[0093] Multi-level frequency loss is decomposed using the Laplacian pyramid to calculate the differences between the restored image and the real image at different frequency components, measuring the consistency of the image in terms of structure and detail.

[0094] The parameters of the generating and discriminating units are iteratively updated through backpropagation and optimization algorithms until the loss converges, generating the final restored image.

[0095] Furthermore, the composite loss function is obtained by weighted summation of adversarial loss, reconstruction loss, and multi-level frequency loss;

[0096] Furthermore, the adversarial loss adopts the binary cross-entropy loss based on standard GAN, which is divided into two parts: discriminator and generator. The discriminator loss is the sum of the losses of all parallel sub-discriminators, including the real image pair part and the repaired image pair part. Specifically, the real image pair is input into the discriminator, the logarithm of its scalar probability is calculated, and then the expectation is taken for the batch samples to encourage the discriminator output to be close to 1; the repaired image pair is input into the discriminator, 1 minus the logarithm of its scalar probability is calculated, and then the expectation is taken for the batch samples to encourage the discriminator output to be close to 0.

[0097] Furthermore, the generator loss is similar to the loss calculation process for the real image pair, except that the input is the repaired image pair, and the discriminator output is encouraged to be close to 1;

[0098] Furthermore, the reconstruction loss uses the L1 norm to calculate the pixel-level difference between the restored image and the real image; the multi-level frequency loss uses the L1 norm to calculate the difference between the restored image and the real image at different frequency components.

[0099] Furthermore, while the Laplacian pyramid primarily captures resolution-related high-frequency details (such as edges) and low-frequency structures (such as smooth regions), it fails to capture details and directional features of local textures. Therefore, multi-level frequency loss introduces discrete wavelet transform decomposition to decompose the restored image and the real image into multi-scale high-frequency and low-frequency components. The L1 loss of the high-frequency and low-frequency components at each scale is calculated and added to the multi-level frequency loss, thereby simultaneously optimizing global structure and local directional textures and improving the realism of the restored image in multiple frequencies and directions.

[0100] By using a composite loss function to comprehensively optimize the quality of the restored image, the shortcomings of traditional GAN ​​models in balancing global semantics and local details are addressed. By calculating multi-level frequency loss through Laplacian pyramid decomposition, the restored image is ensured to be consistent with the real image at different frequency components, significantly improving the problem of poor texture and structure restoration in traditional methods when restoring large areas.

[0101] The output unit is used to output the final repaired image after training and optimization.

[0102] Furthermore, before outputting the final repaired image after training and optimization, the output unit applies super-resolution technology to the repaired image to meet the resolution requirements;

[0103] Furthermore, the output unit denormalizes the high-resolution restored image and saves it as a standard image format for output.

[0104] By applying super-resolution technology before outputting the final restored image, the high resolution requirement is met while overcoming the problems of insufficient resolution or blurred details after restoring large-area defects using traditional methods, thereby improving the image restoration quality.

[0105] This embodiment proposes an image inpainting system combining an attention mechanism and a generative adversarial network (GAN). The input unit receives and preprocesses the input image and mask image. The generation unit generates the inpainted image through an encoding layer, a dual-stream parallel attention bottleneck layer, and a decoding layer. The dual-stream parallel attention bottleneck layer utilizes a global contextual attention module to generate global structural features, while a mask-guided sparse attention module extracts local texture features. These two features are then fused using an adaptive gating fusion module. The discrimination unit employs a multi-scale structure, using parallel sub-discriminators to determine the authenticity of images at different resolutions and introducing gradient maps as additional channels. The training unit optimizes the GAN using a composite loss function, which includes adversarial loss, reconstruction loss, and multi-level frequency loss. The output unit outputs the final inpainted image. This system effectively improves the structural consistency and texture fidelity of image inpainting.

[0106] Example 2:

[0107] This embodiment uses the restoration of old photos in a cultural heritage digitization project as an example to illustrate the application process of an image restoration system that combines attention mechanism and generative adversarial network proposed in this invention.

[0108] The input unit receives the old photo to be restored and the mask image drawn by the user; the mask image is generated by the user manually marking the missing areas on the digital platform using a mouse or touch screen, with missing areas marked as 0 and non-missing areas marked as 255;

[0109] The input old photo and mask image are adjusted to a resolution of 512×512 pixels using bilinear interpolation. Then, the pixel values ​​of the input image and mask image are normalized to generate a preprocessed image, which is then passed to the generation unit.

[0110] The generation unit receives the preprocessed image and generates the repaired image through the encoding layer, the dual-stream parallel attention bottleneck layer, and the decoding layer;

[0111] The encoding layer downsamples the preprocessed image through three 3×3 convolution operations with a stride of 2, generating a low-dimensional feature representation with a spatial resolution of 64×64 and 256 channels. After receiving the low-dimensional feature representation, the dual-stream parallel attention bottleneck layer generates global structural features and local texture features through a global context attention module and a mask-guided sparse attention module, respectively. The spatial resolution and number of channels are consistent with the low-dimensional feature representation. Then, an adaptive gated fusion module is used for dynamic fusion. The decoding layer upsamples the fused features through three 3×3 transposed convolution operations with a stride of 2, gradually restoring the resolution to 512×512, generating a repaired image with pixel values ​​in the range of [0, 1].

[0112] The discrimination unit adopts a multi-scale structure design based on gradient correlation, receives the restored image and the real reference image, and performs authenticity judgment.

[0113] First, the Sobel operator is used to calculate the horizontal and vertical gradient maps of the restored image and the real image, forming a five-channel image pair. The image pair is downsampled twice to generate image pairs with three resolutions: 512×512, 256×256, and 128×128. Then, three parallel sub-discriminators are used to process the image pairs with different resolutions. Each sub-discriminator consists of four convolutional layers and the LeakyReLU activation function, and outputs a true / false probability map to guide the optimization of the generation unit.

[0114] The training units optimize the generative adversarial network (GAN) using a composite loss function, namely the generator and the discriminator. Backpropagation is performed using the Adam optimizer with a learning rate of 0.0002 and a hyperparameter of 0.5. The parameters of the generator and the discriminator are iteratively updated. The training is conducted for 100 epochs with a batch size of 16 until the loss converges.

[0115] The output unit receives the repaired image after training and optimization, applies super-resolution technology to improve the resolution to 1024×1024, and then inversely normalizes the pixel values ​​of the repaired image from [0, 1] to [0, 255], and saves it as a PNG format for output.

[0116] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image inpainting system combining attention mechanisms and generative adversarial networks, characterized in that, include: The input unit is used to receive the input image and the corresponding mask image, and preprocess them to obtain a preprocessed image; The generation unit includes an encoding layer, a dual-stream parallel attention bottleneck layer, and a decoding layer, used to receive a preprocessed image and generate a repaired image. The dual-stream parallel attention bottleneck layer generates structural features through a global context attention module, extracts texture features from known regions through a mask-guided sparse attention module, and fuses structural and texture features through an adaptive gated fusion module to generate the repaired image. The process of determining the known regions involves the user drawing the region to be repaired in the input image, generating a binary mask image based on the region to be repaired, where 0 represents the region to be repaired and 255 represents the known region. The discrimination unit employs a gradient correlation-based multi-scale structure design to receive the restored image and the original image. This gradient correlation-based multi-scale structure design applies the Sobel operator to both the restored and original images, using horizontal and vertical Sobel kernels to calculate gradients in corresponding directions. The gradient maps are then paired with the original images, and stride convolution is used to downsample the image pairs, generating image pairs at different resolutions. A parallel sub-discriminator then judges the authenticity of the images at different resolutions, using the gradient maps of the restored and original images as additional channels for the discriminator. The training unit is used to train the generative adversarial network using a composite loss function, which includes: adversarial loss, reconstruction loss, and multi-level frequency loss. The adversarial loss is the sum of the losses of all parallel sub-discriminators, and the multi-level frequency loss is calculated using Laplacian pyramid decomposition. The output unit is used to output the final repaired image after training and optimization.

2. The image inpainting system combining attention mechanism and generative adversarial network according to claim 1, characterized in that, The preprocessing of the input image and the corresponding mask image includes: The input unit receives the input image and the corresponding mask image; Adjust the size of the input image and the mask image to the resolution required by the generation unit; The input image and the mask image are normalized to scale the pixel values ​​to the range of [0, 1] to meet the input requirements of the generation unit. The normalized input image and mask image are used as preprocessed images and passed to the generation unit for further processing.

3. The image inpainting system combining attention mechanism and generative adversarial network according to claim 1, characterized in that, The generation unit receives the preprocessed image, and the process of generating the repaired image includes: The coding layer downsamples the preprocessed image through multiple convolutional operations to extract low-dimensional feature representations to capture the semantic information of the image; The dual-stream parallel attention bottleneck layer processes the output features of the coding layer, generates global structural features through the global context attention module, generates local texture features through the mask-guided sparse attention module, and fuses the two features through the adaptive gating fusion module. The decoding layer upsamples the fused features through multiple deconvolution operations to generate a repaired image with the same resolution as the input image.

4. The image inpainting system combining attention mechanism and generative adversarial network according to claim 3, characterized in that, The process by which the dual-stream parallel attention bottleneck layer processes the output features of the coding layer includes: The global context attention module calculates the global correlation of the output features of the encoding layer through a self-attention mechanism, generating global structural features; The mask-guided sparse attention module uses the region to be repaired as the query object based on the mask image, obtains key objects and value objects from the known regions, performs local attention calculations, and extracts local texture features. The adaptive gating fusion module learns adaptive weights to perform weighted fusion of global structural features and local texture features to generate fused features.

5. The image inpainting system combining attention mechanism and generative adversarial network according to claim 1, characterized in that, The process by which the discrimination unit judges the authenticity of the restored image and the real image includes: Calculate the gradient maps of the restored image and the real image, and pair them with the corresponding restored image and the real image to form image pairs; Image pairs are downsampled and then real or fake image pairs at different resolutions are distinguished by parallel sub-discriminators. Each sub-discriminator processes image features at the corresponding scale. The outputs of all parallel sub-discriminators are aggregated to generate the final true / false result to guide the optimization of the generation unit.

6. The image inpainting system combining attention mechanism and generative adversarial network according to claim 1, characterized in that, The training process of the training unit includes: Generative adversarial networks are optimized using a composite loss function, which includes adversarial loss, reconstruction loss, and multi-level frequency loss. Adversarial loss measures the consistency of the distribution between the restored image and the real image by calculating the sum of the losses of all parallel sub-discriminators; Reconstruction loss measures the pixel accuracy of the restored image by comparing the pixel-level differences between the restored image and the original image. Multi-level frequency loss is decomposed using the Laplacian pyramid to calculate the differences between the restored image and the real image at different frequency components, measuring the consistency of the image in terms of structure and detail. The parameters of the generating and discriminating units are iteratively updated through backpropagation and optimization algorithms until the loss converges, generating the final restored image.

7. The image inpainting system combining attention mechanism and generative adversarial network according to claim 1, characterized in that, Before outputting the final repaired image after training and optimization, the output unit applies super-resolution technology to the repaired image to meet the resolution requirements.

Images