Image inpainting method, model and device based on attention modulation adversarial network

Abstract

The application discloses an image restoration method, a model and a device based on an attention modulation adversarial network, and a model construction method, and comprises the following steps: step 1, collecting multiple images of tomb murals; step 2, dividing the images with actual damage into a training set, a test set and a verification set; step 3, taking the images without actual damage as the training set of a restoration model; step 4, sending the training set, the test set and the position of the image damage area into an improved detection network for training; step 5, sending the training set of the restoration model and a randomly generated binary mask image into an improved modulation adversarial generation network for training; step 6, sending the verification set into a detection model and then converting the verification set into a binary image; and step 7, sending the mural image and the binary image into the trained restoration network to obtain a restored image. The application can more effectively extract the features of the tomb murals, improve the accuracy of damage area detection, and reconstruct high-quality tomb mural images by using the structure and texture information of the undamaged images.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an image restoration method, a model and a device based on an attention modulation adversarial network. BACKGROUND

[0002] Chinese ancient murals, as a unique carrier in the splendid cultural heritage of China, not only have high artistic value, but also are important data for studying the social customs and production and living style of ancient China. However, the damage phenomenon of the discovered tombs murals is very common at present, and the primary problem for further research is to repair and protect the damaged tombs murals. In recent years, the mural repair work is mostly artificial repair, and various analysis detection instruments and new materials are used to upgrade the traditional mural repair method. However, the mural repair method upgraded by means of new technology still has many problems. The tombs murals subjected to theft and damage often have serious cracking and peeling phenomenon, which leads to the problems of large artificial repair workload, complex process, and long repair cycle. For different burial environments and different preservation conditions of the tombs murals, the repair method needs to be adjusted again to avoid secondary damage. Therefore, the artificial repair of the tombs murals is difficult, and the technical accomplishment of the repair personnel is also required.

[0003] The virtual mural repair method provides a new solution for the problems of the artificial mural repair method. The virtual mural repair method separates the mural information from the cultural relics body through digital repair of the high-resolution damaged mural image, restores the original appearance of the damaged mural without operating the original mural, reduces the artificial repeated workload, and quickly restores the artistic works to the world, which meets the "minimum intervention" concept, avoids the secondary damage phenomenon of the mural due to the current technical problems, preserves the original information of the mural, and provides support for mural protection work.

[0004] For the repair work of the damaged tombs murals, it can be divided into two aspects of damaged area detection and damaged area repair. After the damaged area of the picture is successfully detected, a mask can be generated according to the damaged area, and only according to the mask and the original picture can the picture be repaired. Therefore, how to accurately detect the damaged area is a necessary prerequisite for the repair of the picture.

[0005] In the damaged area detection part of the tomb mural, the existing two-stage and query-based instance segmentation methods have achieved remarkable results. However, their segmentation masks are still very rough. Therefore, some scholars have proposed a method for high-quality and efficient instance segmentation. This method does not operate on conventional dense tensors, but rather decomposes and represents image regions as quad trees. Although these sparse pixels only account for a small portion of the total, they are crucial to the final mask quality. This allows the method to predict highly accurate instance masks at a lower computational cost.

[0006] In the damaged area repair part of the tomb mural, existing image inpainting methods can be roughly divided into two categories: traditional image inpainting methods and deep learning-based image inpainting methods. Traditional image inpainting methods use low-level image priors such as texture and structure to repair small damaged areas, but their performance is poor when the damaged area is large. Deep learning-based image inpainting methods can learn deep image semantic features through an end-to-end approach, making them more effective at image repair and outperforming traditional methods. Early deep repair networks were limited to repairing rectangular holes and often used convolutional filters directly on damaged images, ignoring the interference of missing areas on the repair results, resulting in texture loss, color distortion, artificial edges, and various types of visual artifacts in the repaired image. In recent years, generative adversarial networks have shown excellent performance in image editing and image generation. Some scholars have applied generative adversarial networks to image inpainting by adding collaborative modulation and other improvements to the generative adversarial network, achieving good results in image inpainting. The key to the collaborative modulation-based generative adversarial network repair method is to co-modulate the image of the undamaged part and the latent random vector, utilizing their linear correlation in the style domain to improve the generative effect of the generative adversarial network and thus achieve image inpainting.

[0007] Tomb murals often have complex structures and varying degrees of damage. The lack of structural information can result in the loss of a large amount of texture information, severely affecting image detection and repair results. Existing deep learning-based image detection and repair methods cannot balance the detection and repair of image structure and texture well, resulting in excessive false detection and positioning errors in damaged areas, improper structure repair, and missing details in the repaired image. These problems limit the application of existing image detection and repair algorithms in the digital restoration of tomb murals. Currently, there are no integrated methods, models, and devices for damaged area detection and repair for tomb murals in China. SUMMARY

[0008] In view of the defects and deficiencies in the prior art, the application provides an image restoration method, model and device based on an attention modulation adversarial network, which establishes a damaged area detection model and an image restoration model, and uses the trained model for damaged area detection and restoration of a tomb mural. In addition, the two stages of damaged area detection and restoration of the tomb mural are integrated, solving the problem of separate detection and restoration in the current industry and improving the process and intelligence of tomb mural protection. At the same time, the problems of excessive false detection and positioning error of the damaged area in the existing deep learning technology applied directly to the tomb mural image, and the problem of detail loss of the restored tomb mural image caused by improper image structure restoration are overcome, thereby providing technical support for the application and promotion of deep learning in the digital protection of cultural heritage in the future.

[0009] To achieve the above-mentioned purpose, the application adopts the following technical solutions:

[0010] On the one hand, an image restoration model construction method based on an attention modulation adversarial network is provided, which specifically includes the following steps:

[0011] Step 1: Collect multiple images of tomb murals to obtain a data set of tomb murals, and divide the data set into two parts of tomb murals with actual damage and tomb murals without actual damage;

[0012] Step 2: Divide the tomb mural images with actual damage into a training set, a test set and a validation set, and use a labeling tool to label the damaged areas of the tomb murals in the training set and the test set for subsequent detection model training;

[0013] Step 3: Use the tomb mural images without actual damage as the training set of the restoration model, and process these images for subsequent restoration model training;

[0014] Step 4: Send the training set and test set obtained in step 2 and the corresponding labeled image damaged area position data into an improved damaged area detection network based on a multiple attention mechanism for training; the improved damaged area detection network based on the multiple attention mechanism includes a multiple attention module and a quadtree structure transformer module;

[0015] Step 5: Send the training set of the restoration model in step 3 and a randomly generated binary mask image into an improved modulation adversarial generation network for training;

[0016] Step 6: Process the validation set in step 2 to obtain a tomb mural image with a damaged area with a size of 512*512, and send the image into the trained tomb mural damaged area detection model obtained in step 4, which temporarily stores the detected damaged area position information and directly converts it into a mask binary image;

[0017] Step 7: The image of the tomb mural obtained by processing in step 6 is combined with the mask binary obtained by detecting the image. Figure One Step 8: The trained tomb mural damage repair network obtained in step 5 is inputted to obtain the final repaired tomb mural image.

[0018] Further, in step 4, the multi-attention module is used to implement the following process:

[0019] Step 411: The input features are divided into G base groups to control the connection between the input and the output, and each base group is further divided into R subgroups, so the total number of subgroups is GR; a series of transformations are applied to each subgroup The change process of each subgroup is represented as Where X represents the input feature information, i∈{1,2,…,GR};

[0020] Step 412: In the Split Attention module of each base group, the combination of the G base groups of input features can be obtained by element-wise summation across multiple subgroups, g∈{1,2,…,G}, and the gth base group is represented as:

[0021]

[0022] Where, R represents the number of subgroups within a base group, i.e., H, W and C' represent the size of the sub-group block output feature map;

[0023] Then, the average value of each sub-group block feature map is calculated using global average pooling across the channel and spatial dimensions:

[0024]

[0025] Where represents the cth component of the gth base group in c∈{1,2,…,C' / G};

[0026] Step 413: Each base group component in step 412 is weighted and fused, and all base groups are concatenated along the channel dimension through the Concat function;

[0027] Step 414: The concatenated data is compressed through a 1x1 convolution layer to feature maps N=[N 1 ,N 2 ,…,N G ], which has the same size as the input feature map; then, the output of the multi-attention module is generated through a shortcut connection in residual learning, which is referred to as the coarse mask of the input damaged image.

[0028] Further, in step 4, the quadtree structure transformer module is used to implement the following process:

[0029] Step 421, the feature pyramid takes the input damaged image and its estimated coarse mask as input, and outputs three scale feature maps through a regular convolutional network layer;

[0030] Step 422, for each scale of the cascaded pyramid feature map, the incoherent detector identifies and refines the incoherent region by using the coarse mask and the region of interest feature, and outputs a node sequence; according to the detected incoherent region, a quadtree pyramid is constructed in the form of three-level feature maps; if a pixel in a higher level feature map is predicted to be a discontinuous node, the node is decomposed into four quadrant pixels of adjacent lower level feature maps through upsampling; a node sequence is constructed by collecting all incoherent nodes from three levels of the quadtree pyramid;

[0031] Step 423, the transformer module is used to encode each quadtree node in the node sequence obtained in step 422 to obtain an encoded sequence;

[0032] Step 424, the pixel decoder is used to decode the output encoded sequence to predict the fine mask of the input damaged image; the pixel decoder is a simple multilayer perceptron with two hidden layers.

[0033] Further, in step 4, the learning rate is set to 0.00015625, and the iteration model at 329999 iterations is used as the final tomb mural damage area detection model.

[0034] Further, in step 5, the improved modulation generative adversarial network structure diagram is composed of an image encoder, a generator and a discriminator connected in turn; wherein:

[0035] The image encoder is composed of a grouping gated convolution block and a linear mapping block; the input image generates a random vector, which is then linearly generated into a random style vector through linear mapping; while the input image together with the fine mask is input into the grouping gated convolution block, followed by another linear mapping for generating a conditional style vector; finally, the random style vector and the conditional style vector are connected through the Concat function as the input of the generator;

[0036] The formula of the grouping gated convolution is as follows:

[0037] Gating y,x =∑∑W g ·I

[0038] Feature y,x =∑∑W f ·I

[0039]

[0040] where, sigma is a sigmoid function, so the output gate value is between zero and one; may be any activation function; W g and W f are two different convolution filters.

[0041] Further, in step 5, the initial learning rate is set to 0.0002, the random optimization method Adam with adaptive momentum is used for training, and the iteration model at 500000 iterations is taken as the final tomb mural damaged area repair model.

[0042] In another aspect, the application provides an image restoration model construction device based on attention modulation adversarial network, comprising:

[0043] A data acquisition module is configured to acquire a tomb mural data set, which includes actual damaged tomb murals and actual undamaged tomb murals, wherein the actual damaged tomb mural data set is used for training of a detection model, and the actual undamaged tomb mural data set is used for training of a restoration model.

[0044] A tomb mural damage detection model training module is configured to preprocess training set data and input the preprocessed training set into a detection network training model to obtain a tomb mural damage detection model.

[0045] A tomb mural damage detection result conversion module is configured to acquire a damage area binary image output by the damage detection model.

[0046] A tomb mural damage repair model training module is configured to preprocess training set data and input the preprocessed training set into a restoration network training model to obtain a tomb mural damage repair model, and repair a damage area binary image output by the damage detection model and an original image.

[0047] In a third aspect, the application provides an image restoration method based on attention modulation adversarial network, wherein a tomb mural image to be repaired is input into a trained tomb mural detection model to obtain a detected damage area binary image, and then the damage area binary image is input into a trained tomb mural restoration model to obtain a restored tomb mural image; the trained tomb mural detection model and the trained tomb mural restoration model are obtained by the image restoration model construction method based on attention modulation adversarial network of the application.

[0048] In a fourth aspect, the present application provides a computer device comprising a memory and a processor, wherein the memory stores the image inpainting model construction method based on the attention modulation adversarial network of the present application, and the processor is configured to execute the image inpainting model construction method based on the attention modulation adversarial network.

[0049] In a fifth aspect, the present application provides a computer readable storage medium for storing program instructions, wherein the program instructions can be executed by a processor to implement the image inpainting model construction method based on the attention modulation adversarial network of the present application.

[0050] Compared with the prior art, the present application has the beneficial technical effects that:

[0051] 1. The damaged area detection network in the present application can more effectively capture cross-channel relationships by using a multiple attention mechanism, and more feature information of the tomb mural image can be extracted by using this advantage, which helps to solve the problems of excessive false detection and positioning errors in the prior art, and finally the performance of the network model constructed in the present application is superior to that of the prior art method, and the detection effect of the damaged area of the tomb mural is improved.

[0052] 2. The damaged area detection network in the present application can directly generate a mask binary graph result of the detected damaged area of the tomb mural, which integrates the inference processes of the damaged area detection model and the repair model, and realizes the intelligentization of the damaged area of the tomb mural from detection to repair.

[0053] 3. The damaged area repair network in the present application effectively utilizes the texture and structure of the undamaged mural image, combines the advantages of the restriction collaborative network and the gated convolution in image repair, and realizes high-quality repair of the tomb mural at the pixel level, and improves the repair effect of the large damaged area. Finally, the performance of the network model constructed in the present application is superior to that of the prior art method, and the repair effect of the tomb mural is improved.

[0054] In summary, the present application can more effectively extract feature information of the tomb mural, improve the accuracy of the damaged area detection, and utilize the structure and texture information of the undamaged image to reconstruct a high-quality tomb mural image. Experimental results show that the model constructed in the present application is superior to the prior art method in performance, has better subjective and objective effects, and fully verifies the effectiveness and superiority of introducing the multiple attention mechanism and the grouped gated convolution. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 The figure is a flow chart of the damaged area detection and repair method of the tomb mural of the present application.

[0056] Figure 2 The figure is a structural schematic diagram of the damaged area detection module network of the present application.

[0057] Figure 3The structure diagram of the broken area repair module network of the application.

[0058] Figure 4 The effect display diagram of the real tomb mural processing of the application; wherein, (a) is an image to be repaired, (b) is a binary image of the detected broken area, and (c) is a repaired image.

[0059] Figure 5 The effect display diagram of the real tomb mural processing of the application; wherein, (a) is an image to be repaired, (b) is a binary image of the detected broken area, and (c) is a repaired image.

[0060] The application will be further explained in combination with the drawings and embodiments. DETAILED DESCRIPTION

[0061] Embodiment one:

[0062] As shown in the figure, the image repair model construction method based on attention modulation adversarial network provided by the embodiment includes broken area detection and broken area repair. Figures 1 to 3 As shown in the figure, the image repair model construction method based on attention modulation adversarial network provided by the embodiment includes broken area detection and broken area repair. Figure 1 The broken area detection stage first introduces a multi-path attention mechanism to estimate the coarse mask of the missing area in the input damaged image, and then uses a quadtree-structured transformer module to correct the mask. The broken area repair stage uses a modulation adversarial generation network introducing a grouping gated convolution to repair the fine mask. Specifically, the following steps are included:

[0063] Step 1: Collect multiple images of tomb murals to obtain a data set of tomb murals, and divide the data set into two parts: tomb murals with actual damage and tomb murals without actual damage.

[0064] Step 2: Divide the tomb mural images with actual damage into a training set, a test set, and a validation set according to the proportions of 80%, 15%, and 5%, and use a labeling tool to label the damaged areas of the tomb murals in the training set and the test set for subsequent detection model training.

[0065] Step 3: Use the tomb mural images without actual damage as the training set of the repair model, and process these images to obtain pictures with a size of 512*512 for subsequent repair model training.

[0066] Step 4: Send the training set and test set obtained in step 2 and the corresponding labeled image damaged area position data into the improved broken area detection network based on the multi-path attention mechanism for training.

[0067] The improved multi-attention mechanism based damage area detection network structure is shown in Figure 2 which includes a multi-path attention module and a quadtree-structured transformer module.

[0068] The multi-path attention module first estimates a rough mask of the damaged area in the input image, and then the quadtree-structured transformer module corrects the segmentation error of the rough mask to generate a fine mask. The segmentation error generated by the multi-path attention module usually occurs along the object boundary or high-frequency area. These areas are identified as incoherent areas, and their masks cannot be correctly reconstructed due to the reduction of spatial resolution caused by the first downsampling and then upsampling. Therefore, the quadtree-structured transformer module for correcting the error of the rough mask only needs to be performed on these incoherent areas. Since the object boundary only occupies a small part of the image, the quadtree representation of the incoherent area can significantly reduce the computational complexity of the transformer.

[0069] The multi-path attention module (Multi-Path Attention) is used to implement the following process:

[0070] Step 411, the input features are divided into G cardinal arrays (Cardinal 1~Cardinal G) to control the connection between the input and the output, and each cardinal array is further divided into R subgroups (Split 1~Split R), so the total number of subgroups is GR. A series of transformations are applied to each subgroup The change process of each subgroup is represented as where X represents the input feature information, i∈{1,2,…,GR}.

[0071] Step 412, in the Split Attention module of each cardinal array, the combination of the G cardinal arrays of input features can be obtained by element-wise summation across multiple subgroups, g∈{1,2,…,G}, and the gth cardinal array is represented as:

[0072]

[0073] where, R represents the number of subgroups within a cardinal array, i.e., H, W and C' represent the size of the sub-group output feature map;

[0074] Then, the average value of each sub-group feature map is calculated using global pooling across the channel and spatial dimensions, as follows:

[0075]

[0076] where denotes the c-th component of the g-th basis vector in c e {1,2,…,C' / G}.

[0077] Step 413, weighted fusion is performed on each basis vector component in step 412, and all basis vectors are concatenated along the channel dimension by the Concat function.

[0078] Step 414, the data after concatenation in step 413 is compressed to feature map N = [N 1 ,N 2 ,…,N G ] with the same size as the input feature map through a 1x1 convolution layer. The output of the multi-attention module is generated by a shortcut connection in residual learning, which is called the coarse mask of the input damaged image.

[0079] The Quadtree-Structured Transformer module is used to implement the following process:

[0080] Step 421, the Feature Pyramid takes the input damaged image (Input) and its estimated coarse mask (obtained by the multi-attention module) as input, and outputs three scale feature maps through a regular convolutional network layer.

[0081] Step 422, for each scale of the pyramid feature map in the concatenation mode, the Incoherence Detector identifies and refines the incoherent region by using the coarse mask (CoarseMask) and the region of interest (ROI Feature) feature, and outputs a node sequence (Node Sequence). According to the detected incoherent region, a quadtree pyramid is constructed in the form of three-level feature maps. If a pixel in a higher-level feature map is predicted to be a non-continuous node, the node is decomposed into four quadrant pixels of adjacent lower-level feature maps through up-sampling (Up-sample). The node sequence (Node Sequence) is constructed by collecting all incoherent nodes from three levels of the quadtree pyramid. The order of the node sequence is not important because the transformer architecture is invariant. This step outputs the node sequence (Node Sequence).

[0082] Step 423, the Transformer Encoder is used to encode each quadtree node in the node sequence obtained in step 422 to obtain an encoded sequence. Specifically as follows:

[0083] The Transformer Encoder includes a Node Encoder and a Sequence Encoder for node sequence encoding.

[0084] The Node Encoder is used to encode each quadtree node by using four different information collection features: 1) fine-grained features selected from the corresponding level and position of the quadtree pyramid; 2) coarse masks estimated by the multi-attention mechanism module, providing semantic information of the damaged area; 3) relative position embeddings in each ROI, including relative spatial distance and relationship between nodes; 4) neighboring context of each node, containing rich local information. For each node, first extract fine-grained features, coarse masks and neighboring context from the neighborhood, connect them through a fully connected layer, and then add the generated feature vector to the relative position.

[0085] After encoding by the Node Encoder, the Sequence Encoder jointly processes all levels of encoded nodes in the quadtree to perform global spatial and multi-scale reasoning. In addition, the Sequence Encoder has the same architecture as the standard Transformer, which consists of a Multi-head Attention, a Residuleblock, a Normalization and a fully connected Feed Forward Network (FFN). The output of this step is the Encoded Sequence.

[0086] Step 424, the output encoded sequence is decoded by the Pixel Decoder to predict the fine mask of the input damaged image. The Pixel Decoder is a simple Multi-Layer Perceptron (MLP) with two hidden layers.

[0087] In addition, during the training of the tomb mural damaged area detection model, the learning rate is set to 0.00015625, and the repair effect is best at 329999 iterations, and the model at this iteration is taken as the final tomb mural damaged area detection model.

[0088] Step 5: The training set of the repair model and the randomly generated binary mask image in step 3 are sent into the improved modulation adversarial generative network for training.

[0089] The structure diagram of the improved modulation adversarial generative network is as follows: Figure 3As shown, in order to improve the generalization ability and accuracy of the network model, we design a modulation adversarial generative network to repair the damaged area of the tomb mural. The network can balance the collaborative constraints between conditional and random style representations. In addition, considering that the standard convolution treats all input pixels equally regardless of their location in the damaged or effective area, we propose a group-gated convolution to control the spatial channel connection between the input and output to improve the performance of the damaged area repair. At this stage, the modulation adversarial generative network inputs the damaged tomb mural image and its corresponding binary mask, and then outputs the repaired image.

[0090] The improved modulation adversarial generative network is composed of three modules connected in turn, namely image encoder, generator and discriminator; wherein:

[0091] The image encoder is composed of a group-gated convolution block and a linear mapping block. The input image generates a random vector, which is then linearly mapped to a random style vector through a linear mapping. The input image is input into the group-gated convolution block together with the fine mask, followed by another linear mapping for generating a conditional style vector. Finally, the random style vector and the conditional style vector are connected through a Concat function as the input of the generator.

[0092] Among them, the group-gated convolution adds a group of feature vectors to judge the effectiveness of the current pixel when calculating, so as to avoid the disadvantages of ordinary convolution in the field of image repair. The formula of group-gated convolution is as follows:

[0093] Gating y,x =∑∑W g ·I Equation VIII

[0094] Feature y,x =∑∑W f ·I Equation IX

[0095]

[0096] Among them, σ is the sigmoid function, so the output gating value is between zero and one; It can be any activation function (for example, ReLU, ELU and LeakyReLU); Wg and W f are two different convolution filters.

[0097] The Generator part is composed of ordinary convolution layers to perform the image inpainting process, outputting the inpainted image (Output)

[0098] The Discriminator is an ordinary convolutional neural network that takes the output of the Generator as input and tries to distinguish between fake and real images.

[0099] In addition, during the training of the tomb mural damage repair model, the initial learning rate is set to 0.0002, and the adaptive momentum random optimization method Adam (Adaptive momentum) is used for training. The repair effect is best at 500,000 iterations, and this iteration model is used as the final tomb mural damage repair model.

[0100] Step 6: Process the validation set in step 2 to obtain a tomb mural image with a damaged area of size 512*512, and input the image into the trained tomb mural damage detection model obtained in step 4. The detection model temporarily stores the detected damage area position information and directly converts it to a mask binary image.

[0101] Step 7: Input the tomb mural image processed in step 6 and the mask binary image obtained after detecting the image into the trained tomb mural damage repair network obtained in step 5 to obtain the final repaired tomb mural image. Figure One

[0102] Example Two:

[0103] The embodiment provides an image inpainting model construction device based on an attention modulation adversarial network, which comprises:

[0104] A data acquisition module is configured to acquire a tomb mural data set, which includes actual damage-free tomb murals and actual damaged tomb murals. The actual damaged tomb mural data set is used for training of a detection model, and the actual damage-free tomb mural data set is used for training of a repair model.

[0105] A tomb mural damage detection model training module is configured to preprocess training set data and input the preprocessed training set into a detection network training model to obtain a tomb mural damage detection model.

[0106] A tomb mural damage detection result conversion module is configured to acquire a damage area binary image output by the damage detection model.

[0107] ​The tomb mural damage repair model training module is configured to preprocess the training set data and input the preprocessed training set into the repair network training model, obtain the tomb mural damage repair model, and repair the damage area binary image and the original image output by the damage detection model.

[0108] Embodiment three:

[0109] The embodiment provides an image repair method based on an attention modulation adversarial network. A tomb mural image to be repaired is input into a trained tomb mural detection model to obtain a detected damage area binary image. The damage area binary image is input into a trained tomb mural repair model to obtain a repaired tomb mural image. The trained tomb mural detection model and the tomb mural repair model are obtained by the image repair model construction method based on the attention modulation adversarial network.

[0110] Embodiment four:

[0111] The embodiment provides a computer device including a memory and a processor. The memory stores a computer program. The processor implements the steps of the image repair model construction method based on the attention modulation adversarial network when executing the computer program.

[0112] Embodiment five:

[0113] The embodiment provides a computer readable storage medium for storing program instructions. The program instructions can be executed by a processor to implement the steps of the image repair model construction method based on the attention modulation adversarial network.

[0114] The image repair method based on the attention modulation adversarial network can be implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product and the like.

[0115] The various functional units in the various embodiments can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. When the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a volatile or non-volatile computer readable storage medium executable by a processor. Based on such understanding, the present scheme essentially or the part that contributes to the prior art or part of the technical scheme can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0116] Experimental verification:

[0117] In this embodiment, the image to be repaired shown in Figure 4 (a) and Figure 5 (a) is input into the damaged area detection model of the tomb mural, and the detection result of Figure 4 (b) and Figure 5 (b) is obtained, wherein Figure 4 (a) is Figure 4 (b), Figure 5 (a) is Figure 5 (b), and Figure 4 (b) and Figure 5 (b) shown in the damaged area binary image is input into the damaged area repair model of the tomb mural, and the repair result of Figure 4 (c) and Figure 5 (c) is obtained, wherein Figure 4 (b) is Figure 4 (c), Figure 5 (b) is Figure 5 (c).

[0118] In this experiment, the detection method provided by the present application is compared with the prior art Mask transfiner using the tomb mural data set, and the comparison is made on the average precision AP, AP50, AP75, Aps, APm, APl of the labeled frame bbox. The six indicators are the main indicators for evaluating the coco data set. The comparison of these indicators for measuring the detection effect is shown in Table 1, wherein the larger the MAP, the higher the correct rate;

[0119] The repair method provided by the application is compared with the Co-Mod GAN method in the prior art, and the two indicators for measuring the repair effect, peak signal-to-noise ratio (PSNR) and image diversity and quality (FID), are compared in Table 2. The greater the PSNR, the better the image quality. The smaller the FID, the closer the two distributions, which means that the generated image has high quality and good diversity.

[0120] Table 1 Comparison of the detection method of the application with the existing detection method

[0121] Method AP↑ AP50↑ AP75↑ Aps↑ APm↑ APl↑ Mask transfiner 19.625 35.577 17.579 12.292 29.102 40.200 The method 24.302 42.453 22.511 16.934 32.771 45.895

[0122] Table 2 Comparison of the repair method of the application with the existing repair method

[0123] Method FID↓ PSNR↑ Co-Mod GAN 37.1145 27.3425 The method 35.4257 30.6397

[0124] The experimental results show that the model built by the application is better than the prior art method in performance, which fully verifies the effectiveness and superiority of the method proposed by the application.

Claims

1. A method for constructing an image inpainting model based on attention modulation adversarial networks, characterized in that, Specifically, the following steps are included: Step 1: Collect images of multiple tomb murals to obtain a dataset of tomb murals. Divide the dataset into two parts: tomb murals with actual damage and tomb murals without actual damage. Step 2: Divide the actual damaged tomb mural images into training set, test set, and validation set, and use the annotation tool to mark the damaged areas of the tomb murals in the training set and test set for subsequent detection model training; Step 3: Use images of tomb murals without actual damage as the training set for the restoration model, and process these images for subsequent restoration model training. Step 4: The training set and test set obtained in Step 2, along with the corresponding labeled image damage area location data, are fed into the improved damage area detection network based on the multi-attention mechanism for training; the improved damage area detection network based on the multi-attention mechanism includes a multi-attention module and a quadtree structure transformer module; The multi-attention module first estimates a coarse mask of the damaged region in the input image, and then the quadtree structure transformer module corrects the segmentation error of the coarse mask to generate a fine mask; Step 5: The training set of the repair model described in Step 3 and the randomly generated binary mask image are fed into the improved modulation adversarial generative network for training. The improved modulation adversarial generative network structure consists of an image encoder, a generator, and a discriminator connected in sequence. The image encoder consists of a grouped gated convolutional block and a linear mapping block. The input image generates a random vector, which is then linearly generated into a random pattern vector through a linear mapping. The input image, along with the fine mask, is input into the grouped gated convolutional block, followed by another linear mapping to generate a conditional pattern vector. Finally, the random pattern vector and the conditional pattern vector are connected through the Concat function and used as the input to the generator. Step 6: The validation set in Step 2 is processed to obtain a 512*512 image of a tomb mural with a damaged area. This image is then fed into the trained damaged area detection network obtained in Step 4. The detection network temporarily stores the detected damaged area location information and then directly converts it into a mask binary image. Step 7: Input the processed tomb mural image obtained in Step 6 and the mask binary image obtained after detection into the trained and improved modulation adversarial generative network obtained in Step 5 to obtain the final restored tomb mural image.

2. The method for constructing an image inpainting model based on attention modulation adversarial networks as described in claim 1, characterized in that, In step 4, the multi-attention module is used to implement the following process: Step 411: Divide the input features into G basis arrays to control the connection between the input and output. Each basis array is further divided into R subgroups, resulting in a total of GR subgroups. Apply a series of transformations to each subgroup individually. The change process of each subgroup can be represented as follows: Where X represents the input feature information, ; Step 412, in the Split Attention module of each basis array, the combination of the input feature G basis arrays can be obtained by element-wise summation across multiple subgroups. Then the g-th basis array is represented as: in, R represents the number of subarrays within a given base array. , and Indicates the size of the sub-block output feature map; Then, the average value of the feature map for each subgroup is calculated using global average pooling across the channel spatial dimensions: in express The c-th component of the g-th base array; Step 413: Perform weighted fusion on each basis array component from step 412, and concatenate all basis arrays along the channel dimension using the Concat function; Step 414: Compress the concatenated data into a feature map using a 1x1 convolutional layer. It has the same size as the input feature map; then the output of the multiple attention module is generated through a shortcut connection in residual learning, which is called a coarse mask of the input damaged image.

3. The method for constructing an image inpainting model based on attention modulation adversarial networks as described in claim 1, characterized in that, In step 4, the quadtree structure transformer module is used to implement the following process: Step 421: The feature pyramid takes the input damaged image and its estimated coarse mask as input, and outputs feature maps of three scales through a regular convolutional network layer. Step 422: For each scale of the cascaded pyramid feature map, the incoherent detector identifies and refines the incoherent regions using a coarse mask and region of interest features, outputting a node sequence; based on the detected incoherent regions, a quadtree pyramid is constructed in the form of three-level feature maps; if a pixel in a higher-level feature map is predicted as a non-contiguous node, the node is decomposed into four quadrant pixels of adjacent lower-level feature maps through upsampling; a node sequence is constructed by collecting all incoherent nodes from the three levels of the quadtree pyramid. Step 423: Use the transformer module to encode each quadtree node in the node sequence obtained in step 422 to obtain the encoded sequence; Step 424: The output encoded sequence is decoded using a pixel decoder to predict the fine mask of the input damaged image; The pixel decoder is a simple multilayer perceptron with two hidden layers.

4. The method for constructing an image inpainting model based on attention modulation adversarial networks as described in claim 1, characterized in that, In step 4, the learning rate is set to 0.00015625, and the iterative model at 329999 iterations is used as the final detection model for the damaged area of ​​the tomb mural.

5. The method for constructing an image inpainting model based on attention modulation adversarial networks as described in claim 1, characterized in that, In step 5, the grouped gated convolution formula is as follows: Here, σ is the sigmoid function, so the output gate value is between zero and one; It can be any activation function; and These are two different convolutional filters.

6. The method for constructing an image inpainting model based on attention modulation adversarial networks as described in claim 1, characterized in that, In step 5, the initial learning rate is set to 0.0002, and the Adam adaptive momentum stochastic optimization method is used for training. The iterative model at 500,000 iterations is used as the final model for the restoration of the damaged area of ​​the tomb mural.

7. An image inpainting model construction device based on attention modulation adversarial networks, characterized in that, include: The data acquisition module is used to acquire a dataset of tomb murals, which includes tomb murals without actual damage and tomb murals with actual damage. The dataset of tomb murals with actual damage is used for training the detection model, and the dataset of tomb murals without actual damage is used for training the restoration model. The training module for the tomb mural damage detection model is used to preprocess the training set data and input the preprocessed training set into the improved damage area detection network training model based on the multi-attention mechanism to obtain the trained damage area detection network; the improved damage area detection network based on the multi-attention mechanism includes a multi-attention module and a quadtree structure transformer module. The multi-attention module first estimates a coarse mask of the damaged region in the input image, and then the quadtree structure transformer module corrects the segmentation error of the coarse mask to generate a fine mask; The module for converting the detection results of tomb mural damage is used to obtain the binary map of the damaged area output by the damaged area detection network; The training module for the tomb mural damage restoration model is used to preprocess the training set data and input the preprocessed training set into an improved modulation adversarial generative network for training, obtaining a trained improved modulation adversarial generative network, and restoring the binary image of the damaged area output by the damage detection model and the original image. The improved modulation adversarial generative network structure consists of an image encoder, a generator, and a discriminator connected in sequence. The image encoder consists of a grouped gated convolutional block and a linear mapping block. The input image generates a random vector, which is then linearly generated into a random pattern vector through a linear mapping. The input image, along with a fine mask, is input into the grouped gated convolutional block, followed by another linear mapping to generate a conditional pattern vector. Finally, the random pattern vector and the conditional pattern vector are concatenated through the Concat function and used as the input to the generator.

8. An image inpainting method based on attention modulation adversarial networks, characterized in that, The image of the mural to be repaired is input into a trained mural detection model to obtain a binary image of the detected damaged area. Then, the binary image of the damaged area is input into a trained mural restoration model to obtain a restored mural image. The trained mural detection model and mural restoration model are obtained by the image restoration model construction method based on attention modulation adversarial network as described in any one of claims 1 to 6.

9. A computer device, characterized in that, The device includes a memory and a processor, wherein the memory stores the image inpainting model construction method based on attention modulation adversarial network as described in any one of claims 1 to 6, and the processor is used to execute the image inpainting model construction method based on attention modulation adversarial network.

10. A computer-readable storage medium, characterized in that, Used to store program instructions that can be executed by a processor to implement the image inpainting model construction method based on attention modulation adversarial networks as described in any one of claims 1 to 6.

Images