Structure-guided wall painting image large-area defect repairing method and device

Abstract

The invention provides a structure-guided wall painting image large-area defect repairing method and device. A semi-interactive line draft interactive repair module is designed to provide structure prior information closer to an original picture for a repair network, and the ability of the network to understand the complex structure of the mural is improved; multi-scale structure prior information is injected into a texture repairing module through a structure guiding repairing module to guide repairing, and structure guiding is provided in the repairing process; meanwhile, a gating convolution module, a jump connection module, a multi-scale Fourier convolution module and the like are introduced into a multi-scale texture restoration module, so that the network can perform structure-guided restoration on an irregular and large-area damaged hot tribute mural image in a more targeted manner; and finally, designing a discriminator combining an overall image and edge details, and finely recovering complex image structures and details while ensuring that the overall consistency of the image is kept. Therefore, the damaged wall painting can be effectively recovered, the color consistency can be improved, and the artistry and authenticity of the repairing effect are ensured.

Description

Technical Field

[0001] The present application relates to a method and device for repairing large-area defects in mural images guided by structure, belonging to the technical field of image processing. Background Art

[0002] With the continuous maturity of digital image repair technology, the derived mural image repair technology can repair mural images to a certain extent. Compared with the time-consuming and laborious manual repair by painters, algorithmic repair can complete a large amount of repair work at low cost and quickly.

[0003] However, the current research on digital mural image repair mainly focuses on Dunhuang murals and mainly repairs damaged images with relatively small defects and relatively simple structures. Although the current image repair algorithms have achieved good results, when applied to the Thangka mural dataset with complex structural textures, large damages, and small datasets, phenomena such as unreasonable structures and blurred textures are likely to occur. It is difficult to train a robust and effective repair model, which makes there are still some technical difficulties to be overcome in the Thangka mural image repair method based on deep learning. Summary of the Invention

[0004] The present application provides a method and device for repairing large-area defects in mural images guided by structure to solve the problem that the existing mural image repair method based on deep learning cannot be well applied to the repair of Thangka murals.

[0005] In a first aspect, an embodiment of the present application provides a method for repairing large-area defects in mural images guided by structure, including: Training a pre-constructed network for repairing large-area defects in mural images guided by structure using training samples to obtain a repair model; Using the repair model to repair the defects in the mural image to be repaired; Wherein, the network for repairing large-area defects in mural images guided by structure includes a generator and a discriminator; the generator includes a line drawing interactive repair module, a structure-guided repair module, and a multi-scale texture repair module; the discriminator includes an image discriminator and an edge discriminator; The line drawing interactive repair module is used to extract a damaged line drawing from the defective mural image through the SketchKeras algorithm, and obtain the line drawing repair information manually added by an expert for the defective area of the defective mural image, and obtain a repaired complete line drawing based on the damaged line drawing and the line drawing repair information; and when training the repair model, considering the data volume, using the complete line drawing extracted by the SketchKeras algorithm as the interactively obtained complete line drawing; The structure-guided repair module is used to extract features from the repaired complete line drawing to obtain multi-scale structural features; The multi-scale texture restoration module includes a downsampling module, a multi-scale Fourier convolution module, and an upsampling module connected in sequence. The downsampling module extracts the feature information of the damaged mural image through reflection padding and multiple downsamplings, and selects the pixels of the input image based on a gating mechanism. The multi-scale Fourier convolution module performs frequency domain conversion on part of the input data by integrating frequency domain analysis and spatial domain analysis, and can effectively capture the global information and local information of the image. The upsampling module is used to upsample the input based on the gating mechanism to obtain a reconstructed image. The input of the upsampling module includes the output of the multi-scale Fourier convolution module and the output features transmitted through skip connections of the downsampling module, and the multi-scale structural features obtained by the structure-guided repair module are also input into the upsampling module layer by layer. The image discriminator is used for the overall image discrimination of the reconstructed image, and the edge discriminator is used for the discrimination of the edge information of the reconstructed image.

[0006] Based on the above method, optionally, the processing process of the line drawing interactive repair module includes: Extracting a damaged line drawing from the damaged mural image through the SketchKeras algorithm:

[0007] Among them, I gt represents the original image, E gt and E m represent the line drawings of the complete image and the damaged image respectively; M represents the simulated mask, the missing area is 1, and the non-missing area represents 0. Under the guidance of professionals, accurately repair the damaged area of the line drawing to obtain line drawing repair information E i , and obtain a repaired complete line drawing based on the damaged line drawing and the line drawing repair information:

[0008] Among them, E represents the repaired complete line drawing.

[0009] Based on the above method, optionally, the structure-guided repair module aims to retain low-level guidance information while achieving deep fusion with texture information to optimize the image repair process based on deep learning. Among them, the structure-guided repair module embeds control information into each level of the repair network decoder to perform more accurate structure-guided processing in the feature space of the convolutional neural network. In this process, an additive rather than a concatenated connection method is used to avoid the problem of unreasonable generation results that may be caused by overly strong guidance information, thereby ensuring the high-quality completion of image repair.

[0010] Based on the above method, optionally, the multi-scale Fourier convolution module performs fast Fourier convolution on three-quarter dimensional data, transforms the image into the frequency domain for processing to effectively capture the global structural information of the image, increases the input dimension and feature representation ability of the model, enabling the model to better learn the potential laws and features in the image; performs local Fourier convolution on one-quarter dimensional data through the local Fourier convolution module to obtain the local features of the image, so as to make up for the deficiency of fast Fourier convolution in fine texture synthesis; wherein, the multi-scale Fourier convolution module adopts SpectralTransform, and realizes the effective fusion of frequency domain and spatial domain features by transforming the input data into the frequency domain and extracting local spectral features.

[0011] Based on the above method, optionally, both the image discriminator and the edge discriminator are discriminators based on the PatchGAN architecture. The input of the edge discriminator is the edge images of the real image and the predicted image obtained by the Canny algorithm. The edge discriminator performs adversarial training with the generator, prompting the generator to more finely control the generation of the image structure, making the edges of the predicted image sharper and more accurate, approaching the real image.

[0012] Based on the above method, optionally, the model loss calculated during the training of the large-area defect repair network for mural images based on structure guidance includes: L 1 Loss: L 1 =(1 - M)║I pre - I gt ║ 1 In the formula, I pre and I gt represent the predicted image and the real image respectively; where M is a 0-1 mask, and 1 represents the masked area, that is, the part that needs to be repaired; Adversarial loss:

[0013] LG = EIpre [log D(Ipre ))]

[0014]

[0015] In the formula, D(x) represents the predicted probability of the discriminator for the image x; E represents the expectation, L GP represents the gradient penalty, λ GP= 1e-3; where, by adding gradient penalty, the range of gradient change of the discriminator can be restricted, thereby maintaining the stability of the generator gradient, preventing gradient explosion and gradient disappearance, and at the same time helping the network to converge to a better solution faster; High receptive field matching loss:

[0016] In the formula, φ hrf represents the high receptive field perception loss obtained by training the pre-trained segmentation ResNet50 with dilated convolution; Color consistency loss;

[0017] In the formula, n is the total number of pixels participating in the calculation, m i is the value of the mask at pixel i, 0 means known, 1 means missing, known and missing are the weights of the known area and the missing area, 10 and 5 respectively, p i R , p i G , p i B are the red, green, and blue channel values of the predicted image at pixel i respectively, t i R , t i G , t i B are the red, green, and blue channel values of the target image at pixel i respectively; The total loss is:

[0018] where the weights are λ L1 = 10, λ adv = 10, λ hrf = 30, λ CCL = 15.

[0019] Second aspect, the embodiments of the present application further provide a structure-guided large-area defect repair device for mural images, which includes: A model training unit, configured to train a pre-constructed structure-guided large-area defect repair network for mural images using training samples to obtain a repair model; An image repair unit, configured to use the repair model to repair the defects of the mural image to be repaired; Wherein, the structure-guided large-area defect repair network for mural images includes a generator and a discriminator; the generator includes a line drawing interactive repair module, a structure-guided repair module, and a multi-scale texture repair module; the discriminator includes an image discriminator and an edge discriminator; The line drawing interactive repair module is configured to extract a damaged line drawing from the defective mural image through the SketchKeras algorithm, and obtain the line drawing repair information manually added by an expert for the defective area of the defective mural image, and obtain a repaired complete line drawing based on the damaged line drawing and the line drawing repair information; and when training the repair model, considering the data volume, the complete line drawing extracted by the SketchKeras algorithm is used as the interactive complete line drawing; The structure-guided repair module is configured to extract multi-scale structure features from the repaired complete line drawing; The multi-scale texture repair module includes a downsampling module, a multi-scale Fourier convolution module, and an upsampling module connected in sequence; the downsampling module extracts the feature information of the defective mural image through reflection filling and multiple downsamplings, and selects the pixels of the input image based on a gating mechanism; the multi-scale Fourier convolution module performs frequency domain conversion on part of the input data by integrating frequency domain analysis and spatial domain analysis, and can effectively capture the global information and local information of the image; the upsampling module is configured to perform upsampling on the input based on the gating mechanism to obtain a reconstructed image, the input of the upsampling module includes the output of the multi-scale Fourier convolution module, and the output features transmitted through skip connections of the downsampling module, and the multi-scale structure features obtained by the structure-guided repair module are also input into the upsampling module layer by layer; The image discriminator is configured to perform overall image discrimination on the reconstructed image, and the edge discriminator is configured to perform discrimination on the edge information of the reconstructed image.

[0020] In the method and device for repairing large - area defects of mural images guided by structure provided by this application, the damaged mural image is repaired by the trained network for repairing large - area defects of mural images guided by structure. Among them, first, a semi - interactive line drawing interactive repair module is designed to provide the repair network with structural prior information closer to the original image, and this step significantly improves the network's ability to understand the complex structure of murals; then, through the structure - guided repair module, multi - scale structural prior information is injected into the texture repair module to guide the repair, so as to provide strong structural guidance during the repair process; at the same time, gated convolution, skip connections, and multi - scale Fourier convolution modules are introduced in the multi - scale texture repair module, enabling the network to more specifically repair the irregular and large - area damaged Thangka mural images under structural guidance; finally, a discriminator that combines the overall image and edge details is designed to ensure that while maintaining the overall consistency of the image, the complex image structure and details are finely restored. Based on this, using the method of this application can not only effectively restore damaged murals, but also improve color consistency, ensure the artistic and authentic nature of the repair effect, and make the repair effect closer to real mural images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application. In addition, these drawings and the text description are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments.

[0022] Figure 1 Schematic diagram of the architecture of the network for repairing large - area defects of mural images guided by structure provided by an embodiment of this application; Figure 2 Schematic diagram of the structure of the multi - scale texture repair module provided by an embodiment of this application; Figure 3 Schematic diagram of the structure of the multi - scale Fourier convolution module provided by an embodiment of this application; Figure 4 Schematic diagram of the mask graph provided by an embodiment of this application; Figure 5 Visualization results of comparative experiments of different repair algorithms on the Thangka mural image dataset provided by an embodiment of this application; Figure 6 Visualization results of comparative experiments of different repair algorithms on the CelebA dataset provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following will, in combination with the embodiments of this application, clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the described embodiments are some, rather than all, of the embodiments of this application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope of protection of this application. Without conflict, the following embodiments and the features in the embodiments may be combined with each other.

[0024] Currently, the research on digital mural image restoration is mainly focused on Dunhuang murals: The "Digital Mural Restoration Method and Device Based on Deep Learning" disclosed in Patent No. CN202211102976.0 proposes a digital mural restoration method based on deep learning. The core idea of this method is to first repair the line drawing extracted by the Canny algorithm through a composite Unet network, and then perform color filling and refinement of the original damaged image based on the repaired line drawing. However, it mainly relies on the U-Net network and Transformer Block for automatic line drawing repair, lacking the introduction of expert experience. For mural images with complex lines and a small dataset, it is easy to obtain an incorrect structure line drawing. For images with less complex structures and less damage, the effect may be better, but for Thangka murals with numerous elements, complex structures, and greater damage, the predicted image obtained by color filling under the guidance of an incorrect structure will deviate more from the real mural image.

[0025] The "Mural Digital Restoration System and Method Based on Diffusion Model" disclosed in Patent No. CN202311365641.2 proposes a systematic mural digital restoration system according to different degrees of damage and damaged areas. It is mentioned in this patent that during the repair training of the line drawing module, a manual confirmation step is added. However, it mainly adjusts the training dataset, parameters, etc. and performs iterations according to the results after model training, which is also not applicable to Thangka mural images with numerous elements and a small dataset. In Thangka mural images with numerous elements, the damaged areas of images with large defects will simultaneously involve flowers, plants, costumes, lotus seats, mountains, etc. of different styles. At the same time, their painting styles are not consistent and the dataset is small. Simple manual confirmation during training is not very helpful for the restoration of Thangka mural images.

[0026] To address the above problems, this solution proposes a structure-guided large-area defect repair solution for mural images. In this solution, considering the complex structure, numerous details, and low repeatability of Thangka mural images, and based on a relatively small dataset, in order to repair large damaged Thangka mural images to be closer to the original images. First, a structure-guided large-area defect repair network for mural images is constructed, and then the pre-constructed structure-guided large-area defect repair network for mural images is trained using training samples to obtain a repair model; finally, the repair model is used to repair the defects of the mural images to be repaired.

[0027] Among them, for the structure-guided large-area defect repair network for mural images, first, an interactive line drawing repair module is designed to repair the large damaged complex line drawing with the help of expert experience, providing a relatively accurate structural prior; then, the line drawing is passed through the structure-guided repair module to better fuse with the texture information while avoiding the weakening of low-level guidance information. This module injects the encoded structural information layer by layer into the multi-scale repair network and designs an edge generation discriminator for constraint, making the repaired structure closer to the real mural image; the multi-scale feature repair network introduces a Fourier convolution aggregation module and a gated convolution module to obtain a larger receptive field to restore large and irregular defect areas, and losses such as multi-scale feature loss are added to ensure the reasonable fusion of the structure and texture of the repaired image.

[0028] Specifically, referring to Figure 1 As shown, the overall structure-guided large-area defect repair network for mural images adopts a generative adversarial architecture, including a generator and a discriminator; the generator includes a line drawing interactive repair module, a structure-guided repair module, and a multi-scale texture repair module; the discriminator includes an image discriminator and an edge discriminator.

[0029] The line drawing interactive repair module is used to extract the damaged line drawing from the defective mural image through the SketchKeras algorithm, and obtain the line drawing repair information manually added by experts for the defective area of the defective mural image, and obtain the repaired complete line drawing based on the damaged line drawing and the line drawing repair information.

[0030] Specifically, the automatic image restoration algorithm has a poor effect in restoring images such as Thangka murals with numerous and fine lines, complex textures, and rich connotations. There are often phenomena such as structural disorder and texture blur. At the same time, as a unique art form, murals need to follow specific norms and standards, and the automatic restoration method may lead to the restoration result not conforming to the mural content. To address this issue, this embodiment proposes the idea of line drawing interactive restoration, leveraging the experience of experts such as Thangka art mural inheritors to ensure the accuracy of the overall structure of the image, thereby providing accurate structural prior information for subsequent restoration work and generating a mural image closer to the original. The line drawing interactive restoration module uses the SketchKeras algorithm (denoted as SK) to extract the line drawing of the damaged mural image, and this algorithm can better simulate the real line drawing of the mural image. If I gt represents the original image, E gt and E m represent the line drawing diagrams of the complete image and the damaged image respectively, then E m is expressed as:

[0031] In the formula, M represents the simulated mask (the missing area is 1, and the non-missing area is 0). Under the guidance of professionals, the damaged area of the line drawing is accurately restored to obtain E i , ensuring the integrity and accuracy of the line drawing. Finally, the repaired complete line drawing diagram E is obtained:

[0032] Subsequently, the completed line drawing diagram is used by the structure-guided restoration module for feature extraction, and the extracted multi-scale structural features are injected layer by layer into the decoder of the multi-scale texture restoration module, thereby more effectively guiding the image restoration process.

[0033] It should be noted that when training the restoration model, considering the large amount of data, it is difficult to obtain line drawing restoration information provided by experts for all training data. Therefore, when training the restoration model considering the data volume, the complete line drawing extracted using the SketchKeras algorithm (instead of the damaged line drawing) is used as the complete line drawing diagram obtained through interaction, that is, directly used as training data.

[0034] The structure-guided repair module is used to extract features from the patched complete line drawing to obtain multi-scale structural features. Specifically, existing image repair methods guided by prior information are mainly limited to fusing the input image with prior information at the image level as the initial input of the neural network. However, the guiding effect of these low-level inputs gradually weakens in the feature space, and the generated image content may have problems such as structural chaos, missing details, or inconsistent with expected logic. Therefore, this application designs a structure-guided repair module, which aims to retain low-level guiding information while achieving deep fusion with texture information to optimize the image repair process based on deep learning. Different from directly attaching low-level control information to the input image, these control information are embedded in each level of the decoder of the repair network, so as to perform more accurate structure-guided processing within the feature space of the convolutional neural network. During this process, an addition rather than a concatenation connection method is adopted to avoid the problem of unreasonable generation results that may be caused by overly strong guiding information, thus ensuring the high-quality completion of image repair. As Figure 2 shown.

[0035] Meanwhile, the damaged mural image is input into the multi-scale texture repair module. The multi-scale texture repair module includes a downsampling module, a multi-scale Fourier convolution module, and an upsampling module connected in sequence. The downsampling module extracts the feature information of the damaged mural image through reflection padding and multiple downsamplings, and selects the pixels of the input image based on the gating mechanism. The multi-scale Fourier convolution module performs frequency domain conversion on part of the input data by integrating frequency domain analysis and spatial domain analysis, and can effectively capture the global information and local information of the image. The upsampling module is used to upsample the input based on the gating mechanism to obtain the reconstructed image. The input of the upsampling module includes the output of the multi-scale Fourier convolution module, and is skip-connected to the downsampling module to directly transfer the features obtained by the downsampling module to the upsampling module. Moreover, the multi-scale structural features obtained by the structure-guided repair module are also input into the upsampling module layer by layer.

[0036] Specifically, the damaged mural image is input into the multi-scale texture repair module. First, it passes through the ReflectionPad and the down-convolution module. While keeping the image information complete, it selectively processes the input pixels through the gating convolution mechanism, so as to more effectively handle irregular damaged areas and significantly reduce the artifacts in the repair results.

[0037] More specifically, the Regong mural dataset has the characteristics of complex elements, diverse styles, and bright colors. However, the dataset size is relatively small, which poses challenges for the restoration of images with large - area defects. When current mainstream large - area defect restoration algorithms process such datasets, they often fail to fully learn image features, resulting in unsatisfactory restoration results. Specifically, the restored area is inconsistent with the original image in terms of texture, structure, and style, and artifacts or blurring may occur. This application designs a multi - scale texture restoration module that combines gated convolution and multi - scale Fourier convolution modules, as shown in Figure 2 shown. The gated convolution mechanism is used to dynamically adjust the feature flow and enhance the model's ability to capture key information. Subsequently, the multi - scale Fourier convolution module effectively captures the global and local information of the image through frequency - domain analysis. The specific structure is shown in Figure 3 shown. Fourier convolution processes the image by transforming it into the frequency domain, which can more effectively capture the global structural information of the image. Especially when the dataset is small, the model often has difficulty learning sufficient features from limited training samples. Fourier convolution increases the input dimension and feature representation ability of the model by introducing frequency - domain features, enabling the model to better learn the potential rules and features in the image. In some embodiments, to make up for the deficiency of fast Fourier convolution in fine - texture synthesis, the multi - scale Fourier convolution module obtains the local features of the image by passing a quarter of the dimension data through the local Fourier convolution module, which can to some extent make up for the deficiency of fast Fourier convolution in fine - texture synthesis. Among them, SpectralTransform is adopted to effectively fuse frequency - domain and spatial - domain features by transforming the input data into the frequency domain and extracting local spectral features.

[0038] In addition, in order to better repair and synthesize detailed textures, skip connections are introduced into the entire texture restoration network. The features obtained from the down - convolution module are directly passed to the up - convolution module. At the same time, the multi - scale structural features obtained from the structure - guiding module are also injected layer by layer as part of the decoder to enhance the model's ability to restore local details. While retaining the global structural information, it ensures the accurate reconstruction of local features. When dealing with large - area damage, it can maintain the overall consistency and coherence of the image and generate a restoration result more consistent with the original image.

[0039] Finally, the discriminator is used to judge the quality of the restored image. The two discriminators focus on judging the overall content and edge features of the image respectively. While maintaining the overall consistency of the image, they finely restore the complex image structure and details, thus significantly improving the restoration effect of images with complex structures.

[0040] Among them, the image discriminator is used to reconstruct the overall image discrimination of the image, and the edge discriminator is used to reconstruct the edge information discrimination of the image.

[0041] Specifically, this application uses two discriminators based on the PatchGAN architecture, which are respectively used to distinguish the overall quality of the original image and the generated image, as well as their edge details. PatchGAN is a special convolutional neural network that judges the authenticity of the entire image by classifying local regions (patches) in the image. While reducing the computational amount, it also enables the discriminator to capture fine textures and local features in the image. This design allows PatchGAN to focus on the local details of the image, enabling the model to more meticulously capture the detailed features of the image during the training process.

[0042] Based on the traditional method of using one discriminator to judge the overall image quality, this application adds another discriminator for edge details, which helps to accurately restore the edge information, thereby maintaining the coherence and visual quality of the image. Its input is the edge images of the real image and the predicted image obtained through the Canny algorithm, and it conducts adversarial training with the generator, prompting the generator to more finely control the generation of the image structure, making the edges of the predicted image sharper and more accurate, approaching the real image.

[0043] In addition, the model loss calculated during the training of the large-area defect repair network for mural images based on structure guidance includes: ① L 1 Loss: When calculating the L 1 loss, only the pixels marked as invalid (i.e., need to be repaired) in the mask M are considered, enabling the model to focus more on the areas that need to be repaired while ignoring the known background or areas that do not need to be repaired, thereby improving the model's attention to the repair quality of these areas and further enhancing the overall repair effect. The L 1 loss is expressed as: L 1 =(1 - M)║I pre - I gt ║ 1 In the formula, I pre and I gt represent the predicted image and the real image respectively; where M is a 0 - 1 mask, and 1 represents the masked area, that is, the part that needs to be repaired.

[0044] ② Adversarial loss: The adversarial loss consists of the feature extraction loss L D and the generator loss L G . The RNN discriminator based on PatchGAN is denoted as D, while the entire large-area defect repair method for mural images based on structure guidance is regarded as the generator G. Therefore, the adversarial loss can be expressed as:

[0045] LG = EIpre [log D(Ipre ))]

[0046]

[0047] wherein, D(x) represents the predicted probability of the discriminator for the image x; E represents the expectation, L GP represents the gradient penalty, λ GP = 1e-3. By adding the gradient penalty, the range of the discriminator's gradient change can be restricted, thereby maintaining the stability of the generator's gradient, preventing gradient explosion and gradient disappearance, and at the same time helping the network to converge to a better solution faster.

[0048] ③ High receptive field matching loss: The traditional perceptual loss evaluates the distance between the features extracted from the predicted image and the target image through a pre-trained base network. It does not require precise reconstruction and allows changes in the reconstructed image. The focus of large mask repair needs to be shifted to the understanding of the global structure. Therefore, a high receptive field perceptual loss L hrf is introduced. Different from the traditional perceptual loss, L hrf is implemented using dilated convolutions to improve the quality of large mask repair:

[0049] wherein, φ hrf represents the high receptive field perceptual loss trained by the pre-trained segmentation ResNet50 with dilated convolutions.

[0050] ④ Color consistency loss: The proposed color consistency loss function modifies the L 2 loss function to make it pay more attention to color consistency. First, the image is divided into three channels: red (R), green (G), and blue (B), and the squared differences between the predicted image and the target image on each channel are calculated separately. This step ensures that the color consistency loss can take into account the color information in the image, rather than just the brightness or grayscale values. A mask is used to distinguish the known area and the missing area, and different weights are applied. This allows the loss function to be more flexible in focusing on the parts of the image that need to be repaired during training. Calculate the average of the differences on all color channels, then multiply by the pixel weights, and finally take the average as the final loss value. The color consistency loss function can be expressed as:

[0051] wherein,n is the total number of pixels involved in the calculation, m i is the value of the mask at pixel i, where 0 indicates known and 1 indicates missing, known and missing are the weights of the known region and the missing region respectively, which are 10 and 5 respectively, p i R , p i G , p i B are respectively the red, green, and blue channel values of the predicted image at pixel i ; t i R , t i G , t i B are respectively the red, green, and blue channel values of the target image at pixel i ; Based on this, the total loss is:

[0052] where the weights are λ L1 = 10, λ adv = 10, λ hrf = 30, λ CCL = 15. The weights are determined according to the existing literature combined with experiments.

[0053] The main innovations and contributions of the solution provided by this application are as follows: (1) A multi-scale restoration network for Thangka mural images based on structure guidance is proposed, which repairs the defective line drawing in a semi-interactive manner and provides structural prior information for mural images with complex structures.

[0054] (2) A structure-guided restoration module and an edge discriminator are designed. The edge discriminator ensures that the generator restores an image with consistent structure, and the structure-guided restoration module guides texture restoration with multi-scale structural prior information, avoiding the weakening of structural prior information during network transmission and achieving deep fusion with texture information while preventing structural disorder.

[0055] (3) By introducing gated convolution, skip connections, multi-scale Fourier convolution modules, etc., the network can effectively extract information, effectively repair irregular and severely damaged Regong mural images, fuse strongly guided structural prior information, retain the original mural art style, and repair Regong mural images with clear structural textures and uniform colors.

[0056] To verify the effectiveness and generalization of the Regong mural image restoration network proposed in this embodiment, experimental verification was carried out.

[0057] 1. Datasets In this embodiment, two datasets were constructed, namely the Regong mural image dataset and the CelebA image dataset. First, the Regong mural image dataset is introduced. Due to the limitations of preservation conditions and digital acquisition technology, the obtained mural images have problems such as inclination and uneven illumination. Therefore, it is necessary to preprocess the dataset. In this embodiment, the images are adjusted by perspective cropping, adjusting contrast and light-dark relationship. After preprocessing, a total of 547 Regong mural images are obtained. Considering the scarcity of the building image dataset, in this embodiment, the dataset is expanded to 19,570 Regong mural images with a size of 256×256 pixels and meeting the training conditions by cropping. At the same time, the dataset is divided into a training set, a validation set, and a test set according to the ratio of 8:1:1. In the training stage, due to the large amount of data, if line drawings are drawn one by one, the workload is huge and there may be deviations. Therefore, the SketchKeras algorithm is used to extract line drawings from the cropped pictures to obtain the corresponding line drawing dataset. In the actual application stage, it is guided by Regong mural artists for drawing.

[0058] Different from ordinary mask datasets, combining the characteristics of mural damage in reality: including but not limited to paint peeling off along creases, some paint peeling off in blocks or dots, cracks in irregular directions, etc. In this embodiment, Photoshop is used to simulate these several damage shapes, make a mask dataset, and crop it into mask images with a size of 256×256, as Figure 4 shown, a total of 5320 mask images are obtained. Randomly divided into a training set and a test set according to 5:1.

[0059] At the same time, to verify the generalization of the method in this embodiment, the public dataset CelebA is used for generalization experiments. In this embodiment, 25,000 images are randomly selected and divided into a training set, a test set, and a validation set according to the ratio of 8:1:1.

[0060] 2. Experimental settings This experiment adopts a supervised training mode, with a generator learning rate of 1e-3, a discriminator learning rate of 1e-4, a batch size of 8, and the Adam optimizer is used. This experiment is completed on a cloud server. Hardware environment: Intel(R) Xeon(R) Gold 6330 CPU, NVIDIA RTX 3090 GPU 24GB; Software environment: Python = 3.8, Pytroch = 1.8.1, CUDA = 11.1.

[0061] 3. Evaluation Metrics In the quantitative comparison experiment, the Peak Signal to Noise Ratio (PSNR), Structural Similarity (SSIM), Fréchet Inception Distance (FID), and Learned Perceptual Image Patch Similarity (LPIPS) are used. PSNR can better reflect the objective quality of the image and mainly measure the overall distortion degree of the image. SSIM is used to measure the similarity between two images, including three aspects: brightness, contrast, and structure. The closer the SSIM value is to 1, the more similar the structural information of the restored image is to the original image. FID measures the image quality by calculating the distance between the feature vectors of the restored image and the real image in the Inception network. The smaller the FID, the closer the distribution of the restored image is to the original image distribution. LPIPS is a deep learning-based method that can better capture the visual perception characteristics of the human eye, is more in line with human evaluation of image quality, and is more sensitive to the distortion of details and textures.

[0062] 4. Experimental Results The mural dataset and the CelebA dataset are used to conduct comparative experiments on image inpainting. The comparative methods include several advanced image inpainting methods in recent years: CTSDG, AOTGAN, LaMa, ZITS, Deflocnet, SketchEdit, and SketchRefiner, to verify the reliability and superiority of the method in this embodiment for mural image inpainting. CTSDG models the structure-constrained texture synthesis and texture-guided structure reconstruction in a coupled manner so that the two generation tasks can better promote each other to obtain more accurate results. AOTGAN proposes an aggregated context transform generative adversarial network for high-resolution image inpainting, which captures information-rich remote context for context reasoning, and the model helps the generator synthesize fine-grained textures. LaMa proposes a network based on a fast Fourier convolution module for large-area missing and high-resolution image inpainting and proposes a high receptive field perception loss, which can achieve the consistency of the structure and texture of the inpainting results. ZITS proposes a new enhanced Transformer structure based on ZeroRA, which uses a powerful attention-based Transformer model in a fixed low-resolution sketch space to restore the overall image structure, and combines modules such as masked position encoding Fourier CNN texture restoration to repair an image with clear texture and reasonable structure. During the inpainting process, Deflocnet, SketchEdit, and SketchRefiner all utilize the interactive information provided by the user. Deflocnet directly injects the color and structure control information provided by the user into each structure generation block to refine sketch lines and propagate colors in the CNN feature space, effectively transmitting the user's intention and obtaining the desired editing result. SketchEdit proposes an image editing network that can achieve local modification of the image only with a simple sketch input from the user. SketchRefiner proposes using a cross-correlation loss function to robustly calibrate and refine the sketch, learning to extract information features from the abstract sketch in the feature space in the second stage, and adjusting the inpainting process.

[0063] 4.1. Quantitative Analysis Table 1 and Table 2 respectively list the PSNR, SSIM, FID, and LPIPS of different restoration algorithms on the Regong mural image dataset and the CelebA dataset. The results show that the method proposed in this embodiment performs well, demonstrating the superiority of the method in this embodiment. Compared with the evaluation index scores of other restoration algorithms, the PSNR, SSIM, FID, and LPIPS of the method in this embodiment on the Regong mural dataset are significantly improved, increasing by 8.385, 0.129, 73.688, and 0.098 respectively; on the CelebA dataset, the PSNR, SSIM, FID, and LPIPS of the method in this embodiment are increased by 7.862, 0.082, 17.115, and 0.068 respectively. It shows that the restoration results of the method in this embodiment are closer to the real images in terms of image pixel structure, distribution similarity, and perceptual similarity.

[0064] Table 1 Evaluation Index Results of Different Restoration Algorithms on the Regong Mural Image Dataset

[0065] Table 2 Evaluation Index Results of Different Restoration Algorithms on the CelebA Dataset

[0066] 4.2. Qualitative Analysis The visualization results of the restoration experiment on the Regong mural image dataset are as Figure 5 shown. It can be seen that the method in this embodiment performs better than the other seven types of restoration algorithms. Although CTSDG, AOTGAN, LaMa, and ZITS have proposed algorithms to improve the restoration results of large-area defects or high-resolution images, for the Regong mural dataset with complex structures, numerous elements, and a small dataset, without the guidance of line drawings, the structures of the defect areas cannot be restored. For example, the feet of the Buddha and the parts of the flower leaves in the first row show blurred structures; although Deflocnet combines the given sketches and color information of the user, it cannot make good use of this information. As can be seen from the figure, there will still be cases of distorted structural textures. In contrast, the restoration algorithm in this embodiment can obtain results with more reasonable and clear structures, higher texture quality, and visual quality.

[0067] The visualization results of the restoration experiment on the CelebA dataset are as Figure 6As shown, the method of this embodiment also has better performance compared with other seven types of restoration algorithms. AOTGAN and LaMa are relatively lacking in reconstructing the structure. For example, the nose in the first row can hardly be restored, and at the same time, serious artifacts appear in AOTGAN; CTSDG and ZITS have better results in terms of structure restoration, but for large-area missing parts, the restored structure is still unreasonable; Deflocnet can better restore reasonable structures and textures on face datasets with relatively simple structures and elements, but there will still be some distorted textures due to the failure to make good use of the interaction information between structure and color, such as the nose parts in the first and second rows; the restoration results of SketchEdit show serious distortions and fail to perfectly integrate structure and texture information; SketchRefiner; the method of this embodiment can integrate structure-guided information and texture features to restore better visual quality results.

[0068] 4.2. Subjective Evaluation Due to the complexity and particularity of Regong mural images, it is incomplete and inaccurate to evaluate the quality of the restoration results of Regong mural images only using the general evaluation indicators for restoring natural images. In order to comprehensively evaluate the restoration results, a subjective evaluation experiment session is designed for the restoration results of Regong mural images in this embodiment.

[0069] In this experiment, 15 participants were invited to subjectively evaluate the restoration results of Regong mural images. They were divided into three groups according to the categories of those who have never been exposed to Regong murals, those who have some understanding of Regong murals, and experts in Regong mural restoration or inheritors of Regong mural art, with five people in each group. Each group was respectively recorded as experimental group 1, experimental group 2, and experimental group 3. The method of this embodiment and the comparative methods were anonymously shown to the three groups of participants. The participants were required to score the restoration results from five aspects: structural coherence, color restoration, detail retention, cultural element accuracy, and overall visual beauty, with a full score of 5 points. The three groups weighted the preliminary scoring results according to a weight of 1:3:6 to obtain the final subjective evaluation results, as shown in Table 3. The experimental results show that the score of the method of this embodiment is significantly higher than other methods, indicating that the proposed method for restoring Regong mural images is feasible and effective and relatively conforms to human aesthetics.

[0070] Table 3 Subjective Evaluation Results of Different Restoration Methods for Regong Mural Dataset

[0071] 5. Ablation Experiment To verify the effectiveness of the three modules for interactive line drawing restoration, ablation experiments were conducted in this paper. The experiments were carried out under four conditions: removing interactive line drawing restoration, removing the edge discriminator, removing skip connections and convolutional layers, and removing color consistency loss. All ablation experiments were carried out under the same parameters, hardware, and software conditions. Table 4 shows the PSNR, SSIM, FID, and LPIPS results on the mural dataset under three conditions. It can also be seen from the experimental metric results that the quality of the restored images has been significantly improved after adding each module.

[0072] Table 4 Quantitative Results of Ablation Experiments

[0073] The following conclusions can be drawn from the above experiments: Existing traditional image restoration methods and deep learning-based automatic image restoration methods often produce unreasonable results that do not conform to the texture structure of murals when restoring murals. The rich composition and texture information of mural images make it difficult for the model to learn their unique structural and texture features, and it is difficult to ensure the rationality of the restoration results. Therefore, this paper designs a structure-guided method for repairing large-area defects in mural images. Under the professional guidance of mural artists, the damaged line drawing is accurately restored, ensuring the integrity of the line drawing structure and retaining its unique artistic style; the edge discriminator can better maintain the structural integrity and visual coherence of the restoration results; the design of convolutions such as skip connections and gated convolutions ensures the integrity of feature information and the ability to express details during the restoration process; in addition, the introduction of color consistency loss helps the restoration network maintain color consistency with the surrounding areas when restoring the defective area. Both the comparative experiment and the ablation experiment verify the superiority of the method in this paper. It can not only effectively restore damaged murals, but also ensure the artistic and authentic restoration effects. The restoration effect is closer to real mural images, which is of great significance for the protection and inheritance of intangible cultural heritage.

[0074] In addition, an embodiment of this application provides a structure-guided device for repairing large-area defects in mural images, including: A model training unit, configured to train a pre-constructed structure-guided large-area defect repair network for mural images using training samples to obtain a repair model; An image repair unit, configured to use the repair model to repair the defects of the mural image to be repaired; Among them, the structure-guided large-area defect repair network for mural images includes a generator and a discriminator; the generator includes a line drawing interactive repair module, a structure-guided repair module, and a multi-scale texture repair module; the discriminator includes an image discriminator and an edge discriminator; The line drawing interactive repair module is used to extract a damaged line drawing from the damaged mural image through the SketchKeras algorithm, and obtain the line drawing repair information manually added by experts for the damaged area of the damaged mural image, and obtain a completely repaired line drawing based on the damaged line drawing and the line drawing repair information; and when training the repair model, considering the data volume, the complete line drawing extracted by the SketchKeras algorithm is used as the interactively obtained complete line drawing; The structure-guided repair module is used to extract features from the completely repaired line drawing to obtain multi-scale structural features; The multi-scale texture repair module includes a downsampling module, a multi-scale Fourier convolution module, and an upsampling module connected in sequence; the downsampling module extracts the feature information of the damaged mural image through reflection padding and multiple downsamplings, and selects the pixels of the input image based on the gating mechanism; the multi-scale Fourier convolution module performs frequency domain conversion on part of the input data by integrating frequency domain analysis and spatial domain analysis, and can effectively capture the global information and local information of the image; the upsampling module is used to upsample the input based on the gating mechanism to obtain a reconstructed image, the input of the upsampling module includes the output of the multi-scale Fourier convolution module, and the output features transmitted through the skip connection of the downsampling module, and the multi-scale structural features obtained by the structure-guided repair module are also input layer by layer into the upsampling module; The image discriminator is used for the overall image discrimination of the reconstructed image, and the edge discriminator is used for the discrimination of the edge information of the reconstructed image.

[0075] Among them, regarding the specific implementation methods of each module of the above-mentioned structure-guided large-area damage repair device for mural images, reference can be made to the corresponding content in the foregoing method embodiments, and details are not described herein again.

[0076] It can be understood that the same or similar parts in the above embodiments can be referred to each other, and the content not detailed in some embodiments can be seen in the same or similar content in other embodiments.

[0077] It should be noted that in the description of the present invention, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" refers to at least two.

[0078] Any process or method description depicted in the flowchart or otherwise described herein can be understood to represent a module, segment, or portion of code including one or more executable instructions for implementing a specific logical function or process. The scope of the preferred embodiments of the present invention includes additional implementations where functions may be executed not in the order shown or discussed, including in a substantially simultaneous manner according to the involved functions or in a reverse order, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0079] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following techniques well-known in the art can be used: discrete logic circuits with logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits with appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0080] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the methods for implementing the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable storage medium. When this program is executed, it includes one or a combination of the steps of the method embodiments.

[0081] In addition, in each embodiment of the present invention, the functional units can be integrated into one processing module, or each unit can exist physically alone, or two or more units can be integrated into one module. The above integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The above-mentioned storage medium can be a read-only memory, a magnetic disk, an optical disc, etc.

[0082] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0083] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A structure-guided method for repairing large-area defects in mural images, characterized in that: include: The training samples are used to train the pre-built structure-guided large-area defect repair network of mural images to obtain the repair model; Using the restoration model to repair defects in the mural image to be repaired; The structure-guided large-area defect repair network for mural images includes a generator and a discriminator; the generator includes a line draft interactive repair module, a structure-guided repair module and a multi-scale texture repair module; the discriminator includes an image discriminator and an edge discriminator; The line drawing interactive repair module is used to extract a damaged line drawing from a damaged mural image through the SketchKeras algorithm, and obtain line drawing repair information manually added by an expert for the damaged area of ​​the damaged mural image, and obtain a repaired complete line drawing based on the damaged line drawing and the line drawing repair information; and the amount of data is taken into consideration when training the repair model, and the complete line drawing extracted based on the SketchKeras algorithm is used as the complete line drawing obtained through interaction; The structure-guided repair module is used to extract features from the repaired line drawing to obtain multi-scale structural features; The multi-scale texture restoration module includes a down-sampling module, a multi-scale Fourier convolution module and an up-sampling module which are connected in sequence; the down-sampling module extracts the characteristic information of the defective mural image through reflection filling and multiple down-sampling, and selects the pixels of the input image based on the gating mechanism; the multi-scale Fourier convolution module performs frequency domain conversion on part of the input data by integrating frequency domain analysis and spatial domain analysis, and can effectively capture the global information and local information of the image; the up-sampling module is used to up-sample the input based on the gating mechanism to obtain a reconstructed image, the input of the up-sampling module includes the output of the multi-scale Fourier convolution module, and the output features of the down-sampling module transmitted through the jump connection, and the multi-scale structural features obtained by the structure-guided restoration module are also input into the up-sampling module layer by layer; the image discriminator is used for overall image discrimination of the reconstructed image, and the edge discriminator is used for discrimination of edge information of the reconstructed image.

2. The method according to claim 1, characterized in that The processing process of the line draft interactive repair module includes: The damaged line drawing is extracted from the defective mural image using the SketchKeras algorithm: E m =SK(I gt ×M) Among them, I gt represents the original image, E gt and E m They represent the line drawings of the complete image and the damaged image respectively; M represents the simulated mask, where the missing area is 1 and the non-missing area is 0; Under the guidance of professionals, the damaged area of ​​the line drawing is accurately repaired to obtain the line drawing repair information E i , based on the damaged line drawing and the line drawing repair information, a repaired line drawing is obtained: E=E m +E i Among them, E represents the repaired complete line drawing.

3. The method according to claim 1, characterized in that The structure-guided repair module aims to retain low-level guidance information while achieving deep fusion with texture information to optimize the image repair process based on deep learning; wherein, the structure-guided repair module embeds control information into each level of the repair network decoder to perform more accurate structure guidance processing in the feature space of the convolutional neural network. In this process, an addition rather than splicing connection method is adopted to avoid the problem of unreasonable generation results that may be caused by excessively strong guidance information, thereby ensuring high-quality completion of image repair.

4. The method according to claim 1, characterized in that: The multi-scale Fourier convolution module uses fast Fourier convolution to convert the three-quarter dimensional data into the frequency domain for processing to effectively capture the global structural information of the image, thereby increasing the input dimension and feature representation capability of the model, and enabling the model to better learn the potential laws and features in the image; the one-quarter dimensional data is passed through a local Fourier convolution module to obtain the local features of the image to make up for the shortcomings of fast Fourier convolution in fine texture synthesis; wherein, the multi-scale Fourier convolution module adopts SpectralTransform, which realizes the effective fusion of frequency domain and spatial domain features by converting the input data into the frequency domain and extracting local spectral features.

5. The method according to claim 1, characterized in that The image discriminator and the edge discriminator are both discriminators based on the PatchGAN architecture; the input of the edge discriminator is the edge image of the real image and the predicted image obtained by the Canny algorithm, and the edge discriminator and the generator are adversarially trained to prompt the generator to more finely control the generation of the image structure, so that the edge of the predicted image is sharper and more accurate, and close to the real image.

6. The method according to any one of claims 1 to 5, characterized in that: The model losses calculated when training the structure-guided large-area defect repair network for mural images include: L1 loss: L1=(1-M)║I pre -I gt ║1 In the formula, I pre and I gt Represent the predicted image and the real image respectively; where M is a 0-1 mask, where 1 represents the mask area, that is, the part that needs to be repaired; Fighting Loss: IT G =E Ipre [logD(I pre ))] L adv =L D +L G +λ GP ×L GP In the formula, D(x) represents the predicted probability of the discriminator for image x; E represents the expectation, L GP represents the gradient penalty, λ GP =1e-3; By adding a gradient penalty, the range of gradient changes of the discriminator can be limited, thereby maintaining the stability of the generator gradient, preventing gradient explosion and gradient disappearance, and helping the network converge to a better solution faster; High receptive field matching loss: L hrf =E([φ hrf (I gt )-φ hrf (I pre )] 2 ) In the formula, φ hrf represents the high receptive field perceptual loss trained by the pre-trained segmentation ResNet50 with dilated convolutions; loss of color consistency; Where n is the total number of pixels involved in the calculation, m i is the value of the mask at pixel i, 0 means known, 1 means missing, ω known and ω missing are the weights of known areas and missing areas, which are 10 and 5 respectively. They are the red, green, and blue channel values ​​of the predicted image at pixel i, are the red, green, and blue channel values ​​of the target image at pixel i, respectively; The total loss is: L final =λ L1 L L1 +λ adv L adv +λ hrf L hrf +λ CCL L CCL Among them, the weight is λ L1 =10,λ adv =10,λ hrf =30,λ CCL =15.

7. A structure-guided large-area defect repair device for mural images, characterized in that: include: A model training unit is used to train a pre-built large-area defect repair network for mural images based on structure guidance using training samples to obtain a repair model; An image restoration unit, used for restoring defects of the mural image to be restored by using the restoration model; The structure-guided large-area defect repair network for mural images includes a generator and a discriminator; the generator includes a line draft interactive repair module, a structure-guided repair module and a multi-scale texture repair module; the discriminator includes an image discriminator and an edge discriminator; The line drawing interactive repair module is used to extract a damaged line drawing from a damaged mural image through the SketchKeras algorithm, and obtain line drawing repair information manually added by an expert for the damaged area of ​​the damaged mural image, and obtain a repaired complete line drawing based on the damaged line drawing and the line drawing repair information; and the amount of data is taken into consideration when training the repair model, and the complete line drawing extracted based on the SketchKeras algorithm is used as the complete line drawing obtained through interaction; The structure-guided repair module is used to extract features from the repaired line drawing to obtain multi-scale structural features; The multi-scale texture restoration module includes a down-sampling module, a multi-scale Fourier convolution module and an up-sampling module which are connected in sequence; the down-sampling module extracts the characteristic information of the defective mural image through reflection filling and multiple down-sampling, and selects the pixels of the input image based on the gating mechanism; the multi-scale Fourier convolution module performs frequency domain conversion on part of the input data by integrating frequency domain analysis and spatial domain analysis, and can effectively capture the global information and local information of the image; the up-sampling module is used to up-sample the input based on the gating mechanism to obtain a reconstructed image, the input of the up-sampling module includes the output of the multi-scale Fourier convolution module, and the output features of the down-sampling module transmitted through the jump connection, and the multi-scale structural features obtained by the structure-guided restoration module are also input into the up-sampling module layer by layer; the image discriminator is used for overall image discrimination of the reconstructed image, and the edge discriminator is used for discrimination of edge information of the reconstructed image.

Images