Progressive damaged fresco image restoration method

Abstract

The invention provides a progressive damaged fresco image restoration method, which comprises the following steps of: processing first splicing data by an edge structure restoration module constructed based on a Transform model to obtain an edge restoration fresco line draft graph; processing the damaged mural image by a mural image restoration module constructed based on a Stable Diffusion model, and generating a first restoration mural image by taking the edge restoration mural line draft graph as a control condition of the mural image restoration module; and a color restoration module constructed based on the conditional generative adversarial network processes second splicing data obtained by splicing the first restoration mural image and the damaged area mask channel to obtain a second restoration mural image. According to the progressive damaged fresco image restoration method, the damaged fresco image is cooperatively restored from the two aspects of structure restoration and color restoration through cooperation of multiple modules, the restored fresco image with the good restoration effect can be obtained, and compared with the prior art, the progressive damaged fresco image restoration method has the advantages of being good in restoration effect and the like.

Description

A progressive method for restoring damaged mural images Technical Field

[0001] This invention relates to the field of image restoration, specifically to a method for progressively restoring damaged mural images. Background Technology

[0002] In the field of mural restoration, using digital technology to restore mural images is currently one of the mainstream restoration methods. Image restoration has made significant progress in recent years, but its application in mural restoration still has some shortcomings, such as difficulty in balancing the relationship between edges and structure, and unsatisfactory color deviation restoration results. Summary of the Invention

[0003] This invention provides a progressive method for restoring damaged mural images. By utilizing the cooperation of multiple modules, the damaged mural image is repaired in two aspects: structural repair and color restoration. This method can obtain a restored mural image with better restoration effect and has advantages such as better restoration effect compared with the prior art.

[0004] This invention provides a progressive method for restoring damaged mural images, comprising: channel stitching of a damaged mural image, a damaged mural edge map, and a damaged area mask to obtain first stitched data, wherein the damaged mural edge map is generated by edge extraction of the damaged mural image, and the damaged area mask corresponds to the damaged area of ​​the damaged mural image; an edge structure restoration module based on a Transformer model processes the first stitched data to obtain an edge-restored mural line drawing; a mural image restoration module based on a Stable Diffusion model processes the damaged mural image and uses the edge-restored mural line drawing as a control condition for the mural image restoration module to generate a first restored mural image; and a color restoration module based on a conditional generative adversarial network processes the second stitched data obtained by channel stitching the first restored mural image and the damaged area mask to obtain a second restored mural image.

[0005] In an optional implementation, the edge structure restoration module includes sequentially connected... CNN downsampling layer Layer Transformer unit and Layer CNN upsampling layer; where, , .

[0006] In an optional implementation, each layer of the Transformer unit includes an axial attention block and a standard self-attention block.

[0007] In an optional implementation, the edge structure restoration module is trained based on a first training set, which includes several first data sets. Each first data set includes first input data and first output data. The first input data includes a damage mask, an actively damaged mural image after the damage mask is applied to a complete mural image, and an actively damaged mural edge map corresponding to the actively damaged mural image. The first output data includes a complete mural line drawing corresponding to the complete mural image. During the training process of the edge structure restoration module, the third stitched data obtained by stitching the channels of the damage mask, the actively damaged mural image, and the actively damaged mural edge map is used as the input of the edge structure restoration module, and the complete mural line drawing is used as the ideal output.

[0008] In an optional implementation, the complete mural line drawing is obtained by processing the corresponding complete mural image using the Canny edge detector.

[0009] In an optional implementation, the mural image restoration module includes a Stable Diffusion unit and a ControlNet module corresponding to the Stable Diffusion unit, wherein the ControlNet module operates on the U-Net structure of the Stable Diffusion unit; using the edge-restoring mural line drawing as the control condition for the mural image restoration module includes: inputting the edge-restoring mural line drawing into the ControlNet module.

[0010] In an optional implementation, the mural image restoration module is trained based on a second training set, which includes several sets of second data groups. Each set of second data groups includes second input data and second output data. The second input data includes an actively damaged mural image obtained by processing a complete mural image through a damage mask and a complete mural line drawing corresponding to the complete mural image. The second output data includes the complete mural image. During the training process of the mural image restoration module, the actively damaged mural image is used as input, the complete mural line drawing is used as control conditions, and the complete mural image is used as the ideal output.

[0011] In an optional implementation, the color restoration module processing includes a generator and a discriminator; the color restoration module based on a conditional generative adversarial network processes the second stitched data obtained by stitching the first restored mural image and the mask channel of the damaged area, including: inputting the second stitched data into the generator, and the generator generating the second restored mural image.

[0012] In an optional implementation, the generator comprises sequentially connected... First generator convolutional block, self-attention block A dilated convolution fusion residual block and A second generator convolutional block, the The first generator convolutional block and A second generator convolutional block skip connection; and / or the discriminator includes... Each discriminator convolutional block; among them... , , .

[0013] In an optional implementation, the color restoration module is trained based on a third training set, which includes several sets of third data groups. Each set of third data groups includes third input data and third output data. The third input data includes fourth stitching data. The first stitching data is obtained by stitching the restored image and the damaged mask channel. The restored image is obtained by restoring an actively damaged mural image through the mural image restoration module. The actively damaged mural image is then processed by the damaged mask to obtain a complete mural image. The third output data includes a complete mural image corresponding to the restored image. During the training process of the color restoration module, the fourth stitching data is used as the input data of the generator, and the complete mural image is used as the ideal output.

[0014] The edge structure restoration module generates a line drawing of the mural for edge restoration. The line drawing of the mural for edge restoration controls the image generation process of the mural image restoration module, thereby obtaining a first restored mural image with high structural integrity. Finally, the color restoration module is used to restore the color of the first restored mural image.

[0015] This invention provides a progressive method for restoring damaged mural images. This method restores images from two aspects: line structure and color correction. First, an edge structure recovery module based on a Transformer model is used to learn a normalized grayscale sketch space to predict the global edge and line relationships of the damaged mural image. The damaged mural image is then input into a mural image restoration module built on a Transformer model, and the predicted edge-restoring mural line drawing is input as a dataset into a Control Net to strictly control the first restored mural image generated by the mural image restoration module. Finally, the first restored image and the mask channel of the damaged area are stitched together and input into a color restoration module for processing. The generator in the color restoration module performs local color correction and restoration. After testing and comparison, the progressive method for restoring damaged mural images provided by this invention can effectively restore damaged mural images, and the restored mural images have advantages such as structural integrity and good color consistency. Attached Figure Description

[0016] Figure 1 is a flowchart of the progressive damaged mural image restoration method according to an embodiment of the present invention.

[0017] Figure 2 is a schematic diagram of the function of the edge structure restoration module in an embodiment of the present invention.

[0018] Figure 3 is a schematic diagram of the edge structure restoration module according to an embodiment of the present invention.

[0019] Figure 4 is a schematic diagram of the Transformer unit structure according to an embodiment of the present invention.

[0020] Figure 5 is a schematic diagram of the mural image restoration module according to an embodiment of the present invention.

[0021] Figure 6 is a schematic diagram of the operating principle of U-Net according to an embodiment of the present invention.

[0022] Figure 7 is a schematic diagram of the principle of the color restoration module in an embodiment of the present invention.

[0023] Figure 8 is a schematic diagram of the generator structure according to an embodiment of the present invention.

[0024] Figure 9 is a schematic diagram of the discriminator structure according to an embodiment of the present invention.

[0025] Figure 10 is a schematic diagram of the restoration results of damaged mural images using different image processing methods.

[0026] Figure 11 is a schematic diagram of the ablation experiment results of an embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Figure 1 is a flowchart of the progressive damaged mural image restoration method according to an embodiment of the present invention.

[0029] This invention provides a method for progressively repairing damaged mural images, including: S101: An edge structure restoration module based on the Transformer model processes the first stitched data to obtain an edge-repaired mural line drawing; Figure 2 shows a schematic diagram of the function of the edge structure restoration module in an embodiment of this invention.

[0030] The damaged mural image, the damaged mural edge map, and the damaged area mask are stitched together to obtain the first stitched data. The damaged mural edge map is generated by edge extraction of the damaged mural image, and the damaged area mask corresponds to the damaged area of ​​the damaged mural image.

[0031] Specifically, the edges in the mural line drawing for edge restoration correspond to the concept of "edge" in image restoration. An edge refers to an area in an image where brightness, color, or texture undergoes significant discontinuous changes; it is essentially the boundary between regions. For example, when a dark object is placed against a light background, the pixel values ​​at the object's outline will change abruptly, thus forming an edge.

[0032] The core mechanism of the Transformer model is self-attention. Compared to traditional convolutional neural network methods, self-attention allows each pixel patch in an image to directly interact and perform weighted calculations with all other pixel patches in the image. For example, when it is necessary to repair an occluded area in an image (e.g., a person's face is blocked by an object in a photo), the Transformer model can "see" the entire image. It can look for clues from the other side of the image (such as a person's hair, ears, or symmetrical structures in the background) to infer what the occluded part should look like. This is crucial for recovering large-scale, structural defects and has powerful global and long-distance dependency modeling capabilities. In addition, thanks to global information, the repair content generated by the Transformer model is not only rich in texture details but also more accurate in semantics and structure, avoiding inconsistencies. The generation of the structure is such that, for example, when restoring a face, it doesn't draw an eye on the forehead because it connects the overall layout of the face information through global information. In addition, the Transformer model is highly adaptable to irregular missing regions. Compared with convolutional neural networks, the attention mechanism of the Transformer is content-driven rather than position-fixed. Whether the missing region is a regular rectangle (a common assumption of traditional restoration) or an irregular arbitrary shape (such as scratches, graffiti, or object removal), the Transformer model can handle the relationship between all pixel blocks equally, thus performing robust restoration. In addition to the above, the Transformer model has many advantages in the direction of edge restoration. This enables the edge structure restoration module of this embodiment to perform edge structure restoration on damaged mural images and obtain edge-restored mural line drawings with better restoration effects.

[0033] It should be noted that the edge restoration mural line drawing mainly reflects the edge structure of the mural image. Its essence is to express the structural composition of the image. The restoration logic relies on the self-attention mechanism of the Transformer model. Under the premise of ensuring that the image composition is reasonable, a restored edge restoration mural line drawing is obtained from the damaged mural image.

[0034] S102: The mural image restoration module, built based on the Stable Diffusion model, processes the damaged mural image and uses the edge restoration mural line drawing as the control condition for the mural image restoration module to generate a first restored mural image. The execution process of the mural image restoration module is an image-to-image process, that is, a process that takes the damaged mural image as input and the first restored mural image as output. In this step, since the input is the damaged mural image, the edge restoration mural line drawing obtained in step S101 is used to control the image structure during the image-to-image generation process.

[0035] Correspondingly, in this step, the main function of the mural image restoration module is to restore the damaged mural image into a complete first restored mural image, mainly using the edge restoration mural line drawing as a structural guide to perform preliminary color restoration on the damaged area.

[0036] It should be noted that since the edge restoration mural line drawing can be understood as the dividing line of color blocks, the process of generating the image in this step can logically be regarded as the process of filling in color blocks (the actual data processing process is more complex and will be explained later). This step mainly utilizes the characteristics of the Stable Diffusion model in terms of good image generation effect, low resource consumption, and controllable generation process to obtain a relatively complete first restored mural image without obvious image meaning defects.

[0037] S103: The color restoration module based on the conditional generative adversarial network processes the second stitching data obtained by stitching the first restored mural image and the mask channel of the damaged area to obtain the second restored mural image.

[0038] The color restoration module is built on a conditional generative adversarial network. The generator used in the trained color restoration module can be logically considered as a replication module. The first mural image to be restored is input into the generator for replication. The generator restores the colors in the first mural image by learning the replication technique (model data learned during training), thereby replicating a second mural image that is close to the real mural image.

[0039] In summary, the progressive damaged mural image restoration method provided in this embodiment of the invention first uses an edge structure restoration module to generate an edge-repaired mural line drawing. This edge structure restoration module is built based on the Transformer model, whose attention mechanism effectively combines the specific content of the mural for structural restoration, ensuring the restoration effect. The restored edge-repaired mural line drawing is used to control the image-generating process of the mural image restoration module. This module is based on the Stable Diffusion model. The Diffusion model is a highly efficient open-source image generation tool. Compared to traditional image generation tools, it allows for precise control of the generated images through plugins such as ControlNet. In this embodiment of the invention, a mural image restoration module is used to perform preliminary restoration on damaged mural images. During the restoration process, the edge restoration line drawing of the damaged mural is used as a control condition to ensure the structural integrity of the first restored mural image. Furthermore, preliminary color restoration is performed on the damaged areas of the damaged mural image, resulting in a first restored mural image with high integrity. Finally, a color restoration module is used to perform color restoration on the first restored mural image. The color restoration module is constructed based on a conditional generative adversarial network and utilizes the imitation technology learned by the generator to recreate a second restored mural image with more accurate colors based on the first restored mural image. The modules involved in steps S101 to S103 are further explained below.

[0040] Figure 3 is a schematic diagram of the edge structure recovery module in an embodiment of the present invention, and Figure 4 is a schematic diagram of the Transformer unit in an embodiment of the present invention.

[0041] Basically, the edge structure restoration module is built based on the Transformer model. In this embodiment of the invention, the edge structure restoration module includes a CNN encoder, a position encoder, a Transformer module and a CNN decoder connected in sequence.

[0042] The CNN encoder includes sequentially connected components. The CNN downsampling layer consists of a Transformer module consisting of sequentially connected layers. Layer Transformer units, CNN decoders consist of sequentially connected layers. A CNN upsampling layer. In this embodiment of the invention... , .

[0043] Specifically, CNN downsampling layers have three main functions: expanding the receptive field, suppressing redundant information (enhancing distortion prevention), and reducing computational cost, transforming image processing from pixel-level processing to processing an abstract object of the image. The specific object representing this abstract image is determined by the number of CNN downsampling layers. It's important to note that the image output from a CNN downsampling layer has the same or reduced resolution compared to the input image, but with an increased number of channels.

[0044] Specifically, the position encoder is actually used to encode the position of the data processed by the CNN encoder. The Transformer model's self-attention mechanism has a permutation invariance property, meaning that if the order of the input sequence is shuffled (e.g., image patches are randomly arranged), the output sequence calculated by the self-attention mechanism will not change in content, only the order will be shuffled. However, in image processing, the positions of image patches cannot be arbitrarily shuffled. Therefore, the image data obtained after processing by the CNN downsampling layer needs to have an absolute positional encoding added to each pixel to ensure that the Transformer module can consider the pixel's positional information when calculating attention.

[0045] Specifically, the input content of the Transformer unit is essentially a sequence of vectors (blocks) composed of image pixels. In this embodiment of the invention, each layer of the Transformer unit includes an axial attention block and a standard self-attention block (Vanila Attention).

[0046] Specifically, the basic components of a Transformer unit are an attention block and a feedforward neural network block. Each block is surrounded by residual connections and layer normalization to ensure training stability.

[0047] Specifically, for each block in the input sequence, the attention block calculates attention weights with all other blocks in the sequence. These weights determine how much information should be “borrowed” from each other block when repairing the current block. The feedforward neural network module is used to perform a non-linear transformation on the output of the self-attention block for further data processing.

[0048] In this embodiment of the invention, the attention blocks in the Transformer unit include two types: axial attention blocks and standard self-attention blocks. Accordingly, feedforward neural network blocks are added to the axial attention blocks and standard self-attention blocks respectively, resulting in the specific model structure of each Transformer unit in this embodiment of the invention.

[0049] Specifically, the Transformer unit in this embodiment of the invention includes an axial attention block, a first connection block, a first normalization block, a first feedforward neural network block, a second connection block, a second normalization block, a standard attention block, a third connection block, a third normalization block, a third feedforward neural network block, and a fourth connection block connected in sequence. The input of the axial attention block is connected to the first connection block, the output of the first connection block is also connected to the second connection block, the output of the second connection block is also connected to the third connection block, and the output of the third connection block is also connected to the fourth connection block.

[0050] Specifically, the CNN upsampling layer has the opposite function of the CNN downsampling layer. It restores and refines the low-resolution features containing global information obtained by the Transformer module to the original image size.

[0051] Specifically, after designing the edge structure restoration module, it is necessary to train the module to meet practical application requirements. Specifically, the edge structure restoration module is trained based on a first training set, which includes several sets of first data groups. Each set of first data groups includes first input data and first output data. The first input data includes a damaged mask, an actively damaged mural image after the damaged mask is applied to a complete mural image, and an actively damaged mural edge map corresponding to the actively damaged mural image. The first output data includes a complete mural line drawing corresponding to the complete mural image. During the training process of the edge structure restoration module, the third stitched data obtained by stitching the channels of the damaged mask, the actively damaged mural image, and the actively damaged mural edge map is used as the input to the edge structure restoration module, and the complete mural line drawing is used as the ideal output.

[0052] Specifically, in the training process of this embodiment of the invention, the data related to the mural images comes from the DhMurals1714 dataset, which includes 525 digital photographs and 1189 professional reproductions of the Mogao Grottoes murals. In this embodiment, the dataset is divided into a training set (1,564 images), a validation set (100 images), and a test set (50 images). Additionally, the data related to the masks comes from the Quick Render Irregular Masking Dataset (QD-IMD).

[0053] Specifically, the damaged mask, actively damaged mural image, actively damaged mural edge map, and complete mural line drawing involved in the first training set are all obtained through corresponding processing methods; specifically, the damaged mask is taken from the fast drawing irregular mask dataset; the actively damaged mural image is obtained by applying the damaged mask to the complete mural image, and the complete mural image comes from the DhMurals1714 dataset; the actively damaged mural edge map is obtained by edge extraction of the actively damaged mural image; the complete mural line drawing is obtained by edge extraction of the complete mural line drawing.

[0054] Specifically, the edge extraction tool can be the Canny edge detector.

[0055] After obtaining the first training set data, the edge structure recovery module of this embodiment of the invention is trained according to the training method disclosed in the prior art until the training is completed; the training effect can be verified using data in the verification set after the data is processed by the corresponding data processing method.

[0056] Figure 5 is a schematic diagram of the structure of the mural image restoration module according to an embodiment of the present invention, and Figure 6 is a schematic diagram of the U-Net operating principle according to an embodiment of the present invention.

[0057] Basically, the mural image restoration module of this invention can be understood as a StableDiffusion model that has undergone specific training. Specifically, the mural image restoration module includes a Stable Diffusion unit and a ControlNet module corresponding to the Stable Diffusion unit, wherein the ControlNet module operates on the U-Net structure of the StableDiffusion unit.

[0058] Specifically, the Stable Diffusion unit itself consists of three core modules: VAE, U-Net, and TextEncoder.

[0059] Specifically, VAE stands for Variational Autoencoder, responsible for transforming images between pixel space and latent space. In this embodiment of the invention, when an image is input into the mural image restoration module, it first needs to be compressed by VAE to a low-dimensional latent space for representation (latent data). This latent data is much smaller than the original image, preserving the core semantic information (image meaning) of the image. This processing can greatly reduce the memory and computational overhead of subsequent U-Net computations. After the data processing in the latent space dimension is completed, the final output image is generated through decoding.

[0060] U-Net can be functionally understood as a denoiser, and its essential structure is a convolutional neural network with an encoder-decoder architecture. Specifically, U-Net predicts the noise in the noisy latent variable (the noisy latent data, obtained by adding noise to the latent data corresponding to the input image) based on the text prompt words. Then, it denoises the noisy latent variable with this predicted noise, resulting in a slightly cleaner noisy latent variable (one step less than the input noisy latent variable). The number of repetitions is expressed as the step count, and by repeating this process multiple times, an output image (represented by the latent variable) that is different from the input image is gradually generated.

[0061] The Text Encoder is a text understander whose purpose is to receive text prompts from the user and convert them into text embedding vectors. This allows it to control and guide the generation of noise during the noise estimation process, telling U-Net what to generate.

[0062] Specifically, in actual operation, when a damaged mural image is input into the Stable Diffusion unit, the VAE first converts the damaged mural image into latent data, calculates the amount of initial noise to be added based on the user-set "denoising intensity" (between 0 and 1), and obtains the noisy latent variables and the number of iterations.

[0063] The noisy latent variable, the text prompt received by the Text Encoder, and the current time step are input into U-Net. After U-Net predicts the noise in the current latent variable, it uses a specific algorithm (such as DDIM, DPM-Solver) to subtract a portion of the predicted noise from the current noisy latent variable, resulting in a latent variable with less noise (the time step corresponding to this latent variable is the input time step minus 1).

[0064] After iterating through the total number of iterations, the cleanest latent variable is obtained. Then, the latent variable is decoded into the pixel space using a VAE to obtain the final output image.

[0065] In this embodiment of the invention, the mural restoration module further includes the ControlNet module corresponding to the Stable Diffusion unit. Specifically, the ControlNet module is an auxiliary module of the Stable Diffusion unit. The core idea of ​​the ControlNet module is to "lock" the original Stable Diffusion model (usually called "lock encoder") and create a trainable "copy" or "bypass branch" for U-Net. This branch is specifically designed to learn to process additional conditional inputs. Finally, the image control signal extracted by the ControlNet module is injected into U-Net through a zero convolutional layer to guide image generation. The degree (weight) of the image control signal injection into U-Net is controlled by a zero convolutional layer with initial weights of zero.

[0066] Therefore, the ControlNet module can be understood as a module structure that runs parallel to the Text Encoder. It consists of two independent conditional input sources for U-Net, which work together to finely guide U-Net to generate controllable and high-quality images.

[0067] Specifically, in this embodiment of the invention, using the edge-restoring mural line drawing as the control condition for the mural image restoration module includes: inputting the edge-restoring mural line drawing into the ControlNet module. As described above, the edge-restoring mural line drawing is actually an edge map, the purpose of which is to control the generated contour of the image. Therefore, the denoising of U-Net can be guided by using the edge-restoring mural line drawing as the control condition.

[0068] Specifically, after determining the structure of the mural image restoration module, the mural image restoration module is trained based on a second training set. The second training set includes several sets of second data groups, each set of second data groups including second input data and second output data. The second input data includes an actively damaged mural image obtained by processing a complete mural image through a damage mask and a complete mural line drawing corresponding to the complete mural image. The second output data includes the complete mural image. During the training process of the mural image restoration module, the actively damaged mural image is used as input, the complete mural line drawing is used as control conditions, and the complete mural image is used as the ideal output.

[0069] Similarly, using the DhMurals1714 dataset and the Quick Render Irregular Masking dataset (QD-IMD) as data sources, the complete mural images are from the DhMurals1714 dataset, the damaged masks are from the Quick Render Irregular Masking dataset, and the complete mural line drawings are obtained by edge extraction from the complete mural images (preferably using the Canny edge detector).

[0070] It should be noted that, based on the explanation of the principle of the Stable Diffusion unit, the Stable Diffusion unit requires text prompts when outputting images. Since the purpose of the Stable Diffusion unit in this embodiment of the invention is image restoration, the input and output images are of the same type. Therefore, the text prompts can be prompts describing the input image or descriptions such as "similar to the input image".

[0071] Specifically, since this embodiment of the invention also includes a ControlNet module, at the start of training, since the weights are zero (the initial weights of the zero convolutional layer are zero), the output of the ControlNet module is also zero. This means that it will not affect the main U-Net, and the behavior of the main U-Net is exactly the same as the original Stable Diffusion, ensuring that the knowledge of the pre-trained model will not be destroyed in the early stage of training. During the training process, the weights gradually learn from zero, and the ControlNet learns to generate meaningful control signals and slowly injects them into the main U-Net.

[0072] Figure 7 is a schematic diagram of the principle of the color restoration module in an embodiment of the present invention, Figure 8 is a schematic diagram of the generator structure in an embodiment of the present invention, and Figure 9 is a schematic diagram of the discriminator structure in an embodiment of the present invention.

[0073] Specifically, the color restoration module processing includes a generator and a discriminator; the color restoration module based on conditional generative adversarial network processes the second stitched data obtained by stitching the first restored mural image and the mask channel of the damaged area, including: inputting the second stitched data into the generator, and the generator generating the second restored mural image.

[0074] Specifically, the generator includes sequentially connected... The first generator convolutional block (down sanple), and the self-attention block. Each dilated convolution fused residual block and The second generator convolutional block (Up sample), the The first generator convolutional block and A second generator convolutional block skip connection; and / or the discriminator includes... Each discriminator convolutional block; among them... , , .

[0075] Specifically, since the main object of the color restoration module in this embodiment of the invention is the color of the damaged area, and the colors of other positions are real colors, the following method can be adopted when outputting the image: Specifically, the real area image in the first restored mural image is retained (referred to as the real area image), the damaged area image in the output image obtained by the generator is retained (referred to as the damaged area image), and the real area image and the damaged area image are merged and then output to obtain the final required second restored mural image.

[0076] Specifically, in terms of data processing format, the damaged area mask is represented by "mask", the first restored mural image is represented by "input", and the output image directly obtained by the generator is represented by "output". Therefore, the final second restored mural image can be represented as follows: .

[0077] Specifically, the conventional first restored mural image data can be understood as a set of three-channel (RGB) data, the damaged area mask is a set of single-channel data, and after channel stitching, the second stitched data is four-channel data. The generator uses the four-channel second stitched data to generate and output a three-channel second restored mural image.

[0078] During training, the generator learns the input image based on the data provided by the damaged region mask. For the data outside the mask (i.e., the non-repaired area), it learns and constructs an identity mapping, directly copying the content of the input image. For the data inside the mask (i.e., the repaired area), it learns a context-based generative mapping, that is, based on the pixel information outside the mask, it infers what should look like inside the mask.

[0079] To achieve this function, the calculation of the loss function needs to be limited to a specific image region. Therefore, the typical loss function of the color restoration module in this embodiment of the invention is:

[0080] in, To rebuild the losses, In order to perceive loss, To combat the losses, , and For the corresponding weighting coefficients, specifically, , This serves as a mask for the damaged region; however, when calculating the reconstruction loss, this loss only considers the differences in the unmasked regions (M=0), forcing the generator to accurately replicate the content of the unrepaired areas. For the repaired regions (M=1), other losses are used to constrain the loss. In this typical loss function, by improving... The weight values ​​can be used to strictly control the image of the non-repaired areas to remain unchanged.

[0081] Specifically, the color restoration module is trained based on a third training set, which includes several sets of third data groups. Each set of third data groups includes third input data and third output data. The third input data includes fourth stitching data. The first stitching data is obtained by stitching the restored image and the damaged mask channel. The restored image is obtained by restoring an actively damaged mural image through the mural image restoration module. The actively damaged mural image is then processed by the damaged mask to obtain a complete mural image. The third output data includes a complete mural image corresponding to the restored image. During the training process of the color restoration module, the fourth stitching data is used as the input data of the generator, and the complete mural image is used as the ideal output.

[0082] Specifically, regarding the data in the third input data group, during the training process, the repaired image is obtained by repairing the actively damaged mural image through the mural image repair module. The damage mask is consistent with the damage mask corresponding to the actively damaged mural image, and the complete mural image is consistent with the complete mural image corresponding to the actively damaged mural image.

[0083] It should be noted that the edge structure restoration module, mural image restoration module, and color restoration module involved in the embodiments of the present invention each have corresponding training and application phases. In the training phase, the module's input data is the corresponding training data. The module's output data needs to be compared with the corresponding training data, and the module's internal parameters are adjusted through corresponding adjustment logic to achieve module learning. In the application phase, the module, after learning, outputs corresponding data based on the input data. In both phases, the module has uniform requirements for the format of the input data; therefore, in Figures 2, 5, and 6, the input data for the module can be either training data or actual data. In the embodiments of the present invention, in order to simulate the input damaged mural image, an actively damaged mural image is obtained by fusing the complete mural image and the damaged mask. The actively damaged mural image and the damaged mural image are images of the same type.

[0084] To evaluate the restoration effect of the progressive damaged mural image restoration method provided in this embodiment of the invention, data from a test set were used for verification. Specifically, the selected comparison objects include the following restoration methods and their combinations: 1. SD-inpainting (SD image restoration): Fine-tuning Stable Diffusion by using a randomly erased mask.

[0085] 2. SRN+CCN (from reference Li Luxi, Zou Qin, Zhang Fan. Line drawing guided progressive inpainting of mural damages. In arXiv preprint arXiv:2211.06649, 2025.): Based on known edges, this method uses two generative adversarial networks to achieve phased restoration of image structure and color respectively.

[0086] 3. EdgeConnect (EC, from reference Kamyar Nazeri, Eric Ng, Tony Joseph. EdgeConnect: Structure guided image inpainting using edge prediction. In Proceedings of the IEEE / CVF International Conference on Computer Vision Workshops, 2019.): This method uses two dilated convolution-based generative adversarial networks to first generate image edges and then perform image content inpainting.

[0087] 4. ControlNet (from reference Hsieh Tienyi, Zhao Qijun, Pan Fan. Text and Edge Guided Thangka Image Inpainting with Diffusion Model. In 2024 IEEE International Conference on Multimedia and Expo (ICME), 2024, 1: 1–10.): uses image edges as conditional input to a diffusion model to perform constrained restoration of damaged images.

[0088] 5. MECNet (from reference Wang Dongsheng, Xie Chaohao, Liu Shaohui, et al. Image inpainting with edge-guided learnable bidirectional attention maps. In arXiv preprint arXiv:2104.12087, 2021.): Based on EdgeConnect, it introduces a multi-scale convolution module to enhance the ability to repair large-scale damaged regions (such as wide masks).

[0089] Figure 10 shows the restoration results of damaged mural images using different image processing methods. Specifically, the comparison results are compared in six dimensions: SSIM, PSNR, MSE, CIEDE2000, FSIMc, and FID. The specific comparison data results are shown in Table 1.

[0090] It should be noted that the method referred to by Ours is the progressive damaged mural image restoration method disclosed in the embodiments of this invention, and sigma is an adjustable parameter of the Canny edge detector.

[0091]

[0092] Table 1 compares the restoration results of damaged mural images using different image processing methods. The damaged mural images are obtained by fusing a complete mural image with a damaged mask. In Figure 9, the first column shows the complete mural image, the second column shows the damaged mask, and subsequent columns show the restoration results of the restoration method of this embodiment (sigma = 1.0, sigma = 1.5, sigma = 2.0) and the restoration structures of six other methods. In Figure 9, methods involving line structures (such as EdgeConnect and MECNet) perform poorly, resulting in severe color inconsistencies; the SRN + CCN method tends to produce artifacts and color distortion; the MECNet + SD + CC method achieves relatively good restoration compared to other methods (except for the restoration method provided in this embodiment), but the results still lack vibrant colors and clear textures; the SDI method can generate relatively reasonable image content, but there are obvious artifacts. In contrast, the progressive damaged mural image restoration method provided in this embodiment achieves complete structural restoration, with sharp details and consistent overall color.

[0093] Furthermore, reasonable qualitative and quantitative evaluations can be performed from the data. Specifically, the Structural Similarity Index (SSIM), Peak Signal-to-Noise Ratio (PSNR), Mean Squared Error (MSE), Fréchet Inception Distance (FID), CIEDE200, and FSIMc are used as evaluation metrics for all comparison methods. The progressive damaged mural image restoration method provided by this invention can effectively learn the global structure in the sparse tensor space, providing key support for improving image restoration performance. The results show that the progressive damaged mural image restoration method provided by this invention significantly outperforms other methods in mainstream image restoration and generation methods, demonstrating obvious advantages and competitiveness.

[0094] In addition, to measure the performance of each module in the progressive damaged mural image restoration method provided in this embodiment of the invention, an ablation experiment was also conducted to evaluate the contribution of each module to the restoration of the damaged mural image.

[0095] Figure 11 is a schematic diagram of the ablation experiment results of the embodiment of the present invention, and Table 2 is a data table of the ablation experiment evaluation implemented in the present invention.

[0096]

[0097] Table 2 specifically states that the ablation experiment refers to the process of progressively damaged mural image restoration method provided in the embodiments of the present invention, in which the edge structure restoration module (TSR), color restoration module (CCN) are removed respectively, and the edge structure restoration module and color restoration module are removed simultaneously, respectively, to obtain the restored mural image without the assistance of each module.

[0098] In Figure 10, the damaged mural image is obtained by fusing a complete mural image and a damaged mask. In Figure 10, the first column shows the complete mural image, the second column shows the damaged mask, the third column shows the restored mural image obtained by the progressive damaged mural image restoration method of this embodiment, the fourth column shows the restored mural image obtained after removing the Structure Restore (TSR) module, the fifth column shows the restored mural image obtained after removing the color restoration module, and the sixth column shows the restored mural image obtained after simultaneously removing both the TSR and color restoration modules. Combining visual observation and evaluation data, the progressive damaged mural image restoration method provided by this embodiment significantly improves color consistency and fidelity.

[0099] In summary, this invention provides a progressive method for restoring damaged mural images. This method restores images from two aspects: line structure and color correction. First, an edge structure recovery module based on a Transformer model is used to learn a normalized grayscale sketch space to predict the global edge and line relationships of the damaged mural image. The damaged mural image is then input into a mural image restoration module built based on a Transformer model, and the predicted edge-restoring mural line drawing is input as a dataset into a Control Net to strictly control the first restored mural image generated by the mural image restoration module. Finally, the first restored image and the mask channel of the damaged area are stitched together and input into a color restoration module for processing. The generator in the color restoration module performs local color correction and restoration. After testing and comparison, the progressive method for restoring damaged mural images provided by this invention can effectively restore damaged mural images, and the restored mural images have advantages such as structural integrity and good color consistency.

[0100] The above provides a detailed description of a progressive method for restoring damaged mural images according to embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for progressively restoring damaged mural images, characterized in that, include: The damaged mural image, the damaged mural edge map, and the damaged area mask are stitched together to obtain the first stitched data. The damaged mural edge map is generated by edge extraction of the damaged mural image, and the damaged area mask corresponds to the damaged area of ​​the damaged mural image. The edge structure restoration module based on the Transformer model processes the first stitched data to obtain the edge-repaired mural line drawing; the mural image restoration module based on the Stable Diffusion model processes the damaged mural image and uses the edge-repaired mural line drawing as the control condition of the mural image restoration module to generate the first restored mural image. A color restoration module based on a conditional generative adversarial network processes the first restored mural image and the second stitched data obtained by stitching the mask channel of the damaged area to obtain the second restored mural image.

2. The method for progressive restoration of damaged mural images as described in claim 1, characterized in that, The edge structure recovery module includes a CNN encoder, a position encoder, a Transformer module, and a CNN decoder connected in sequence.

3. The method for progressive restoration of damaged mural images as described in claim 2, characterized in that, Each layer of the Transformer unit includes an axial attention block and a standard self-attention block.

4. The method for progressive restoration of damaged mural images as described in claim 1, characterized in that, The edge structure restoration module is trained based on a first training set, which includes several sets of first data groups. Each set of first data groups includes first input data and first output data. The first input data includes a damaged mask, an actively damaged mural image after the damaged mask is applied to a complete mural image, and an actively damaged mural edge map corresponding to the actively damaged mural image. The first output data includes a complete mural line drawing corresponding to the complete mural image. During the training process of the edge structure restoration module, the third stitched data obtained by stitching the channels of the damaged mask, the actively damaged mural image, and the actively damaged mural edge map is used as the input of the edge structure restoration module, and the complete mural line drawing is used as the ideal output.

5. The method for progressive restoration of damaged mural images as described in claim 4, characterized in that, The complete mural line drawing is obtained by processing the corresponding complete mural image using the Canny edge detector.

6. The method for progressive restoration of damaged mural images as described in claim 1, characterized in that, The mural image restoration module includes a Stable Diffusion unit and a ControlNet module corresponding to the Stable Diffusion unit. The ControlNet module operates on the U-Net structure of the Stable Diffusion unit. Using the edge-repaired mural line drawing as the control condition for the mural image restoration module includes: inputting the edge-repaired mural line drawing into the ControlNet module.

7. The method for progressive restoration of damaged mural images as described in claim 1, characterized in that, The mural image restoration module is trained based on a second training set, which includes several sets of second data groups. Each set of second data groups includes second input data and second output data. The second input data includes an actively damaged mural image obtained by processing the complete mural image through a damaged mask and a complete mural line drawing corresponding to the complete mural image. The second output data includes the complete mural image. During the training process of the mural image restoration module, the actively damaged mural image is used as input, the complete mural line drawing is used as control condition, and the complete mural image is used as ideal output.

8. The method for progressive restoration of damaged mural images as described in claim 1, characterized in that, The color restoration module includes a generator and a discriminator; the color restoration module based on a conditional generative adversarial network processes the second stitched data obtained by stitching the first restored mural image and the mask channel of the damaged area, including: inputting the second stitched data into the generator, and the generator generating the second restored mural image.

9. The method for progressive restoration of damaged mural images as described in claim 8, characterized in that, The generator includes sequentially connected... First generator convolutional block, self-attention block A dilated convolution fusion residual block and A second generator convolutional block, the The first generator convolutional block and A second generator convolutional block skip connection; and / or the discriminator includes... Each discriminator convolutional block; among them... ， ， 。 10. The method for progressive restoration of damaged mural images as described in claim 8, characterized in that, The color restoration module is trained based on a third training set, which includes several sets of third data sets. Each set of third data sets includes third input data and third output data. The third input data includes fourth stitching data. The first stitching data is obtained by stitching the restored image and the damaged mask channel. The restored image is obtained by restoring the actively damaged mural image through the mural image restoration module. The actively damaged mural image is obtained by processing the complete mural image through the damaged mask. The third output data includes a complete mural image corresponding to the restored image; During the training process of the color restoration module, the fourth stitching data is used as the input data of the generator, and the complete mural image is used as the ideal output.

Images