Mural image restoration method based on multilayer generative adversarial network
By constructing a multi-layer generative adversarial network model, the problems of incomplete data sets and large differences in characteristics in mural image restoration are solved, and high-quality repair of mural images is achieved, especially in terms of tone and details.
Patent Information
- Application Number
- CN202510248160.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-04
- Publication Date
- 2025-06-17
AI Technical Summary
The prior art has problems in the repair of mural images, such as incomplete data sets, large differences in image features, and unsatisfactory repair effects, especially when dealing with complex and grand mural scenes.
The mural image repair method based on a multi-layer generative adversarial network is adopted. By constructing an adversarial network model of a multi-layer generator and discriminator, the Dunhuang mural image data set is used for training to achieve the repair of damaged mural images in any area.
High-quality restoration of mural images was achieved, especially in terms of tone and details, with significantly improved restoration effects, and improved PSNR and SSIM values.
Smart Images
Figure CN120163740A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of digital ancient mural image restoration technology, and specifically relates to an image restoration method for damaged mural images based on a multi-layer generative adversarial network. Background Art
[0002] For thousands of years, most murals have been restricted by harsh natural weather conditions, leading to urgent corrosion and aging problems such as mildew, cracks, smoke, and large-scale peeling. However, in the process of restoring damaged murals, simply relying on painting experience for manual restoration easily causes restorative damages such as color differences and misunderstandings.
[0003] Therefore, using digital image processing technology to solve the problem of automatic restoration of ancient mural heritage through a data-driven method has become a research hotspot in the field of image processing. Currently, many restoration methods have been proposed, which are divided into traditional methods and deep learning methods. The relevant research on the former has been relatively mature, making it difficult to make progress, and the quality of the generated images is far inferior to that of deep learning methods. Deep learning restoration methods are generally divided into those based on the Autoencoder (AE) and the Generative Adversarial Networks (GAN).
[0004] (1) Image Restoration Method Based on AE
[0005] The core principle is to repair the damaged part by automatically learning image features. The autoencoder consists of an encoder and a decoder. The encoder is responsible for compressing the input image into a low-dimensional feature vector, and the decoder then reconstructs the complete image based on these feature vectors. Ballester et al. proposed a method of jointly interpolating the image gray level and the vector field to fill the area to be restored. This method has certain effectiveness in texture restoration, but it is powerless when faced with large-scale damaged images. To overcome this problem, Shen et al. innovatively introduced the Total Variation (TV) model into the image restoration task, organically combining it with the PDE, using the idea of anisotropic diffusion, and promoting the restoration of damaged images by minimizing the energy functional. Although this method has achieved some results in boundary restoration, it still has limitations in dealing with straight-line restoration problems.
[0006] (2) Image Restoration Method Based on GAN
[0007] The Generative Adversarial Network (GAN) is a deep learning model that consists of two main components: a Generator and a Discriminator. The goal of the Generator is to generate data that is as realistic as possible, making it indistinguishable from real data. The Discriminator, on the other hand, is responsible for determining whether the input data is real or fake data generated by the Generator. During the training process, the Generator continuously optimizes itself to deceive the Discriminator, while the Discriminator also improves its discrimination ability. The two compete with each other and co-evolve, ultimately reaching a balanced state where the data generated by the Generator approaches the real data distribution. The Generative Adversarial Network (GAN) proposed by Goodfellow in 2014 has made groundbreaking progress. In the process of image inpainting, the Generative Adversarial Network can better fit the data compared to the encoder-decoder, with a faster fitting speed and sharper generated samples. However, this method also has many drawbacks, such as unstable data training, uncontrollable model freedom, and training crashes.
[0008] Although deep learning has been widely applied and achieved remarkable results in the field of image restoration, due to the uniqueness of mural images, there are still many unresolved problems in their restoration work.
[0009] Deep learning image restoration commonly uses datasets of human faces, cars, or natural scenes. These datasets have simple structures and consistent features, which are very different from mural datasets. The murals to be restored in our country cover works from multiple eras with significant style differences. Moreover, the mural scenes are complex and grand, and it is difficult to find similar features in different parts of the same image. Scarce mural images with unified styles, clarity, and integrity lead to incomplete mural restoration datasets. Summary of the Invention
[0010] Aiming at the problems of the existing technology, the present invention proposes a digital mural image restoration method based on a multi-layer generator and discriminator adversarial network to achieve the restoration of mural images with damaged areas anywhere.
[0011] To achieve the above objective, the main content of the present invention is as follows:
[0012] A mural image restoration method based on a multi-layer generative adversarial network, comprising the following steps;
[0013] S1, Select the Dunhuang mural image dataset, perform cropping and screening to construct the training set required for the experiment; artificially construct a mask to simulate damaged mural images and construct the test set required for the experiment;
[0014] S2: Construct a GAN-based digital mural restoration model, and use the training set to train the model until the model generates relatively realistic and clear mural images, and save the model parameters; the GAN-based digital mural restoration model includes a generator and a discriminator with basically symmetric structures, including two layers of generators, namely U-net and Resnet50 respectively, and three layers of discriminators, focusing on the global, local, and pixel parts respectively;
[0015] S3: Utilize the model parameters saved in S2, use the simulated damaged mural images in the test set, update the model parameters by calculating the loss function, and iterate multiple times until the image restoration is completed.
[0016] 2. In order to unify the image styles in the dataset, the following preprocessing was performed on the collected mural images:
[0017] S1: Screen the mural images, only select the facial parts of the figures to obtain figure mural images, and use them as the training set;
[0018] S2: The images generated by the GAN network are all of the same size, and the inputs are also images of the same size. Therefore, the mural images are cropped into image blocks of K*K.
[0019] S3: Simulate the artificial construction of a mask, and overlay the mask and the intact mural image to simulate the damage of the mural.
[0020] In the GAN-based mural restoration model, the generator uses the ReLU function, and each layer in the discriminator uses the Leaky ReLU function.
[0021] In the described mural image restoration method, the generator loss function is as follows:
[0022]
[0023] Composed of adversarial loss , feature loss , artistic loss , perceptual loss and pixel loss respectively, , , , , are the corresponding weights respectively.
[0024] Specifically, the adversarial loss formula is as follows:
[0025]
[0026] Where is from the prior noise distribution The noise vector obtained by medium sampling, represents the sample generated by the generator according to the noise is the discriminative output of the discriminator for the generated sample, and its value range is usually between The closer it is to 1, the more real the discriminator thinks the sample is.
[0027] The feature loss formula is as follows:
[0028]
[0029] Among them, is the number of convolutional layers of the discriminator, is the number of feature maps in the th activation layer, is the feature map of the layer of the discriminator, is the original mural image,
[0030] The art loss formula is as follows:
[0031]
[0032] Among them, is the Gram matrix of size constructed from the th feature map.
[0033] The perceptual loss formula is as follows:
[0034]
[0035] Among them, is the feature map of the th layer of the pre-trained network.
[0036] The pixel loss formula is as follows:
[0037] ;
[0038] In the mural image restoration method described above, the calculation formula of the discriminator loss function is as follows:
[0039]
[0040] Among them , , are the global, local, and pixel discrimination losses respectively, , , are the corresponding weights respectively.
[0042] After experimental verification, when , , , , , , are 1, 120, 2, 1, 0.2, 0.4, and 0.4 respectively, a better repair effect can be obtained.
[0043] In the described mural image restoration method, when the generator and discriminator satisfy the following calculation formula:
[0044]
[0045] where denotes taking the expectation of on the original mural image (which follows the probability distribution ); denotes taking the expectation of on the image after adding the mask (which follows the probability distribution , and after the generator generates G(z) and then the discriminator
[0047] The beneficial effects of the present invention are:
[0048] First, use the intact image to train the GAN network, and then update the generator parameters according to the image restoration loss function to obtain the restored image. During the loss calculation process, by adding weights, the network attention can be effectively made more concentrated. This method has remarkable effects in the restoration of ancient murals, especially showing good performance in aspects such as color tone and details. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 is the process of constructing the artificial mask in the present invention;
[0050] Figure 2 is the specific structure of the GAN network in the present invention;
[0051] Figure 3 is the training process of the GAN network in the present invention;
[0052] Figure 4 is the comparison of the mural image restoration results in the present invention.
[0053] SPECIFIC IMPLEMENTATION EXAMPLES
[0054] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings. A digital mural image restoration method based on a multi-layer generator and discriminator adversarial network provided by the present invention specifically includes the following steps:
[0055] (1) Collect images of the human parts in Dunhuang murals to construct a dataset with unified style, clear and complete. By drawing a mask image and superimposing it on the original image, artificially simulate the damaged effect of the mural, and use the processed image as the training set for image restoration. The construction process is shown in Figure 1 .
[0056] (2) Construct a digital mural restoration model based on GAN, and use the training set to train the model until the model generates relatively real and clear mural images, and save the model parameters.
[0057] The construction process of the mural restoration model based on GAN is as follows:
[0058] S1: Construct a dataset of mural images with clear and unified style. Use the intact mural images in the training set to train the GAN network. After multiple iterations, until the network can generate relatively real mural images, save the model parameters. The basic structure of the GAN network is as Figure 2 , the discriminator and the generator are basically symmetric. Some of the generator encoding parameters are as follows:
[0059] The encoding process is mainly composed of multiple downsampling modules, and each downsampling module contains a series of convolutional layers, attention modules, normalization (BN) processing, and activation function (Relu) operations.
[0060] Similarly, some of the generator decoding parameters are as follows:
[0061] The decoding process is mainly composed of multiple upsampling modules, and each upsampling module contains convolutional operations to restore the details and size of the image.
[0062] Furthermore, the discriminator is divided into three layers: global, local, and pixel. Next, some of its parameters will be shown respectively.
[0063]
[0064]
[0065] The model training process is the same as the conventional training of the generative adversarial network. The training process is as follows Figure 3, first initialize the network parameters of the discriminator and the generator; then extract n samples from the training set, and the generator generates n samples based on the noise; subsequently, fix the parameters of the generator and train the discriminator to enable it to accurately distinguish between real and fake samples; then fix the discriminator and train the generator to make it difficult for the discriminator to distinguish between real and fake samples; after multiple iterations, ideally, the discriminator cannot determine whether the image comes from the generator or the real dataset.
[0066] S2: Use the model saved in S1, obtain the output image through the generator, input it into the discriminator, and the discrimination result is true (1) or false (0).
[0067] S3: Calculate its loss function, and calculate the generator loss and the discriminator loss respectively.
[0068] S4: Update the model parameters through the calculation results of the loss function, and continuously iterate (that is: repeat steps 2, 3, and 4) until a repaired result similar to the original damaged image is generated.
[0069] Experimental result analysis:
[0070] Figure 4 To save better experimental result images, Figure 4 (a) is the original image of the mural, Figure 4 (b) is the image simulating the damaged effect of the mural, Figure 4 (c)~(e) are the comparison methods, Figure 4 (f) is the method of this article, where Figure 4 (c) is FRRN, Figure 4 (d) is DeepFillv2, Figure 4 (e) is MEDEF. Subjectively, the repair results perform well in terms of detail clarity, color tone, and consistency. In terms of details, the damaged areas are clearly restored, and there is no large-area blur affecting the visual experience; in terms of color tone, the colors of the original image are highly restored, with almost no visual deviation; in terms of consistency, the repaired part is well integrated with the surrounding pixels, the overall image is smooth, and there are no obvious abrupt changes in tone or color.
[0071] However, it is difficult to give the most correct judgment solely based on subjective feelings, so it is necessary to combine objective evaluation. The present invention uses the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM) values for evaluation.
[0072] PSNR
[0073] SSIM
[0074] It can be obtained from the experimental data that both the PSNR and SSIM values are improved after image restoration. Among them, the PSNR value is above 30 dB, indicating that the mural restoration result is good, the image distortion is not obvious, and it is within the acceptable range of human visual perception. Similarly, the average value of the structural similarity index reaches 0.96, which is highly similar to the original image and achieves the expected restoration effect.
[0075] The specific embodiments of the present invention are only illustrative examples of the present invention. Those skilled in the art can make various modifications, supplements or similar substitutions thereto without departing from the spirit of the present invention and the scope defined by the appended claims.
Claims
1. A method for restoring ancient murals based on a multi-layer generator and discriminator adversarial network, characterized in that: The following steps are involved: S1: Select the Dunhuang mural image dataset, cut and filter it, and construct the training set required for the experiment; artificially construct a mask to simulate the damaged mural image and construct the test set required for the experiment; S2: Build a GAN-based digital mural restoration model, train the model using the training set, until the model generates a relatively realistic and clear mural image, and save the model parameters; The GAN-based digital mural restoration model includes a generator and a discriminator with a basically symmetrical structure, including two layers of generators, using U-net and Resnet50 respectively, and three layers of discriminators, focusing on the global, local, and pixel parts respectively; S3: Using the model parameters saved in S2 and the damaged mural images simulated in the test set, the model parameters are updated by calculating the loss function, and the process is repeated multiple times until the image restoration is completed.
2. According to claim 1, a method for restoring ancient mural images based on a multi-layer adversarial network is characterized in that: In the mural image dataset S1, in order to make the image style in the dataset uniform, the collected mural images were preprocessed as follows: S1: In order to unify the style, the mural images are screened and only the facial parts of the characters are selected to obtain the mural images of the characters, which are used as the training set; S2: The images generated by the GAN network are all of the same size, and the inputs are also images of the same size, so the mural image is cut into K*K image blocks; S3: Simulate the artificial construction of a mask and superimpose the mask with the intact mural image to simulate the damage of the mural.
3. The mural image restoration method based on a multi-layer adversarial network according to claim 1 is characterized in that: In the GAN-based mural restoration model, the generator uses the ReLU function, and each layer in the discriminator uses the Leaky ReLU function.
4. The mural image restoration method based on a multi-layer adversarial network according to claim 1 is characterized in that: The generator loss function in S2 is as follows: The adversarial loss , feature loss , artistic loss , Perceptual Loss and pixel loss composition, , , , , are the corresponding weights respectively.
5. The mural image restoration method based on a multi-layer adversarial network according to claim 1 is characterized in that: The calculation formula of the discriminator loss function in S3 is as follows: in , , They are global, local, and pixel discrimination losses, , , are the corresponding weights respectively.
6. The ancient mural image restoration method based on a multi-layer adversarial network according to claim 1 is characterized in that: In the mural image restoration method, when the generator and the discriminator satisfy the following calculation formula: in, , indicating that in the original mural image (Its probability distribution ) Seek expectations; Represents the image after adding the mask (Its probability distribution , Through the generator Generate G(z), and then the discriminator Judgment) To meet expectations means to achieve the best results.
7. The ancient mural image restoration method based on a multi-layer adversarial network according to claim 4 is characterized by: The adversarial loss formula is as follows: in is the prior noise distribution The noise vector sampled from Represents the generator according to the noise The generated samples, is the discriminant output of the generated sample, and its value range is usually The closer it is to 1, the more real the discriminator thinks the sample is; The feature loss formula is as follows: in, is the number of convolutional layers of the discriminator, It is The number of feature maps in the activation layer, It is the discriminator The feature map of the layer, It is the original mural image. This is the restoration result of the damaged murals; The formula for art loss is as follows: in, By The size of the feature map constructed is The Gram matrix of The perceptual loss formula is as follows: in, It is the pre-trained network Feature map of the layer; The pixel loss formula is as follows: 。