Digital restoration method of Dunhuang murals based on adversarial diffusion model and feature fusion module

Abstract

The invention discloses a digital restoration method for Dunhuang murals based on an adversarial diffusion model and a feature fusion module, and relates to the technical field of image restoration. After the mural image denoising preprocessing task is completed, the method uses a self-designed adversarial diffusion model to optimize the predicted results at different time steps, enhances the semantic consistency between masked and unmasked areas, and adds a variational transformer layer (VTL) to ensure that the denoised image and the predicted results are consistent in structural semantics; then uses a self-designed multi-scale feature aggregation module (MCFA) to encode rich semantic features at multiple scales, enhance the correlation between mural contexts, and reconstruct murals that fit the original content and style.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a digital restoration method for Dunhuang murals based on an adversarial diffusion model and a feature fusion module. Background Art

[0002] Dunhuang murals are famous for their rich content, bright colors and unique craftsmanship, and have important scientific, historical and artistic value. However, wind, sun and human activities have caused the murals to suffer various damages, such as fading, scratches, breaks and mildew. The manual restoration of murals not only requires professional painting skills, but may also lead to irreversible restoration results or even secondary damage. Non-contact digital restoration technology has become a better choice for mural restoration.

[0003] Traditional methods usually use geometric or patch methods to use the semantic information of the unmasked area to restore the masked part of the image. Reconstruct the damaged area. However, these methods have difficulty capturing high-level semantic context and are not effective for images containing non-repeating structures or complex backgrounds. In recent years, image restoration technology based on convolutional neural networks (CNN) and generative adversarial networks (GAN) has made great progress. Most modern image restoration uses a generative adversarial model, combined with a CNN codec and a discriminator, to restore the defective area by understanding the image context, and use the discriminator to enhance texture details and generate visually realistic content.

[0004] However, when processing masked areas, convolutional neural networks often only focus on the local semantics of the surrounding area, ignoring the bridging and utilization of information on a global scale. In addition, the unpredictability of the masked area of ​​the input image causes the convolution to be unable to accurately capture effective information. This uncertainty may lead to information loss during image restoration and even cause distortion in unmasked image areas.

[0005] As an emerging generative model, the Denoising Diffusion Probabilistic Model (DDPM) performs well in image denoising, but there are still two problems when used for mural restoration: First, it ignores the semantic consistency between masked and non-masked areas, which may cause the background after restoration to be distorted. Second, although it can reconstruct the defective area, it is difficult to ensure that the restored mural fits the original content and style.

[0006] To solve the above problems, the present invention proposes a Dunhuang mural restoration method based on an adversarial diffusion model and a multi-scale feature aggregation module. After completing the mural image denoising preprocessing task, the method uses a self-designed adversarial diffusion model to optimize the predicted results at different time steps, enhance the semantic consistency between masked and unmasked areas, and add a variational transformer layer (VTL) to ensure that the denoised image and the predicted results are consistent in structural semantics. Then, the self-designed multi-scale feature aggregation module (MCFA) is used to encode rich semantic features at multiple scales, enhance the correlation between the mural contexts, and reconstruct murals that fit the original content and style. Summary of the invention

[0007] The purpose of the present invention is to solve the above-mentioned problems and to provide a digital restoration method for Dunhuang murals based on an anti-diffusion model and a feature fusion module.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] According to one aspect of the present invention, a method for digital restoration of Dunhuang murals based on an anti-diffusion model and a feature fusion module is provided, which is used for restoration of Dunhuang murals and includes the following steps:

[0010] Step 1: Image denoising preprocessing through denoising diffusion probability model

[0011] In the forward diffusion stage T In time steps, the input image x 0Add with specific variance Gaussian noise:

[0012] (1)

[0013] (2)

[0014] In the formula, is the conditional probability density function, I is the identity matrix, N represents a normal distribution, and Represent the state of the previous moment and the current moment respectively, with the time step t The increase in input image x 0 gradually becomes independent Gaussian white noise x T Since the noise diffusion follows a Markov process, any time step t The noise added result It can be defined as:

[0015] (3)

[0016] In the formula, , and , is a random variable of standard normal distribution, which represents the noise added in the diffusion process. Reconstruct images from noise and use Markov chains and Bayesian formulas to train neural networks to predict posterior conditional distributions :

[0017] (4)

[0018] In the formula is a parameterized conditional probability density function with a mean and variance The calculation formula is as follows:

[0019] (5)

[0020] (6)

[0021] Then by inverse solution To get the new mean:

[0022] (7)

[0023] Finally, train the parameterized model To predict from arrive Noise accumulation , the training objective is:

[0024] (8)

[0025] In the formula, represents the expected operation. During the test phase, the random noise x T Start by applying the model ,from x t Iterate to x 0;

[0026] Step 2: Use the adversarial diffusion model to enforce semantic consistency between masked and unmasked regions

[0027] After completing the image denoising task, the adversarial diffusion model is used to solve the semantic error problem between the masked and non-masked areas of the mural. In the forward diffusion process, real murals are randomly selected. , according to formula (3), each step of the forward process is as follows:

[0028] (9)

[0029] The random noise , , and ;

[0030] In the reverse denoising process, it is no longer predicted arrive The noise accumulated in the middle is not through a defective mural , a binary mask M and a line drawing As conditional input, directly predict and restore the mural , that is, Instead of predicting the noise in formula (7):

[0031] (10)

[0032] Where ADM is a prediction network, which consists of a prediction backbone network, two conditional encoding networks and a discriminator. In the inference stage, Gaussian noise is randomly obtained. X T ,in T is the number of sampling steps, and then, under the conditions of a defective mural, a line drawing and a binary mask, the diffusion process is reversed to obtain the final restored mural.

[0033] Step 3: Add a variational transformer layer to enhance the interaction of semantic feature space

[0034] The variational transformer layer consists of a transformer encoder and a CNN decoder, which is used for the spatial interaction of different semantic features and fuses the line drawing and damaged mural features in the latent space to ensure the structural and semantic consistency of the denoised image;

[0035] First, from the given damaged mural , binary mask , noisy murals and line drawing murals The encoder learns multi-scale visual information; where h and w Represents the number of pixels in the vertical and horizontal dimensions of the image respectively:

[0036]

[0037]

[0038] (11)

[0039] In the formula, f d en , f z en , f m en are different semantic features with the same spatial size, Enc g and Enc d is an encoder consisting of two convolutional stages, Enc m It consists of gated convolution and extracts effective features from different damaged murals, which can update the mask and renormalize the convolution feature map to better extract effective features from different damaged murals;

[0040] Specifically given different semantic features f d en , f z en , f m en ,VTL embeds each feature through the encoder, generator Dec a Use parameterization techniques to reconstruct multimodal features that may have lost content:

[0041]

[0042] (12)

[0043] In the formula, m and s The encoders are Enc a The estimated mean and deviation, Z is a randomly generated vector under standard normal distribution, is a feature reconstructed based on the decoder's potential distribution. and After that, continue to apply formula (12) to obtain the final reconstructed features ;

[0044] Step 4: Use multi-scale feature aggregation module to ensure global context consistency

[0045] The multi-scale feature aggregation module is used to capture the relevance of the mural context, focusing on generating reasonable mural content through multi-scale feature splicing. The multi-scale feature aggregation module can encode rich semantic features at multiple scales, automatically learn and understand the structural texture and color features of the visible area, and use them to repair and fill the damaged area.

[0046] First, given different semantic features obtained from VTL and , concatenate them as input F :

[0047] (13)

[0048] In the formula, Represents the feature concatenation operation, and then uses three sets of dilated convolutional layers with different dilation rates to capture multi-scale semantic features:

[0049] (14)

[0050] In the formula, The expansion rate is j ∈{1, 2, 3} dilated convolutional layers;

[0051] There is a problem of effective information loss for sparse dilated convolution kernels with a pixel-level weight map. By predicting the weights of different dilated convolutions, the convolution features from different kernels are effectively aggregated:

[0052] (15)

[0053] In the formula, is the convolution operation, is channel-wise Softmax, It is a channel-wise slice; multi-scale semantic features are aggregated and weighted element by element to produce a refined feature map :

[0054] (16)

[0055] In order to adaptively incorporate multi-scale semantic features into image decoding features, a spatial adaptive normalization method is used to reduce the conflict between different branches. W 4 As input, the image features are transformed pixel by pixel, and the process can be written as:

[0056] (17)

[0057] In the formula, and are two variables output by the two convolutional layers. Finally, a skip connection is used to prevent semantic damage caused by the patch shift operation, and the output result is Embedded into modular architecture to improve computational efficiency;

[0058] (18)

[0059] In the formula, Represents non-parametric instance normalization; in this way, MCFA is able to reduce information loss while expanding the range of feature fusion, and adaptively refine the semantic encoding features of the mural, thereby improving the accuracy of mural restoration through context perception.

[0060] Furthermore, the following loss function is included to make the restored murals consistent with the ground truth mural distribution, structure and style;

[0061] Diffusion loss function: To ensure the restoration of murals and real murals The diffusion loss function set by the present invention is as follows:

[0062] (19)

[0063] Where ADM is a parameterized network, where represents the absolute value, To add noise to the mural, t is the time step, For line drawing, To damage the murals and M is a binary mask.

[0064] Perceptual loss and style loss function: Perceptual loss and style loss The features of VGG16 are used to learn high-level structures and overall style information, ensuring the style consistency between the reconstructed murals and the ground truth murals in a high-dimensional feature space.

[0065] The perceptual loss is calculated as follows:

[0066] (20)

[0067] The style loss is calculated as follows:

[0068] (twenty one)

[0069] In the formula, It is i A VGG16 layer network. G represents the Gram matrix on the feature map, .

[0070] Hole loss and effective loss function: Through hole loss calculate and The L1 distance of the mask area in the middle. By effective loss calculate and In this way, the L1 distance of the background area is maintained. and Content consistency between:

[0071] (twenty two)

[0072] (twenty three)

[0073] Edge loss function: In order to make the reconstructed mural have overall coherent lines, the present invention proposes an edge loss function To maintain the consistency of the overall structure:

[0074] (twenty four)

[0075] In the formula, represents the Sobel operator.

[0076] Adversarial loss function: This function is used to match the design of adversarial diffusion model to directly predict the repair results. , designed to fill damaged areas of a mural with realistic content:

[0077] (25)

[0078] In the formula D is the discriminator, G For the generator, is a logarithmic function.

[0079] KL divergence loss function: The present invention also optimizes the KL divergence to constrain the distance between the learning posterior distribution and the prior distribution:

[0080] (26)

[0081] In the formula, , and Responsible for projecting multi-semantic features into the normal distribution space, E 1. E 2 and E 3 are the encoding layers of the three VTL modules. Z1, Z2 and Z3 are the eigenvectors in the normal distribution space.

[0082] Finally, the above losses are combined to obtain the training loss of the prediction network:

[0083] (27)

[0084] In the formula, and is the loss function hyperparameter.

[0085] Compared with the prior art, the present invention has the following beneficial effects:

[0086] The adversarial diffusion model created by the present invention is different from other diffusion-based restoration models. It can directly predict the restoration results of murals, optimize the predicted mural content at different time steps, and use line drawings and a variety of different loss functions to constrain the restoration process to approximate the original structure and style of the murals. At the same time, a variational transformer layer (VTL) is added to fuse the line drawings and defective mural features in the latent space to generate content close to the original murals.

[0087] The multi-scale feature aggregation (MCFA) module proposed in the present invention can further capture spatial multi-scale semantic features using the dilated convolutional layer on the basis of the adversarial diffusion model, prevent semantic damage caused by patch shift operations through jump connections of the convolutional layer output variables, ensure the overall quality of the restored murals through global context interaction, and make the structural style of the reconstructed murals consistent with the ground truth murals by reasonably configuring the loss function.

[0088] The present invention is based on the forefront of digital restoration technology for Dunhuang murals. The adversarial diffusion model and multi-scale feature aggregation module in the technical system play a key role. It is organically combined with the denoising diffusion probability model, variational transformer layer and loss function to jointly solve the connotation problem of Dunhuang mural restoration, thus forming a set of technical solutions for Dunhuang mural reconstruction. The comparative experimental results show that the method of the present invention achieves better mural restoration effects than the currently popular methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is the technical route and flow chart of the present invention;

[0090] Figure 2 Schematic diagram of the variational transformer layer;

[0091] Figure 3 This is a schematic diagram of the multi-scale feature aggregation module framework;

[0092] Figure 4 A comparison of the visual effects of five image restoration methods. DETAILED DESCRIPTION

[0093] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0094] The present invention provides a digital restoration method for Dunhuang murals based on an anti-diffusion model and a feature fusion module. The implementation steps and key algorithms are as follows:

[0095] Step 1: Image denoising preprocessing through denoising diffusion probability model

[0096] Aiming at the specific task of restoring Dunhuang murals, the present invention improves the denoising diffusion probability model. T In time steps, the input image x 0Add with specific variance Gaussian noise:

[0097]

[0098] (2)

[0099] In the formula, is the conditional probability density function, I is the identity matrix, N represents a normal distribution, and Represent the state of the previous moment and the current moment respectively. t The increase in input image x 0 gradually becomes independent Gaussian white noise x T Since the noise diffusion follows a Markov process, any time step t The noise added result It can be defined as:

[0100] (3)

[0101] In the formula, , and . is a random variable with standard normal distribution, which represents the noise added in the diffusion process. The reverse denoising process aims to reverse the forward diffusion and calculate the probability distribution Reconstruct the image from the noise. Since the initial image cannot be obtained The true distribution of Therefore, the present invention uses Markov chain and Bayesian formula to train neural network to predict the posterior conditional distribution :

[0102] (4)

[0103] In the formula is a parameterized conditional probability density function with a mean and variance The calculation formula is as follows:

[0104] (5)

[0105] (6)

[0106] Then by inverse solution To get the new mean:

[0107] (7)

[0108] Finally, train the parameterized model To predict from arrive Noise accumulation , the training objective is:

[0109] (8)

[0110] In the formula, represents the expected operation. In the test phase, from random noise x T Start by applying the model ,from x t Iterate to x 0.

[0111] Step 2: Use the adversarial diffusion model to enforce semantic consistency between masked and unmasked regions

[0112] After completing the image denoising task, the present invention uses the adversarial diffusion model to solve the semantic error problem between the masked and unmasked areas of the mural. The model directly predicts the complete content of the mural, and optimizes the predicted results at different time steps through the discriminator to ensure that the filling content of the missing area of ​​the mural is not only reasonable, but also consistent with the semantics of the visible area. In the forward diffusion process, the real mural is randomly selected. , according to formula (3), each step of the forward process is as follows:

[0113] (9)

[0114] The random noise , , and .

[0115] In the reverse denoising process, it is no longer predicted arrive The noise accumulated in the middle is not through a defective mural , a binary mask M and a line drawing As conditional input, directly predict and restore the mural , that is, Instead of predicting the noise in formula (7):

[0116] (10)

[0117] In the formula, ADM is a prediction network, which consists of a prediction backbone network, two conditional encoding networks and a discriminator. It generates content that is consistent with the context semantics and fits the original mural structure style. In the inference stage, Gaussian noise is randomly obtained. X T ,in T is the number of sampling steps. Then, under the condition of a defective mural, a line drawing and a binary mask, the diffusion process is reversed to obtain the final restored mural.

[0118] Step 3: Add a variational transformer layer to enhance the interaction of semantic feature space

[0119] The variational transformer layer (VTL) consists of a transformer encoder and a CNN decoder, such as Figure 2 It is designed for spatial interaction of different semantic features and can fuse line drawing and damaged mural features in latent space to ensure the structural and semantic consistency of the denoised image. In particular, the transformer-based encoder can explicitly simulate the global long-range interaction of different semantic features and enhance the overall effect of mural restoration.

[0120] First, from the given damaged mural , binary mask , noisy murals and line drawing murals The encoder of is used to learn multi-scale visual information. h and w Represents the number of pixels in the vertical and horizontal dimensions of the image respectively:

[0121]

[0122]

[0123] (11)

[0124] In the formula, f d en , f z en , f m en are different semantic features with the same spatial size. Enc g and Enc d is an encoder consisting of two convolutional stages, Enc m It consists of gated convolutions and extracts effective features from different damaged murals.

[0125] features, which can update the mask and renormalize the convolutional feature map to better extract effective features from different damaged murals.

[0126] Specifically given different semantic features f d en , f z en , f m en ,VTL embeds each feature through the encoder, generator Dec a Use parameterization techniques to reconstruct multimodal features that may have lost content:

[0127]

[0128] (12)

[0129] In the formula, m and s The encoders are Enc a The estimated mean and deviation, Z is a randomly generated vector under standard normal distribution, is a feature reconstructed based on the decoder’s latent distribution. and After that, continue to apply formula (12) to obtain the final reconstructed features In this way, the learned prior semantic information can be effectively integrated into a unified representation to achieve semantic complementarity for the damaged murals.

[0130] Step 4: Use multi-scale feature aggregation module to ensure global context consistency

[0131] The multi-scale feature aggregation module (MCFA) is designed to capture the relevance of the mural context, focusing on generating reasonable mural content through multi-scale feature splicing. The module framework is as follows Figure 3As shown in Figure 2, this module can encode rich semantic features at multiple scales, automatically learn to understand the structural, texture and color features of the visible area, and use them to repair and fill in damaged areas. The reconstructed mural is both contextually consistent and fits the structure and style of the original mural.

[0132] First, given different semantic features obtained from VTL and , concatenate them as input F :

[0133] (13)

[0134] In the formula, Represents the feature concatenation operation. Three sets of dilated convolutional layers with different dilation rates are then used to capture multi-scale semantic features:

[0135] (14)

[0136] In the formula, The expansion rate is j ∈{1, 2, 3} of the dilated convolutional layer.

[0137] To address the problem that sparse dilated convolution kernels may lead to loss of effective information, this paper designs a pixel-level weight map, which effectively aggregates convolution features from different kernels by predicting the weights of different dilated convolutions:

[0138] (15)

[0139] In the formula, is the convolution operation, is channel-wise Softmax, It is a channel-wise slice. Multi-scale semantic features are aggregated and weighted element by element to produce a refined feature map. :

[0140] (16)

[0141] In order to adaptively incorporate multi-scale semantic features into image decoding features, the present invention adopts a spatial adaptive normalization method to reduce the conflict between different branches. W 4 As input, the image features are transformed pixel by pixel. The process can be written as:

[0142] (17)

[0143] In the formula, and are two variables output by the two convolutional layers. Finally, a skip connection is used to prevent semantic damage caused by the patch shift operation, and the output result is Embedded into the module architecture to improve computational efficiency.

[0144] (18)

[0145] In the formula, In this way, MCFA can reduce information loss while expanding the range of feature fusion, and adaptively refine the mural semantic encoding features, thereby improving the accuracy of mural restoration through context perception.

[0146] (0) Setting the loss function

[0147] In view of the characteristics of Dunhuang mural restoration, the present invention sets a series of loss functions to ensure that the restored murals are consistent with the distribution, structure and style of the ground truth murals.

[0148] Diffusion loss function: To ensure the restoration of murals and real murals The diffusion loss function set by the present invention is as follows:

[0149] (19)

[0150] Where ADM is a parameterized network, where represents the absolute value, To add noise to the mural, t is the time step, For line drawing, To damage the murals and M is a binary mask.

[0151] Perceptual loss and style loss function: Perceptual loss and style loss The features of VGG16 are used to learn high-level structures and overall style information, ensuring the style consistency between the reconstructed murals and the ground truth murals in a high-dimensional feature space.

[0152] The perceptual loss is calculated as follows:

[0153] (20)

[0154] The style loss is calculated as follows:

[0155] (twenty one)

[0156] In the formula, It is i A VGG16 layer network.G represents the Gram matrix on the feature map, .

[0157] Hole loss and effective loss function: Through hole loss calculate and The L1 distance of the mask area in the middle. By effective loss calculate and In this way, the L1 distance of the background area is maintained. and Content consistency between:

[0158] (twenty two)

[0159] (twenty three)

[0160] Edge loss function: In order to make the reconstructed mural have overall coherent lines, the present invention proposes an edge loss function To maintain the consistency of the overall structure:

[0161] (twenty four)

[0162] In the formula, represents the Sobel operator.

[0163] Adversarial loss function: This function is used to match the design of adversarial diffusion model to directly predict the repair results. , designed to fill damaged areas of a mural with realistic content:

[0164] (25)

[0165] In the formula D is the discriminator, G For the generator, is a logarithmic function.

[0166] KL divergence loss function: The present invention also optimizes the KL divergence to constrain the distance between the learning posterior distribution and the prior distribution:

[0167] (26)

[0168] In the formula, , and Responsible for projecting multi-semantic features into the normal distribution space, E 1. E 2 and E3 are the encoding layers of the three VTL modules. Z1, Z2 and Z3 are the eigenvectors in the normal distribution space.

[0169] Finally, the above losses are combined to obtain the training loss of the prediction network:

[0170] (27)

[0171] In the formula, and is the loss function hyperparameter.

[0172] Comparison of experimental data and evaluation results

[0173] The method of the present invention is tested and verified on the Dunhuang murals dataset. According to the proportion of the mask area, the test mask is divided into 6 types, namely 1-10%, 10-20%, 20-30%, 30-40%, 40-50% and 50-60%. The effectiveness of the method of the present invention is verified by comparing with five other representative algorithms. The five comparison algorithms are FcF-Inpainting, StrDiffusion, RePaint, MAPConv and Spa-former. The image quality evaluation index adopts several common indicators in image restoration: L1 error, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), Frechette distance (FID) and learning perceptual image block similarity (LPIPS). The comparative experimental results on the Dunhuang murals dataset are shown in Table 1.

[0174] Table 1: Comparison of experimental results of Dunhuang mural dataset

[0175]

[0176] It can be seen from the experimental data in Table 1 that in different proportions of image missing areas, the method of the present invention is superior to other methods in the data results of the five evaluation indicators of L1, PSNR, SSIM, FID and LPIPS, indicating that the repaired image obtained by the method of the present invention is closer to the original image, and can not only restore the damaged area of ​​the mural, but also maintain the semantic consistency between the restored content and the visual content. The comparative experimental results show that the method of the present invention is effective.

[0177] Figure 4A comparison chart of the visual effects of five image restoration methods is given. The experimental gallery is a Dunhuang mural dataset. Among them, the first column of images are real images; the second column of images are defective images; the third column of images are restoration images obtained by the FcF-Inpainting algorithm; the fourth column of images are restoration images obtained by the StrDiffusion algorithm; the fifth column of images are restoration images obtained by the RePaint algorithm; the sixth column of images are restoration images obtained by the MAPConv algorithm; the seventh column of images are restoration images obtained by the Spa-former algorithm; and the eighth column of images are restoration images obtained by the method of the present invention.

[0178] In a comparative experiment based on the Dunhuang mural dataset, the FcF-Inpainting method can restore the overall style of the murals, but the generated image details are blurred. This method can infer the reasonable color and structure of the damaged area, but it cannot generate textures with clear details. RePaint, MAPConv and Spa-former infer plausible backgrounds from surrounding information, and their repaired images usually contain blurred textures and distorted structures. StrDiffusion can generate reasonable structures and details for the missing areas, but it cannot restore the content of the original murals. Compared with the above methods, the method of the present invention can restore content that is globally semantically consistent and close to the structure and style of the original murals.

[0179] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0180] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A digital restoration method for Dunhuang murals based on an adversarial diffusion model and a feature fusion module is used for restoration of Dunhuang murals, and is characterized by: The following steps are involved: Step 1: Use the adversarial diffusion model to enforce semantic consistency between masked and unmasked regions The adversarial diffusion model is proposed to solve the semantic error problem between the masked and non-masked areas of the mural. In the forward diffusion process, a real mural x is randomly selected. g , each step forward process is as follows: Where random noise ∈~N(0,I), β t ∈(0,1), and In the reverse denoising process, it is no longer predicted arrive The noise accumulated in the middle is not through a defective mural x m , a binary mask M and a line drawing x d As a conditional input, directly predict the restored mural x out : Where ADM is a prediction network, which consists of a prediction backbone network, two conditional encoding networks and a discriminator. In the inference stage, Gaussian noise X is randomly obtained. T , where T is the number of sampling steps. Then, under the conditions of a defective mural, a line drawing and a binary mask, the diffusion process is reversed to obtain the final restored mural; Step 2: Add a variational transformer layer to enhance the semantic feature space interaction The variational transformer layer consists of a transformer encoder and a CNN decoder, which is used for the spatial interaction of different semantic features and fuses the line drawing and damaged mural features in the latent space to ensure the structural and semantic consistency of the denoised image; First, from the given damaged mural Binary Mask Noise mural and line drawing murals The multi-scale visual information is learned from the encoder of , where h and w represent the number of pixels in the vertical and horizontal dimensions of the image, respectively: In the formula, are different semantic features with the same spatial size, Enc g and Enc d It is an encoder composed of two convolutional stages, Enc m It consists of gated convolution and extracts effective features from different damaged murals, which can update the mask and renormalize the convolution feature map to better extract effective features from different damaged murals; Specifically given different semantic features VTL embeds each feature through the encoder, and the generator Dec a Use parameterization techniques to reconstruct multimodal features that may have lost content: Where μ and σ are the encoder Enc a The estimated mean and deviation, Z is a randomly generated vector under the standard normal distribution, is a feature reconstructed based on the decoder's potential distribution. and After that, continue to apply formula (12) to obtain the final reconstructed features Step 3: Use multi-scale feature aggregation module to ensure global context consistency The multi-scale feature aggregation module is used to capture the relevance of the mural context, focusing on generating reasonable mural content through multi-scale feature splicing. The multi-scale feature aggregation module can encode rich semantic features at multiple scales, automatically learn and understand the structural texture and color features of the visible area, and use them to repair and fill the damaged area. First, given different semantic features obtained from VTL and Concatenate them as input F: In the formula, Concat(·) represents the feature concatenation operation, and then three sets of dilated convolutional layers with different dilation rates are used to capture multi-scale semantic features: Where Dconv k (·) denotes a dilated convolutional layer with dilation rate j∈{1, 2, 3}; There is a problem of effective information loss for sparse dilated convolution kernels with a pixel-level weight map. By predicting the weights of different dilated convolutions, the convolution features from different kernels are effectively aggregated: W n =Slice(Softmax(Conv(F))),n∈(1,2,3,4) (15) In the formula, Conv(·) is a convolution operation, Softmax(·) is a channel-wise Softmax, and Slice(·) is a channel-wise slice. Multi-scale semantic features are aggregated and weighted element by element to produce a refined feature map F. o : In order to adaptively incorporate multi-scale semantic features into image decoding features, a spatial adaptive normalization method is used to reduce the conflict between different branches. 4 As input, the image features are transformed pixel by pixel, and the process can be written as: [γ,β]=Conv(W 4 ) (17) In the formula, γ and β are two variables output by the two convolutional layers. Finally, a skip connection is used to prevent the semantic damage caused by the patch shift operation, and the output result F out Embedded into modular architecture to improve computational efficiency; Where IN(·) represents non-parametric instance normalization. In this way, MCFA can reduce information loss while expanding the range of feature fusion, and adaptively refine the semantic encoding features of murals, thereby improving the accuracy of mural restoration through context perception.

2. The method for digital restoration of Dunhuang murals based on the counter-diffusion model and feature fusion module as claimed in claim 1, characterized in that: The following loss functions are also included to make the restored murals consistent with the ground truth mural distribution, structure and style; Diffusion loss function: To ensure the restoration of the mural x out and real murals x g The distribution consistency of , the diffusion loss function is as follows: Where ADM is a parameterized network, where ||·|| represents the absolute value, is the noise mural, t is the time step, x d For line drawing, x m is the damaged mural and M is the binary mask; Perceptual loss and style loss function: Perceptual loss and style loss The features of VGG16 are used to learn high-level structures and overall style information, ensuring the style consistency between the reconstructed murals and the ground truth murals in a high-dimensional feature space; The perceptual loss is calculated as follows: The style loss is calculated as follows: In the formula, is the i-th VGG16 layer network, G represents the Gram matrix on the feature map, Hole loss and effective loss function: Through hole loss Calculate x out and x g L1 distance in the mask area; through the effective loss Calculate x out and x g The L1 distance of the background area in the middle; in this way, x out and x g Content consistency between: Edge loss function: In order to make the reconstructed mural have overall coherent lines, an edge loss function is proposed To maintain the consistency of the overall structure: Where Φ(·) represents the Sobel operator; Adversarial loss function: This function is used to match the design of adversarial diffusion model to directly predict the repair results. Designed to fill damaged areas of a mural with realistic content: Where D is the discriminator, G is the generator, and log(·) is the logarithmic function; KL divergence loss function: Optimize the KL divergence to constrain the distance between the learning posterior distribution and the prior distribution: In the formula, and Responsible for projecting multi-semantic features into the normal distribution space. E1, E2 and E3 are the encoding layers of the three VTL modules respectively; Z1, Z2 and Z3 are the feature vectors in the normal distribution space respectively; Finally, the above losses are combined to obtain the training loss of the prediction network: In the formula, λ d , per , style , rec , edge , adv and λ kl is the loss function hyperparameter.

Images