Dunhuang mural digital repairing method based on adversarial diffusion model and feature fusion module

Abstract

The invention discloses a Dunhuang mural digital restoration method based on an adversarial diffusion model and a feature fusion module, and relates to the technical field of image restoration. According to the method, after a mural image de-noising preprocessing task is completed, a self-designed adversarial diffusion model is used, a prediction result is optimized at different time steps, and the image de-noising performance is improved. The semantic consistency between the shielded and unshielded areas is enhanced, and a variational transform layer (VTL) is added to ensure that the de-noised image and the prediction result are kept consistent in structural semantics; and then a self-designed multi-scale feature aggregation module (MCFA) is utilized to encode rich semantic features on multiple scales, the correlation between mural contexts is enhanced, and the mural fitting the original content and style is reconstructed.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to a digital restoration method for Dunhuang murals based on an adversarial diffusion model and a feature fusion module. Background Art

[0002] Dunhuang murals are famous for their rich content, bright colors and unique craftsmanship. They record Buddhist stories, social life and cultural customs, and have important scientific, historical and artistic value. However, wind, sun and human activities have caused the murals to suffer various damages, such as fading, scratches, breaks and mildew. The manual restoration of murals not only requires professional painting skills, but may also lead to irreversible restoration results or even secondary damage. Non-contact digital restoration technology has become a better choice for mural restoration.

[0003] Traditional methods usually use geometric or patch methods to use the semantic information of the unmasked area to restore the masked part of the image. Reconstruct the damaged area. However, these methods have difficulty capturing high-level semantic context and are not effective for images containing non-repeating structures or complex backgrounds. In recent years, image restoration technology based on convolutional neural networks (CNN) and generative adversarial networks (GAN) has made great progress. Most modern image restoration uses a generative adversarial model, combined with a CNN codec and a discriminator, to restore the defective area by understanding the image context, and use the discriminator to enhance texture details and generate visually realistic content.

[0004] However, when processing masked areas, convolutional neural networks often only focus on the local semantics of the surrounding area, ignoring the bridging and utilization of information on a global scale. In addition, the unpredictability of the masked area of ​​the input image causes the convolution to be unable to accurately capture effective information. This uncertainty may lead to information loss during image restoration and even cause distortion in unmasked image areas.

[0005] As an emerging generative model, the Denoising Diffusion Probabilistic Model (DDPM) performs well in image denoising, but there are still two problems when used for mural restoration: First, it ignores the semantic consistency between masked and non-masked areas, which may cause the background after restoration to be distorted. Second, although it can reconstruct the defective area, it is difficult to ensure that the restored mural fits the original content and style.

[0006] To solve the above problems, the present invention proposes a Dunhuang mural restoration method based on an adversarial diffusion model and a multi-scale feature aggregation module. After completing the mural image denoising preprocessing task, the method uses a self-designed adversarial diffusion model to optimize the predicted results at different time steps, enhance the semantic consistency between masked and unmasked areas, and add a variational transformer layer (VTL) to ensure that the denoised image and the predicted results are consistent in structural semantics. Then, the self-designed multi-scale feature aggregation module (MCFA) is used to encode rich semantic features at multiple scales, enhance the correlation between the mural contexts, and reconstruct murals that fit the original content and style. Summary of the invention

[0007] The purpose of the present invention is to solve the above-mentioned problems and to provide a digital restoration method for Dunhuang murals based on an anti-diffusion model and a feature fusion module.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] According to one aspect of the present invention, a method for digital restoration of Dunhuang murals based on an anti-diffusion model and a feature fusion module is provided, which is used for restoration of Dunhuang murals and includes the following steps:

[0010] Step 1: Image denoising preprocessing through denoising diffusion probability model

[0011] In the forward diffusion phase, T time steps, a vector with a certain variance β is added to the input image x0. t ∈(0,1) Gaussian noise:

[0012]

[0013] Where q(·) is the conditional probability density function, I is the identity matrix, N represents the normal distribution, and x t-1 and x t Represent the state of the previous moment and the current moment respectively. As the time step t increases, the input image x0 gradually becomes an independent Gaussian white noise x T Since the noise diffusion follows a Markov process, the noise addition result x at any time step t is t It can be defined as:

[0014]

[0015] In the formula, α t =1-β t , and ∈~N(0,I), ∈ is a random variable of standard normal distribution, which represents the noise added in the diffusion process. By calculating the probability distribution q(x t-1 |xt ) reconstructs the image from the noise and uses the Markov chain and Bayesian formula to train the neural network to predict the posterior conditional distribution p θ (x t-1 |x0,x t ):

[0016] p θ (x t-1 |x t ,x0)=N(x t-1 ;μ θ (x t , t), ∑ θ (x t ,t)) (4)

[0017] Where p θ (·) is a parameterized conditional probability density function with mean μ θ (x t , t) and variance ∑ θ (x t , t) is calculated as follows:

[0018]

[0019] Then we can get the new mean by inversely solving for x0:

[0020]

[0021] Finally, train the parameterized model ∈ θ To predict from x0 to x t The noise accumulation ∈ t , the training objective is:

[0022]

[0023] In the formula, represents the expected operation. In the test phase, from the random noise x T To begin, apply model ∈ θ , from x t Iterate to x0;

[0024] Step 2: Use the adversarial diffusion model to enforce semantic consistency between masked and unmasked regions

[0025] After completing the image denoising task, the adversarial diffusion model is used to solve the semantic error problem in the masked and non-masked areas of the mural. In the forward diffusion process, a real mural x is randomly selected. g , according to formula (3), each step of the forward process is as follows:

[0026]

[0027] Where random noise ∈~N(0,I), β t ∈(0,1),α t =1-β t and

[0028] In the reverse denoising process, it is no longer predicted arrive The noise accumulated in the middle is not through a defective mural x m , a binary mask M and a line drawing x d As a conditional input, directly predict the restored mural x out , that is, x0 in formula (5) instead of predicting the noise in formula (7):

[0029]

[0030] Where ADM is a prediction network, which consists of a prediction backbone network, two conditional encoding networks and a discriminator. In the inference stage, Gaussian noise X is randomly obtained. T , where T is the number of sampling steps. Then, under the conditions of a defective mural, a line drawing and a binary mask, the diffusion process is reversed to obtain the final restored mural.

[0031] Step 3: Add a variational transformer layer to enhance the interaction of semantic feature space

[0032] The variational transformer layer consists of a transformer encoder and a CNN decoder, which is used for the spatial interaction of different semantic features and fuses the line drawing and damaged mural features in the latent space to ensure the structural and semantic consistency of the denoised image;

[0033] First, from the given damaged mural Binary Mask Noise mural and line drawing murals The multi-scale visual information is learned from the encoder of , where h and w represent the number of pixels in the vertical and horizontal dimensions of the image, respectively:

[0034]

[0035] In the formula, are different semantic features with the same spatial size, Enc g and Enc d It is an encoder composed of two convolutional stages, Enc mIt consists of gated convolution and extracts effective features from different damaged murals, which can update the mask and renormalize the convolution feature map to better extract effective features from different damaged murals;

[0036] Specifically given different semantic features VTL embeds each feature through the encoder, and the generator Dec a Use parameterization techniques to reconstruct multimodal features that may have lost content:

[0037]

[0038] Where μ and σ are the encoder Enc a The estimated mean and deviation, Z is a randomly generated vector under the standard normal distribution, is a feature reconstructed based on the decoder's potential distribution. and After that, continue to apply formula (12) to obtain the final reconstructed features

[0039] Step 4: Use multi-scale feature aggregation module to ensure global context consistency

[0040] The multi-scale feature aggregation module is used to capture the relevance of the mural context, focusing on generating reasonable mural content through multi-scale feature splicing. The multi-scale feature aggregation module can encode rich semantic features at multiple scales, automatically learn and understand the structural texture and color features of the visible area, and use them to repair and fill the damaged area.

[0041] First, given different semantic features obtained from VTL and Concatenate them as input F:

[0042]

[0043] In the formula, Concat(·) represents the feature concatenation operation, and then three sets of dilated convolutional layers with different dilation rates are used to capture multi-scale semantic features:

[0044]

[0045] Where Dconv k (·) denotes a dilated convolutional layer with dilation rate j∈{1, 2, 3};

[0046] There is a problem of effective information loss for sparse dilated convolution kernels with a pixel-level weight map. By predicting the weights of different dilated convolutions, the convolution features from different kernels are effectively aggregated:

[0047] W n=Slice(Softmax(Conv(F))),n∈(1,2,3,4) (15)

[0048] In the formula, Conv(·) is a convolution operation, Softmax(·) is a channel-wise Softmax, and Slice(·) is a channel-wise slice. Multi-scale semantic features are aggregated and weighted element by element to produce a refined feature map F. o :

[0049]

[0050] In order to adaptively incorporate multi-scale semantic features into image decoding features, a spatial adaptive normalization method is used to reduce the conflict between different branches. 4 As input, the image features are transformed pixel by pixel, and the process can be written as:

[0051] [γ,β]=Conv(W 4 ) (17)

[0052] In the formula, γ and β are two variables output by the two convolutional layers. Finally, a skip connection is used to prevent the semantic damage caused by the patch shift operation, and the output result F out Embedded into modular architecture to improve computational efficiency;

[0053]

[0054] Where IN(·) represents non-parametric instance normalization. In this way, MCFA can reduce information loss while expanding the range of feature fusion, and adaptively refine the semantic encoding features of murals, thereby improving the accuracy of mural restoration through context perception.

[0055] Furthermore, the following loss function is included to make the restored murals consistent with the ground truth mural distribution, structure and style;

[0056] Diffusion loss function: To ensure that the mural x is restored out and real murals x g The distribution consistency of the diffusion loss function set by the present invention is as follows:

[0057]

[0058] Where ADM is a parameterized network, where ||·|| represents the absolute value, is the noise mural, t is the time step, x d For line drawing, x m is the damage mural and M is the binary mask.

[0059] Perceptual loss and style loss function: Perceptual loss and style loss The features of VGG16 are used to learn high-level structures and overall style information, ensuring the style consistency between the reconstructed murals and the ground truth murals in a high-dimensional feature space.

[0060] The perceptual loss is calculated as follows:

[0061]

[0062] The style loss is calculated as follows:

[0063]

[0064] In the formula, is the i-th VGG16 layer network. G represents the Gram matrix on the feature map,

[0065] Hole loss and effective loss function: Through hole loss Calculate x out and x g The L1 distance of the mask area in the middle. By effective loss Calculate x out and x g In this way, x is kept out and x g Content consistency between:

[0066]

[0067] Edge loss function: In order to make the reconstructed mural have overall coherent lines, the present invention proposes an edge loss function To maintain the consistency of the overall structure:

[0068]

[0069] Where Φ(·) represents the Sobel operator.

[0070] Adversarial loss function: This function is used to match the design of adversarial diffusion model to directly predict the repair results. Designed to fill damaged areas of a mural with realistic content:

[0071]

[0072] Where D is the discriminator, G is the generator, and log(·) is the logarithmic function.

[0073] KL divergence loss function: The present invention also optimizes the KL divergence to constrain the distance between the learning posterior distribution and the prior distribution:

[0074]

[0075] In the formula, and Responsible for projecting multi-semantic features into the normal distribution space, E1, E2 and E3 are the encoding layers of the three VTL modules respectively. Z1, Z2 and Z3 are the feature vectors in the normal distribution space respectively.

[0076] Finally, the above losses are combined to obtain the training loss of the prediction network:

[0077]

[0078] In the formula, λ d , per , style , rec , edge , adv and λ kl is the loss function hyperparameter.

[0079] Compared with the prior art, the present invention has the following beneficial effects:

[0080] The adversarial diffusion model created by the present invention is different from other diffusion-based restoration models. It can directly predict the restoration results of murals, optimize the predicted mural content at different time steps, and use line drawings and a variety of different loss functions to constrain the restoration process to approximate the original structure and style of the murals. At the same time, a variational transformer layer (VTL) is added to fuse the line drawings and defective mural features in the latent space to generate content close to the original murals.

[0081] The multi-scale feature aggregation (MCFA) module proposed in the present invention can further capture spatial multi-scale semantic features using the dilated convolutional layer on the basis of the adversarial diffusion model, prevent semantic damage caused by patch shift operations through jump connections of the convolutional layer output variables, ensure the overall quality of the restored murals through global context interaction, and make the structural style of the reconstructed murals consistent with the ground truth murals by reasonably configuring the loss function.

[0082] The present invention is based on the forefront of digital restoration technology for Dunhuang murals. The adversarial diffusion model and multi-scale feature aggregation module in the technical system play a key role. It is organically combined with the denoising diffusion probability model, variational transformer layer and loss function to jointly solve the connotation problem of Dunhuang mural restoration, thus forming a set of technical solutions for Dunhuang mural reconstruction. The comparative experimental results show that the method of the present invention achieves better mural restoration effects than the currently popular methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is the technical route and flow chart of the present invention;

[0084] Figure 2 Schematic diagram of the variational transformer layer;

[0085] Figure 3 This is a schematic diagram of the multi-scale feature aggregation module framework;

[0086] Figure 4 A comparison of the visual effects of five image restoration methods. DETAILED DESCRIPTION

[0087] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0088] The present invention provides a digital restoration method for Dunhuang murals based on an anti-diffusion model and a feature fusion module. The implementation steps and key algorithms are as follows:

[0089] Step 1: Image denoising preprocessing through denoising diffusion probability model

[0090] Aiming at the specific task of Dunhuang mural restoration, the present invention improves the denoising diffusion probability model and adds a specific variance β to the input image x0 within T time steps in the forward diffusion stage. t ∈(0,1) Gaussian noise:

[0091]

[0092] Where q(·) is the conditional probability density function, I is the identity matrix, N represents the normal distribution, and x t-1 and x t Represent the state of the previous moment and the current moment respectively. As the time step t increases, the input image x0 gradually becomes an independent Gaussian white noise x T Since the noise diffusion follows a Markov process, the noise addition result x at any time step t is t It can be defined as:

[0093]

[0094] In the formula, α t =1-β t , and ∈~N(0,I). ∈ is a random variable of standard normal distribution, which represents the noise added in the diffusion process. The reverse denoising process aims to reverse the forward diffusion and calculate the probability distribution q(x t-1 |x t ) reconstructs the image from the noise. Since it is impossible to obtain the true distribution of the initial image x0, directly fitting q(xt-1 |x t ) is not feasible. Therefore, the present invention uses Markov chain and Bayesian formula to train neural network to predict the posterior conditional distribution p θ (x t-1 |x0,x t ):

[0095] p θ (x t-1 |x t ,x0)=N(x t-1 ;μ θ (x t , t), Σ θ (x t ,t)) (4)

[0096] Where p θ (·) is a parameterized conditional probability density function with mean μ θ (x t , t) and variance Σ θ (x t , t) is calculated as follows:

[0097]

[0098] Then we can get the new mean by inversely solving for x0:

[0099]

[0100] Finally, train the parameterized model ∈ θ To predict from x0 to x t The noise accumulation ∈ t , the training objective is:

[0101]

[0102] In the formula, represents the expected operation. In the test phase, from the random noise x T To begin, apply model ∈ θ , from x t Iterate to x0.

[0103] Step 2: Use the adversarial diffusion model to enforce semantic consistency between masked and unmasked regions

[0104] After completing the image denoising task, the present invention uses the adversarial diffusion model to solve the semantic error problem between the masked and unmasked areas of the mural. The model directly predicts the complete content of the mural and optimizes the predicted results at different time steps through the discriminator to ensure that the filling content of the missing area of ​​the mural is not only reasonable but also consistent with the semantics of the visible area. In the forward diffusion process, a real mural x is randomly selected.g , according to formula (3), each step of the forward process is as follows:

[0105]

[0106] Where random noise ∈~N(0,I), β t ∈(0,1),α t =1-β t , and

[0107] In the reverse denoising process, it is no longer predicted arrive The noise accumulated in the middle is not through a defective mural x m , a binary mask M and a line drawing x d As a conditional input, directly predict the restored mural x out , that is, x0 in formula (5) instead of predicting the noise in formula (7):

[0108]

[0109] In the formula, ADM is a prediction network, which consists of a prediction backbone network, two conditional encoding networks and a discriminator. It generates content that is consistent with the context semantics and fits the original mural structure style. In the inference stage, randomly obtain Gaussian noise X T , where T is the number of sampling steps. Then, under the conditions of a defective mural, a line drawing and a binary mask, the diffusion process is reversed to obtain the final restored mural.

[0110] Step 3: Add a variational transformer layer to enhance the interaction of semantic feature space The variational transformer layer (VTL) consists of a transformer encoder and a CNN decoder, such as Figure 2 It is designed for spatial interaction of different semantic features and can fuse line drawing and damaged mural features in latent space to ensure the structural and semantic consistency of the denoised image. In particular, the transformer-based encoder can explicitly simulate the global long-range interaction of different semantic features and enhance the overall effect of mural restoration.

[0112] First, from the given damaged mural Binary Mask Noise mural and line drawing murals The multi-scale visual information is learned in the encoder of . Where h and w represent the number of pixels in the vertical and horizontal dimensions of the image, respectively:

[0113]

[0114] In the formula, are different semantic features with the same spatial size. g and Enc d It is an encoder composed of two convolutional stages, Enc m It consists of gated convolutions and extracts effective features from different damaged murals.

[0115] features, which can update the mask and renormalize the convolutional feature map to better extract effective features from different damaged murals.

[0116] Specifically given different semantic features VTL embeds each feature through the encoder, and the generator Dec a Use parameterization techniques to reconstruct multimodal features that may have lost content:

[0117]

[0118] Where μ and σ are the encoder Enc a The estimated mean and deviation, Z is a randomly generated vector under the standard normal distribution, is a feature reconstructed based on the decoder’s latent distribution. and After that, continue to apply formula (12) to obtain the final reconstructed features In this way, the learned prior semantic information can be effectively integrated into a unified representation to achieve semantic complementarity of the damaged murals.

[0119] Step 4: Use multi-scale feature aggregation module to ensure global context consistency

[0120] The multi-scale feature aggregation module (MCFA) is designed to capture the relevance of the mural context, focusing on generating reasonable mural content through multi-scale feature splicing. The module framework is as follows Figure 3 As shown in Figure 2, this module can encode rich semantic features at multiple scales, automatically learn to understand the structural, texture and color features of the visible area, and use them to repair and fill in damaged areas. The reconstructed mural is both contextually consistent and fits the structure and style of the original mural.

[0121] First, given different semantic features obtained from VTL and Concatenate them as input F:

[0122]

[0123] In the formula, Concat(·) represents the feature concatenation operation. Then three groups of dilated convolutional layers with different dilation rates are used to capture multi-scale semantic features:

[0124]

[0125] Where Dconv k (·) denotes a dilated convolutional layer with dilation rate j∈{1, 2, 3}.

[0126] To address the problem that sparse dilated convolution kernels may lead to loss of effective information, this paper designs a pixel-level weight map, which effectively aggregates convolution features from different kernels by predicting the weights of different dilated convolutions:

[0127] W n =Slice(Softmax(Conv(F))),n∈(1,2,3,4) (15)

[0128] In the formula, Conv(·) is a convolution operation, Softmax(·) is a channel-wise Softmax, and Slice(·) is a channel-wise slice. Multi-scale semantic features are aggregated and weighted element by element to produce a refined feature map F o :

[0129]

[0130] In order to adaptively incorporate multi-scale semantic features into image decoding features, the present invention adopts a spatial adaptive normalization method to reduce the conflict between different branches. 4 As input, the image features are transformed pixel by pixel. The process can be written as:

[0131] [γ,β]=Conv(W 4 ) (17)

[0132] In the formula, γ and β are two variables output by the two convolutional layers. Finally, a skip connection is used to prevent the semantic damage caused by the patch shift operation, and the output result F out Embedded into the module architecture to improve computational efficiency.

[0133]

[0134] In the formula, IN(·) represents non-parametric instance normalization. In this way, MCFA can reduce information loss while expanding the range of feature fusion, and adaptively refine the semantic encoding features of murals, thereby improving the accuracy of mural restoration through context perception.

[0135] (3) Setting of loss function

[0136] In view of the characteristics of Dunhuang mural restoration, the present invention sets a series of loss functions to ensure that the restored murals are consistent with the distribution, structure and style of the ground truth murals.

[0137] Diffusion loss function: To ensure that the mural x is restored out and real murals x g The distribution consistency of the diffusion loss function set by the present invention is as follows:

[0138]

[0139] Where ADM is a parameterized network, where ||·|| represents the absolute value, is the noise mural, t is the time step, x d For line drawing, x m is the damage mural and M is the binary mask.

[0140] Perceptual loss and style loss function: Perceptual loss and style loss The features of VGG16 are used to learn high-level structures and overall style information, ensuring the style consistency between the reconstructed murals and the ground truth murals in a high-dimensional feature space.

[0141] The perceptual loss is calculated as follows:

[0142]

[0143] The style loss is calculated as follows:

[0144]

[0145] In the formula, is the i-th VGG16 layer network. G represents the Gram matrix on the feature map,

[0146] Hole loss and effective loss function: Through hole loss Calculate x out and x g The L1 distance of the mask area in the middle. By effective loss Calculate x out and x g In this way, x is kept out and x g Content consistency between:

[0147]

[0148] Edge loss function: In order to make the reconstructed mural have overall coherent lines, the present invention proposes an edge loss function To maintain the consistency of the overall structure:

[0149]

[0150] Where Φ(·) represents the Sobel operator.

[0151] Adversarial loss function: This function is used to match the design of adversarial diffusion model to directly predict the repair results. Designed to fill damaged areas of a mural with realistic content:

[0152]

[0153] Where D is the discriminator, G is the generator, and log(·) is the logarithmic function.

[0154] KL divergence loss function: The present invention also optimizes the KL divergence to constrain the distance between the learning posterior distribution and the prior distribution:

[0155]

[0156] In the formula, and Responsible for projecting multi-semantic features into the normal distribution space, E1, E2 and E3 are the encoding layers of the three VTL modules respectively. Z1, Z2 and Z3 are the feature vectors in the normal distribution space respectively.

[0157] Finally, the above losses are combined to obtain the training loss of the prediction network:

[0158]

[0159] In the formula, λ d , per , style , rec , edge , adv and λ kl is the loss function hyperparameter.

[0160] Comparison of experimental data and evaluation results

[0161] The method of the present invention is tested and verified on the Dunhuang murals dataset. According to the proportion of the mask area, the test masks are divided into 6 types, namely 1-10%, 10-20%, 20-30%, 30-40%, 40-50% and 50-60%. The effectiveness of the method of the present invention is verified by comparing with five other representative algorithms. The five comparison algorithms are FcF-Inpainting, StrDiffusion, RePaint, MAPConv and Spa-former. The image quality evaluation index adopts several common indicators in image restoration: L1 error, peak signal-to-noise ratio (PSNR), structural similarity (SSIM), Frechette distance (FID) and learning perceptual image block similarity (LPIPS). The comparative experimental results on the Dunhuang murals dataset are shown in Table 1.

[0162] Table 1: Comparison of experimental results of Dunhuang mural dataset

[0163]

[0164]

[0165] It can be seen from the experimental data in Table 1 that in different proportions of image missing areas, the method of the present invention is superior to other methods in the data results of the five evaluation indicators of L1, PSNR, SSIM, FID and LPIPS, indicating that the repaired image obtained by the method of the present invention is closer to the original image, and can not only restore the damaged area of ​​the mural, but also maintain the semantic consistency between the restored content and the visual content. The comparative experimental results show that the method of the present invention is effective.

[0166] Figure 4 A comparison chart of the visual effects of five image restoration methods is given. The experimental gallery is a Dunhuang mural dataset. Among them, the first column of images are real images; the second column of images are defective images; the third column of images are restoration images obtained by the FcF-Inpainting algorithm; the fourth column of images are restoration images obtained by the StrDiffusion algorithm; the fifth column of images are restoration images obtained by the RePaint algorithm; the sixth column of images are restoration images obtained by the MAPConv algorithm; the seventh column of images are restoration images obtained by the Spa-former algorithm; and the eighth column of images are restoration images obtained by the method of the present invention.

[0167] In a comparative experiment based on the Dunhuang mural dataset, the FcF-Inpainting method can restore the overall style of the murals, but the generated image details are blurred. This method can infer the reasonable color and structure of the damaged area, but cannot generate textures with clear details. RePaint, MAPConv and Spa-former infer plausible backgrounds from surrounding information, and their repaired images usually contain blurred textures and distorted structures. StrDiffusion can generate reasonable structures and details for the missing areas, but cannot restore the content of the original murals. Compared with the above methods, the method of the present invention can restore content that is globally semantically consistent and close to the structure and style of the original murals.

[0168] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0169] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A digital restoration method for Dunhuang murals based on an adversarial diffusion model and a feature fusion module is used for restoration of Dunhuang murals, and is characterized by: The following steps are involved: Step 1: Use the adversarial diffusion model to enforce semantic consistency between masked and unmasked regions The adversarial diffusion model is proposed to solve the semantic error problem between the masked and non-masked areas of the mural. In the forward diffusion process, a real mural x is randomly selected. g , each step forward process is as follows: Where random noise ∈~N(0,I), β t ∈(0,1), and In the reverse denoising process, it is no longer predicted arrive The noise accumulated in the middle is not through a defective mural x m , a binary mask M and a line drawing x d As a conditional input, directly predict the restored mural x out : Where ADM is a prediction network, which consists of a prediction backbone network, two conditional encoding networks and a discriminator. In the inference stage, Gaussian noise X is randomly obtained. T , where T is the number of sampling steps. Then, under the conditions of a defective mural, a line drawing and a binary mask, the diffusion process is reversed to obtain the final restored mural; Step 2: Add a variational transformer layer to enhance the semantic feature space interaction The variational transformer layer consists of a transformer encoder and a CNN decoder, which is used for the spatial interaction of different semantic features and fuses the line drawing and damaged mural features in the latent space to ensure the structural and semantic consistency of the denoised image; First, from the given damaged mural Binary Mask Noise mural and line drawing murals The multi-scale visual information is learned from the encoder of , where h and w represent the number of pixels in the vertical and horizontal dimensions of the image, respectively: In the formula, are different semantic features with the same spatial size, Enc g and Enc d It is an encoder composed of two convolutional stages, Enc m It consists of gated convolution and extracts effective features from different damaged murals, which can update the mask and renormalize the convolution feature map to better extract effective features from different damaged murals; Specifically given different semantic features VTL embeds each feature through the encoder, and the generator Dec a Use parameterization techniques to reconstruct multimodal features that may have lost content: Where μ and σ are the encoder Enc a The estimated mean and deviation, Z is a randomly generated vector under the standard normal distribution, is a feature reconstructed based on the decoder's potential distribution. and After that, continue to apply formula (12) to obtain the final reconstructed features Step 3: Use multi-scale feature aggregation module to ensure global context consistency The multi-scale feature aggregation module is used to capture the relevance of the mural context, focusing on generating reasonable mural content through multi-scale feature splicing. The multi-scale feature aggregation module can encode rich semantic features at multiple scales, automatically learn and understand the structural texture and color features of the visible area, and use them to repair and fill the damaged area. First, given different semantic features obtained from VTL and Concatenate them as input F: In the formula, Concat(·) represents the feature concatenation operation, and then three sets of dilated convolutional layers with different dilation rates are used to capture multi-scale semantic features: Where Dconv k (·) denotes a dilated convolutional layer with dilation rate j∈{1, 2, 3}; There is a problem of effective information loss for sparse dilated convolution kernels with a pixel-level weight map. By predicting the weights of different dilated convolutions, the convolution features from different kernels are effectively aggregated: W n =Slice(Softmax(Conv(F))),n∈(1,2,3,4) (15) In the formula, Conv(·) is a convolution operation, Softmax(·) is a channel-wise Softmax, and Slice(·) is a channel-wise slice. Multi-scale semantic features are aggregated and weighted element by element to produce a refined feature map F. o : In order to adaptively incorporate multi-scale semantic features into image decoding features, a spatial adaptive normalization method is used to reduce the conflict between different branches. 4 As input, the image features are transformed pixel by pixel, and the process can be written as: [γ,β]=Conv(W 4 ) (17) In the formula, γ and β are two variables output by the two convolutional layers. Finally, a skip connection is used to prevent the semantic damage caused by the patch shift operation, and the output result F out Embedded into modular architecture to improve computational efficiency; Where IN(·) represents non-parametric instance normalization. In this way, MCFA can reduce information loss while expanding the range of feature fusion, and adaptively refine the semantic encoding features of murals, thereby improving the accuracy of mural restoration through context perception.

2. The method for digital restoration of Dunhuang murals based on the counter-diffusion model and feature fusion module as claimed in claim 1, characterized in that: The following loss functions are also included to make the restored murals consistent with the ground truth mural distribution, structure and style; Diffusion loss function: To ensure that the mural x is restored out and real murals x g The diffusion loss function set by the present invention is as follows: Where ADM is a parameterized network, where ||·|| represents the absolute value, is the noise mural, t is the time step, x d For line drawing, x m is the damaged mural and M is the binary mask; Perceptual loss and style loss function: Perceptual loss and style loss The features of VGG16 are used to learn high-level structures and overall style information, ensuring the style consistency between the reconstructed murals and the ground truth murals in a high-dimensional feature space; The perceptual loss is calculated as follows: The style loss is calculated as follows: In the formula, is the i-th VGG16 layer network, G represents the Gram matrix on the feature map, Hole loss and effective loss function: Through hole loss Calculate x out and x g L1 distance in the mask area; through the effective loss Calculate x out and x g The L1 distance of the background area in the middle; in this way, x out and x g Content consistency between: Edge loss function: In order to make the reconstructed mural have overall coherent lines, an edge loss function is proposed To maintain the consistency of the overall structure: Where Φ(·) represents the Sobel operator; Adversarial loss function: This function is used to match the design of adversarial diffusion model to directly predict the repair results. Designed to fill damaged areas of a mural with realistic content: Where D is the discriminator, G is the generator, and log(·) is the logarithmic function; KL divergence loss function: Optimize the KL divergence to constrain the distance between the learning posterior distribution and the prior distribution: In the formula, and Responsible for projecting multi-semantic features into the normal distribution space. E1, E2 and E3 are the encoding layers of the three VTL modules respectively; Z1, Z2 and Z3 are the feature vectors in the normal distribution space respectively; Finally, the above losses are combined to obtain the training loss of the prediction network: In the formula, λ d , per , style , rec , edge , adv and λ kl is the loss function hyperparameter.

Images