Ancient mural restoration method based on progressive reconstruction and damage perception self-adaption

Abstract

The invention discloses an ancient fresco restoration method based on progressive reconstruction and damage perception self-adaption, and relates to the technical field of computer vision and image processing, a damaged fresco image is input into a coarse restoration network for initial restoration, in the initial restoration process, a discriminator and the coarse restoration network are adopted for adversarial training, and a damaged fresco image is obtained; generating an initial restoration result of the mural image; inputting the initial repair result and the mask image into a mask-guided local information extraction network to obtain a local optimization result; and inputting the local optimization result into a global information extraction network to obtain an overall restoration result of the mural image. The problem that an existing network is low in efficiency in the aspect of capturing local details and global styles is solved. The method has good applicability to damaged wall paintings; the problem of fuzzy texture in the repairing result is effectively solved; local features are adaptively extracted and fused according to the damage degree of the mural, and the multi-stage residual information distillation module further refines details of the mural on different scales.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing technology, and in particular to an ancient mural restoration method based on progressive reconstruction and damage perception adaptation. Background Art

[0002] As a vital cultural heritage, murals have endured thousands of years of natural erosion and human damage, and are commonly subject to damage such as fading, peeling, and cracking, severely impacting their artistic value and the integrity of their historical information. Traditional restoration methods rely on physical intervention, which carries irreversible risks. Digital restoration technology, with its non-invasive and reversible nature, has become a key solution for cultural heritage preservation.

[0003] Existing digital mural restoration methods primarily include: traditional image restoration algorithms (such as diffusion-based or image patch-based methods), which rely on local pixel or texture matching but struggle to infer missing content in complex scenes, resulting in semantic and structural inconsistencies between the restored area and the original mural; deep learning-based restoration methods, which utilize convolutional neural networks (CNNs) to extract high-level features, have made some progress in mural restoration. For example, multi-generator collaborative restoration improves restoration quality by refining local details; and structural or semantically guided restoration utilizes prior information about the mural (such as edges and colors) to optimize the restoration results.

[0004] However, existing technologies still have the following key technical bottlenecks:

[0005] First, current networks are inefficient at capturing both local details and the global style of murals. Existing methods often employ a single-stage encoder-decoder architecture, which has limited information capture and struggles to simultaneously optimize both local details and global style. While multi-stage inpainting strategies exist (such as hierarchical loss functions and coarse-to-fine progressive inpainting), the dynamic balance between local and global features remains unresolved, leading to color distortion and stylistic deviation in the inpainted areas.

[0006] Second, existing methods struggle to effectively restore fine local textures in murals, often resulting in blurred textures in the repaired area. Due to the complex details of the murals and the irregular shapes of the damaged areas, existing methods (such as partial convolution (PConv) and gated convolution (GConv)) can dynamically adjust the range of the convolution kernel, but they do not fully consider the sparsity of hole boundary information, resulting in blurred textures in the repaired area. Furthermore, validity transferable convolution (VMC) relies on background information transfer, but if the damaged area is not accurately identified, the repair efficiency will drop significantly.

[0007] Third, existing methods often lead to structural distortions in the restoration results. When repairing large areas of missing or weakly correlated areas, existing methods are prone to structural distortions. Although some technologies (such as multi-scale dilated convolution and large receptive field networks) attempt to aggregate long-range contextual information, they lack an understanding of the overall composition of the murals and cannot guarantee the topological coherence of the restored murals. This is especially true for murals with complex compositions (such as group portraits and architectural structures). Summary of the Invention

[0008] 1) Technical problems solved

[0009] The purpose of the present invention is to make up for the shortcomings of the existing technology and provide an ancient mural restoration method based on progressive reconstruction and damage perception adaptation.

[0010] 2) Technical solution

[0011] To achieve the above objectives, the present invention provides the following technical solution: a method for restoring ancient murals based on progressive reconstruction and damage-aware self-adaptation, characterized by comprising the following specific steps:

[0012] Step 1: Input the damaged mural image and the mask image into the coarse restoration network for initial restoration. During the initial restoration process, a discriminator is used to perform adversarial training with the coarse restoration network to generate an initial restoration result for the mural image.

[0013] Step 2: Inputting the initial restoration result and the mask image into a mask-guided local information extraction network to extract and optimize the local features of the mural image after the initial restoration to obtain a local optimization result;

[0014] Step 3: Input the local optimization results into the global information extraction network to capture the global artistic style of the mural image and obtain the overall restoration result of the mural image.

[0015] Furthermore, the coarse repair network includes a U-type codec structure with skip connections.

[0016] Furthermore, the adversarial training is performed using pixel-by-pixel reconstruction loss and adversarial loss;

[0017] The pixel-by-pixel reconstruction loss uses weighted L1 loss for pixel-by-pixel reconstruction, which is expressed as:

[0018]

[0019] in, represents the effective area loss of the coarse repair network; represents the damage area loss of the coarse repair network; I gt Represents the correct label image; represents the restoration result of the coarse restoration network; ⊙ represents the element-by-element product operation; M represents the mask used to distinguish the valid area and the missing area in the image; sum(M) represents the number of non-zero elements in M; sum(1-M) represents the number of zero elements in M;

[0020] The pixel-by-pixel loss Defined as:

[0021]

[0022] Among them, λ h represents the balance factor;

[0023] The adversarial loss Using least squares loss, define the corresponding coarse repair network loss function and the discriminator loss function

[0024] The discriminator loss function Defined as:

[0025]

[0026] Among them, I in represents the input image, represents the merged image; E represents expectation; represents the expectation of a real image, sampled from the real data distribution; represents the expectation of the merged image, sampled from the distribution of generated images; D represents the discriminator;

[0027] The coarse repair network loss function for:

[0028]

[0029] Among them, λ g Represents the weighting coefficient of the adversarial loss.

[0030] Furthermore, the mask-guided local information extraction network includes:

[0031] A mask-guided adaptive module, configured to dynamically adjust the attention weights of the features of the input initial restoration result according to a damage weight matrix, wherein the mask-guided adaptive module comprises a standard convolution layer, an instance normalization layer, a LeakyReLU activation function, and a mask-guided adaptive convolution layer arranged in sequence;

[0032] The multi-level residual information distillation module is used to extract and fuse local detail features of the initial restoration results at different scales through a multi-level residual connection structure.

[0033] Furthermore, the mask-guided adaptive convolution layer includes:

[0034] (1) For the input damaged feature F∈R C×H×W , a set of convolution filters are used to filter the pixels on Calculate the damage degree p of the input damage feature;

[0035]

[0036] Where σ represents the sigmoid function, which normalizes the damage degree to the interval [0,1]; R o represents the receptive field corresponding to the convolution filter; x i Represents the receptive field R o All pixels within;

[0037] Use another set of convolution filters To obtain features, its output is X c It is expressed as follows:

[0038]

[0039] (2) Based on the damage degree p, the feature expression is enhanced through the self-attention mechanism, and its output feature X a It is expressed as follows:

[0040]

[0041] Where <·,·> represents the composite operation of dot product and softmax; q is the query vector; k is the key vector; v is the value vector; x o represents the feature vector of the center pixel; ω represents the learnable weight matrix;

[0042] (3) For the mask perception part, for the input feature x i and input mask m i , input feature x i Project to x with half the number of channels but the same size, and perform pixel shuffling downsampling operations on x and m:

[0043]

[0044] Among them, PD represents pixel shuffling downsampling; Represents the image features after pixel shuffling and downsampling; Represents the mask features after pixel shuffling and downsampling;

[0045] (4) Due to The missing pixels in are discontinuous between channels, while Each channel of The positions of valid pixels and invalid pixels in Act accurately on the corresponding channel and slice and Scattered and connected on the channel, we get

[0046]

[0047] Among them, Slice(·) represents slicing by channel; || represents splicing by channel; % represents the modulo operator;

[0048] (5) Using separable convolutional layers The feature and mask pair are encoded to learn the correct local prior from the features represented by the mask and model the effective information in the visible area; the mask perception part fuses the attention and mask prior through matrix multiplication, and its output feature X o The expression is as follows:

[0049]

[0050] (6) Obtain the output feature X of the mask-guided adaptive convolution out , the output feature X out is the expectation based on the damage level p:

[0051] X out =p·X c +(1-p)·X o ;

[0052] Furthermore, the execution of the multi-level residual information distillation module includes:

[0053] (7) Given input feature X in , in each step, a channel splitting operation is performed on the previous features, the input features are divided into two parts, one part of the coarse features is retained, and the other part of the distilled features is passed to the next distillation step:

[0054]

[0055] Among them, Split1 represents the first channel cutting operation; Split2 represents the second channel cutting operation; Split3 represents the third channel cutting operation; P represents the pyramid squeeze attention module, ReLU represents the activation function, Represents the first distilled feature that will be further processed by subsequent layers; Represents the second distilled feature that will be further processed by subsequent layers; Represents the third distilled feature that will be further processed by subsequent layers; Indicates the coarse features retained in the first layer; Indicates the coarse features retained in the second layer; Indicates the coarse features retained in the third layer;

[0056] (8) The extracted multi-level coarse features and distilled features are fused as output:

[0057]

[0058] Among them, Conv represents convolution, and Concat represents the connection operation along the channel dimension;

[0059] Furthermore, the design of the pyramid squeeze attention module is as follows:

[0060] A multi-scale parallel feature extraction strategy is adopted, and by building a parallel multi-branch network architecture, the contextual information of the input feature map is extracted synchronously under different receptive fields:

[0061] X1=Conv(k 3×3 )(X in );

[0062] X2=Conv(k 5×5 )(X in );

[0063] X3=Conv(k 7×7 )(X in );

[0064] X4=Conv(k 9×9 )(X in );

[0065] Among them, k 3×3 represents a 3×3 convolution kernel; k 5×5 represents a 5×5 convolution kernel; k 7×7 represents a 7×7 convolution kernel; k 9×9 Represents a 9×9 convolution kernel; X1 represents the features obtained after 3×3 convolution; X2 represents the features obtained after 5×5 convolution; X3 represents the features obtained after 7×7 convolution; X4 represents the features obtained after 9×9 convolution;

[0066] The entire multi-scale extraction feature map is obtained by cascading:

[0067] X all =Concat([X1,X2,X3,X4]);

[0068] The channel attention mechanism is learned from feature maps of different scales to generate channel weight distribution corresponding to the features of each scale. The vector of attention weight is expressed as:

[0069] Zi =SEWeight(X i ),i=1,2,3,4;

[0070] Among them, SEWeight is used to obtain attention weights from feature maps of different scales;

[0071] The vector of the entire multi-scale channel attention weights is obtained in a cascade manner:

[0072] Z = Concat([Z1, Z2, Z3, Z4]);

[0073] Recalibrate multi-scale channel attention weights using Softmax:

[0074]

[0075] The calibrated multi-scale channel attention weights Attn i With the corresponding scale X i Multiply the feature maps of:

[0076] Y i =Attn i ⊙X i ,i=1,2,3,4;

[0077] Among them, ⊙ represents the element-by-element multiplication operation, Y i Represents a feature map containing multi-scale channel attention weights; the output of the pyramid squeeze attention module is expressed as:

[0078] P = Concat([Y1, Y2, Y3, Y4]).

[0079] Furthermore, the loss function of the mask-guided local information extraction network is designed as follows:

[0080] Define the first term as weighted reconstruction loss The formula is:

[0081]

[0082] in, and They represent the effective region loss and damaged region loss of the mask-guided local information extraction network respectively; I gt Represents the correct label image; represents the restoration result of the mask-guided local information extraction network; ⊙ is the element-by-element product operation; M represents the mask used to distinguish the valid area and the missing area in the image; sum(M) represents the number of non-zero elements in M; λ h represents the balance factor;

[0083] Use the total variation loss to smooth the penalty term:

[0084]

[0085] in, represents the merged image; represents the total variational loss; i represents the row index; j represents the column index;

[0086] The perceptual loss is used to compare the perceptual similarity of the restored image and the original image in local areas to guide the restoration process:

[0087]

[0088] in, represents the feature map of the i-th layer of the pre-trained VGG-16 network; Indicates perceived loss;

[0089] Use style loss to measure the style difference between the restored image and the real image:

[0090]

[0091] Among them, G i (·) represents the Gram matrix, represents style loss;

[0092] The total loss of the mask-guided local information extraction network is:

[0093]

[0094] Among them, λ tv represents the weighting coefficient of the total variational loss; λ per Represents the weighting coefficient of perceptual loss; λ sty Represents the weighting coefficient of style loss.

[0095] Furthermore, the global information extraction network includes a Transformer network and a UNet network;

[0096] The implementation of the Transformer network includes:

[0097] Remove the layer normalization in the standard Transformer module, use feature concatenation instead of residual connection, and use a fully connected layer to connect the input and output of the multi-head self-attention; The output of the fully connected layer Expressed as:

[0098]

[0099]

[0100] Among them, MSA stands for multi-head self-attention; MLP stands for multi-layer perceptron; Indicates the The output of the MLP, Indicates the The output of the layer Transformer block.

[0101] Furthermore, the loss function of the global information extraction network is designed as follows:

[0102] The loss function of the global information extraction network is expressed as:

[0103]

[0104] Therefore, the final total training loss of the network is:

[0105]

[0106] Beneficial effects:

[0107] Compared with the existing technology, this ancient mural restoration method based on progressive reconstruction and damage-aware self-adaptation has the following beneficial effects:

[0108] 1) This paper proposes a mural restoration network based on a progressive strategy. This network extracts mural features in stages, solving the problem of existing networks being inefficient in capturing both local details and global style. It has good applicability for damaged murals.

[0109] 2) This paper proposes a mask-guided local information extraction network, which effectively solves the problem of texture blur in the restoration results; the mask-guided adaptive module adaptively extracts and fuses local features according to the degree of damage to the mural, while the multi-level residual information distillation module further refines the details of the mural at different scales.

[0110] 3) Based on a hybrid architecture of Transformer blocks and UNet, this paper designs a global information extraction network to extract the global features of the mural, ensuring the coherence of the overall structure and the coordination of the style. Utilizing the global information of the mural can further improve the visual quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0111] Figure 1 A flowchart of an ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to an embodiment of the present application is shown;

[0112] Figure 2 A system block diagram of an ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to an embodiment of the present application is shown;

[0113] Figure 3 A schematic structural diagram of a method for restoring ancient murals according to an embodiment of the present application is shown;

[0114] Figure 4 A schematic structural diagram of a mask-guided adaptive convolution operator according to an embodiment of the present application is shown;

[0115] Figure 5 A schematic structural diagram of a multi-stage residual information distillation module according to an embodiment of the present application is shown;

[0116] Figure 6 A comparison diagram of the results according to an embodiment of the present application is shown, wherein: a is the damaged image before repair; a1 is an enlarged image of the point taken at the damaged part of a; b is image a repaired by the CTSDG method; b1 is an enlarged image of the point taken on b after repair; c is image a repaired by the MISF method; c1 is an enlarged image of the point taken on c after repair; d is image a repaired by the LGNet method; d1 is an enlarged image of the point taken on d after repair; e is image a repaired by the CANet method; e1 is an enlarged image of the point taken on e; f is image a repaired by the CoordFill method; f1 is an enlarged image of the point taken on f; g is image a repaired by the StrDiffusion method; g1 is an enlarged image of the point taken on g; h is image a repaired by the technical solution of the present application; h1 is an enlarged image of the point taken on h; i is the original mural; i1 is an enlarged image of the point taken on i. DETAILED DESCRIPTION

[0117] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0118] like Figure 1-2 As shown, the present invention provides a technical solution: a method for restoring ancient murals based on progressive reconstruction and damage-aware adaptation.

[0119] The implementation principle of an ancient mural restoration method based on progressive reconstruction and damage-aware self-adaptation is as follows:

[0120] S1: data preprocessing;

[0121] The collected ancient mural images and binary mask images were uniformly set to 256×256 size. The mask images were used to accurately annotate damaged areas of the murals. A training set was created, containing 1,564 image-mask pairs, while the test set contained 150 image-mask pairs.

[0122] S2: Network architecture design;

[0123] like Figure 3 As shown in the figure, a neural network based on a progressive restoration strategy is constructed. It includes three sub-networks: a coarse restoration network, a mask-guided local information extraction network, and a global information extraction network. The coarse restoration network uses a U-shaped codec structure with skip connections, with the input being the damaged mural image and the mask image, and the output being the initial restoration result. The mask-guided local information extraction network consists of a mask-guided adaptive module and a multi-level residual information distillation module, which is used to dynamically extract and optimize local features. The global information extraction network is based on a Transformer-UNet (a combination of Transformer and Unet) hybrid architecture, combining the feature extraction advantages of CNN and the global modeling capabilities of Transformer to capture the global artistic style characteristics of the mural.

[0124] S3: Progressive Repair Process

[0125] Step 3.1: Coarse inpainting network: This consists of a U-shaped encoder-decoder structure with skip connections, including eight downsampling and upsampling modules. The damaged mural image and the mask image are fed into the coarse inpainting network. A spectral normalization-based block discriminator is used for adversarial training against the coarse inpainting network. The discriminator distinguishes authenticity from counterfeit images in local regions of the image and generates an initial inpainting result.

[0126] Furthermore, the pixel-by-pixel reconstruction loss uses weighted L1 loss (weighted mean absolute error) for pixel-by-pixel reconstruction:

[0127]

[0128] in, represents the effective area loss of the coarse repair network; represents the damage area loss of the coarse repair network; I gt Represents the correct label image (ground-truth image); represents the restoration result of the coarse restoration network; ⊙ represents the element-by-element product operation; M represents the mask used to distinguish the valid area and the missing area in the image; sum(M) represents the number of non-zero elements in M; sum(1-M) represents the number of zero elements in M;

[0129] The pixel-by-pixel loss Defined as;

[0130]

[0131] Among them, λ h represents the balance factor;

[0132] For GAN loss (Generative Adversarial Networks Loss, adversarial loss) Using least squares loss, the corresponding coarse repair network loss function and the discriminator loss function Defined as:

[0133]

[0134] Among them, I in represents the input image, represents the merged image; E represents expectation;

[0135] represents the expectation of a real image, sampled from the real data distribution;

[0136] represents the expectation of the merged image, sampled from the distribution of generated images; D represents the discriminator.

[0137] Total loss of the coarse repair network for:

[0138]

[0139] Among them, λ g represents the weighted coefficient of adversarial loss; λ h =6,λ g =0.1.

[0140] Step 3.2: Mask-guided local information extraction network: The initial restoration result and the mask image are used as the input of the mask-guided local information extraction network, where the mask-guided adaptive module dynamically adjusts the attention weight of the feature map according to the damage weight matrix; the multi-level residual information distillation module extracts and fuses local detail features at different scales through a multi-level residual connection structure.

[0141] The mask-guided adaptive module contains a standard convolutional layer, an instance normalization layer, a LeakyReLU activation function, and a mask-guided adaptive convolutional layer. Figure 4 As shown in the embodiment, the specific design of the mask-guided adaptive convolution layer is as follows:

[0142] First, for the input corrupted feature F∈R C×H×W vector (pixels) on which a set of filters ω is applied pTo calculate the degree of damage of the input, denoted by p.

[0143]

[0144] Where σ represents the sigmoid function, which normalizes the damage degree to the interval [0,1]; x i Represents the input features; R o Represents the receptive field corresponding to the convolution filter.

[0145] Furthermore, F represents a feature map (or image) with C channels, height H, and width W as input;

[0146] Apply another set of filters ω c To obtain features, the output is as follows:

[0147]

[0148] Then, based on the damage level p, the feature expression is enhanced through the self-attention mechanism, and its output feature X a It is expressed as follows:

[0149]

[0150] Where <·,·> represents the composite operation of dot product and softmax; q is the query vector; k is the key vector; v is the value vector; x o represents the feature vector of the center pixel; ω represents the learnable weight matrix;

[0151] For the mask perception part, for a given input feature x i And input mask m, project the features into x with half the number of channels but the same size, and perform pixel shuffle downsampling (PD) operation on x and m:

[0152]

[0153] Among them, PD represents pixel shuffling downsampling; Represents the image features after pixel shuffling and downsampling; Represents the mask features after pixel shuffling and downsampling;

[0154] because The missing pixels in are discontinuous between channels, while Each channel of In order to make the effective pixels and invalid pixels in Act accurately on the corresponding channel and slice and Scattered and connected on the channel, we get

[0155]

[0156] Among them, Slice(·) is slicing by channel, || is splicing by channel, and % represents the modulo operator. Therefore, each feature has a paired mask as an indicator.

[0157] Next, using the separable convolutional layer The feature and mask pair are encoded to learn the correct local prior from the features represented by the mask and to model the effective information in the visible area. The mask perception part fuses the attention and mask prior through matrix multiplication, and its output is X o The expression is as follows:

[0158]

[0159] Finally, the mask guides the output feature X of the adaptive convolution out It can be regarded as the expectation based on the damage degree p, and its calculation formula is as follows:

[0160] X out =p·X c +(1-p)·X o ;

[0161] Among them, Figure 4 In the equation, wq, wk, and wv are learnable weight matrices (or vectors), which generate the Query vector, Key vector, and Value vector respectively.

[0162] Furthermore, Figure 5 As shown in the embodiment, the specific design of the multi-level residual information distillation module is as follows:

[0163] First, given the input feature X in In each step, a channel splitting operation is performed on the previous features to divide the input features into two parts, one of which is the coarse features and is retained, and the other is the distilled features and is passed to the next distillation step. This process is described as follows:

[0164]

[0165] Among them, Split1 represents the first channel cutting operation; Split2 represents the second channel cutting operation; Split3 represents the third channel cutting operation; P represents the pyramid squeeze attention module, ReLU represents the activation function, Represents the first distilled feature that will be further processed by subsequent layers; Represents the second distilled feature that will be further processed by subsequent layers; Represents the third distilled feature that will be further processed by subsequent layers; Indicates the coarse features retained in the first layer;

[0166] Indicates the coarse features retained in the second layer; Indicates the coarse features retained in the third layer;

[0167] Figure 5 In the expression, Excitation means excitation; Squeeze means compression;

[0168] The extracted multi-level features are fused as output, which can be expressed as:

[0169]

[0170] Among them, Conv represents convolution and Concat represents the connection operation along the channel dimension.

[0171] Specifically, the design of the Pyramid Split Attention (PSA) module is as follows:

[0172] First, a multi-scale parallel feature extraction strategy is adopted to construct a parallel multi-branch network architecture to simultaneously extract the contextual information of the input feature map under different receptive fields:

[0173] X1=Conv(k 3×3 )(X in );

[0174] X2=Conv(k 5×5 )(X in );

[0175] X3=Conv(k 7×7 )(X in );

[0176] X4=Conv(k 9×9 )(X in );

[0177] Among them, k 3×3 represents a 3×3 convolution kernel; k 5×5 represents a 5×5 convolution kernel; k 7×7 represents a 7×7 convolution kernel; k 9×9 Represents a 9×9 convolution kernel; X1 represents the features obtained after 3×3 convolution; X2 represents the features obtained after 5×5 convolution; X3 represents the features obtained after 7×7 convolution; X4 represents the features obtained after 9×9 convolution;

[0178] The entire multi-scale extraction feature map is obtained by cascading:

[0179] Xall =Concat([X1,X2,X3,X4]);

[0180] The channel attention mechanism is learned from feature maps of different scales to generate channel weight distribution corresponding to the features of each scale. The vector of attention weight is represented as:

[0181] Z i =SEWeight(X i ),i=1,2,3,4;

[0182] Among them, SEWeight is used to obtain attention weights from feature maps of different scales. The entire multi-scale channel attention vector is obtained in a cascade manner:

[0183] Z = Concat([Z1, Z2, Z3, Z4]);

[0184] Recalibrate multi-scale channel attention weights using Softmax:

[0185]

[0186] The calibrated multi-scale channel attention weights Attn i With the corresponding scale X i Multiply the feature map of:

[0187] Y i =Attn i ⊙X i ,i=1,2,3,4;

[0188] Among them, ⊙ represents element-by-element multiplication, Y i Represents a feature map containing multi-scale channel attention weights. The output of PSA is expressed as:

[0189] P = Concat([Y1, Y2, Y3, Y4])

[0190] Furthermore, the loss function of the mask-guided local information extraction network is designed as follows:

[0191] The first term of the loss function is the weighted reconstruction loss The formula is:

[0192]

[0193] in, and They represent the effective region loss and damaged region loss of the mask-guided local information extraction network respectively; I gt Represents the correct label image; represents the restoration result of the mask-guided local information extraction network; ⊙ is the element-by-element product operation; represents the mask, which is used to distinguish the valid area and the missing area in the image; sum(M) represents the number of non-zero elements in M; λ h Represents the balance factor.

[0194] Total variation loss (TV loss) is used as a smoothing penalty term. is the merged image. Its formula is:

[0195]

[0196] Among them, i represents the row index; j represents the column index;

[0197] Perceptual Loss guides the restoration process by comparing the perceptual similarity of the restored image and the original image in local areas:

[0198]

[0199] in, is the feature map of layer i of the pre-trained VGG-16 network.

[0200] It's worth noting that VGG-16 uses a series of 3x3 convolution kernels to replace the larger convolution kernels (11x11, 7x7, and 5x5) used in AlexNet (a type of Deep Convolutional Neural Network proposed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton in 2012). For a given receptive field (the local size of the input image relative to the output), stacking small convolution kernels is superior to using large ones because multiple nonlinear layers increase network depth, enabling learning of more complex patterns, at a relatively low cost (fewer parameters). Simply put, in VGG, three 3x3 convolution kernels are used instead of 7x7 convolution kernels, and two 3x3 convolution kernels are used instead of 5x5 convolution kernels. The main purpose is to increase the depth of the network while maintaining the same receptive field, thereby improving the effectiveness of the neural network to a certain extent. For example, the stacking of three 3x3 convolution kernels with a stride of 1 can be seen as a receptive field of size 7 (in fact, it means that three consecutive 3x3 convolutions are equivalent to a 7x7 convolution), and the total number of parameters is 3x(9xC 2 ), if the 7x7 convolution kernel is used directly, the total number of parameters is 49xC 2 , where C refers to the number of input and output channels. Obviously, 27xC 2 Less than 49xC 2, that is, the parameters are reduced; and the 3x3 convolution kernel is conducive to better maintaining the image properties.

[0201] Style Loss is used to measure the style difference between the restored image and the real image. Its formula is:

[0202]

[0203] Among them, G i (·) represents the Gram matrix, represents style loss;

[0204] The total loss of the mask-guided local inpainting network is:

[0205]

[0206] Among them, λ tv represents the weighting coefficient of the total variational loss; λ per Represents the weighting coefficient of perceptual loss; λ sty Represents the weighting coefficient of style loss; λ h =6,λ tv =0.1,λ per =0.05,λ sty =120.

[0207] Step 3.3: Global Information Extraction Network: Figure 3 As shown in the embodiment, the local optimized results and the mask image are used as the input of the global information extraction network. Through the feature extraction capability of CNN and the long-range dependency modeling capability of the Transformer block, the coherence of the overall structure of the mural and the coordination of the style are ensured, and the final restoration result is output.

[0208] exist Figure 3 In the figure, Instance Normalization is instance normalization; LeakyRelu is a leaky rectified linear unit, which is an activation function; Mask-guidedAdaptive Convolution is a mask-guided adaptive convolution; MultiheadSelf-Attention is multi-head self-attention; Fully Connected Layer is a fully connected layer; Multi-Layer Perceptron is a multi-layer perceptron.

[0209] Construct a Transformer hybrid UNet network TU-HNet based on fusion learning to extract the global information of the mural. The standard Transformer block (Transformer module) usually contains two sub-modules: a multi-head self-attention (MSA) module and a multi-layer perceptron (MLP) module. Layer normalization is applied before each module, and a residual connection is used after each module to encourage the model to learn high-frequency content. However, in image restoration, when many labels are invalid, layer normalization may over-amplify these useless labels, so layer normalization (LayerNorm) is removed, and residual learning is replaced by fusion learning (ie, feature splicing). Use the fully connected (FC) layer to connect the input and output of the multi-head self-attention. The output of the layer is expressed as:

[0210]

[0211]

[0212] MSA is multi-head self-attention, MLP is multi-layer perceptron, For the The output of the MLP, For the The output of a Transformer block.

[0213] Furthermore, the global information extraction network loss function is designed as follows:

[0214] The loss function of this network is similar to that of the mask-guided local restoration network:

[0215]

[0216] Therefore, the final training loss of the network is:

[0217]

[0218] This example uses the Adam optimizer to train the model with a batch size of 8. The initial learning rate is set to 0.0002 for the first 100 epochs and then linearly decays to zero over the next 100 epochs. All experiments are trained on NVIDIA 4090 GPUs. The size of the mural image and mask is 256×256.

[0219] This example is mainly compared with six benchmark methods (including classic algorithms and the latest methods), namely: CTSDG (Image Inpainting via Conditional Texture and Structure Dual Generation), which models texture synthesis under structure constraints and texture-guided structure reconstruction; MISF (Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting), a multi-level interactive twin filtering technology that combines semantic filtering and image-level filtering; LGNet (Image Inpainting with Local and Global Refinement), a three-stage image inpainting network based on different receptive fields for local and global optimization; CANet (Context Adaptive Network for Image Inpainting), which generates diverse weights based on different inputs and flexibly inpaints images with various forms of damage; CoordFill (Efficient High-Resolution Image Inpainting via Parameterized Coordinate Refinement), which generates diverse weights based on different inputs and flexibly inpaints images with various forms of damage. Querying, a high-resolution image restoration network), repairs damaged images through parameterized coordinate querying; StrDiffusion (a structure-guided neural network, a diffusion model guided by edge maps and grayscale images), a diffusion model for image restoration.

[0220] Table 1 presents the quantitative comparison results of this embodiment with six other methods. Specifically, in terms of objective evaluation metrics, the present invention excels in peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learning-perceptual image patch similarity (LPIPS). This advantage is primarily attributed to the network architecture based on a progressive strategy, which takes into account both local and global features and incorporates a damage-aware adaptive mechanism, enabling effective processing of mural images with varying degrees of damage. For example, when the damage rate is between 30% and 40%, the peak signal-to-noise ratio (PSNR) value of the present invention is approximately 31% higher than that of the higher-performing MISF, demonstrating a significant advantage in processing murals with high damage rates. However, in cases of less severe damage (damage rates between 0% and 10%), the present invention slightly lags behind CoordFill in terms of learning-perceptual image patch similarity (LPIPS) and structural similarity index (SSIM). However, as the degree of damage increases, the advantages of the present method gradually become apparent, particularly when processing large damaged areas, where its performance advantage becomes even more pronounced.

[0221] Table 1

[0222]

[0223] Figure 6 The following is a qualitative comparison of this example with six other methods, showing the restoration results of damaged murals by different methods and the details after partial magnification. CTSDG imposes dual constraints on the texture and structure of the image, but its ordinary convolution treats damaged and undamaged areas equally, hindering the targeted extraction of effective information from undamaged areas. Figure 6 As shown in the first and fourth rows of , a lot of content is lost when repairing the character ribbon part. MISF proposes a new filtering technology that uses effective semantic information and pixel-level filling to perform high-fidelity restoration, but it still cannot fully capture the fine-grained information of the mural. Figure 6 As can be seen in the third row, the transition between the diamond grids is not smooth enough and details are seriously missing. LGNet uses a progressive strategy for repair, but this method lacks in-depth research on the processing of local details and fails to establish an effective balance mechanism between local features and global features. It cannot fully integrate local detail information with global structure, style and other features, resulting in blurred textures and structural deformation. CANet introduces an attention mechanism, which can generate relatively rich and detailed textures, but the generated edges are not clear enough, for example Figure 6The white line in the third row and the yellow ribbon in the fourth row. CoordFill relies too much on frequency domain information and ignores the spatial domain features, which is not enough to fully restore its structure, and a large number of structural missing problems appear in the repair results. StrDiffusion has certain advantages in texture generation by virtue of the powerful generation ability of the diffusion model. However, it uses traditional edge operators for structural guidance, which cannot fully guide the generation of edges, and the edges of the restored murals are not clear enough. The present invention adaptively evaluates the degree of damage of the murals, and dynamically adjusts the feature extraction and generation process according to the differences in the degree of damage in different areas to maximize the use of effective information in the murals. At the same time, the details of the murals at different scales are optimized to ensure the generation of fine textures and coherent structures. In addition, global information is fully mined to maintain the consistency of the overall style. According to Figure 6 It can be seen that the experimental effect of the ancient mural restoration method of this application is good, and it can restore murals with clear texture, coherent structure and high fidelity.

[0224] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An ancient mural restoration method based on progressive reconstruction and damage-aware adaptation, characterized by: The specific steps include: Step 1: Input the damaged mural image and the mask image into the coarse restoration network for initial restoration. During the initial restoration process, a discriminator is used to perform adversarial training with the coarse restoration network to generate an initial restoration result for the mural image. Step 2: Inputting the initial restoration result and the mask image into a mask-guided local information extraction network to extract and optimize the local features of the mural image after the initial restoration to obtain a local optimization result; Step 3: Input the local optimization results into the global information extraction network to capture the global artistic style of the mural image and obtain the overall restoration result of the mural image.

2. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 1 is characterized in that: The coarse repair network includes a U-type codec structure with skip connections.

3. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 1 is characterized in that: The adversarial training is performed using pixel-by-pixel reconstruction loss and adversarial loss; The pixel-by-pixel reconstruction loss uses weighted L1 loss for pixel-by-pixel reconstruction, which is expressed as: in, represents the effective area loss of the coarse repair network; represents the damage area loss of the coarse repair network; I gt Represents the correct label image; represents the restoration result of the coarse restoration network; ⊙ represents the element-by-element product operation; M represents the mask used to distinguish the valid area and the missing area in the image; sum(M) represents the number of non-zero elements in M; sum(1-M) represents the number of zero elements in M; The pixel-by-pixel loss Defined as: Among them, λ h represents the balance factor; The adversarial loss Using least squares loss, define the corresponding coarse repair network loss function and the discriminator loss function The discriminator loss function Defined as: Among them, I in represents the input image, represents the merged image; E represents expectation; represents the expectation of a real image, sampled from the real data distribution; represents the expectation of the merged image, sampled from the distribution of generated images; D represents the discriminator; The coarse repair network loss function for: Among them, λ g Represents the weighting coefficient of the adversarial loss.

4. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 1 is characterized in that: The mask-guided local information extraction network includes: A mask-guided adaptive module, configured to dynamically adjust the attention weights of the features of the input initial restoration result according to a damage weight matrix, wherein the mask-guided adaptive module comprises a standard convolution layer, an instance normalization layer, a LeakyReLU activation function, and a mask-guided adaptive convolution layer arranged in sequence; The multi-level residual information distillation module is used to extract and fuse local detail features of the initial restoration results at different scales through a multi-level residual connection structure.

5. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 4 is characterized in that: The mask guides the execution of the adaptive convolution layer including: (1) For the input damaged feature F∈R C×H×W , a set of convolution filters are used to filter the pixels on Calculate the damage degree p of the input damage feature; Where σ represents the sigmoid function, which normalizes the damage degree to the interval [0,1]; R o represents the receptive field corresponding to the convolution filter; x i Represents the receptive field R o All pixels within; Use another set of convolution filters To obtain features, its output feature X c It is expressed as follows: (2) Based on the damage degree p, the feature expression is enhanced through the self-attention mechanism, and its output feature X a It is expressed as follows: Where <·,·> represents the composite operation of dot product and softmax; q is the query vector; k is the key vector; v is the value vector; x o represents the feature vector of the center pixel; ω represents the learnable weight matrix; (3) For the mask perception part, for the input feature x i And input mask m, the input feature x i Project to x with half the number of channels but the same size, and perform pixel shuffling downsampling operations on x and m: Among them, PD represents pixel shuffling downsampling; Represents the image features after pixel shuffling and downsampling; Represents the mask features after pixel shuffling and downsampling; (4) Due to The missing pixels in are discontinuous between channels, while Each channel of The positions of valid pixels and invalid pixels in Act accurately on the corresponding channel and slice and Scattered and connected on the channel, we get Among them, Slice(·) represents slicing by channel; || represents splicing by channel; % represents the modulo operator; (5) Using separable convolutional layers The feature and mask pair are encoded to learn the correct local prior from the features represented by the mask and model the effective information in the visible area; the mask perception part fuses the attention and mask prior through matrix multiplication, and its output feature X o The expression is as follows: (6) Obtain the output feature X of the mask-guided adaptive convolution out , the output feature X out is the expectation based on the damage level p: X out =p·X c +(1-p)·X o 。 6. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 4 is characterized in that: The execution of the multi-level residual information distillation module includes: (7) Given input feature X in In each step, a channel splitting operation is performed on the previous features to divide the input features into two parts, one part of the coarse features is retained, and the other part of the distilled features is passed to the next distillation step: Among them, Split1 represents the first channel cutting operation; Split2 represents the second channel cutting operation; Split3 represents the third channel cutting operation; P represents the pyramid squeeze attention module, ReLU represents the activation function, Represents the first distilled feature that will be further processed by subsequent layers; Represents the second distilled feature that will be further processed by subsequent layers; Represents the third distilled feature that will be further processed by subsequent layers; Indicates the coarse features retained in the first layer; Indicates the coarse features retained in the second layer; Indicates the coarse features retained in the third layer; (8) The extracted multi-level coarse features and distilled features are fused as output: Among them, Conv represents convolution and Concat represents the connection operation along the channel dimension.

7. The ancient mural restoration method based on progressive reconstruction and damage-aware self-adaptation according to claim 6 is characterized in that: The design of the pyramid squeeze attention module is as follows: A multi-scale parallel feature extraction strategy is adopted, and by building a parallel multi-branch network architecture, the contextual information of the input feature map is extracted synchronously under different receptive fields: X1=Conv(k 3×3 )(X in ); X2=Conv(k 5×5 )(X in ); X3=Conv(k 7×7 )(X in ); X4=Conv(k 9×9 )(X in ); Among them, k 3×3 represents a 3×3 convolution kernel; k 5×5 represents a 5×5 convolution kernel; k 7×7 represents a 7×7 convolution kernel; k 9×9 Represents a 9×9 convolution kernel; X1 represents the features obtained after 3×3 convolution; X2 represents the features obtained after 5×5 convolution; X3 represents the features obtained after 7×7 convolution; X4 represents the features obtained after 9×9 convolution; The entire multi-scale extraction feature map is obtained by cascading: X all =Concat([X1,X2,X3,X4]); The channel attention mechanism is learned from feature maps of different scales to generate the channel weight distribution corresponding to the features of each scale. The attention weight is expressed as: Z i =SEWeight(X i ),i=1,2,3,4; Among them, SEWeight is used to obtain attention weights from feature maps of different scales; The vector of the entire multi-scale channel attention weights is obtained in a cascade manner: Z = Concat([Z1, Z2, Z3, Z4]); Recalibrate multi-scale channel attention weights using Softmax: The calibrated multi-scale channel attention weights Attn i With the corresponding scale X i Multiply the feature maps of: Y i =Attn i ⊙X i ,i=1,2,3,4; Among them, ⊙ represents the element-by-element multiplication operation, Y i Represents a feature map containing multi-scale channel attention weights; the output of the pyramid squeeze attention module is expressed as: P = Concat([Y1, Y2, Y3, Y4]).

8. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 7 is characterized in that: The loss function of the mask-guided local information extraction network is designed as follows: Define the first term as weighted reconstruction loss The formula is: in, and They represent the effective region loss and damaged region loss of the mask-guided local information extraction network respectively; I gt Represents the correct label image; represents the restoration result of the mask-guided local information extraction network; ⊙ is the element-by-element product operation; M represents the mask used to distinguish the valid area and the missing area in the image; sum(M) represents the number of non-zero elements in M; λ h represents the balance factor; Use the total variation loss to smooth the penalty term: in, represents the merged image; represents the total variational loss; i represents the row index; j represents the column index; The perceptual loss is used to compare the perceptual similarity of the restored image and the original image in local areas to guide the restoration process: in, represents the feature map of the i-th layer of the pre-trained VGG-16 network; Indicates perceived loss; Use style loss to measure the style difference between the restored image and the real image: Among them, G i (·) represents the Gram matrix, represents style loss; The total loss of the mask-guided local information extraction network is: Among them, λ tv represents the weighting coefficient of the total variational loss; λ per Represents the weighting coefficient of perceptual loss; λ sty Represents the weighting coefficient of style loss.

9. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 8 is characterized in that: The global information extraction network includes a Transformer network and a UNet network; The implementation of the Transformer network includes: Remove the layer normalization in the standard Transformer module, use feature concatenation instead of residual connection, and use a fully connected layer to connect the input and output of the multi-head self-attention; The output of the fully connected layer Expressed as: Among them, MSA stands for multi-head self-attention; MLP stands for multi-layer perceptron; FC stands for fully connected layer; Indicates the The output of the MLP, Indicates the The output of the layer Transformer block.

10. The ancient mural restoration method based on progressive reconstruction and damage-aware adaptation according to claim 9 is characterized in that: The loss function of the global information extraction network is designed as follows: The loss function of the global information extraction network is expressed as: Therefore, the final total training loss of the network is:

Images