Progressive fresco restoration method based on multi-scale feature extraction and edge guidance

Abstract

The application belongs to the technical field of image processing, and specifically discloses a progressive mural repair method based on multi-scale feature extraction and edge guidance, which comprises the following steps: collecting mural images to obtain a mural dataset; a generative adversarial network model is established, wherein the coarse network is a codec structure with a skip connection; the fine network introduces image edge information to strengthen the structural information of the image; the global refinement network introduces a self-attention mechanism and a multi-scale feature extraction module to capture more global information; the generative adversarial network model is trained: the training set of the mural dataset is used to train the generative adversarial network model to obtain a three-stage generative adversarial model; and the trained generative adversarial model is used to repair the test set. The mural repair method has the advantages of the edge guidance module, can better obtain the significant structural features of the image by weighted fusion of a saliency map and an edge detection map, and can make the edge structure of the generated image more coherent and consistent with semantic consistency.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and specifically relates to a progressive mural restoration method based on multi-scale feature extraction and edge guidance. Background Technology

[0002] Ancient murals, affected by natural and human environments, as well as the inherent factors of the materials and techniques used in their creation, inevitably suffer from various forms of damage, impacting their preservation and transmission. Therefore, timely restoration of discovered damaged murals is crucial for preserving their economic and cultural value to the greatest extent possible.

[0003] Currently, technologies applicable to mural restoration can be mainly divided into three categories: pixel diffusion-based restoration algorithms, sample matching-based restoration algorithms, and deep learning-based algorithms. The first two categories are traditional restoration algorithms, and related research is relatively mature, making improvement difficult, and the quality of the generated images is far inferior to that of deep learning algorithms. The mainstream deep learning-based mural restoration algorithms are further divided into convolutional neural networks and generative adversarial networks.

[0004] The core advantage of Convolutional Neural Networks (CNNs) lies in their region perception and weight sharing. They utilize convolutional and pooling layers to capture image features, while fully connected layers perform operations such as classification or regression. With the gradual development of deep learning, CNNs have demonstrated powerful performance and have been applied to image restoration tasks, including methods such as GL and PConv. They can automatically learn feature representations in images. Through backpropagation and a large amount of training data, CNNs can learn feature representations suitable for restoration tasks, thereby better restoring image details and structure. However, on the one hand, CNN-based networks typically require a large amount of training data to achieve good performance. For mural restoration tasks, obtaining large-scale mural image data is difficult, leading to low-quality images generated during restoration. On the other hand, as the number of layers in a CNN-based method increases, the gradient vanishing problem may occur, making it almost impossible to update the shallow parameters of the deep network. Therefore, it cannot capture details and local features, resulting in semantic incoherence and blurry artifacts in the generated mural images.

[0005] Generative Adversarial Networks (GANs) have shown remarkable performance in image restoration in recent years, with numerous GAN-based image restoration algorithms emerging. Through adversarial training between the generator and discriminator networks, GANs can learn the distribution of an image and generate restored images with detail and structure. This allows GANs to produce more realistic and natural results in image restoration tasks. However, GAN-based methods struggle to extract salient structural features. Furthermore, because murals possess more complex structures and textures than other natural images, there are few GAN models specifically designed for mural restoration. This is because GAN model architectures primarily focus on the overall distribution and texture of the image, failing to extract higher-level structural features. Moreover, GANs lack explicit constraints or prior knowledge to guide the extraction of structural features, making it relatively difficult to capture complex structural features like those found in murals. Due to the lack of salient structural feature extraction, the generated images often exhibit incomplete edge structures, affecting the clarity of the restored results. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a progressive mural restoration method based on multi-scale feature extraction and edge guidance.

[0007] This invention provides a progressive mural restoration method based on multi-scale feature extraction and edge guidance, comprising the following steps: S1. Establish a generative adversarial network model: The generative adversarial network model includes coarse networks, fine networks, and a globally refined network; The coarse network is used to roughly restore the mural image, resulting in a roughly restored image; The fine network is used to extract edge features from the coarsely restored image, and the extraction results are guided by the edge saliency map of the mural image to obtain a locally refined restored image. The global thinning network performs multi-scale feature extraction on the locally thinned and repaired image to capture more global features; S2. Use the aforementioned generative adversarial network model to restore the mural.

[0008] A further embodiment is that the coarse network includes a first downsampling module and a first upsampling module, and the first downsampling module and the first upsampling module are connected in a skip connection to transmit the mural image information from the encoder to the decoder.

[0009] A further embodiment is that the fine network includes an edge guiding module, a second downsampling module, a residual block, and a second upsampling module arranged sequentially. The edge guidance module includes a Laplace edge detection operator and an edge saliency map, which are used to extract the edge structure information of the coarsely repaired image to obtain an edge map; and use the edge saliency map to guide edge enhancement of the edge map to obtain a locally refined repaired image; The edge saliency map is obtained by processing the mural image using a deep convolutional neural network.

[0010] A further approach is to adopt a coarse network architecture for the global refinement network and introduce a self-attention mechanism and a multi-scale feature extraction module to capture more global information by expanding the range of information captured by neurons.

[0011] A further solution is that the multi-scale feature extraction module includes defolding and folding operations, including five parallel convolutional layers, including two 1*1 convolutions and three dilated convolutions with different dilation rates, namely 2, 4 and 6.

[0012] A further approach involves inputting the first repair result and a random mask into the fine network. Before downsampling, the first repair result is processed by an edge guidance module to extract salient edge structures, resulting in an edge guidance map. The edge guidance map is then batch-normalized and added element-wise to the first repair result. After passing through two upsampling modules, four residual blocks, and two downsampling modules, the second repair result of the fine network is obtained.

[0013] A further solution is that the edge guidance module acquires the edge guidance map. The process is as follows: The first repair result is used to perform edge detection and extraction using the Laplace edge detection operator to obtain the edge map. ; Simultaneously, the weights of the convolutional layers are observed using class activation mapping. The feature map of the last convolutional layer is linearly combined with the weights of the classification layer to generate a saliency map. The calculation process is represented as follows:

[0014] in The weights are represented as those of the k-th feature map. This is represented as the k-th feature map; A saliency map is generated from the feature map output from the last convolutional layer of ResNet-50. ; The edge guidance map is composed of the edge detection map. With saliency map The pixel-by-pixel weighted fusion is calculated as follows:

[0015] in The weighting coefficient has a value of [value]. .

[0016] A further approach involves using the second repair result and the random mask as input to a global refinement network. This network is then processed through eight downsampling modules, with a multi-scale feature extraction module capturing feature information at different scales in the last layer of the downsampling modules to obtain a feature map. The feature map is then upsampled, and three self-attention modules are added to the front end of the decoder to capture more global information by expanding the range of information captured by neurons.

[0017] A further approach is that the process of the multi-scale feature extraction module outputting the feature map is as follows: The second repair result is processed by a convolution kernel of size [size missing]. Convolution extracts shallow, low-level features from the input features. , is represented as:

[0018] The obtained feature map After being unfolded into a high-dimensional feature matrix through a defolding operation, it is then passed through 5 parallel convolutional layers, which are represented as follows:

[0019]

[0020]

[0021]

[0022]

[0023]

[0024] in, This is the second repair result input. Dilation represents dilated convolutions with different dilation rates, where dilation represents the dilation rate. This represents the output features of a regular convolution. This represents the output features of the 1st, 2nd, and 3rd dilated convolutions. This represents the feature map obtained by concatenating the feature maps output by parallel convolutional layers.

[0025] A further embodiment of S1 includes training the established generative adversarial network model.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) This invention utilizes the advantages of the edge guidance module to obtain the significant structural features of the image by weighted fusion of the saliency map and the edge detection map, making the edge structure of the generated image more coherent and consistent with semantics.

[0027] (2) This invention introduces a multi-scale feature extraction module, which uses five parallel sets of convolutions, including ordinary convolutions and dilated convolutions with different dilation rates, to capture features at different scales of the image. By utilizing multi-scale information, the missing details of the generated image are restored and the overall quality of the image is improved. Attached Figure Description

[0028] The following figures are for illustrative purposes only and are not intended to limit the scope of the invention, wherein: Figure 1 : A structural diagram of the generative adversarial network model of this invention; Figure 2 Edge guidance module structure diagram; Figure 3 : Structure diagram of the multi-scale feature extraction module; Figure 4 Images illustrating the restoration effects of different methods on simulated damaged murals; among them Figure 4 (a) is the original image of the mural. Figure 4 (b) To simulate an image of a damaged mural, Figure 4 (c) is the image repaired using the GL method. Figure 4 (d) is the image repaired using the PConv method. Figure 4 (e) is the image repaired using the Edge-Connect method. Figure 4 (f) shows the image restored using the RFR method. Figure 4 (g) is the image restored using the CTSDG method. Figure 4 (h) is the image restored by the method of the present invention; Figure 5 Images showing the restoration effects of different methods on murals with certain defects; among them Figure 5 (a) is the original image of the mural. Figure 5 (b) To simulate an image of a damaged mural, Figure 5 (c) is the image repaired using the GL method. Figure 5 (d) is the image repaired using the PConv method. Figure 5 (e) is the image repaired using the Edge-Connect method. Figure 5 (f) shows the image restored using the RFR method. Figure 5 (g) is the image restored using the CTSDG method. Figure 5 (h) is the image repaired by the method of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, design methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0030] This invention provides a progressive mural restoration method based on multi-scale feature extraction and edge guidance, comprising the following steps: Step 1: Dataset Creation Images of the murals were collected, and the data volume was expanded by applying data augmentation technology to obtain a total of 15,680 mural datasets. 200 of these images were used as the test set, and the rest were used as the training set. Step 2: Establish a generative adversarial network model; Step 3: Train the generative adversarial network model; The generative adversarial network model was trained using the training set of the mural dataset to obtain a three-stage generative adversarial model. Step 4: Use the trained generative adversarial model to restore the mural image.

[0031] Among them, the application of data augmentation techniques includes random cropping, mirroring, flipping and other techniques to make a series of random changes to the image, thereby expanding the scale of the mural dataset.

[0032] like Figure 1 As shown, the aforementioned generative adversarial network model includes a coarse network, a fine network, and a global refinement network. The coarse network is an encoder-decoder structure with skip connections, comprising 8 downsampling modules and 8 upsampling modules. Skip connections are used to transfer information from the encoder to the decoder, thereby recovering information lost during downsampling. When an image is used as input, the coarse network receives the input image. The coarse network outputs a roughly repaired image, which is a binary mask M simulating the missing regions. The repair result from the coarse network is then fed into a fine network, which includes an edge guidance module, two downsampling modules, four residual blocks, and two upsampling modules. By introducing edge information, the structural information of the image is enhanced, ensuring that the structure is not distorted. After processing by the fine network, some visual artifacts are appropriately eliminated under the guidance of surrounding local regions and edge information. When acquiring information from a relatively large surrounding region, to ensure that some missing regions can be further refined, a global refinement network based on attention and multi-scale feature extraction is introduced. This global refinement network adopts the coarse network architecture and expands the range of information captured by neurons to capture more global information by adding multi-scale feature extraction and using an attention mechanism. Three attention modules are added at the front end of the decoder, and a multi-scale feature extraction module is used in the last layer of the encoder to extract multi-scale features to capture the global and local features of the image, so as to better understand the structure and content of the image during the repair process. Specifically, the edge guidance module is used to extract edge structure information from the mural image, including the Laplace edge detection operator and the feature map output from the last convolutional layer of ResNet-50. The multi-scale feature extraction module is used for multi-scale feature extraction, including defolding and folding operations, and five parallel convolutional layers, containing two 1x1 convolutions and three dilated convolutions with different dilation rates of 2, 4, and 6.

[0033] The training steps for a generative adversarial model include: S3.1: Separate the mural datasets and random mask Input generative adversarial model; S3.2: The coarse network contains eight downsampling and upsampling modules, and uses skip connections to pass information from the encoder to the decoder, thereby recovering the information lost during downsampling. The coarse network yields a rough restoration result of the mural image. ; S3.3: The rough repair result and random mask As input to the fine network, the image is used to extract salient edge structures before downsampling to obtain an edge guide map; S3.4: Guide the edge map After batch normalization processing and Element-wise addition is performed, followed by two upsampling modules, four residual blocks, and two downsampling modules to obtain the repair result of the locally refined network. ; S3.5: Refine the repair results of the local network. and random mask As input to the global refinement network, it passes through eight downsampling modules and captures feature information at different scales using a multi-scale feature extraction module in the last layer of the downsampling modules; S3.6: Upsample the feature map and add three self-attention modules to the front end of the decoder to capture more global information by expanding the range of information captured by neurons; S3.7: Output the restored mural image to complete one training cycle; S3.8: Repeat steps S3.1-S3.7 to obtain a three-stage generative adversarial model for restoring mural images.

[0034] like Figure 2 As shown, the edge guidance module obtains the edge guidance map. The process is as follows: The first repair result is used to perform edge detection and extraction using the Laplace edge detection operator to obtain the edge map. ; Simultaneously, class activation mapping (whose main idea is to observe the weights of convolutional layers, linearly combine the feature map of the last convolutional layer with the weights of the classification layer to generate a saliency map) is used. The calculation process is as follows:

[0035] in The weights are represented as those of the k-th feature map. This is represented as the k-th feature map; A saliency map is generated from the feature map output from the last convolutional layer of ResNet-50. The classification layer, also known as the fully connected layer, maps the output features of the convolutional layer to different categories to determine the region of interest, which is obtained through global average pooling. The edge guidance map is composed of the edge detection map. With saliency map The pixel-by-pixel weighted fusion is calculated as follows:

[0036] in The weighting coefficient has a value of [value]. .

[0037] The first repair result is used to perform edge detection and extraction using the Laplace edge detection operator to obtain the edge map. The edge detection map can be obtained by performing edge detection using the Laplacian operator. ; like Figure 3As shown, in step S3.5, the multi-scale feature extraction module includes: Input features First, use a convolution kernel with a size of Convolution extracts shallow, low-level features from the input features. :

[0038] The obtained feature map After being unfolded into a high-dimensional feature matrix through a defolding operation, it is then passed through five parallel convolutional layers, which are represented as follows:

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] in Dilation represents dilated convolutions with different dilation rates, where dilation represents the dilation rate. This represents the output features of a regular convolution. This represents the output features of the 1st, 2nd, and 3rd dilated convolutions. This represents the feature map obtained by concatenating the feature maps output by parallel convolutional layers.

[0045] This embodiment uses two types of mural images to verify the effectiveness of the proposed mural restoration method: one is a relatively well-preserved mural image, and the other is a mural image with a complex background that has been damaged to some extent.

[0046] To better evaluate the practicality of the method, a comparative experiment was conducted in this embodiment, with both image and mask sizes set to... The Adam optimizer is used, where , The experiment requires training for 200 epochs, with a learning rate of [missing information] for the first 100 epochs. The next 100 epochs will be in... The linear coefficient decays. The generative adversarial model is implemented in PyTorch using an NVIDIA GeForce RTX 3090 GPU.

[0047] This embodiment mainly compares with five classic and novel image restoration methods: deep learning-based methods GL and PConv; and generative adversarial network-based methods Edge-Connect, RFR, and CTSDG.

[0048] Analysis of experimental results: Figure 4 To preserve relatively intact experimental results of the murals; Figure 4 (a) is the original image of the mural. Figure 4 (b) An image simulating the effect of damage to the mural. Figure 4 Images (c) through (g) show mural restoration images using five different comparative methods. Figure 4 (h) shows the mural restoration result using this method; by visually comparing the restored image and the original image, it can be seen that... Figure 4 In (c) and (d), the mural images generated by the GL and PConv methods have obvious structural incompleteness and unreasonable filling. The ribbons at the waist of the figures are not generated completely. Figure 4 Although the mural images generated by the Edge-Connect and CTSDG methods in (e)(g) are structurally improved compared to the two methods above, the generated images have certain blurring and artifact problems, resulting in lower image quality. Figure 4 (f) shows more consistent color, but structural information is still not well recovered at the edges. Figure 4 As can be seen in (h), the mural image generated by the present invention has the smallest difference from the original image, and the generated mural image has more delicate details, smoother lines, and a more complete structure.

[0049] Figure 5 These are experimental results images of murals with certain defects. Figure 5 (a) is the original image of the mural. Figure 5 (b) An image simulating the effect of damage to the mural. Figure 5 Images (c) through (g) show mural restoration images using five different comparative methods. Figure 5 (h) shows the mural restoration result using this method; by visually comparing the restored image and the original image, it can be seen that... Figure 5 (c) shows obvious artifacts at the neck of the character, achieved using the GL method. Figure 5 (d) shows the PConv method, where the lines at the right nostril of the figure are discontinuous. While Edge-Connect, RFR, and CTSDG generate images with relatively little blur and artifacts, their performance in generating details remains poor, such as... Figure 5 (e) Figure 5 (f) The character's nose is noticeably missing, and the left eye is poorly formed. Figure 5(g) Although changes were made to the nose to better match the original image, the left eye was still elongated. And from... Figure 5 As can be seen from (h), the mural image generated by the present invention has the least difference from the original image. It not only has a complete structure, but also pays attention to semantic coherence, and suppresses the problem of blur artifacts. At the same time, it displays richer local detail information and has better image quality.

[0050] Table 1

[0051] Subjective evaluation alone is insufficient to accurately assess the restoration results; therefore, objective indicators must be combined for evaluation. This invention employs three objective evaluation metrics—PSNR, SSIM, and LPIPS—to comprehensively evaluate the image. Peak Signal-to-Noise Ratio (PSNR) is a commonly used metric for measuring image quality, comparing the similarity between the original and reconstructed images. A higher PSNR value indicates less distortion and better image quality between the generated and original images. Structural Similarity (SSIM) considers the structural information of the image and is more relevant to human perception of similarity. SSIM assesses the similarity between the original and reconstructed images by comparing their structure, brightness, and contrast. A higher SSIM value indicates richer edge information in the generated image. Perceptual Similarity (LPIPS) utilizes perceptual features learned by deep learning models, rather than relying solely on pixel-level differences. This allows LPIPS to better capture the subjective perception of image quality. A lower LPIPS value indicates a smaller perceptual difference between the generated and original images, meaning they are more visually similar.

[0052] Table 1 shows the objective indicators of the results of different mural restoration methods in the experiment. As can be seen from Table 1, all objective indicators of the present invention are superior to other methods. That is, the mural image generated by the method of the present invention is closest to the original image and can well restore the edge structure information and texture details of the mural image.

[0053] Based on a combination of visual evaluation and objective metrics, the algorithm proposed in this invention can effectively obtain information that combines salient edge structure with texture detail.

[0054] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A progressive mural restoration method based on multi-scale feature extraction and edge guidance, characterized in that, Includes the following steps: S1. Establish a generative adversarial network model; The generative adversarial network model includes coarse networks, fine networks, and a globally refined network; The coarse network includes eight first downsampling modules and eight first upsampling modules. The first downsampling modules and the first upsampling modules transfer mural image information from the encoder to the decoder through skip connections to recover information lost during downsampling. The mural image and a random mask are input into the coarse network, and the mural image is roughly restored through the coarse network to obtain a roughly restored image as the first restoration result. The fine network includes an edge guidance module, two second downsampling modules, four residual blocks, and two second upsampling modules arranged sequentially. The first restoration result and a random mask are used as inputs to the fine network. Before downsampling, the edge guidance module extracts the salient edge structure of the first restoration result to obtain an edge guidance map. The edge guidance map is batch normalized and then added element-wise to the first restoration result. After passing through the two second downsampling modules, four residual blocks, and two second upsampling modules, a locally thinned restoration image, which is the second restoration result, is obtained. The edge guidance module includes a Laplace edge detection operator and an edge saliency map. The process of obtaining the edge guidance map is as follows: The first repair result is used to perform edge detection and extraction using the Laplace edge detection operator to obtain an edge map. And obtain the edge detection map used for weighted fusion. ; Simultaneously, the weights of the convolutional layers are observed using class activation mapping. The feature map of the last convolutional layer is linearly combined with the weights of the classification layer to generate a saliency map. The calculation process is represented as follows: ； in, This represents the weight of the k-th feature map. This represents the k-th feature map; in The weights are represented as those of the k-th feature map. This is represented as the k-th feature map; A saliency map is generated from the feature map output from the last convolutional layer of ResNet-50. ; The edge guidance map is composed of the edge detection map. With saliency map The pixel-by-pixel weighted fusion is calculated as follows: ； in The weighting coefficient has a value of [value]. ; The global refinement network adopts the network architecture of the coarse network and introduces a self-attention mechanism and a multi-scale feature extraction module. The second restoration result and the random mask are used as inputs to the global refinement network. The feature map is obtained by using eight downsampling modules and the multi-scale feature extraction module in the last layer of the downsampling module to capture feature information at different scales. The feature map is then upsampled, and three self-attention modules are added to the front end of the decoder to capture more global information by expanding the range of information captured by neurons, thus obtaining the mural restoration image. S2. Use the aforementioned generative adversarial network model to restore the mural.

2. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 1, characterized in that, The multi-scale feature extraction module includes defolding and folding operations, comprising five parallel convolutional layers, including two 1*1 convolutions and three dilated convolutions with different dilation rates of 2, 4, and 6.

3. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 2, characterized in that, The process by which the multi-scale feature extraction module outputs the feature map is as follows: The second repair result is processed by a convolution kernel of size [size missing]. Convolution extracts shallow, low-level features from the input features. , is represented as: ； The obtained feature map After being unfolded into a high-dimensional feature matrix through a defolding operation, it is then passed through 5 parallel convolutional layers, which are represented as follows: ； ； ； ； ； ； in, This is the second repair result input. Dilation represents dilated convolutions with different dilation rates, where dilation represents the dilation rate. This represents the output features of a regular convolution. This represents the output features of the 1st, 2nd, and 3rd dilated convolutions. This represents the feature map obtained by concatenating the feature maps output by parallel convolutional layers.

4. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 1, characterized in that, S1 further includes training the established generative adversarial network model.

Images