Progressive mural restoration method based on multi-scale feature extraction and edge guidance

By introducing a multi-scale feature extraction and edge-guided generative adversarial network model in mural restoration technology, the problems of low image quality of mural restoration and incomplete edge structure in the prior art are solved, and a higher quality mural restoration effect is achieved.

CN119067887AActive Publication Date: 2024-12-03NORTHWEST UNIV

Patent Information

Application Number
CN202411068718.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-03
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

In the existing mural repair technology, CNN-based methods require a large amount of training data and are prone to gradient disappearance problems, resulting in low quality of generated images; GAN-based methods cannot effectively extract the significant structural features of murals, resulting in incomplete edge structure, affecting restoration. clarity of the results.

Method used

Using a generative adversarial network model based on multi-scale feature extraction and edge guidance, including coarse network, fine network and global refined network, the multi-scale feature and edge information of the image are captured through the Laplace edge detection operator and self-attention mechanism to generate More coherent and clear mural restoration images.

Benefits of technology

By weighted fusion of significant images and edge detection images, the edge structure consistency and semantic consistency of the image are improved, and details are restored using the multi-scale feature extraction module to significantly improve image quality. The generated mural images have richer details and more complete structure.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067887A_ABST
    Figure CN119067887A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of image processing, and particularly discloses a progressive mural restoration method based on multi-scale feature extraction and edge guidance, which comprises the following steps: acquiring a mural image to obtain a mural data set; a generative adversarial network model is established, and a coarse network is of a codec structure with jump connection; the fine network introduces image edge information for enhancing the structure information of the image; a self-attention mechanism and a multi-scale feature extraction module are introduced into the global refinement network, and more global information is captured; training a generative adversarial network model: training the generative adversarial network model by adopting the training set of the mural data set to obtain a generative adversarial model of a three-stage network; and repairing the test set by adopting the trained generative adversarial model. According to the invention, by using the advantages of the edge guiding module, through weighted fusion of the saliency map and the edge detection map, the salient structure features of the image can be better obtained, so that the edge structure of the generated image is more coherent and accords with semantic consistency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a progressive mural restoration method based on multi-scale feature extraction and edge guidance. Background Art

[0002] Ancient murals are inevitably damaged by the natural and human environment, as well as the inherent factors of the materials and craftsmanship of mural production, which have affected the protection and inheritance of murals. Therefore, timely repair of damaged murals is crucial to preserve their economic and cultural value to the greatest extent.

[0003] At present, the technologies that can be used for mural restoration are mainly divided into three categories: restoration algorithms based on pixel diffusion, restoration algorithms based on sample matching and filling, and algorithms based on deep learning. The first two categories are traditional restoration algorithms. The relevant research is relatively mature, and it is difficult to improve them. In addition, the quality of the generated images is far inferior to that of deep learning algorithms. The mainstream algorithms of mural restoration algorithms based on deep learning are divided into convolutional neural networks and generative adversarial networks.

[0004] The core advantage of convolutional neural network (CNN) lies in its perception of regions and sharing of weights. It uses convolutional layers and pooling layers to obtain image features, and uses fully connected layers to perform classification or regression operations. With the gradual development of deep learning, convolutional neural networks have shown powerful performance and have been applied to image restoration tasks, including GL, PConv and other methods. They can automatically learn feature representations in images. Through back propagation algorithms and a large amount of training data, CNN can learn feature representations suitable for restoration tasks, thereby better restoring image details and structures. However, on the one hand, CNN-based networks usually require a large amount of training data to achieve good performance. For mural restoration tasks, it is difficult to obtain large-scale mural image data, which will result in low quality of generated images when mural images are restored; on the other hand, CNN-based methods may have the problem of gradient disappearance as the number of network layers increases, making it almost impossible to update the shallow parameters of the deep network, so it is impossible to capture details and local features, resulting in semantic incoherence and blur artifacts in the generated mural images.

[0005] Generative adversarial networks (GANs) have performed well in the field of image restoration in recent years, and GAN-based image restoration algorithms have emerged one after another. Through adversarial training of the generator network and the discriminator network, GAN can learn the distribution of images and generate restored images with details and structures. This enables GAN to produce more realistic and natural results in image restoration tasks. However, methods based on generative adversarial networks cannot extract significant features of the structure. And because murals have more complex structures and textures than other natural images, there are few GAN models used for mural restoration. The reason is that the GAN model architecture mainly focuses on the overall distribution and texture of the image, and cannot extract more advanced structural features. In addition, GAN has no explicit constraints or prior knowledge to guide the extraction of image structural features, which is relatively difficult to capture for images with complex structural features such as murals. Due to the lack of extraction of significant structural features, the generated images have incomplete edge structures, which affects the clarity of the restoration results. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects in the prior art and provide a progressive mural restoration method based on multi-scale feature extraction and edge guidance.

[0007] The present invention provides a progressive mural restoration method based on multi-scale feature extraction and edge guidance, comprising the following steps:

[0008] S1. Establish a generative adversarial network model:

[0009] The generative adversarial network model includes a coarse network, a fine network and a global refined network;

[0010] The coarse network is used to roughly restore the mural image to obtain a roughly restored image;

[0011] The fine network is used to extract edge features from the rough restoration image, and to guide edge enhancement of the extraction result using the edge saliency map of the mural image to obtain a local refined restoration image;

[0012] The global refinement network performs multi-scale feature extraction on the local refined restoration image to capture more global features;

[0013] S2. Using the generative adversarial network model to perform mural restoration.

[0014] A further solution is that the coarse network includes a first down-sampling module and a first up-sampling module, and the first down-sampling module and the first up-sampling module are jump-connected to transfer the mural image information from the encoder to the decoder.

[0015] A further solution is that the fine network includes an edge guidance module and a second down-sampling module, a residual block and a second up-sampling module arranged in sequence;

[0016] The edge guidance module includes a Laplace edge detection operator and an edge saliency map, which are used to extract edge structure information of the roughly restored image to obtain an edge map; and use the edge saliency map to guide the edge enhancement of the edge map to obtain a locally refined restored image;

[0017] The edge saliency map is obtained by processing the mural image through a deep convolutional neural network.

[0018] A further solution is that the global refinement network adopts the network architecture of the coarse network and introduces a self-attention mechanism and a multi-scale feature extraction module to capture more global information by expanding the range of information captured by neurons.

[0019] A further solution is that the multi-scale feature extraction module includes de-folding and folding operations, including five parallel convolutional layers, including two 1*1 convolutions and three dilated convolutions with different dilation rates, and the dilation rates of the three dilated convolutions are 2, 4, and 6, respectively.

[0020] A further solution is that in S3.2, after the fine network inputs the first repair result and the random mask, the edge-significant structure of the first repair result is extracted through an edge guidance module before downsampling to obtain an edge guidance map, and the edge guidance map is batch normalized and then added element-by-element to the first repair result, and the second repair result of the fine network is obtained after two upsampling modules, four residual blocks and two downsampling modules.

[0021] A further solution is that the edge guidance module obtains the edge guidance graph F fused_image The process is:

[0022] The first restoration result is subjected to edge detection extraction through the Laplace edge detection operator to obtain the edge graph F ege_image ;

[0023] At the same time, the weights of the convolutional layer are observed through the class activation mapping technology, and the feature map of the last convolutional layer is linearly combined with the weights of the classification layer to generate a saliency map. The calculation process is expressed as:

[0024]

[0025] where w k Represented as the weight of the kth feature map, A k Represented as the kth feature map;

[0026] The saliency map F is generated by the feature map output by the last convolutional layer of ResNet-50 saliency_image ;

[0027] The edge guidance map is composed of the edge detection map F edge_map With the saliency map F saliency_image Perform pixel-by-pixel weighted fusion, and the calculation process is expressed as:

[0028] F fused_image =α×F edge_map +(1-α)F saliency_map

[0029] Wherein α is the weight coefficient, and its value is α=0.5.

[0030] A further solution is that in S3.3, the second repair result and the random mask are used as the input of the global refinement network and passed through eight downsampling modules and a multi-scale feature extraction module is used in the last layer of the downsampling module to capture feature information of different scales to obtain a feature map, the feature map is upsampled, and three self-attention modules are added to the front end of the decoder to capture more global information by expanding the range of information captured by neurons.

[0031] A further solution is that in step S3.5, the process of the multi-scale feature extraction module outputting the feature map is:

[0032] The second restoration result is extracted through a convolution with a convolution kernel size of 3×3 to obtain the shallow low-level feature F0 of the input feature, which is expressed as:

[0033] F0=Conv 3×3 (F in_feature )

[0034] The obtained feature map F0 is unfolded into a high-dimensional feature matrix through an unfolding operation and passed through 5 parallel convolutional layers, where these 5 parallel convolutional layers are represented as follows:

[0035] F1=Conv 1×1 (F0)

[0036] F2=DConv 3×3 (F0, dilation = 2)

[0037] F3=DConv 3×3 (F0, dilation = 4)

[0038] F4=DConv 3×3 (F0, dilation = 6)

[0039] F5=Conv 1×1 (F0)

[0040] F6=Conv 1×1 (Concat(F1,F2F3,F4,F5))

[0041] Among them, F in_feature The second repair result of the input, DConv 3×3 (·) represents dilated convolutions with different dilation rates, dilation represents the dilation rate, F1 and F5 represent the output features of ordinary convolutions, F2, F3, and F4 represent the output features of the 1st, 2nd, and 3rd dilated convolutions, and F6 represents the feature map obtained by concatenating the feature maps output by the parallel convolutional layers.

[0042] A further solution is that S1 also includes training the established generative adversarial network model.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention takes advantage of the edge guidance module and can better obtain the significant structural features of the image by weighted fusion of the saliency map and the edge detection map, so that the edge structure of the generated image is more coherent and conforms to semantic consistency.

[0045] (2) The present invention introduces a multi-scale feature extraction module and uses five sets of parallel convolutions, including ordinary convolutions and dilated convolutions with different dilation rates, to capture features of images at different scales. By utilizing multi-scale information, the details of the missing generated image are restored and the overall quality of the image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The following drawings are only used to illustrate and explain the present invention, and are not used to limit the scope of the present invention, wherein:

[0047] Figure 1 : The structure diagram of the generative adversarial network model of the present invention;

[0048] Figure 2 : Edge guidance module structure diagram;

[0049] Figure 3 : Multi-scale feature extraction module structure diagram;

[0050] Figure 4 :The restoration effect diagram of different methods on simulated damaged murals; Figure 4 (a) is the original image of the mural. Figure 4 (b) is a simulated damaged mural image. Figure 4 (c) is the image restored by GL method. Figure 4 (d) is the image restored by PConv method. Figure 4 (e) is the image repaired by the Edge-Connect method. Figure 4 (f) is the image repaired by RFR method. Figure 4 (g) is the image restored by CTSDG method. Figure 4 (h) is the image restored by the method of the present invention;

[0051] Figure 5 :The restoration effect diagram of murals with certain defects by different methods; Figure 5 (a) is the original image of the mural. Figure 5 (b) is a simulated damaged mural image. Figure 5 (c) is the image repaired by GL method. Figure 5 (d) is the image restored by PConv method. Figure 5 (e) is the image repaired by Edge-Connect method. Figure 5 (f) is the image repaired by RFR method. Figure 5 (g) is the image restored by CTSDG method. Figure 5 (h) is the image restored by the method of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] The present invention provides a progressive mural restoration method based on multi-scale feature extraction and edge guidance, comprising the following steps:

[0054] Step 1: Dataset creation

[0055] Collect mural images and expand the data volume by applying data augmentation technology to obtain a total of 15,680 mural datasets, of which 200 images are used as test sets and the rest as training sets;

[0056] Step 2: Establish a generative adversarial network model;

[0057] Step 3: Train the generative adversarial network model;

[0058] The training set of the mural dataset is used to train the generative adversarial network model, and a three-stage network generative adversarial model is obtained;

[0059] Step 4: Use the trained generative adversarial model to repair the mural image.

[0060] Among them, the application of data augmentation technology includes random cropping, mirroring, flipping and other technologies to make a series of random changes to the image, thereby expanding the scale of the mural dataset.

[0061] like Figure 1 As shown in Figure 1, the above-mentioned generative adversarial network model includes a coarse network, a fine network, and a global fine network. The coarse network is a codec structure with a skip connection, including 8 downsampling modules and 8 upsampling modules. The skip connection is used to transfer information from the encoder to the decoder to restore the information lost during downsampling. When the image is taken as input, the coarse network receives the input image I in The output of the coarse network is a roughly restored image, and the restoration result of the coarse network is fed into the fine network as input. The fine network includes an edge-guided module, two downsampling modules, four residual blocks, and two upsampling modules. By introducing edge information, the structural information of the image is enhanced to ensure that the structure is not distorted. After being processed by the fine network, some visual artifacts are appropriately eliminated under the guidance of the surrounding local area and edge information. When obtaining information from a relatively large surrounding area, in order to ensure that some missing areas can be better refined, a global refinement network based on attention and multi-scale feature extraction is introduced. The global refinement network adopts the network architecture of the coarse network, and expands the range of information captured by neurons by adding multi-scale feature extraction and using attention mechanism to capture more global information. Three attention modules are added to the front end of the decoder, and the multi-scale feature extraction module is used to extract multi-scale features in the last layer of the encoder to capture the global and local features of the image, so as to better understand the structure and content of the image during the restoration process. Specifically, the edge guidance module is used to extract the edge structure information of the mural image, including the Laplace edge detection operator and the feature map output by the last convolution layer of ResNet-50. Multi-scale feature extraction module: used for multi-scale feature extraction, including de-folding and folding operations, five parallel convolution layers, including two 1*1 convolutions and three dilated convolutions with different dilation rates, which are 2, 4, and 6 respectively.

[0062] The training steps for generating adversarial models include:

[0063] S3.1: The mural dataset I in And random mask M input to generate adversarial model;

[0064] S3.2: The coarse network contains 8 downsampling modules and upsampling modules, and uses skip connections to pass information from the encoder to the decoder to restore the information lost during downsampling. The coarse network obtains a rough restoration result of the mural image.

[0065] S3.3: Roughly repair the result and random mask M as the input of the fine network, and extract the edge salient structure of the image before downsampling to obtain the edge guidance map;

[0066] S3.4: Edge-guided graph F fused_image After batch normalization, The elements are added one by one, and then two upsampling modules, four residual blocks and two downsampling modules are used to obtain the repair result of the local refinement network.

[0067] S3.5: Repair results of local refinement network The random mask M is used as the input of the global refinement network and passes through eight downsampling modules. In the last layer of the downsampling module, a multi-scale feature extraction module is used to capture feature information of different scales.

[0068] S3.6: Upsample the feature map and add three self-attention modules to the decoder front end to capture more global information by expanding the range of information captured by neurons;

[0069] S3.7: Output the mural restoration image and complete one training;

[0070] S3.8: Repeat steps S3.1-S3.7 to obtain a three-stage generative adversarial model for repairing mural images.

[0071] like Figure 2 As shown, the edge guidance module obtains the edge guidance graph F fused_image The process is:

[0072] The first restoration result is subjected to edge detection extraction through the Laplace edge detection operator to obtain the edge graph F edge_image ;

[0073] At the same time, the weights of the convolutional layer are observed through the class activation mapping technology (the main idea is to linearly combine the feature map of the last convolutional layer with the weight of the classification layer by observing the weight of the convolutional layer to generate a saliency map), and the feature map of the last convolutional layer is linearly combined with the weight of the classification layer to generate a saliency map. The calculation process is expressed as:

[0074]

[0075] where w k Represented as the weight of the kth feature map, A k Represented as the kth feature map;

[0076] The saliency map F is generated by the feature map output by the last convolutional layer of ResNet-50 saliency_image ; The classification layer is the fully connected layer, which maps the output features of the convolutional layer to different categories to determine the focus area, which is obtained through the global average pooling operation;

[0077] The edge guidance map is composed of the edge detection map F edge_map With the saliency map F saliency_image Perform pixel-by-pixel weighted fusion, and the calculation process is expressed as:

[0078] F fused_image =α×F edge_map +(1-α)F saliency_map

[0079] Wherein α is the weight coefficient, and its value is α=0.5.

[0080] The first restoration result is subjected to edge detection extraction through the Laplace edge detection operator to obtain the edge graph F edge_image ; Use the Laplace operator to perform edge detection to obtain the edge detection graph F edge_map ;

[0081] like Figure 3 As shown, in step S3.5, the multi-scale feature extraction module includes:

[0082] The input feature F in_feature First, a convolution with a kernel size of 3×3 is used to extract the shallow low-level feature F0 of the input feature:

[0083] F0=Conv 3×3 (F in_feature )

[0084] The obtained feature map F0 is unfolded into a high-dimensional feature matrix through an unfolding operation, and then passed through 5 parallel convolutional layers, where these 5 parallel convolutional layers are represented as follows:

[0085] F1=Conv 1×1 (F0)

[0086] F2=DConv 3×3 (F0, dilation = 2)

[0087] F3=DConv 3×3 (F0, dilation = 4)

[0088] F4=DConv 3×3 (F0, dilation = 6)

[0089] F5=Conv 1×1 (F0)

[0090] F6=Conv 1×1 (Concat(F1,F2F3,F4,F5))

[0091] Where DConv 3×3(·) represents dilated convolutions with different dilation rates, dilation represents the dilation rate, F1 and F5 represent the output features of ordinary convolutions, F2, F3, and F4 represent the output features of the 1st, 2nd, and 3rd dilated convolutions, and F6 represents the feature map obtained by concatenating the feature maps output by the parallel convolutional layers.

[0092] This embodiment uses two types of mural images to verify the effectiveness of the proposed mural restoration method; one is a mural image that is relatively intact, and the other is a mural image that is eroded by certain diseases and has a complex background.

[0093] In order to better evaluate the practicality of the method, this example conducted a comparative experiment. The image and mask sizes were set to 256×256, and the Adam optimizer was used, where β1=0.5 and β2=0.999. The experiment required 200 epochs of training, and the learning rate for the first 100 epochs was 2×10 -4 , the next 100 epochs will be 10 -1 The generative adversarial model is implemented in PyTorch and uses an NVIDIA GeForce RTX 3090 GPU.

[0094] This embodiment is mainly compared with five classic and novel image restoration methods: deep learning-based methods GL and PConv; generative adversarial network-based methods Edge-Connect, RFR, and CTSDG.

[0095] Experimental effect analysis:

[0096] Figure 4 This is a picture of the experimental results of a relatively well-preserved mural; Figure 4 (a) is the original image of the mural. Figure 4 (b) is an image simulating the damage effect of the mural. Figure 4 (c) to (g) are the restored images of the murals using five different comparison methods. Figure 4 (h) is the mural restoration result of this method; by visually comparing the restored image with the original image, it can be seen that Figure 4 The mural images generated by the GL and PConv methods in (c) and (d) have obvious structural incompleteness and unreasonable filling, and the ribbon on the waist of the figure is not completely generated; Figure 4 Although the mural images generated by the Edge-Connect and CTSDG methods in (e) and (g) have improved structures compared with the above two methods, the generated images have certain blurring and artifact problems, and the generated image quality is low; Figure 4 (f) The colors are more coherent, but the structural information is still not well restored at the edges. Figure 4As can be seen in (h), the mural image generated by the present invention has the smallest difference with the original image, and the generated mural image has finer details, smoother lines, and a more complete structure.

[0097] Figure 5 This is the experimental result of a mural with certain diseases. Figure 5 (a) is the original image of the mural. Figure 5 (b) is an image simulating the damage effect of the mural. Figure 5 (c) to (g) are the restored images of the murals using five different comparison methods. Figure 5 (h) is the mural restoration result of this method; by visually comparing the restored image with the original image, it can be seen that Figure 5 (c) is the GL method, where there are obvious artifacts at the neck of the character. Figure 5 (d) is the PConv method, where the lines on the right nose of the character are discontinuous. The images generated by Edge-Connect, RFR, and CTSDG are relatively free of blur and artifacts, but the generation of details is still poor, such as Figure 5 (e) Figure 5 (f) The character's nose is obviously missing, and the left eye of the character is poorly generated. Figure 5 (g) Although the nose has been changed to be more in line with the original image, the left eye of the character is still lengthened. Figure 5 It can be seen from (h) that the mural image generated by the present invention has the smallest difference with the original image. It is not only structurally complete, but also pays attention to semantic coherence and suppresses blur artifacts. At the same time, it displays richer local detail information and has better image quality.

[0098] Table 1

[0099]

[0100]

[0101] It is difficult to give the most correct judgment on the restoration results by relying solely on subjective evaluation, so it is necessary to combine objective indicators to evaluate the mural restoration results; the present invention uses three objective evaluation indicators, PSNR, SSIM, and LPIPS, to comprehensively evaluate the image; among them, the peak signal-to-noise ratio (PSNR) is an indicator commonly used to measure image quality, which is used to compare the similarity between the original image and the reconstructed image. The larger the PSNR value, the smaller the distortion between the generated image and the original image, and the better the image quality; the structural similarity (SSIM) takes into account the structural information of the image and is more relevant to the similarity perceived by the human eye. SSIM evaluates the similarity between the original image and the reconstructed image by comparing their structure, brightness and contrast. The larger the SSIM value, the richer the edge information contained in the generated image. Perceptual similarity (LPIPS) utilizes the perceptual features learned by the deep learning model, rather than just relying on pixel-level differences. This enables LPIPS to better capture the subjective perception of the human eye to image quality. The smaller the LPIPS value, the smaller the perceptual difference between the generated image and the original image, that is, the more similar they are visually.

[0102] Table 1 shows the objective indicators of the results of different mural restoration methods in the experiment; it can be seen from Table 1 that all objective indicators of the present invention are better than other methods, that is, the mural image generated by the method of the present invention is closest to the original image and can well restore the edge structure information and texture details of the mural image.

[0103] Combining visual evaluation and objective index evaluation, the algorithm proposed in the present invention can well obtain both edge salient structure and texture detail information.

[0104] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A progressive mural restoration method based on multi-scale feature extraction and edge guidance, characterized in that: The following steps are involved: S1. Build a generative adversarial network model: The generative adversarial network model includes a coarse network, a fine network and a global refined network; The coarse network is used to roughly restore the mural image to obtain a roughly restored image; The fine network is used to extract edge features from the rough restoration image, and to guide edge enhancement of the extraction result using the edge saliency map of the mural image to obtain a local refined restoration image; The global refinement network performs multi-scale feature extraction on the local refined restoration image to capture more global features; S2. Using the generative adversarial network model to perform mural restoration.

2. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 1 is characterized in that: The coarse network includes a first down-sampling module and a first up-sampling module, and the first down-sampling module and the first up-sampling module are jump-connected to transfer the mural image information from the encoder to the decoder.

3. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 2 is characterized in that: The fine network includes an edge guidance module and a second down-sampling module, a residual block and a second up-sampling module which are arranged in sequence; The edge guidance module includes a Laplace edge detection operator and an edge saliency map, which are used to extract edge structure information of the roughly restored image to obtain an edge map; and use the edge saliency map to guide the edge enhancement of the edge map to obtain a locally refined restored image; The edge saliency map is obtained by processing the mural image through a deep convolutional neural network.

4. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 3 is characterized in that: The global refinement network adopts the network architecture of the coarse network and introduces a self-attention mechanism and a multi-scale feature extraction module to capture more global information by expanding the range of information captured by neurons.

5. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 4 is characterized in that: The multi-scale feature extraction module includes de-folding and folding operations, including five parallel convolutional layers, including two 1*1 convolutions and three dilated convolutions with different dilation rates, and the dilation rates of the three dilated convolutions are 2, 4, and 6 respectively.

6. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 5 is characterized in that: In S3.2, after the fine network inputs the first restoration result and the random mask, the edge guidance module is used to extract the edge significant structure of the first restoration result before downsampling to obtain an edge guidance map, and the edge guidance map is batch normalized and then added element by element to the first restoration result. The second restoration result of the fine network is obtained after two upsampling modules, four residual blocks and two downsampling modules.

7. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 6 is characterized in that: The edge guidance module obtains the edge guidance graph F fused_image The process is: The first restoration result is subjected to edge detection extraction through the Laplace edge detection operator to obtain the edge graph F edge_image ; At the same time, the weights of the convolutional layer are observed through the class activation mapping technology, and the feature map of the last convolutional layer is linearly combined with the weights of the classification layer to generate a saliency map. The calculation process is expressed as: where w k Represented as the weight of the kth feature map, A k Represented as the kth feature map; The saliency map F is generated by the feature map output by the last convolutional layer of ResNet-50 saliency_image ; The edge guidance map is composed of the edge detection map F edge_map With the saliency map F saliency_image Perform pixel-by-pixel weighted fusion, and the calculation process is expressed as: F fused_image =α×F edge_map +(1-α)F saliency_map Where α is the weight coefficient, and its value is α=0.

5.

8. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 7 is characterized in that: In S3.3, the second repair result and the random mask are used as the input of the global refinement network and passed through eight downsampling modules. A multi-scale feature extraction module is used in the last layer of the downsampling module to capture feature information of different scales to obtain a feature map. The feature map is upsampled, and three self-attention modules are added to the front end of the decoder to capture more global information by expanding the range of information captured by neurons.

9. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 8, characterized in that: In step S3.5, the process of the multi-scale feature extraction module outputting the feature map is as follows: The second restoration result is extracted through a convolution with a convolution kernel size of 3×3 to obtain the shallow low-level feature F0 of the input feature, which is expressed as: F0=Conv 3×3 (F in_feature ) The obtained feature map F0 is unfolded into a high-dimensional feature matrix through an unfolding operation and passed through 5 parallel convolutional layers, where these 5 parallel convolutional layers are represented as follows: F1=Conv 1×1 (F0) F2=DConv 3×3 (F0,dilation=2) F3=DConv 3×3 (F0,dilation=4) F4=DConv 3×3 (F0,dilation=6) F5=Conv 1×1 (F0) F6=Conv 1×1 (Concat(F1,F2F3,F4,F5)) Among them, F in_feature The second repair result of the input, DConv 3×3 (·) represents dilated convolutions with different dilation rates, dilation represents the dilation rate, F1 and F5 represent the output features of ordinary convolutions, F2, F3, and F4 represent the output features of the 1st, 2nd, and 3rd dilated convolutions, and F6 represents the feature map obtained by concatenating the feature maps output by the parallel convolutional layers.

10. The progressive mural restoration method based on multi-scale feature extraction and edge guidance according to claim 1, characterized in that: The S1 also includes training the established generative adversarial network model.

Citation Information

Patent Citations

  • Ancient text image restoration method and system, medium, equipment and terminal

    CN116630778A

  • Method for constructing Chinese landscape painting repair model guided by line draft

    CN118351026A

  • Deep learning network-based automatic delineation method for mediastinal lymphatic drainage region

    WO2022166800A1

Cited By

  • Mural virtual restoration method based on hyperspectral image

    CN119379563A