Method for repairing locally and globally refined Dunhuang mural

Abstract

The invention discloses a local and global refined Dunhuang mural restoration method, and relates to the technical field of image processing. The method comprises the following steps: S1, preprocessing images of a real mural, a damaged mural and a reference mural; s2, inputting the preprocessed data set into a repair model, and training the model through continuous iteration; and S3, performing a repair experiment on the simulated damaged fresco and the real damaged fresco by using the trained repair model to complete the repair of the damaged fresco. The network structure is simple, the channel attention residual block ECA-ResBlock is adopted, the convergence speed of the repair model is increased, and meanwhile the repair performance of the model is improved; a semi-instance normalization block is adopted to ensure similar distribution of a restoration area and a normal area, so that features are effectively enhanced, the modeling capability of the network is further improved, and wall painting restoration is facilitated; and through a reference guide repair method, the generalization ability of the model is improved, and the method plays a role in promoting repair of the large-area damaged mural.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a local and global refinement method for restoring Dunhuang murals. Background Art

[0002] In the dazzling cultural galaxy, the Mogao Grottoes in Dunhuang are the treasures of the art halls of my country and even the world. They are three-dimensional art that combines architecture, sculpture and murals. The Dunhuang murals are rich in content and can be divided into nine categories, including statues, Buddhist historical relics, sutra paintings, Han mythology paintings, and decorative pattern paintings. Among them, sutra paintings are the main part of Dunhuang murals. They are pictures of Sakyamuni preaching Buddhist scriptures. They are mainly based on religious themes. From the perspective of aesthetic style, they can be divided into five styles: sublime and solemn, symmetrical and harmonious, tragic and heroic, elegant, and hard and soft. It is known as the "world's largest art history gallery". The caves are painted and have been popular for thousands of years. The Dunhuang murals are a microcosm of the world's ancient civilizations, recording the long history of the Chinese nation from the 4th to the 14th century AD. The murals are rich in content and large in scale. They vividly record the national customs, religious beliefs, artistic aesthetics, political culture and other information of ancient my country. They have extremely important research value in terms of ethnicity, religion, art, etc., especially for the development history of the Northwest region.

[0003] However, due to the influence of geographical location, climatic environment and human factors, the murals in the caves generally suffer from mildew, cracking, fading and other defects to varying degrees.

[0004] Since traditional mural protection requires extremely high professionalism from protection talents, artists usually use manual copying to restore murals. They need to have profound theoretical knowledge, practical ability and aesthetic literacy about murals, and archaeological, humanistic, historical and architectural knowledge are indispensable. In addition, factors such as long restoration cycle, high operation requirements, irreversible copying results and severe shortage of domestic talents have led to the inability to carry out large-scale copying restoration, which has seriously affected the progress of mural protection projects in my country. In order to make the Dunhuang mural culture go to the world and better protect the cultural development and dissemination of murals, the introduction of digital restoration technology to achieve the protection of Dunhuang mural art has far-reaching significance. Through the digital protection of murals, the security of mural storage is greatly improved, and information sharing is also facilitated. In addition, the use of digital means to repair murals does not require direct contact with the mural body, and will not cause secondary damage to the murals. The digital restoration cycle is short, which greatly improves the restoration efficiency and effectively protects the artistic value of the murals. Therefore, the digital restoration of murals has far-reaching significance for the protection of cultural relics and the inheritance of culture.

[0005] At present, researchers at home and abroad have made a series of improvements and optimizations on the restoration of Dunhuang murals based on deep learning. However, there are still problems such as blurred restoration of large areas of damaged murals, structural fractures, color confusion, and even restoration errors. There is still huge room for improvement in the restoration of Dunhuang murals. Summary of the invention

[0006] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a local and global detailed Dunhuang mural restoration method, thereby solving the problems raised in the above-mentioned background technology.

[0007] In order to solve the above technical problems, the basic concept of the technical solution adopted by the present invention is:

[0008] A local and global detailed Dunhuang mural restoration method comprises the following steps:

[0009] S1: Preprocessing of real murals, damaged murals and reference murals images;

[0010] S2: Input the preprocessed data set into the repair model, and train the model through continuous iteration; the model training specifically includes:

[0011] S21: Encode the input mural into a feature representation of a latent space, and complete the mural restoration through a decoding operation; wherein the specific steps of the decoding operation include:

[0012] S211: A network consisting of eight downsampling and eight upsampling operations;

[0013] S212: The skip connection propagates information from the encoder to the decoder by connecting features of different scales in the channel dimension to recover the feature information lost during network downsampling;

[0014] S22: Use a repair network with a large receptive field to learn the overall structure of the image and complete the rough repair of the damaged image; the rough repair network is based on the damaged mural image I in and a binary mask image M describing the damaged area as input. The input image resolution is 256×256. At the end of the encoder, the receptive field is 766×766, which is larger than the image input resolution, which is conducive to learning the entire structure of the image. c The output of The original damaged mural image I in Merge to get the merged image Replace the corresponding area of ​​the coarse repair network output with the valid area in the original damaged image, and connect the merged image with the binary mask image M as the local refinement network Net L Input;

[0015] S23: Using the residual block of the channel attention to learn the feature information around the damaged area and refine the texture information of the damaged edge, further weakening the influence of the distant invalid features on the repair result; wherein, using the residual block of the channel attention to learn the feature information around the damaged area and refine the texture information of the damaged edge includes:

[0016] S231: ECA-ResBlock, a residual block based on ECA channel attention, effectively implements local cross-channel interaction without dimensionality reduction through one-dimensional convolution, thus improving the restoration effect;

[0017] S232: The feature is input into the ECA module and recorded as feature y after global average pooling on the channel without dimensionality reduction. ECA captures local cross-channel interaction information by considering each channel and its k adjacent channels. Then y i The weight ω i By y i It is calculated by the interaction between its k neighboring channels, and parameters are shared across all channels; ω i The specific formula is:

[0018]

[0019] Where, σ(·) represents the Sigmoid activation function. Represents y i The channel attention ω is realized by a 1D convolution with a convolution kernel of k. The specific formula is:

[0020] ω=σ(C1D K (y))

[0021] Among them, C1D represents one-dimensional convolution. Given the channel dimension C, the size of the convolution kernel is calculated. The specific formula is as follows:

[0022]

[0023] Among them, |t odd To represent the nearest odd integer of t, the values ​​of γ and b are set to 2 and 1;

[0024] S24: A global refinement network with a large receptive field based on a semi-instance normalization block is used to ensure that the repaired area has a similar distribution to the normal area, and further perform feature enhancement; wherein the specific process of the global refinement network with a large receptive field based on a semi-instance normalization block is as follows:

[0025] The semi-instance normalization block is introduced to enhance the features, further improve the modeling ability of the network, ensure that the repaired area has a similar distribution to the normal area, and refine the local network NetL The output of The original damaged mural image I in Merge to get the merged image Then merge the images Connected with the binary mask image M as a global refinement network Net G Input, global refinement network Net G Using Net C The network architecture is improved by adding HIN blocks in the encoder and using context attention modules in the decoder, which enables the network to learn effective features with high matching degree from distant feature spaces.

[0026] S3: Performing restoration experiments on simulated damaged murals and real damaged murals using the trained restoration model to complete the restoration of damaged murals; wherein, performing restoration experiments on simulated damaged murals and real damaged murals using the trained restoration model to complete the restoration of damaged murals includes:

[0027] S31: Perform an ablation experiment on the proposed module and evaluate the restoration results quantitatively and qualitatively; wherein the proposed modules are added to the benchmark model in turn, and the model training is completed through S2;

[0028] S32: Using the training model generated in S2, the optimized restoration model is used to simulate the damaged mural restoration experiment, and the restoration results are evaluated quantitatively and qualitatively;

[0029] S33: Using the training model generated in S2, the optimized restoration model is used to conduct a restoration experiment on a real damaged mural, and the restoration results are evaluated quantitatively and qualitatively;

[0030] S4: Make a quality assessment of the restoration results; after the restoration model training is completed, the training model generated by S2 is used to perform restoration evaluation on the damaged mural data of S1, and the restoration evaluation includes subjective evaluation and objective evaluation.

[0031] After adopting the above technical solution, the present invention has the following beneficial effects compared with the prior art. Of course, any product implementing the present invention does not necessarily need to achieve all the advantages described below at the same time:

[0032] (1) The network structure of the present invention is simple, and the channel attention residual block ECA-ResBlock is used to accelerate the convergence speed of the repair model and improve the repair performance of the model;

[0033] (2) A Half Instance Normalization Block (HIN Block) is used to ensure that the repaired area has a similar distribution to the normal area, effectively enhancing the features and further improving the modeling ability of the network, which is helpful for mural restoration.

[0034] (3) The reference-guided restoration method improves the generalization ability of the model and promotes the restoration of large areas of damaged murals.

[0035] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings described below are only some embodiments. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0037] In the figure:

[0038] Figure 1 A repair model constructed for the present invention;

[0039] Figure 2 The ECA-ResBlock module designed for the present invention;

[0040] Figure 3 This is the HIN module of the present invention.

[0041] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0042] The present invention will now be described in further detail with reference to the accompanying drawings.

[0043] Example 1: Please refer to Figure 1-3 As shown, in this embodiment, a local and global refinement method for repairing Dunhuang murals is provided, including the following steps:

[0044] S1: Preprocessing of real murals, damaged murals and reference murals images;

[0045] S2: Input the preprocessed data set into the repair model, and train the model through continuous iteration; the model training specifically includes:

[0046] S21: Encode the input mural into a feature representation of a latent space, and complete the mural restoration through a decoding operation; wherein the specific steps of the decoding operation include:

[0047] S211: A network consisting of eight downsampling and eight upsampling operations;

[0048] S212: The skip connection propagates information from the encoder to the decoder by connecting features of different scales in the channel dimension to recover the feature information lost during network downsampling;

[0049] S22: Use a repair network with a large receptive field to learn the overall structure of the image and complete the rough repair of the damaged image; the rough repair network is based on the damaged mural image I in and a binary mask image M describing the damaged area as input. The input image resolution is 256×256. At the end of the encoder, the receptive field is 766×766, which is larger than the image input resolution, which is conducive to learning the entire structure of the image. c The output of The original damaged mural image I in Merge to get the merged image Replace the corresponding area of ​​the coarse repair network output with the valid area in the original damaged image, and connect the merged image with the binary mask image M as the local refinement network Net L The local refinement network consists of two downsampling operations, four residual blocks based on ECA channel attention, and two upsampling operations. Due to the shallowness of the network, the network has a small receptive field (109×109 per output neuron). Then the network refines and repairs the local area of ​​the merged image in a sliding window manner. Image I in The combined formula is:

[0050]

[0051] Among them, ⊙ is the pixel-by-pixel product operation;

[0052] S23: Using the residual block of the channel attention to learn the feature information around the damaged area and refine the texture information of the damaged edge, further weakening the influence of the distant invalid features on the repair result; wherein, using the residual block of the channel attention to learn the feature information around the damaged area and refine the texture information of the damaged edge includes:

[0053] S231: ECA-ResBlock, a residual block based on ECA channel attention, effectively implements local cross-channel interaction without dimensionality reduction through one-dimensional convolution, thus improving the restoration effect;

[0054] S232: The feature is input into the ECA module and recorded as feature y after global average pooling on the channel without dimensionality reduction. ECA captures local cross-channel interaction information by considering each channel and its k adjacent channels. Then y iThe weight ω i By y i It is calculated by the interaction between its k neighboring channels, and parameters are shared across all channels; ω i The specific formula is:

[0055]

[0056] Where, σ(·) represents the Sigmoid activation function. Represents y i The channel attention ω is realized by a 1D convolution with a convolution kernel of k. The specific formula is:

[0057] ω=σ(C1D K (y))

[0058] Among them, C1D represents one-dimensional convolution. Given the channel dimension C, the size of the convolution kernel is calculated. The specific formula is as follows:

[0059]

[0060] Among them, |t odd represents the nearest odd integer of t, and the values ​​of γ and b are set to 2 and 1;

[0061] S24: A global refinement network with a large receptive field based on a semi-instance normalization block is used to ensure that the repaired area has a similar distribution to the normal area, and further perform feature enhancement; wherein the specific process of the global refinement network with a large receptive field based on a semi-instance normalization block is as follows:

[0062] The semi-instance normalization block is introduced to enhance the features, further improve the modeling ability of the network, ensure that the repaired area has a similar distribution to the normal area, and refine the local network Net L The output of The original damaged mural image I in Merge to get the merged image Then merge the images Connected with the binary mask image M as a global refinement network Net G Input, global refinement network Net G Using Net C The network architecture is improved by adding HIN blocks in the encoder and using context attention modules in the decoder, which enables the network to learn effective features with high matching degree from distant feature spaces.

[0063] S3: Performing restoration experiments on simulated damaged murals and real damaged murals using the trained restoration model to complete the restoration of damaged murals; wherein, performing restoration experiments on simulated damaged murals and real damaged murals using the trained restoration model to complete the restoration of damaged murals includes:

[0064] S31: Perform ablation experiments on the proposed modules and evaluate the restoration results quantitatively and qualitatively; wherein the proposed modules are added to the benchmark model in turn, and the model training is completed through S2;

[0065] S32: Using the training model generated in S2, the optimized restoration model is used to simulate the restoration experiment of damaged murals, and the restoration results are evaluated quantitatively and qualitatively;

[0066] S33: Using the training model generated in S2, the optimized restoration model is used to conduct a restoration experiment on a real damaged mural, and the restoration results are evaluated quantitatively and qualitatively;

[0067] S4: Make a quality assessment of the restoration results; after the restoration model training is completed, the training model generated by S2 is used to perform restoration evaluation on the damaged mural data of S1, and the restoration evaluation includes subjective evaluation and objective evaluation.

[0068] (1) The network structure of the present invention is simple, and the channel attention residual block ECA-ResBlock is used to accelerate the convergence speed of the repair model and improve the repair performance of the model;

[0069] (2) A Half Instance Normalization Block (HIN Block) is used to ensure that the repaired area has a similar distribution to the normal area, effectively enhancing the features and further improving the modeling ability of the network, which is helpful for mural restoration;

[0070] (3) The reference-guided restoration method improves the generalization ability of the model and promotes the restoration of large areas of damaged murals.

[0071] Example 2: The present invention sends real murals, binary masks and reference murals into the model to complete the training of the model. Among them, the input image size is 256×256, the training data set is 500, the test data set is 100, and the total is 600. According to the proportion of holes in the image, the mask image is divided into four categories (1% to 10%, 10% to 20%, 20% to 30%, 30% to 40%). The experiment was carried out under Ubuntu 21.04, and python 3.7 and pytorch 1.10.0 were used to build a deep learning environment for the experiment. The hardware environment is Intel (R) Core i7-11700K CPU @ 3.60GHz, the GPU is NVIDIA RTX3080Ti, and the comparative experiments are all carried out under the same configuration environment. The Adam optimizer is used to train the model, and a total of 200 epochs are trained. The learning rate of the first 100 epochs is 0.0002, and the learning rate of the last 100 epochs decays linearly to 0.

[0072] like Figure 1 As shown, this embodiment provides a Dunhuang mural restoration method based on local and global refinement, including the following steps:

[0073] 1) Data preprocessing

[0074] The real mural, damaged mural and reference mural are cropped to 256×256 size and fed into the model. The model merges the real mural with the binary mask to simulate the damaged mural.

[0075] 2) Repair model training

[0076] The restoration model is continuously iteratively optimized to learn the characteristics of the mural dataset and optimize the model weight parameters. Figure 1 As shown, the key steps in model training are as follows:

[0077] Step 201) Feature encoding and decoding: The coarse patching network uses a U-Net network architecture with skip connections. The network consists of eight downsampling and upsampling operations. The skip connection propagates information from the encoder to the decoder by connecting features of different scales in the channel dimension to recover the feature information lost during network downsampling.

[0078] The output of the coarse repair network is merged with the original damaged mural image to obtain a merged image, that is, the corresponding area of ​​the coarse repair network output is replaced with the valid (undamaged) area in the original damaged image, and then the merged image is connected with the binary mask image as the input of the local refinement network. The local refinement network consists of two downsampling operations, four residual blocks based on ECA channel attention, and two upsampling operations. Due to the shallowness of the network, the network has a small receptive field (each output neuron is 109×109). Then the network refines and repairs the local area of ​​the merged image in a sliding window manner.

[0079] Step 202) Alleviate the vanishing gradient: The residual module can speed up the convergence of the repair model and improve the repair performance of the model. Studies have shown that avoiding dimensionality reduction is very important for learning channel attention. Appropriate cross-channel interaction can significantly reduce the complexity of the model while maintaining performance. The residual block ECA-ResBlock based on ECA channel attention effectively implements local cross-channel interaction without dimensionality reduction through one-dimensional convolution, improving the repair effect. The network structure of the ECA-ResBlock module is as follows: Figure 2 As shown;

[0080] Step 203) Semi-instance normalization: After the coarse repair result is repaired by the local refinement network based on channel attention, some visual artifacts are properly eliminated under the guidance of the local area around the damage. However, when the network captures feature information from a relatively large surrounding area, large structures or long-distance details can be better refined. To this end, we adopt a global refinement network based on contextual attention with a large receptive field (such as Figure 2 In the third stage, the range of information captured by neurons is expanded. At the same time, in order to ensure that the repaired area has a similar distribution to the normal area, the Half Instance Normalization Block (HIN Block) is introduced for feature enhancement to further improve the modeling ability of the network.

[0081] 3) Damaged mural restoration experiment

[0082] In order to verify the generalization of the restoration model, a restoration experiment was conducted on the Dunhuang murals dataset, and objective analysis was performed using peak signal-to-noise ratio (PSNR), structural similarity (SSIM), L1, and perceptual similarity (LPIPS). The larger the PSNR and SSIM values, the better the restoration effect, and the smaller the L1, LPIPS, and BRISQUE values, the better the restoration effect.

[0083] Specifically, step 3) includes the following steps:

[0084] 301) Conduct ablation experiments on the proposed module and evaluate the repair results quantitatively and qualitatively;

[0085] 302) using the training model generated in step 2), using the optimized restoration model to simulate the damaged mural restoration experiment, and evaluating the restoration results quantitatively and qualitatively;

[0086] 303) Using the training model generated in step 2), the optimized restoration model is used to conduct a real damaged mural restoration experiment, and the restoration results are evaluated quantitatively and qualitatively.

[0087] 4) Quality assessment of repair results

[0088] After the restoration model training is completed, the training model generated in step 2) is used to perform restoration evaluation on the damaged mural data in step 1), including subjective evaluation and objective evaluation. Subjective evaluation is to evaluate the restoration effect of the mural through human eye observation. Objective evaluation is to evaluate the restored mural through a general image evaluation algorithm. The evaluation indicators include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), L1 mean error, and perceptual similarity (LPIPS) to evaluate the restoration results of the real damaged mural.

[0089] The present invention is not limited to the above-mentioned embodiments. Anyone should be aware that any structural changes made under the enlightenment of the present invention, and any technical solutions that are the same or similar to the present invention, fall within the protection scope of the present invention. The technology, shape, and structural parts not described in detail in the present invention are all well-known technologies.

Claims

1. A local and global refinement method for restoring Dunhuang murals, characterized in that: The steps include: S1: Preprocessing of real murals, damaged murals and reference murals images; S2: Input the preprocessed data set into the repair model and train the model through continuous iteration; S3: Use the trained restoration model to conduct restoration experiments on simulated damaged murals and real damaged murals to complete the restoration of damaged murals; S4: Make a quality assessment of the restoration results.

2. The method for restoring Dunhuang murals with local and global refinement according to claim 1, characterized in that: In S2, training the model specifically includes: S21: Encode the input mural into the feature representation of the latent space, and complete the mural restoration through decoding operation; S22: Use a repair network with a large receptive field to learn the overall structure of the image and complete the rough repair of the damaged image; S23: Use the residual block of channel attention to learn the feature information around the damaged area and refine the texture information of the damaged edge, further weakening the impact of invalid features in the distance on the repair result; S24: A global refinement network with a large receptive field based on a semi-instance normalization block is used to ensure that the repaired area has a similar distribution to the normal area, and further perform feature enhancement.

3. The method for local and global refinement of Dunhuang mural restoration according to claim 2 is characterized in that: In S21, the specific steps of the decoding operation include: S211: A network consisting of eight downsampling and eight upsampling operations; S212: Skip connections propagate information from the encoder to the decoder by connecting features of different scales in the channel dimension to recover the feature information lost during network downsampling.

4. The method for repairing Dunhuang murals with local and global refinement according to claim 2 is characterized in that: In step S22, the rough restoration network is based on the damaged mural image I in and a binary mask image M describing the damaged area as input. The input image resolution is 256×256. At the end of the encoder, the receptive field is 766×766, which is larger than the image input resolution, which is conducive to learning the entire structure of the image. c The output of The original damaged mural image I in Merge to get the merged image Replace the corresponding area of ​​the coarse repair network output with the valid area in the original damaged image, and connect the merged image with the binary mask image M as the local refinement network Net L Input.

5. The method for repairing Dunhuang murals with local and global refinement according to claim 2 is characterized in that: In S23, the specific steps of using the residual block of the channel attention to learn the feature information around the damaged area and refine the texture information of the damaged edge include: S231: ECA-ResBlock, a residual block based on ECA channel attention, effectively implements local cross-channel interaction without dimensionality reduction through one-dimensional convolution, thus improving the restoration effect; S232: The feature is input into the ECA module and recorded as feature y after global average pooling on the channel without dimensionality reduction. ECA captures local cross-channel interaction information by considering each channel and its k adjacent channels. Then y i The weight ω i By y i The computation is based on the interactions between its k neighboring channels, and parameters are shared across all channels.

6. The method for repairing Dunhuang murals with local and global refinement according to claim 5, characterized in that: ω i The specific formula is: Where, σ(·) represents the Sigmoid activation function. Represents y i The channel attention ω is realized by a 1D convolution with a convolution kernel of k. The specific formula is: ω=σ(C1D K (y)) Among them, C1D represents one-dimensional convolution. Given the channel dimension C, the size of the convolution kernel is calculated. The specific formula is as follows: Among them, |t odd To represent the nearest odd integer of t, the values ​​of γ and b are set to 2 and 1.

7. The method for local and global refinement of Dunhuang murals restoration according to claim 2, characterized in that: In S24, the specific process of the global refinement network with a large receptive field based on the semi-instance normalization block is: The semi-instance normalization block is introduced to enhance the features, further improve the modeling ability of the network, ensure that the repaired area has a similar distribution to the normal area, and refine the local network Net L The output of The original damaged mural image I in Merge to get the merged image Then merge the images Connected with the binary mask image M as a global refinement network Net G Input, global refinement network Net G Using Net C The network architecture is modified, and a HIN block is added to the encoder. The context attention module is used in the decoder, so that the network can learn effective features with high matching degree from farther feature space.

8. The method for local and global refinement of Dunhuang mural restoration according to claim 1 is characterized in that: In S3, the trained restoration model is used to perform restoration experiments on simulated damaged murals and real damaged murals. The specific steps to complete the restoration of damaged murals include: S31: Conduct ablation experiments on the proposed module and evaluate the restoration results quantitatively and qualitatively; S32: Using the training model generated in S2, the optimized restoration model is used to simulate the restoration experiment of damaged murals, and the restoration results are evaluated quantitatively and qualitatively; S33: Using the training model generated in S2, the optimized restoration model is used to conduct a real damaged mural restoration experiment, and the restoration results are evaluated quantitatively and qualitatively.

9. The method for repairing Dunhuang murals with local and global refinement according to claim 8, characterized in that: In S31, the proposed modules are added to the baseline model in sequence, and the model training is completed through S2.

10. The method for local and global refinement of Dunhuang mural restoration according to claim 1, characterized in that: In S4, after the training of the restoration model is completed, the training model generated by S2 is used to perform restoration evaluation on the damaged mural data of S1, and the restoration evaluation includes subjective evaluation and objective evaluation.

Images