Interactive Thangka image restoration method based on line manuscript restoration

Abstract

The invention relates to the technical field of image restoration, and provides an interactive Thangka image restoration method based on line manuscript restoration, which comprises the following steps of: 1, an interactive mask drawing and line manuscript restoration stage: drawing a mask according to a damaged area of Thangka, generating a corresponding line manuscript graph, and performing interactive restoration on the damaged area of the line manuscript; the damaged Thangka image, the repaired complete line draft image and the mask image are input into a style texture repair network together, the original artistic style and texture features of the Thangka are recovered through style texture repair, and then a fine repair network is used for processing detail parts to obtain a final repair result. According to the invention, Thangka image restoration can be well carried out.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an interactive thangka image restoration method based on line draft repair. Background Art

[0002] After years of development, Regong art has gradually formed an artistic style with unique regional characteristics. Regong art has rich and diverse forms of expression, including thangka, murals, embroidery, clay sculptures, architectural painting and other painting and plastic arts. These works are mainly based on Buddhist stories, historical figures and myths and legends. With their unique aesthetic concepts and painting styles, they not only occupy an important position in the field of Tibetan Buddhist art, but also have a profound historical background and cultural value in the fields of folk art and architectural art.

[0003] Regong art has gradually won the love of art lovers around the world with its exquisite composition, gorgeous colors and delicate lines. As an important part of Regong art, Thangka art has appeared in large numbers and spread widely. On the one hand, it is because the raw materials of Thangka canvas, such as silk and satin, can be easily obtained. On the other hand, it is related to the religious belief of the people in the Tibetan cultural area and their nomadic lifestyle, because the easy-to-carry characteristics of Thangka can meet their daily religious needs. Thangka mainly uses natural mineral materials as pigments, including gold, silver, cinnabar, sapphire, agate, etc., so the Thangka painted is bright in color and beautiful in lines. If properly preserved, it can remain bright for a long time without fading. Most painters engaged in Thangka art have learned Thangka painting since childhood. Because there are a lot of long lines in Thangka images, painters need to spend a lot of time practicing and forming memories to ensure that they can complete it in one stroke during the Thangka painting process. A good Thangka work often requires multiple processes and is completed through the cooperation of many people after several years of painting. Based on the above reasons, Thangkas often have high historical and cultural value, artistic value and economic value, and therefore require digitalization and other means to protect and pass on.

[0004] During the field research, due to the use of natural mineral pigments, if not stored properly, the surface of the thangka is prone to cracks; and for some ancient temples, water seepage is prone to occur on rainy days, causing rainwater to drip onto the thangka. If not handled in time, it may cause water stains, discoloration, mold and other problems on the thangka. For these damaged and contaminated thangkas, locals usually use traditional manual repair methods to repair them, based on the content and texture of the undamaged area on the thangka and the experience of the painter. There are only a handful of painters who can repair thangkas, which requires the painter to accumulate experience for a long time, and the repair efficiency is extremely low, there are risks, and it is easy to cause the thangka to be damaged again.

[0005] The above problems have seriously hindered the dissemination and development of Regong art. In addition, thangkas also carry the history and culture of the Regong region and record all aspects of the lives of local people. Through the study of thangkas, we can better understand, protect and inherit Regong culture. With the continuous development of computer vision, we can use digital methods to restore thangka images and realize the protection and inheritance of Regong art.

[0006] Traditional image restoration methods are mainly based on two methods: patch and pixel diffusion. These two methods usually fill the damaged area according to the pixels around the damaged area. When repairing, problems such as unreasonable repair content and blurred texture are prone to occur. Automatic image restoration based on deep learning is mainly divided into three categories: methods based on convolutional neural networks, methods based on generative adversarial networks, and methods based on Transformer networks. After the thangka image is repaired using the currently popular enhancement and restoration algorithms, the texture lines will be blurred and the restoration will be dislocated. The complex and diverse texture structure of the thangka makes it impossible for the model to learn its unique structure and texture features. In addition, there are significant differences between natural images and thangkas in content and color, and the number of samples of natural images far exceeds that of thangka digital images. In addition, given the particularity of thangka painting and Buddhism, directly applying deep learning restoration algorithms may produce unreasonable results that do not meet cultural and traditional requirements. Therefore, in the process of thangka image restoration, some human-computer interaction or the introduction of professional knowledge are used to guide the restoration of textures, edges, etc. in advance to improve the restoration effect of thangka images. Summary of the invention

[0007] The present invention provides an interactive thangka image restoration method based on line drawing repair, which can overcome certain defects of the prior art.

[0008] The interactive thangka image restoration method based on line draft repair according to the present invention comprises the following steps:

[0009] 1. Interactive mask drawing and line drawing repair stage: Draw a mask according to the damaged area of ​​the thangka, generate the corresponding line drawing, and manually repair the damaged area of ​​the line drawing;

[0010] 2. Restoration stage: The damaged thangka image, the repaired complete line drawing and the mask image are input into the style texture restoration network together. First, the style texture restoration is performed to restore the original artistic style and texture characteristics of the thangka. Then, the fine restoration network is used to process the details to obtain the final restoration result.

[0011] Preferably, in step one, specifically: for the input damaged thangka image I, firstly draw a mask map M according to the damaged area to clarify the range that needs to be repaired; then, based on the mask map, extract the image edge of the undamaged area through the Canny algorithm to generate the corresponding line drawing image; finally, interactively repair the damaged area of ​​the line drawing to finally obtain the repaired complete line drawing E.

[0012] Preferably, in step 2, specifically: the input images are the damaged thangka image I, the mask image M obtained in the first stage, and the repaired line drawing E; first, the three images are fused and input into the style texture restoration network to obtain a restoration result with unified texture style, and then the features are fused with the line drawing and input into the fine restoration network for refined restoration to obtain the final restoration result.

[0013] As a preferred method, in the style texture restoration network, first, two layers of gated convolution are used for preprocessing, and then an encoder including transposed convolution, normalization layer, ReLU activation function and pooling layer is constructed to extract key features in the image;

[0014] A spatial style attention module is introduced in the jump connection part to focus on the key spatial areas in the current restoration task.

[0015] Preferably, in the fine restoration network, the line drawing E and the restoration image generated by the style texture restoration part are firstly feature fused again, and then these feature information are input into the fine restoration module; the fine restoration network introduces a linear attention mechanism in the downsampling stage, aiming to capture the local and global dependencies of the input feature map; the encoding part combines depthwise separable convolution and ordinary convolution;

[0016] Add residual blocks in the skip connection part to create channels across layers.

[0017] Preferably, the loss function includes reconstruction loss, style loss, perceptual loss and adversarial loss;

[0018] Reconstruction loss L re Refers to the output image I out Compared with the real image I gt The average of the absolute differences between , which is expressed as follows:

[0019] L re =||I out -I gt ||1

[0020] Style loss L style as follows:

[0021]

[0022] in, It means averaging the losses of different feature layers j to get the average value, Represented by the corresponding activation function φ j Composition of C j ×C j The Gram matrix:

[0023]

[0024] Among them, h represents the feature map height index, w represents the feature map width index, c, c' represents the channel index of the feature map, C j , H j , W j They represent the number of channels, height and width of the feature map on the jth layer of VGG19 pre-trained on ImageNet, respectively, j (x) h,w,c represents the output at position (h, w, c) of the jth layer of VGG19;

[0025] Perceptual loss L per as follows:

[0026]

[0027] in, Represents the average perceptual loss of the entire image to obtain the overall loss, N i represents the number of elements in the feature map of the i-th layer, φ i is the activation function of the i-th layer of VGG19;

[0028] Adversarial loss L adv as follows:

[0029]

[0030] in, and They represent the real image I gt And generate image I out The expectation of , D(·) represents the output of the discriminator;

[0031] In summary, the total loss function is as follows:

[0032] L total =λ a L adv +λ b L re +λ c L per +λ d L style

[0033] Among them, setting the scale parameter λ a =0.1,λ b =λ c =1,λ d =250.

[0034] The present invention designs a thangka image restoration network, which is divided into two stages. The first stage is interactive mask drawing and line draft repair; the second stage is divided into a style texture restoration part and a fine restoration part, wherein a spatial style module is added to the style texture restoration part to learn the overall style and texture features of the thangka image, and then the style texture restoration result is fused with the line draft map, and the fused feature map is input into the fine restoration part to eliminate the restoration traces, further improve the image restoration effect, and obtain the final restoration result. Through experimental comparative analysis with several popular enhancement and restoration methods, the present invention has achieved the best results in the three indicators of PSNR, SSIM and LPIPS, and has a good visualization effect. In addition, an ablation experiment is also carried out to further verify the effectiveness of the three modules of interactive line draft restoration, style texture restoration and fine restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of an interactive thangka image restoration method based on line draft repair in an embodiment;

[0036] Figure 2 Schematic diagram of a style texture restoration network in an embodiment;

[0037] Figure 3 A schematic diagram of a fine repair network in an embodiment;

[0038] Figure 4 The mask diagram in the embodiment (from left to right: crack fold, flake damage, water stain and mildew spot);

[0039] Figure 5 This is a schematic diagram of the visual comparison results in the embodiment;

[0040] Figure 6 This is a visualization result diagram of the ablation experiment in the embodiment. DETAILED DESCRIPTION

[0041] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments are only for explaining the present invention and are not intended to limit it.

[0042] Example

[0043] like Figure 1 As shown, this embodiment provides an interactive thangka image restoration method based on line draft repair, which includes the following steps:

[0044] 1. Interactive mask drawing and line drawing repair stage: Draw a mask according to the damaged area of ​​the thangka, generate the corresponding line drawing, and perform manual interactive repair on the damaged area of ​​the line drawing under the guidance of thangka intangible cultural heritage inheritors or thangka artists such as arts and crafts masters;

[0045] 2. Restoration stage: The damaged thangka image, the repaired complete line drawing and the mask image are input into the style texture restoration network together. First, the style texture restoration is performed to restore the original artistic style and texture characteristics of the thangka. Then, the fine restoration network is used to process the details to obtain the final restoration result.

[0046] Interactive mask drawing and line drawing repair module

[0047] Due to the complex and varied lines in thangka images, and the different painting styles displayed by different painters, it is difficult for current automatic restoration technology to accurately restore the edge texture of thangka images. In addition, as a unique art form, thangka has its own inherent norms and standards. If the restoration method of ordinary scenes is used, it is likely that the restoration result will not match the content of the thangka and cannot maintain its original artistic characteristics.

[0048] Therefore, an interactive method was used to complete the restoration of the line drawings. In this process, the guidance of thangka artists such as thangka intangible cultural heritage inheritors or arts and crafts masters played a vital role. With their profound artistic attainments and rich experience, they were able to accurately repair the line drawings of the damaged areas, ensuring that the repaired line drawings were not only structurally complete, but also consistent in style, laying the foundation for subsequent restoration work, thereby achieving the best restoration effect.

[0049] For the damaged thangka image I, a mask map M is first drawn based on the damaged area to clarify the scope that needs to be repaired. Then, based on the mask map, the image edge of the undamaged area is extracted through the Canny algorithm to generate the corresponding line drawing image. Finally, under the guidance of thangka intangible cultural heritage inheritors or thangka artists such as arts and crafts masters, the damaged area of ​​the line drawing is further carefully repaired to ensure the integrity and accuracy of the line drawing. Finally, the complete repaired line drawing E is obtained.

[0050] Image Inpainting Network

[0051] The whole network is divided into two parts: style texture restoration network and fine restoration network. The input images are the damaged thangka image I, the mask image M obtained in the first stage, and the repaired line drawing E. First, the three images are fused and input into the style texture restoration network to obtain a restoration result with unified texture style. Then, the features are fused with the line drawing and input into the fine restoration network for refined restoration to obtain the final restoration result.

[0052] Style Texture Inpainting Network

[0053] like Figure 2 As shown in the figure, first, two layers of gated convolution are used for preprocessing. In the repair task, the pixels in the mask area are regarded as invalid pixels and do not need special processing. The uniqueness of gated convolution is that it can dynamically adjust the activation values ​​of different positions during the convolution process, making them more sensitive to the current repair task. This mechanism gives the network stronger learning ability, enabling it to capture more complex feature representations, thereby improving the nonlinear expression ability of the model.

[0054] Next, we build an encoder that includes transposed convolution, normalization layer, ReLU activation function and pooling layer to extract key features in the image. Upsampling in the decoder helps to recover high-quality image details from the features extracted in the encoding stage.

[0055] The spatial style attention module is introduced in the jump connection part to focus on the key spatial areas in the current restoration task. By efficiently aggregating spatial information, the representation ability of the model is significantly enhanced, enabling it to more accurately capture the complex structures and patterns in the image, and further learn the overall style characteristics of the thangka image.

[0056] As the training progresses, this module automatically forms a sophisticated hierarchical structure, making the image representation more efficient and accurate. It can effectively represent the details of the image, thereby extracting more valuable information and realizing a multi-level attention mechanism, which helps the model better capture the complex texture structure and patterns in thangka images.

[0057] In general, the spatial style attention module performs preliminary and effective repair of damaged areas by learning the overall style and texture features of thangka images. This module can not only repair the approximate image outline of the damaged area, but also significantly improve the performance and accuracy of the model. In addition, combining channel attention and fully connected layers helps to further capture global information and perform feature fusion. This way of fusing contextual information enables the model to more accurately predict the content of the mask area, thereby further improving the repair effect. Finally, the network obtains the preliminary repair result I1.

[0058] Fine Repair Network

[0059] like Figure 3As shown in the figure, the line drawing E and the repaired image generated by the style texture repair part are firstly feature fused again, and then these feature information are input into the fine repair module; to further improve the repair quality, make the texture more delicate, and eliminate the traces generated in the repair process as much as possible. In addition, the network introduces a linear attention mechanism in the downsampling stage, aiming to capture the local and global dependencies of the input feature map. This attention mechanism can efficiently solve the association problem between features of different scales and realize the effective transmission of global information, which helps the model to understand and represent the content of thangka images more deeply. At the same time, the attention mechanism significantly enhances the representation ability of the feature map, enabling the model to more accurately capture the complex texture structure features in the image. By combining the attention mechanism with the downsampling technology, not only more image details and contextual information are retained while reducing the spatial size of the feature map, but also the robustness of the model is improved. In addition, the learnable parameters in the attention mechanism allow the model to adaptively adjust the degree of association between features of different scales according to task requirements, so as to learn more useful feature representations.

[0060] In short, the introduction of the linear attention mechanism in the downsampling process not only effectively enhances the representation ability and robustness of the model, but also provides richer and more accurate feature information for subsequent repair tasks. In addition, the encoding part combines depthwise separable convolution and ordinary convolution. This hybrid method significantly reduces the amount of calculation, thereby speeding up the calculation speed of the model and making the entire repair process more efficient.

[0061] As the depth of the network increases, the expressive power of the model will be enhanced, but at the same time it may be affected by problems such as gradient vanishing or gradient exploding, so residual blocks are added to the skip connection. By introducing residual blocks, a certain number of layers can be effectively skipped to achieve direct connection, which greatly alleviates these problems and avoids the performance degradation caused by training deep neural networks.

[0062] The residual block creates a cross-layer channel in the network, allowing the signal to be transmitted more smoothly. In this way, the network can use multiple convolution kernels to extract information from the image at different scales and fuse this information to obtain a richer and more accurate image representation. The residual learning method enables the network to better fuse feature information at different levels. This fusion of multi-scale information is crucial for the model to understand and process complex thangka image features. In addition, the introduction of the residual block also brings an additional benefit, which can improve the performance of the model to a certain extent and speed up the training speed, which enables the network to learn and optimize more efficiently when repairing thangka images.

[0063] Discriminator

[0064] The discriminator uses the PatchGAN discriminator. The PatchGAN discriminator is a discriminant model based on convolutional neural networks, which is used for tasks such as image generation and style transfer. Compared with traditional classification models, the structure of the PatchGAN discriminator is completely composed of convolutional layers, and the output is an n×n matrix rather than a single scalar value. Each element of this output matrix corresponds to a receptive field in the input image, representing the probability of whether the receptive field is real or false. Therefore, the PatchGAN discriminator can be regarded as a fine texture style loss to ensure that the generated image maintains high resolution and delicate texture. The PatchGAN discriminator divides the input image into multiple receptive fields, and performs independent binary classification judgment on each receptive field to identify whether it has the texture features of the target image. This design enables the PatchGAN discriminator to effectively supervise the generated image at a pixel-by-pixel level, thereby improving the quality and resolution of the generated image. At the same time, since the PatchGAN discriminator mainly focuses on the texture feature information of the local area, it can better adapt to image inputs of different sizes and shapes, and has strong generalization ability.

[0065] Loss Function

[0066] The loss functions include reconstruction loss, style loss, perceptual loss, and adversarial loss;

[0067] Reconstruction loss L re Refers to the output image I out Compared with the real image I gt The average of the absolute differences between , which is expressed as follows:

[0068] L re =||I out -I gt ||1

[0069] The core role of style loss is to ensure that the restored image is highly consistent with the original image in style. The Gram matrix is ​​used to accurately measure the subtle differences between different feature maps to ensure the consistency of style between images. Specifically, style loss focuses on the characteristics expressed by the feature maps output by each layer in the image. In a deep convolutional neural network, the feature maps of each layer show diverse characteristics, among which the shallow feature maps mainly capture detailed information, while the deep feature maps tend to express abstract concepts.

[0070] Style loss ensures that the restored image is consistent with the original image in terms of style by carefully comparing the differences between the feature maps of the image to be restored and the original image at multiple different levels. In the image restoration process, style loss plays a vital role, helping the network to better preserve the texture and detail information of the image, making the restoration result more natural and closer to human visual aesthetics. Combining style loss with perceptual loss can further improve the visual quality and perceptual quality of the restoration result. Style loss L style as follows:

[0071]

[0072] in, It means averaging the losses of different feature layers j to get the average value, Represented by the corresponding activation function φ j Composition of C j ×C j The Gram matrix:

[0073]

[0074] Among them, h represents the feature map height index, w represents the feature map width index, c, c' represents the channel index of the feature map, C j , H j , W j They represent the number of channels, height and width of the feature map on the jth layer of VGG19 pre-trained on ImageNet, respectively, j (x) h,w,c Represents the output at the (h, w, c)th position of the jth layer of VGG19.

[0075] Perceptual loss is committed to improving the perceived quality of images, rather than just being limited to pixel-level differences. In the image restoration process, perceptual loss plays a vital role, as it can effectively preserve the rich details and delicate textures in the image, thereby significantly improving the visual effect of the restoration result.

[0076] In addition, perceptual loss also helps the network learn more semantically meaningful feature representations, which makes the restoration results more natural and smooth, and more in line with human visual aesthetics. Compared with the traditional pixel-based loss function, perceptual loss shows more excellent performance in processing complex structures and detailed information in images, which can further improve the accuracy and overall quality of restoration. per as follows:

[0077]

[0078] in, Represents the average perceptual loss of the entire image to obtain the overall loss, N i represents the number of elements in the feature map of the i-th layer, φ i is the activation function in the i-th layer of VGG19.

[0079] Adversarial loss L adv as follows:

[0080]

[0081] in, and They represent the real image I gt And generate image I out , and D(·) represents the output of the discriminator.

[0082] In summary, the total loss function is as follows:

[0083] L total =λ a L adv +λ b L re +λ c L per +λ d L style

[0084] Among them, setting the scale parameter λ a =0.1,λ b =λ c =1,λ d =250.

[0085] experiment

[0086] The thangka data used in this experiment also comes from thangka images collected during field research. 300 images with high imaging quality were selected from the captured images, and then these images were cropped and corrected: the redundant parts were removed, and then these thangka images were cut into 256×256 resolution images, resulting in a total of 25,000 images, which were divided into training set, test set and validation set at a ratio of 8:1:1. The Canny algorithm was used to extract the edges of the cropped images to obtain the corresponding edge line draft data set.

[0087] Different from ordinary mask datasets, thangka images usually have the following types of damage and pollution: crack-like creases, point-like damage, sheet-like damage, water stains, mildew spots, etc. Therefore, this embodiment uses Photoshop to simulate these types of damage shapes, create a mask dataset, and cut it into a 256×256 size mask image, as shown in the figure. Figure 4As shown in the figure, a total of 5000 mask images are obtained. At the same time, 1000 more mask images are selected from the public mask dataset, totaling 6000 mask images, which are randomly divided into training set and test set at a ratio of 5:1.

[0088] This experiment uses supervised training mode, with generator learning rate of 1e-4, discriminator learning rate of 1e-5, batch size = 8, and Adam optimizer. This experiment was completed on a cloud server, with hardware environment: Intel (R) Xeon (R) Gold 6330CPU, NVIDIA RTX 3090GPU 24GB; software environment: python = 3.8, pytroch = 1.8.1, CUDA = 11.1.

[0089] A thangka dataset was used for image restoration comparison experiments. The comparison methods included several advanced image restoration methods in recent years: DeepFillv2, EdgeConnect, DFNet, HiFill, and T-Former, to verify the reliability and superiority of the method of this embodiment in thangka restoration.

[0090] Quantitative comparison

[0091] This experiment uses three evaluation indicators: PSNR, SSIM, and LPIPS. PSNR is used to represent the ratio between the maximum possible power of a signal and the destructive noise power of its representation accuracy. The larger the value, the closer the generated image is to the original image. SSIM is an indicator that measures the similarity of three features of brightness, contrast, and structure in two images. The larger the value, the smaller the image distortion. LPIPS evaluates the perceptual similarity of two images by comparing their distance in the feature space of a deep neural network. The smaller the value, the more similar the generated image is to the original image.

[0092] Table 1 PSNR, SSIM, LPIPS values ​​of different methods on the Tangka dataset

[0093] method PSNR SSIM LPIPS DeepFillv2 30.4801 0.9434 0.044 EdgeConnect 30.6045 0.9467 0.040 DFNet 30.3620 0.9417 0.053 HiFill 27.4012 0.9125 0.066 T-Former 31.4755 0.9513 0.038 Our 34.7967 0.9685 0.016

[0094] Table 1 lists the PSNR, SSIM and LPIPS values ​​of different methods on the Thangka dataset. Through data analysis, it can be found that on the Thangka dataset, the method of this embodiment has achieved the best results in all three indicators, indicating that the restoration result of this embodiment is closer to the original image, proving that the method proposed in this embodiment has achieved a good restoration effect on the Thangka dataset.

[0095] Qualitative comparison

[0096] Figure 5The results of repairing different methods on the thangka dataset are shown. From the results, it can be seen that DeepFillv2 has obvious blurring in the repair of the 3rd and 4th images, and the texture is not clear. There are obvious repair marks in the 5th image. EdgeConnect also has obvious unclear texture in the repair of the 3rd and 4th images. This is mainly because the algorithm adopts the method of automatically repairing the line draft. Unlike natural images, the edge styles of thangka images are diverse, and automatic repair cannot accurately repair the edge of the line draft. DFNet and HiFill have obvious repair marks, and the repaired part does not conform to the thangka image. T-Former has obvious blurring in the repair of the 3rd, 4th and 6th images, and the texture is not clear. The method of this embodiment uses interactive line draft repair to constrain the repair result. Therefore, compared with the algorithm without adding line draft constraints, the repair result of this embodiment has coherent and clear texture, no obvious repair marks, and better visual effect; because EdgeConnect adopts the method of automatic line draft repair, it may produce unreasonable line draft repair results, resulting in poor repair effect. This embodiment adopts interactive line draft repair, so the repair result is closer to the real thangka image.

[0097] Ablation experiment

[0098] In order to verify the effectiveness of adding interactive line draft repair, style texture repair and fine repair in the method of this embodiment, this embodiment conducted an ablation experiment, in which the experiments were conducted under three conditions: removing interactive line draft repair, removing style texture repair and removing fine repair. The ablation experiments were conducted under the same parameters, hardware and software conditions. Table 2 shows the PSNR, SSIM and LPIPS results on the thangka dataset under three conditions. From the experimental results, it can be seen that after adding interactive line draft repair, style texture repair and fine repair, the quality of the repaired image is significantly improved:

[0099] Table 2 Quantitative results of ablation experiments

[0100]

[0101]

[0102] Figure 6The restoration results of different combinations are shown. It can be found that without adding line draft constraints, the restoration results have obvious blurring and unclear textures. After adding line draft constraints, due to the line draft constraints, the restoration results have clear textures and no other features that do not conform to the thangka image. The line draft plus style texture restoration method has obvious mask-like shape to constrain the restoration results, making its texture features closer to the real thangka texture. The style texture restoration part can learn the style texture features of thangka, and the fine restoration further improves the restoration quality. The combination of the three modules that eliminate restoration traces makes the final restored thangka image content texture clear and without restoration traces, thereby verifying the effectiveness of each module.

[0103] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. An interactive thangka image restoration method based on line draft repair, characterized by: The following steps are involved:

1. Interactive mask drawing and line drawing repair stage: Draw a mask according to the damaged area of ​​the thangka, generate the corresponding line drawing, and interactively repair the damaged area of ​​the line drawing; 2. Restoration stage: The damaged thangka image, the repaired complete line drawing and the mask image are input into the style texture restoration network together. First, the style texture restoration is performed to restore the original artistic style and texture characteristics of the thangka. Then, the fine restoration network is used to process the details to obtain the final restoration result.

2. The interactive thangka image restoration method based on line draft repair according to claim 1 is characterized in that: In step one, specifically: for the input damaged thangka image I, first draw a mask map M according to the damaged area to clarify the range that needs to be repaired; then, based on the mask map, extract the image edge of the undamaged area through the Canny algorithm to generate the corresponding line drawing image; finally, interactively repair the damaged area of ​​the line drawing to finally obtain the repaired complete line drawing E.

3. The interactive thangka image restoration method based on line draft repair according to claim 2 is characterized in that: In step 2, specifically: the input images are the damaged thangka image I, the mask image M obtained in the first stage, and the repaired line drawing E; first, the three images are fused and input into the style texture restoration network to obtain a restoration result with unified texture style, and then the features are fused with the line drawing and input into the fine restoration network for refined restoration to obtain the final restoration result.

4. The interactive thangka image restoration method based on line draft repair according to claim 3 is characterized in that: In the style texture restoration network, first, two layers of gated convolution are used for preprocessing, and then an encoder including transposed convolution, normalization layer, ReLU activation function and pooling layer is constructed to extract key features in the image; A spatial style attention module is introduced in the jump connection part to focus on the key spatial areas in the current restoration task.

5. The interactive thangka image restoration method based on line draft repair according to claim 4 is characterized in that: In the fine restoration network, the line drawing E and the restoration image generated by the style texture restoration part are firstly fused again, and then these feature information are input into the fine restoration module; the fine restoration network introduces a linear attention mechanism in the downsampling stage to capture the local and global dependencies of the input feature map; the encoding part combines depthwise separable convolution and ordinary convolution; Add residual blocks in the skip connection part to create channels across layers.

6. The interactive thangka image restoration method based on line draft repair according to claim 5 is characterized in that: The loss functions include reconstruction loss, style loss, perceptual loss, and adversarial loss; Reconstruction loss L re Refers to the output image I out Compared with the real image I gt The average of the absolute differences between , which is expressed as follows: Style loss L style as follows: in, It means averaging the losses of different feature layers j to get the average value. Represented by the corresponding activation function φ j Composition of C j ×C j The Gram matrix: Among them, h represents the feature map height index, w represents the feature map width index, c, c' represents the channel index of the feature map, C j , H j , W j They represent the number of channels, height and width of the feature map on the jth layer of VGG19 pre-trained on ImageNet, respectively, j (x) h,w,c represents the output at position (h, w, c) of the jth layer of VGG19; Perceptual loss L per as follows: in, Represents the average perceptual loss of the entire image to obtain the overall loss, N i represents the number of elements in the feature map of the i-th layer, φ i is the activation function of the i-th layer of VGG19; Adversarial loss L adv as follows: in, and They represent the real image I gt And generate image I out The expectation of , D(·) represents the output of the discriminator; In summary, the total loss function is as follows: L total =λ a L adv +λ b L re +λ c L per +λ d L style Among them, setting the scale parameter λ a =0.1,λ b =λ c =1,λ d =250.

Images