Interactive thangka image restoration method based on line drawing repair

Abstract

The present invention relates to the field of image restoration technology and provides an interactive thangka image restoration method based on line drawing repair. The method comprises the following steps: 1. An interactive mask drawing and line drawing repair phase: a mask is drawn according to the damaged area of ​​the thangka, a corresponding line drawing is generated, and the damaged line drawing area is interactively repaired; 2. A repair phase: the damaged thangka image, the repaired complete line drawing, and the mask image are input into a style and texture restoration network. The style and texture restoration is first performed to restore the thangka's original artistic style and texture features. Subsequently, a fine repair network is used to further process the details to obtain the final restoration result. The present invention can effectively perform thangka image restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and in particular to an interactive thangka image restoration method based on line draft repair. Background Art

[0002] Over the years, Regong art has gradually developed a unique regional artistic style. Regong art encompasses a rich variety of forms, including thangkas, murals, embroidery, clay sculptures, and architectural paintings. These works primarily draw on Buddhist Jataka stories, historical figures, and myths and legends. With their unique aesthetic concepts and painting styles, they not only hold a significant position in Tibetan Buddhist art but also possess a profound historical heritage and cultural value in fields such as folk art and architecture.

[0003] Regong art, with its exquisite composition, vibrant colors, and delicate lines, has gradually won the admiration of art lovers worldwide. Thangka art, a crucial component of Regong art, has emerged and spread in large numbers. This is partly due to the readily available materials, such as silk and satin, used as canvases for thangkas. It is also related to the widespread religious devotion and nomadic lifestyle of the Tibetan people, as thangkas' portability meets their daily religious needs. Thangkas primarily use natural mineral pigments, including gold, silver, cinnabar, sapphire, and agate. This results in vibrant colors and elegant lines that, when properly preserved, retain their vibrant color over time without fading. Most thangka artists begin learning to paint from a young age. Because thangka images often feature long lines, artists must devote considerable time to practice and memorize the strokes, ensuring they can be executed with a single stroke. A well-crafted thangka often requires multiple steps, years of painting, and the collaborative efforts of many individuals. Based on the above reasons, Thangkas often have high historical and cultural value, artistic value and economic value, and therefore need to be protected and passed on through means such as digitization.

[0004] During the field research, it was discovered that due to the use of natural mineral pigments, thangkas are prone to cracking if not stored properly. Furthermore, some older temples are prone to water seepage on rainy days, causing rainwater to drip onto the thangkas. If not addressed promptly, this can lead to water stains, discoloration, and mold. For these damaged and contaminated thangkas, locals typically use traditional hand-repair methods, based on the texture of the intact areas and the artist's experience. However, there are only a handful of artists who can perform this repair, requiring a long period of experience. Furthermore, the repair process is extremely inefficient, presents risks, and can easily lead to secondary damage to the thangkas.

[0005] These issues have severely hampered the spread and development of Regong art. Furthermore, thangkas carry the history and culture of the Regong region, documenting every aspect of local life. Research on thangkas can help better understand, protect, and pass on Regong culture. With the continuous advancement of computer vision, digital restoration of thangka images is now possible, contributing to the preservation and inheritance of Regong art.

[0006] Traditional image restoration methods primarily rely on patching and pixel diffusion. These methods typically fill damaged areas based on surrounding pixels, which can lead to issues like illogical restoration content and blurred textures. Deep learning-based automatic image restoration can be categorized into three main categories: convolutional neural network-based methods, generative adversarial network-based methods, and Transformer network-based methods. When using currently popular enhancement and restoration algorithms to restore thangka images, artifacts such as blurred texture lines and misaligned restorations can occur. The complex and diverse texture structure of thangkas prevents the model from learning their unique structural and texture features. Furthermore, natural images differ significantly from thangkas in content and color, and the number of natural image samples far exceeds that of digital thangka images. Furthermore, given the unique nature of thangka painting and Buddhism, directly applying deep learning restoration algorithms can produce illogical results that are inconsistent with cultural and traditional requirements. Therefore, in the thangka image restoration process, human-computer interaction or the introduction of specialized knowledge can be used to pre-process and guide restoration of textures, edges, and other aspects to improve the restoration results. Summary of the Invention

[0007] The present invention provides an interactive thangka image restoration method based on line drawing repair, which can overcome certain defects of the prior art.

[0008] The interactive thangka image restoration method based on line drawing repair according to the present invention comprises the following steps:

[0009] 1. Interactive mask drawing and line drawing repair stage: Draw a mask based on the damaged area of ​​the thangka, generate the corresponding line drawing, and manually repair the damaged area of ​​the line drawing;

[0010] 2. Restoration stage: The damaged thangka image, the repaired complete line drawing and the mask image are input into the style texture restoration network. First, the style texture restoration is performed to restore the original artistic style and texture characteristics of the thangka. Then, the fine restoration network is used to further process the details to obtain the final restoration result.

[0011] Preferably, in step one, specifically: for the input damaged thangka image I, first draw a mask map M according to the damaged area to clarify the range that needs to be repaired; then, based on the mask map, extract the image edge of the undamaged area through the Canny algorithm to generate the corresponding line drawing image; finally, perform interactive repair on the damaged area of ​​the line drawing, and finally obtain the repaired complete line drawing E.

[0012] Preferably, in step 2, specifically: the input images are the damaged thangka image I, the mask image M obtained in the first stage, and the repaired line drawing E; first, the three images are fused and input into the style texture restoration network to obtain a restoration result with unified texture style, and then the features are fused with the line drawing and input into the fine restoration network for refined restoration to obtain the final restoration result.

[0013] As a preferred approach, in the style texture restoration network, first, two layers of gated convolution are used for preprocessing. Next, an encoder is constructed that includes a transposed convolution, a normalization layer, a ReLU activation function, and a pooling layer to extract key features from the image.

[0014] A spatial style attention module is introduced in the jump connection part to focus on the key spatial areas in the current restoration task.

[0015] As a preferred approach, in the fine restoration network, the line drawing E is firstly fused with the restored image generated by the style and texture restoration part, and then these feature information is input into the fine restoration module; the fine restoration network introduces a linear attention mechanism in the downsampling stage to capture the local and global dependencies of the input feature map; the encoding part combines depthwise separable convolution and ordinary convolution;

[0016] Add residual blocks to the skip connection part to create cross-layer channels.

[0017] Preferably, the loss function includes reconstruction loss, style loss, perceptual loss and adversarial loss;

[0018] Reconstruction loss L re Refers to the output image I out Compared with the real image I gt The average of the absolute differences between , which is expressed as follows:

[0019] L re =||I out -I gt ||1

[0020] Style loss L style as follows:

[0021]

[0022] in, Indicates averaging the losses of different feature layers j to obtain the average value, Represented by the corresponding activation function φ j Composition of C j ×C j Gram matrix:

[0023]

[0024] Among them, h represents the feature map height index, w represents the feature map width index, c, c' represents the channel index of the feature map, C j , H j , W j denote the number of channels, height, and width of the feature map on the jth layer of VGG19 pre-trained on ImageNet, respectively, j (x) h,w,c represents the output at position (h, w, c) of the jth layer of VGG19;

[0025] Perceptual loss L per as follows:

[0026]

[0027] in, Represents the average perceptual loss of the entire image to obtain the overall loss, N i Indicates the number of elements in the feature map of layer i, φ i is the activation function of the i-th layer of VGG19;

[0028] Adversarial loss L adv as follows:

[0029]

[0030] in, and They represent the real image I gt and output image I out The expectation of , D(·) represents the output of the discriminator;

[0031] In summary, the total loss function is as follows:

[0032] L total =λ a L adv +λ b L re +λ c L per +λ d L style

[0033] Among them, setting the scale parameter λ a =0.1,λ b =λ c =1,λ d =250.

[0034] This paper designs a thangka image restoration network, which is divided into two stages. The first stage is interactive mask drawing and line drawing repair; the second stage is divided into style and texture restoration and fine restoration. The style and texture restoration part incorporates a spatial style module to learn the overall style and texture characteristics of the thangka image. The style and texture restoration result is then fused with the line drawing. The fused feature map is input into the fine restoration part to eliminate the restoration traces, further improve the image restoration effect, and obtain the final restoration result. Through experimental comparative analysis with several currently popular enhancement and restoration methods, this paper achieves the best results in the three indicators of PSNR, SSIM, and LPIPS, and has excellent visualization effects. In addition, ablation experiments are conducted to further verify the effectiveness of the three modules of interactive line drawing restoration, style and texture restoration, and fine restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Flowchart of an interactive thangka image restoration method based on line drawing repair in an embodiment;

[0036] Figure 2 Schematic diagram of a style texture restoration network in an embodiment;

[0037] Figure 3 Schematic diagram of a fine repair network in an embodiment;

[0038] Figure 4 The mask diagram in the embodiment (from left to right: crack crease, flake damage, water stain and mildew spot);

[0039] Figure 5 This is a schematic diagram of the visual comparison results in the embodiment;

[0040] Figure 6 This is a visualization result diagram of the ablation experiment in the embodiment. DETAILED DESCRIPTION

[0041] In order to further understand the content of the present invention, the present invention is described in detail with reference to the accompanying drawings and embodiments. It should be understood that the embodiments are merely for explaining the present invention and are not intended to limit the present invention.

[0042] Example

[0043] like Figure 1 As shown, this embodiment provides an interactive thangka image restoration method based on line drawing repair, which includes the following steps:

[0044] 1. Interactive mask drawing and line drawing repair stage: Draw a mask based on the damaged area of ​​the thangka to generate the corresponding line drawing. Under the guidance of thangka intangible cultural heritage inheritors or thangka artists such as arts and crafts masters, the damaged area of ​​the line drawing is manually repaired.

[0045] 2. Restoration stage: The damaged thangka image, the repaired complete line drawing and the mask image are input into the style texture restoration network. First, the style texture restoration is performed to restore the original artistic style and texture characteristics of the thangka. Then, the fine restoration network is used to further process the details to obtain the final restoration result.

[0046] Interactive mask drawing and line drawing repair module

[0047] Due to the complex and varied lines in thangka images, and the vastly different painting styles displayed by different artists, current automatic restoration techniques struggle to accurately restore the edge textures of thangka images. Furthermore, as a unique art form, thangkas possess inherent norms and standards. Using common restoration methods for common scenes would likely result in a restoration that is inconsistent with the thangka's content and fails to preserve its original artistic characteristics.

[0048] Therefore, an interactive approach was adopted to complete the restoration of the line drawings. The guidance of thangka artists, such as inheritors of the thangka intangible cultural heritage or arts and crafts masters, played a crucial role in this process. Their profound artistic attainments and rich experience enabled them to accurately repair the damaged line drawings, ensuring that the restored line drawings were not only structurally complete but also consistent in style. This laid the foundation for subsequent restoration work, ultimately achieving the best possible results.

[0049] For a damaged thangka image I, a mask M is first drawn based on the damaged areas to identify the areas requiring repair. Based on the mask, the Canny algorithm is used to extract the edges of the intact areas, generating a corresponding line drawing. Finally, under the guidance of thangka artists, such as inheritors of the intangible cultural heritage of thangka or craftsmen, the damaged areas of the line drawing are meticulously repaired to ensure their integrity and accuracy. The final result is the complete repaired line drawing E.

[0050] Image Inpainting Network

[0051] The network as a whole is divided into two parts: the style and texture restoration network and the fine restoration network. The input images are the damaged thangka image I, the mask image M obtained in the first stage, and the repaired line drawing E. First, the three images are fused and input into the style and texture restoration network to obtain a restoration result with unified texture style. Then, the features are fused with the line drawing and input into the fine restoration network for refined restoration to obtain the final restoration result.

[0052] Style Texture Inpainting Network

[0053] like Figure 2 As shown in the figure, first, two layers of gated convolution are used for preprocessing. In the restoration task, pixels in the masked area are considered invalid pixels and do not require special processing. The unique feature of gated convolution is that it can dynamically adjust the activation values ​​of different positions during the convolution process, making them more sensitive to the current restoration task. This mechanism gives the network stronger learning ability, enabling it to capture more complex feature representations, thereby improving the model's nonlinear expression capabilities.

[0054] Next, we build an encoder consisting of transposed convolutions, normalization layers, ReLU activation functions, and pooling layers to extract key features from the image. Upsampling in the decoder helps recover high-quality image details from the features extracted in the encoder phase.

[0055] A spatial style attention module is introduced within the skip connection to focus on key spatial regions in the restoration task. By efficiently aggregating spatial information, the model's representational capabilities are significantly enhanced, enabling it to more accurately capture complex structures and patterns in images, thereby deeply learning the overall stylistic characteristics of thangka images.

[0056] As training progresses, this module automatically forms a sophisticated hierarchical structure, making image representation more efficient and accurate. It can effectively represent image details, thereby extracting more valuable information and realizing a multi-level attention mechanism, which helps the model better capture the complex texture structures and patterns in thangka images.

[0057] In summary, the spatial style attention module learns the overall style and texture characteristics of thangka images to perform preliminary and effective restoration of damaged areas. This module not only restores the approximate image outline of the damaged area but also significantly improves model performance and accuracy. Furthermore, combining channel attention with fully connected layers helps further capture global information and perform feature fusion. This integration of contextual information enables the model to more accurately predict the content of the masked area, further improving the restoration effect. Ultimately, the network achieves preliminary restoration result I1.

[0058] Fine Restoration Network

[0059] like Figure 3As shown, the line drawing E is first combined with the inpainted image generated by the style and texture restoration component for feature fusion. This feature information is then fed into the fine restoration module to further improve restoration quality, refine textures, and minimize artifacts from the restoration process. Furthermore, the network introduces a linear attention mechanism during the downsampling phase to capture local and global dependencies within the input feature map. This attention mechanism efficiently addresses the correlation between features at different scales and effectively transfers global information, helping the model to gain a deeper understanding and representation of thangka image content. Furthermore, the attention mechanism significantly enhances the representational power of the feature map, enabling the model to more accurately capture the complex texture structures within the image. Combining the attention mechanism with downsampling techniques not only reduces the spatial size of the feature map while preserving more image detail and contextual information, but also improves the model's robustness. Furthermore, the learnable parameters in the attention mechanism allow the model to adaptively adjust the correlation between features at different scales based on task requirements, thereby learning more useful feature representations.

[0060] In summary, the introduction of a linear attention mechanism during the downsampling process not only effectively enhances the model's representational capabilities and robustness, but also provides richer and more accurate feature information for subsequent inpainting tasks. Furthermore, the encoding process combines depthwise separable convolution with ordinary convolution. This hybrid approach significantly reduces computational effort, thereby speeding up the model's computation and making the entire inpainting process more efficient.

[0061] As network depth increases, the model's expressive power increases, but it can also be affected by issues like vanishing or exploding gradients. Therefore, residual blocks are added to skip connections. By introducing residual blocks, a certain number of layers can be effectively skipped, enabling direct connections. This significantly alleviates these issues and avoids the performance degradation associated with training deep neural networks.

[0062] The residual block creates a cross-layer channel in the network, allowing signals to be transmitted more smoothly. In this way, the network can use multiple convolution kernels to extract information from the image at different scales and fuse this information to obtain a richer and more accurate image representation. The residual learning method enables the network to better integrate feature information from different layers. This fusion of multi-scale information is crucial for the model to understand and process the complex features of thangka images. In addition, the introduction of the residual block also brings an additional benefit, which can improve the performance of the model to a certain extent and speed up the training speed, which enables the network to learn and optimize more efficiently when repairing thangka images.

[0063] Discriminator

[0064] The discriminator uses the PatchGAN discriminator. The PatchGAN discriminator is a discriminative model based on convolutional neural networks, used for tasks such as image generation and style transfer. Compared to traditional classification models, the PatchGAN discriminator's structure consists entirely of convolutional layers, and its output is an n×n matrix rather than a single scalar value. Each element of this output matrix corresponds to a receptive field in the input image, representing the probability of that receptive field being real or fake. Therefore, the PatchGAN discriminator can be viewed as a sophisticated texture style loss, ensuring that the generated images maintain high resolution and fine textures. The PatchGAN discriminator divides the input image into multiple receptive fields and performs an independent binary classification on each receptive field to determine whether it possesses the texture features of the target image. This design enables the PatchGAN discriminator to effectively supervise the generated images at a pixel-by-pixel level, thereby improving the quality and resolution of the generated images. Furthermore, because the PatchGAN discriminator focuses on texture features in local regions, it can better adapt to image inputs of varying sizes and shapes, demonstrating strong generalization capabilities.

[0065] Loss Function

[0066] The loss functions include reconstruction loss, style loss, perceptual loss and adversarial loss;

[0067] Reconstruction loss L re Refers to the output image I out Compared with the real image I gt The average of the absolute differences between , which is expressed as follows:

[0068] L re =||I out -I gt ||1

[0069] The core function of style loss is to ensure that the restored image closely matches the original image's style. The Gram matrix is ​​used to accurately measure subtle differences between feature maps, thereby ensuring stylistic consistency across images. Specifically, style loss focuses on the characteristics expressed by the feature maps output by each layer of the image. In deep convolutional neural networks, feature maps at each layer exhibit diverse characteristics. Shallow feature maps primarily capture detailed information, while deeper feature maps tend to express more abstract concepts.

[0070] Style loss ensures that the restored image is consistent with the original image in terms of style by carefully comparing the differences between the feature maps of the image to be restored and the original image at multiple different levels. In the image restoration process, style loss plays a vital role, helping the network to better preserve the texture and detail information of the image, making the restoration result more natural and closer to human visual aesthetics. Combining style loss with perceptual loss can further improve the visual quality and perceptual quality of the restoration result. Style loss L style as follows:

[0071]

[0072] in, Indicates averaging the losses of different feature layers j to obtain the average value, Represented by the corresponding activation function φ j Composition of C j ×C j Gram matrix:

[0073]

[0074] Among them, h represents the feature map height index, w represents the feature map width index, c, c' represents the channel index of the feature map, C j , H j , W j denote the number of channels, height, and width of the feature map on the jth layer of VGG19 pre-trained on ImageNet, respectively, j (x) h,w,c Represents the output at the (h, w, c)th position of the jth layer in VGG19.

[0075] Perceptual loss aims to improve the perceived quality of an image, rather than focusing solely on pixel-level differences. It plays a crucial role in image restoration, effectively preserving rich details and delicate textures in the image, significantly improving the visual quality of the restoration result.

[0076] In addition, perceptual loss also helps the network learn more semantically meaningful feature representations, which makes the restoration results more natural and smooth, and more in line with human visual aesthetics. Compared with traditional pixel-based loss functions, perceptual loss shows better performance in processing complex structures and detailed information in images, which can further improve the accuracy and overall quality of restoration. per as follows:

[0077]

[0078] in, Represents the average perceptual loss of the entire image to obtain the overall loss, N i Indicates the number of elements in the feature map of layer i, φ i is the activation function in the i-th layer of VGG19.

[0079] Adversarial loss L adv as follows:

[0080]

[0081] in, and They represent the real image I gt and output image I out , and D(·) represents the output of the discriminator.

[0082] In summary, the total loss function is as follows:

[0083] L total =λ a L adv +λ b L re +λ c L per +λ d L style

[0084] Among them, setting the scale parameter λ a =0.1,λ b =λ c =1,λ d =250.

[0085] experiment

[0086] The thangka data used in this experiment also comes from thangka images collected during field research. We selected 300 high-quality images from these images and then cropped and corrected them to remove excess content. These thangka images were then cut into 256×256 resolution images, resulting in a total of 25,000 images. These images were then divided into training, test, and validation sets in an 8:1:1 ratio. The Canny algorithm was used to extract edges from the cropped images to generate the corresponding edge line drawing dataset.

[0087] Unlike common mask datasets, thangka images usually have the following types of damage and pollution: crack-like creases, point-like damage, sheet-like damage, water stains, mildew spots, etc. Therefore, this example uses Photoshop to simulate these types of damage shapes, create a mask dataset, and crop it into a 256×256 size mask image, as shown in the following figure. Figure 4As shown in Figure 2, a total of 5,000 mask images were obtained. At the same time, another 1,000 mask images were selected from the public mask dataset, totaling 6,000 mask images, which were randomly divided into training and test sets at a ratio of 5:1.

[0088] This experiment uses supervised training with a generator learning rate of 1e-4, a discriminator learning rate of 1e-5, a batch size of 8, and the Adam optimizer. All experiments were conducted on a cloud server with an Intel(R) Xeon(R) Gold 6330 CPU and a 24GB NVIDIA RTX 3090 GPU. The software environment includes Python 3.8, PyTorch 1.8.1, and CUDA 11.1.

[0089] A comparative experiment on image restoration was conducted using a thangka dataset. The comparison methods included several advanced image restoration methods in recent years: DeepFillv2, EdgeConnect, DFNet, HiFill, and T-Former, to verify the reliability and superiority of the method in this embodiment in thangka restoration.

[0090] Quantitative comparison

[0091] This experiment used three evaluation metrics: PSNR, SSIM, and LPIPS. PSNR represents the ratio of the maximum possible signal power to the destructive noise power that represents its accuracy. A larger value indicates that the generated image is more similar to the original image. SSIM measures the similarity of brightness, contrast, and structure between two images. A larger value indicates less image distortion. LPIPS evaluates the perceptual similarity of two images by comparing their distance in the feature space of a deep neural network. A smaller value indicates that the generated image is more similar to the original image.

[0092] Table 1 PSNR, SSIM, LPIPS values ​​of different methods on the Thangka dataset

[0093] method PSNR SSIM LPIPS DeepFillv2 30.4801 0.9434 0.044 EdgeConnect 30.6045 0.9467 0.040 DFNet 30.3620 0.9417 0.053 HiFill 27.4012 0.9125 0.066 T-Former 31.4755 0.9513 0.038 Our 34.7967 0.9685 0.016

[0094] Table 1 lists the PSNR, SSIM, and LPIPS values ​​of different methods on the Thangka dataset. Through data analysis, it can be found that on the Thangka dataset, the method of this embodiment achieved the best results in all three indicators, indicating that the restoration result of this embodiment is closer to the original image, proving that the method proposed in this embodiment achieved a good restoration effect on the Thangka dataset.

[0095] Qualitative comparison

[0096] Figure 5The results of different methods on the thangka dataset are presented. DeepFillv2 exhibits noticeable blurring and unclear textures in images 3 and 4, and noticeable restoration artifacts in image 5. EdgeConnect also exhibits noticeable texture artifacts in images 3 and 4, primarily due to its automatic line art restoration. Unlike natural images, thangka images have diverse edge styles, making automatic restoration inaccurate. DFNet and HiFill exhibit noticeable restoration artifacts, and the restorations do not conform to the thangka image. T-Former exhibits noticeable blurring and unclear textures in images 3, 4, and 6. Because the method in this embodiment uses interactive line art restoration to constrain the restoration results, the restoration results in this embodiment exhibit coherent and clear textures, lack obvious restoration artifacts, and offer superior visual quality compared to algorithms without line art constraints. EdgeConnect's automatic line art restoration can produce illogical line art restoration results, resulting in poor restoration results. This embodiment uses interactive line art restoration, resulting in restoration results that are closer to the true thangka image.

[0097] Ablation experiments

[0098] In order to verify the effectiveness of the method of this embodiment in adding interactive line draft repair, style texture repair and fine repair, this embodiment conducted an ablation experiment, and conducted experiments under three conditions: removing interactive line draft repair, removing style texture repair, and removing fine repair. The ablation experiments were all conducted under the same parameters, hardware and software conditions. Table 2 shows the PSNR, SSIM and LPIPS results on the thangka dataset under the three conditions. From the experimental results, it can be seen that after adding interactive line draft repair, style texture repair and fine repair, the quality of the repaired image is significantly improved:

[0099] Table 2 Quantitative results of ablation experiments

[0100]

[0101]

[0102] Figure 6The restoration results of different combinations are shown. It can be found that without adding line draft constraints, the restoration results have obvious blurring and unclear textures. After adding line draft constraints, due to the line draft constraints, the restoration results have clear textures and no other features that do not conform to the thangka image. The line draft plus style texture restoration method has obvious mask-like shapes to constrain the restoration results, making its texture features closer to the real thangka texture. The style texture restoration part can learn the style texture features of thangka, and the fine restoration further improves the restoration quality and eliminates the restoration traces. The combination of the three modules makes the final restored thangka image content texture clear and without restoration traces, thereby verifying the effectiveness of each module.

[0103] The above is a schematic description of the present invention and its embodiments, which is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs a structure and embodiment similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. An interactive thangka image restoration method based on line drawing repair, characterized by: The following steps are involved:

1. Interactive mask drawing and line drawing repair stage: Draw a mask based on the damaged area of ​​the thangka, generate the corresponding line drawing image, and interactively repair the damaged area of ​​the line drawing; Restoration phase: The damaged thangka image, the repaired complete line drawing E, and the mask image are input into the style texture restoration network. First, the style texture restoration is performed to restore the thangka's original artistic style and texture characteristics. Then, the fine restoration network is used to further process the details to obtain the final restoration result. In step 2, the input images are the damaged thangka image I, the mask image M obtained in the first stage, and the repaired line drawing E. First, the three images are fused and input into the style texture restoration network to obtain a restoration result with unified texture style. Then, the features are fused with the line drawing E and input into the fine restoration network for refined restoration to obtain the final restoration result. In the style and texture restoration network, two layers of gated convolution are first used for preprocessing. Next, an encoder is constructed, which includes a transposed convolution layer, a normalization layer, a ReLU activation function, and a pooling layer to extract key features from the image. A spatial style attention module is introduced in the skip connection part to focus on the key spatial areas in the current restoration task; In the fine restoration network, the line drawing E is first fused with the restored image generated by the style and texture restoration part, and then this feature information is input into the fine restoration module. The fine restoration network introduces a linear attention mechanism in the downsampling stage to capture the local and global dependencies of the input feature map. The encoding part combines depthwise separable convolution and ordinary convolution. Add residual blocks to the skip connection part to create cross-layer channels.

2. The interactive thangka image restoration method based on line drawing repair according to claim 1 is characterized in that: In step one, specifically: for the input damaged thangka image I, first draw a mask map M based on the damaged area to clarify the range that needs to be repaired; then, based on the mask map, use the Canny algorithm to extract the image edges of the undamaged area to generate the corresponding line drawing image; finally, interactively repair the damaged area of ​​the line drawing to finally obtain the repaired complete line drawing E.

3. The interactive thangka image restoration method based on line drawing repair according to claim 2 is characterized in that: The loss functions include reconstruction loss, style loss, perceptual loss and adversarial loss; Reconstruction loss L re Refers to the output image I out Compared with the real image I gt The average of the absolute differences between , which is expressed as follows: L re =||I out -I gt ||1 Style loss L style as follows: in, Indicates averaging the losses of different feature layers j to obtain the average value, Represented by the corresponding activation function φ j Composition of C j ×C j Gram matrix: Among them, h represents the feature map height index, w represents the feature map width index, c, c' represents the channel index of the feature map, C j , H j , W j denote the number of channels, height, and width of the feature map on the jth layer of VGG19 pre-trained on ImageNet, respectively, j (x) h,w,c represents the output at position (h, w, c) of the jth layer of VGG19; Perceptual loss L per as follows: in, Represents the average perceptual loss of the entire image to obtain the overall loss, N i Indicates the number of elements in the feature map of layer i, φ i is the activation function of the i-th layer of VGG19; Adversarial loss L adv as follows: in, and They represent the real image I gt and output image I out The expectation of , D(·) represents the output of the discriminator; In summary, the total loss function is as follows: L total =λ a L adv +λ b L re +λ c L per +λ d L style Among them, setting the scale parameter λ a =0.1,λ b =λ c =1,λ d =250.

Images