A method, system, device and storage medium for repairing a Thangka mural

Abstract

The application relates to a Thangka mural picture repairing method, system, device and storage medium. The method comprises the following steps: obtaining a reference image data set by screening from an initial image data set; extracting key features and value features of each reference image in the reference image data set and a damaged area feature of a mural picture to be repaired; calculating a target attention weight matrix based on the key features of each reference image in the reference image data set and the damaged area feature of the mural picture to be repaired; fusing the target attention weight matrix and the value features of each reference image in the reference image data set to obtain matching features; and performing image repairing on the mural picture to be repaired based on the damaged area feature of the mural picture to be repaired and the matching features to obtain a target repaired mural picture. The method can solve the problems of insufficient semantic information caused by repairing without reference image guidance, incapability of extracting effective matching features in a large-area damaged and complex semantic environment, reference mismatch and style inconsistency, and comprehensively improve the repairing effect through the synergistic effect among various dimensions.

Description

Technical Field

[0001] This application relates to the fields of computer vision and digital cultural heritage protection technology, and in particular to a method, system, device and storage medium for the restoration of Thangka murals. Background Technology

[0002] Digital cultural heritage preservation is an important application area in computer vision and image processing. Thangka murals, as precious digital cultural heritage, carry significant historical and artistic value. However, during long-term preservation, they are prone to damage such as fragmentation, fading, and cracking due to natural aging and human interference. Utilizing advanced image restoration technology to digitally reconstruct and visually restore damaged murals not only preserves the information of these cultural relics in a timely manner but also restores their original artistic appearance with high precision, playing an irreplaceable role in inheriting and promoting national cultural heritage.

[0003] However, when existing image restoration methods are applied to Thangka murals, the unique historical nature of these murals makes it difficult to obtain their original, undamaged state as a reference. This results in a severe deficiency in semantic information extraction for existing methods that rely on context or a single reference. Furthermore, for the large-area damage and complex texture environment commonly found in murals, existing technologies are easily affected by irrelevant features during feature matching, leading to low matching efficiency and poor restoration results. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] The main objective of this disclosure is to propose a method, system, device, and storage medium for the restoration of Thangka murals. This method can solve the problems of insufficient semantic information, inability to extract effective matching features and reference mismatch, and inconsistent styles caused by restoration without reference image guidance. Through the synergistic effect between various dimensions, the restoration effect can be comprehensively improved.

[0006] A first aspect of this application provides a method for restoring Thangka murals, the method comprising: Obtain the mural to be restored and the initial image dataset; the mural to be restored and the initial image dataset originate from the same mural area; A reference image dataset is obtained by filtering from the initial image dataset; The mural to be restored and the reference image dataset are input into the mural restoration network to obtain the target mural to be restored output by the mural restoration network. The process by which the mural restoration network outputs the target mural for restoration is as follows: Extract the key features and value features of each reference image in the reference image dataset, as well as the features of the damaged area of ​​the mural to be restored; Based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored, the target attention weight matrix is ​​calculated; The matching features are obtained by fusing the target attention weight matrix and the value features of each reference image in the reference image dataset; Based on the damaged area features of the mural to be repaired and the matching features, image restoration is performed on the mural to be repaired to obtain the target mural to be repaired.

[0007] In some embodiments of this application, obtaining a reference image dataset from the initial image dataset includes: The overall similarity between each image in the initial image dataset and the mural to be restored is calculated based on a multi-level similarity evaluation model. The multi-level similarity evaluation model includes a low-level structure evaluation layer, a mid-level perception evaluation layer, and a high-level semantic evaluation layer. The low-level structure evaluation layer is used to evaluate the degree of alignment between the geometric structure and the details of the edge texture of the image. The mid-level perception evaluation layer is used to measure the consistency of texture and color style at the visual perception level. The high-level semantic evaluation layer is used to evaluate the overall semantic consistency of the image. Images with a comprehensive similarity greater than a preset similarity threshold are used as reference images to form the reference image dataset.

[0008] In some embodiments of this application, the mural restoration network is trained through the following steps: Obtain the initial training dataset; The training data in the initial training dataset are damaged to generate a damaged training dataset. Based on the initial training dataset and the damaged training dataset, multiple reference image training datasets are constructed; the generation methods of each reference image dataset in the multiple reference image training datasets are different; there is a correspondence between each training image in the reference image training dataset and the damaged training dataset. The damaged training dataset and the multiple reference image training datasets are input into the initial mural restoration network to obtain the initial restoration training mural output by the initial mural restoration network. Based on the initial training dataset and the initial mural restoration training, a composite loss function is constructed to optimize the initial mural restoration network, thereby obtaining the mural restoration network.

[0009] In some embodiments of this application, the multiple reference image datasets include a first reference image dataset, a second reference image dataset, a third reference image dataset, and a fourth reference image dataset. The construction of multiple reference image training datasets based on the initial training dataset and the damaged training dataset includes: The initial training dataset is subjected to data augmentation processing on each training image to obtain the first reference image dataset; the data augmentation processing includes at least one of image cropping, image contrast adjustment, image resolution adjustment, image brightness adjustment, noise addition, and complex composite transformation; The training images in the initial training dataset are overlapped and cropped to obtain the second reference image dataset; the overlapped cropping process includes cropping along the upper left, lower left, upper right, lower right and center positions of the training images; Calculate the overall training similarity between each training image in the initial image dataset and the corresponding training images in the damaged training dataset; The training image corresponding to the maximum value in the training comprehensive similarity is used as the training reference image to form the third reference image dataset; The training images whose overall training similarity is greater than a preset similarity training threshold are used as the training reference images to form the fourth reference image dataset.

[0010] In some embodiments of this application, the composite loss function includes reconstruction loss, perceptual loss, global style loss, reference consistency contrast loss, and broken region style consistency loss.

[0011] In some embodiments of this application, the loss of style consistency in the damaged area is calculated through the following steps: Separate the damaged and intact regions of each damaged image in the damaged training dataset; Extract the regional features of the damaged area and the regional features of the intact area; Based on the regional characteristics of the damaged area, the calculation range of style constraints in the style consistency loss of the damaged area is determined; The regional characteristics of the intact region are used as the style reference for the style consistency loss of the damaged region. The style consistency loss of the damaged region is constructed based on the regional characteristics of the damaged region and the regional characteristics of the intact region.

[0012] In some embodiments of this application, the target attention weight matrix is ​​calculated using the following formula: ; ; in, Let be the target attention weight matrix. The characteristics of the damaged area of ​​the mural to be repaired are as follows: This represents the transpose matrix corresponding to the key features after sparsification. For temperature parameters, The transpose matrix corresponding to the key features of each reference image in the reference image dataset. The key features of each reference image in the reference image dataset correspond to the position index in the feature space. The key features of each reference image in the reference image dataset, Set the preset threshold for filtering matching items.

[0013] The Thangka mural restoration method provided in this embodiment has at least the following beneficial effects: This method obtains a reference image dataset by filtering from an initial image dataset; extracts the key and value features of each reference image in the reference image dataset, as well as the damaged area features of the mural to be restored; calculates a target attention weight matrix based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored; fuses the target attention weight matrix and the value features of each reference image in the reference image dataset to obtain matching features; and performs image restoration on the mural to be restored based on the damaged area features and matching features to obtain the target restored mural. This method can solve the problems of insufficient semantic information, inability to extract effective matching features in large-area damage and complex semantic environments, reference mismatch, and style inconsistency caused by restoration without reference image guidance. Through the synergistic effect between various dimensions, it achieves a comprehensive improvement in restoration effect.

[0014] A second aspect of this application provides a Thangka mural restoration system, the system comprising: The acquisition module is used to acquire the mural to be restored and the initial image dataset; the mural to be restored and the initial image dataset originate from the same mural area; A filtering module is used to filter a reference image dataset from the initial image dataset; The restoration module is used to input the mural to be restored and the reference image dataset into the mural restoration network to obtain the target mural to be restored output by the mural restoration network; The process by which the mural restoration network outputs the target mural for restoration is as follows: Extract the key features and value features of each reference image in the reference image dataset, as well as the features of the damaged area of ​​the mural to be restored; Based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored, the target attention weight matrix is ​​calculated; The matching features are obtained by fusing the target attention weight matrix and the value features of each reference image in the reference image dataset; Based on the damaged area features of the mural to be repaired and the matching features, image restoration is performed on the mural to be repaired to obtain the target mural to be repaired.

[0015] A third aspect of this application provides an electronic device, including at least one controller and a memory for communicatively connecting with the controller; the memory stores instructions executable by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a Thangka mural restoration method as described above.

[0016] In a fourth aspect, this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a Thangka mural restoration method as described above.

[0017] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the Thangka mural restoration method provided in the embodiments of this application; Figure 2 This is a schematic diagram provided in an embodiment of this application; Figure 3 This is another schematic diagram provided in the embodiments of this application; Figure 4 This is yet another schematic diagram provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the Thangka mural restoration system provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0021] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0022] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or function in a specific orientation, and therefore should not be construed as a limitation of this application.

[0023] Image restoration technology plays a crucial role in the field of digital cultural heritage preservation, enabling the digital reconstruction and visual restoration of murals damaged or faded due to natural aging, human-caused damage, and other reasons. Currently, researchers have proposed various image restoration methods, mainly including diffusion methods based on partial differential equations, traditional methods based on sample block matching, and deep learning-based methods such as Context Encoder, EdgeConnect, and LaMa.

[0024] While the aforementioned methods have made significant progress in general image restoration tasks, they still face many inherent limitations when restoring special objects such as Thangka murals: First, Thangka murals are historically unique and irreproducible, making it difficult to obtain their original undamaged state as a restoration reference, resulting in insufficient semantic information in existing methods that rely on context or a single reference; second, the damaged areas of murals are often large and have complex textures, making traditional attention mechanisms susceptible to interference from irrelevant features, resulting in a "global dilution effect" and leading to low matching efficiency and structural distortion; third, cultural heritage restoration must strictly adhere to the principle of "restoring the old as it was," and existing methods are still insufficient in maintaining consistency between the restored area and the original artifact in terms of stylistic elements such as color, brushstrokes, and texture, easily resulting in visual disjointed restoration results.

[0025] like Figure 1 As shown in one embodiment of this application, a method for restoring Thangka murals is provided, the method comprising: Step S110: Obtain the mural to be restored and the initial image dataset; the mural to be restored and the initial image dataset originate from the same mural area; Step S120: Filter the initial image dataset to obtain the reference image dataset; Step S130: Input the mural to be restored and the reference image dataset into the mural restoration network to obtain the target mural to be restored output by the mural restoration network; The process of restoring murals, as output by the mural restoration network, is as follows: Extract the key features and value features of each reference image in the reference image dataset, as well as the features of the damaged areas of the mural to be restored; Based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored, the target attention weight matrix is ​​calculated; The matching features are obtained by fusing the target attention weight matrix and the value features of each reference image in the reference image dataset; Based on the damaged area features and matching features of the mural to be restored, image restoration is performed on the mural to be restored to obtain the target mural to be restored.

[0026] For ease of understanding, the following explains some key terms in this embodiment: The murals to be restored can be Thangka murals that have been damaged due to natural aging, human destruction, etc., and require digital reconstruction and visual restoration. The initial image dataset can be a collection of various Thangka mural related images that originate from the same Thangka mural region as the mural to be restored, have the same structural, textural and stylistic features, and can be used as a reference for restoration. Key features can be a set of core features extracted from the reference image, which characterize the geometric structure, edge texture, semantic attributes, etc. of each region of the reference image. They serve as the benchmark for achieving correlation matching with the features of the damaged area of ​​the mural to be restored. Value features can correspond one-to-one with key features. They are feature sets extracted from reference images and contain detailed texture details, color parameters, style features, etc. of each region. They are information carriers that provide substantial material for the restoration of damaged areas of murals. The features of the damaged area can be extracted from the damaged parts of the mural to be repaired. They can reflect the repair feature requirements of the damaged area and are the core basis for matching with the features of the reference image. The target attention weight matrix can be calculated based on the correlation between the key features of the reference image and the features of the damaged area of ​​the mural to be restored. After sparse selection to retain highly correlated matching terms and temperature scaling to smooth the attention distribution, the weight matrix is ​​used to quantify the matching degree between the features of the reference image and the features of the damaged area, and to provide a weight basis for feature fusion. Specifically, the target attention weight matrix is ​​calculated using the following formula: ; ; in, For the target attention weight matrix, The characteristics of the damaged areas of the mural to be restored. This represents the transpose matrix corresponding to the key features after sparsification. For temperature parameters, This is the transpose matrix of the key features of each reference image in the reference image dataset. For each reference image in the reference image dataset, the key feature corresponds to its position index in the feature space. For the key features of each reference image in the reference image dataset, Set the preset threshold for filtering matching items.

[0027] The matching feature can be obtained by weighted fusion of the target attention weight matrix and the value features of the reference image. This feature filters and integrates effective restoration information in the reference image that is highly related to the damaged area of ​​the mural to be restored, providing accurate reference guidance for the restoration of the damaged area.

[0028] In this step, we first acquire the mural to be restored that has damage such as incompleteness, fading, and cracks, as well as the initial image dataset that comes from the same Thangka mural area as the mural to be restored. This ensures that the images in the initial image dataset have the same structural, textural, and stylistic features as the mural to be restored, laying a foundation for the correlation of subsequent reference image selection.

[0029] Furthermore, based on the pre-trained multi-level similarity evaluation model, the comprehensive similarity between each image in the initial image dataset and the mural to be restored is calculated. Images with a comprehensive similarity greater than a preset similarity threshold are selected and integrated into a reference image dataset, providing reliable semantic, structural and textural guidance information for mural restoration.

[0030] Furthermore, the mural to be restored and the reference image dataset are input into a mural restoration network trained with a composite loss function. The mural restoration network, trained with a composite loss function, outputs the target restored mural. The steps for the mural restoration network to output the target restored mural are as follows: First, a multi-scale feature extraction mechanism is used to extract the key features and value features of each reference image in the reference image dataset, as well as the damaged area features corresponding to the damaged parts of the mural to be restored. Then, based on the key features of the reference images and the damaged area features of the mural to be restored, correlation calculation is performed. A target attention weight matrix is ​​obtained by sparsely selecting highly correlated matching terms and smoothing the attention distribution through temperature scaling. Next, this target attention weight matrix is ​​weighted and fused with the value features of the reference images to obtain matching features containing effective restoration information. Finally, based on the damaged area features of the mural to be restored, combined with the reference guidance information in the matching features, the damaged area of ​​the mural is reconstructed, textured, and styled, completing the full image restoration and outputting a visually natural and stylistically unified target restored mural.

[0031] In some embodiments, the mural restoration network is trained through the following steps: Obtain the initial training dataset; perform damage processing on each training data in the initial training dataset to generate a damaged training dataset; construct multiple reference image training datasets based on the initial training dataset and the damaged training dataset; the generation methods of each reference image dataset in the multiple reference image training datasets are different; there is a correspondence between the training images in the reference image training dataset and the damaged training dataset; input the damaged training dataset and the multiple reference image training datasets into the initial mural restoration network to obtain the initial restoration training mural output by the initial mural restoration network; construct a composite loss function based on the initial training dataset and the initial restoration training mural to optimize the initial mural restoration network, thus obtaining the mural restoration network.

[0032] In this embodiment, the core of the mural restoration network model training is to construct a diverse reference image training dataset that corresponds one-to-one with the damaged training data, use the undamaged initial training dataset as the real label, and use a composite loss function to perform multi-dimensional iterative optimization on the unoptimized initial mural restoration network. This allows the network to learn the feature matching, structural reconstruction, and style fusion rules of the damaged areas of Thangka murals, and finally obtain a mural restoration network that is adaptable to Thangka mural restoration tasks with different degrees of damage and complex textures, and has high precision and high robustness.

[0033] Specifically, the initial training dataset is first obtained. This dataset consists of high-resolution images of Thangka murals that are undamaged, have complete textures, and are consistent in color and style. The images cover diverse textures, colors, compositions, and structural features of different Thangka mural regions. These images serve as ground truth labels for model training, providing a core benchmark for evaluating subsequent restoration effects, calculating losses, and optimizing network parameters.

[0034] Furthermore, for each complete Thangka mural image in the initial training dataset, simulate actual damage scenarios of Thangka murals such as natural aging and human-caused damage. Generate binary damage masks by combining manual annotation or edge detection algorithms with morphological operations to mask the damaged areas of the complete images, generating damaged mural images that correspond one-to-one with the complete images. All damaged images are integrated into a damaged training dataset to simulate the data distribution of real murals to be restored, allowing the network to learn repair logic for damaged areas that fits reality.

[0035] Furthermore, based on the initial training dataset, and combining the damaged images in the damaged training dataset, multiple reference image training datasets with different generation methods are constructed. Each reference image in each of these datasets forms a one-to-one correspondence with the corresponding damaged image in the damaged training dataset and the corresponding complete image in the initial training dataset. The generation methods for these reference images include selection based on a multi-level similarity evaluation model, data augmentation, and multi-position overlap cropping. Through these diverse reference image generation methods, the network learns feature matching and repair strategies under different reference features, improving the network's generalization ability and its ability to effectively utilize reference features.

[0036] In one embodiment, based on the initial training dataset and the damaged training dataset, multiple reference image training datasets are constructed, including: The initial training dataset is augmented with data to obtain a first reference image dataset. This augmentation includes at least one of image cropping, image contrast adjustment, image resolution adjustment, image brightness adjustment, noise addition, and complex composite transformations. The initial training dataset is then overlaid and cropped to obtain a second reference image dataset. This overlay cropping includes cropping along the upper left, lower left, upper right, lower right, and center positions of the training images. The overall training similarity between each training image in the initial image dataset and corresponding training images in the damaged training dataset is calculated. The training image with the maximum overall training similarity is used as the training reference image to form a third reference image dataset. Finally, the training images with an overall training similarity greater than a preset similarity training threshold are used as the training reference images to form a fourth reference image dataset.

[0037] In this embodiment, for the initial training dataset and the simulated damage training dataset, four differentiated generation strategies are used to construct the first reference image training dataset, the second reference image training dataset, the third reference image training dataset, and the fourth reference image training dataset, respectively. The generation methods of each dataset are completely different, and the images in all reference image training datasets are matched one-to-one with the corresponding damaged images in the damage training dataset. This can provide multi-dimensional and diversified reference guidance features for the initial mural restoration network, allowing the network to learn feature matching and restoration logic under different reference features, thereby improving the network's adaptability, feature utilization ability, and restoration accuracy to complex damage scenarios of Thangka murals.

[0038] Specifically, for each complete training image in the initial training dataset, data augmentation processing is performed using at least one of the following methods: image cropping, image contrast adjustment, image resolution adjustment, image brightness adjustment, noise addition, and complex composite transformation. This simulates the image characteristics of Thangka murals under different shooting conditions and preservation environments, generating augmented images corresponding to the original complete training images. All augmented images are integrated to form the first reference image dataset, thereby enriching the feature diversity of the reference images and improving the robustness of the network.

[0039] Furthermore, for each complete training image in the initial training dataset, overlapping cropping is performed along five fixed positions: top left, bottom left, top right, bottom right, and center, to obtain sub-images with overlapping edges. The overlapping regions can improve the local continuity of image features. All cropped sub-images are integrated to form a second reference image dataset, providing the network with reference features from multiple local perspectives.

[0040] Furthermore, through a multi-level similarity evaluation model, for each damaged training image in the damaged training dataset, the comprehensive training similarity between it and all training images in the initial training dataset is calculated. The initial training image with the largest comprehensive training similarity value is selected as the training reference image corresponding to the damaged training image. All training reference images of this type corresponding to the damaged training images are integrated to form a third reference image dataset, providing the network with the best reference features with the highest matching degree with the damaged images.

[0041] Furthermore, based on the same training comprehensive similarity calculation results mentioned above, a preset similarity training threshold is set in advance. For each damaged training image in the damaged training dataset, all training images in the initial training dataset with a training comprehensive similarity value greater than the threshold are used as training reference images corresponding to the damaged training image. All training reference images of this type corresponding to the damaged training images are integrated to form a fourth reference image dataset, providing the network with multiple reference features with high matching degree and enriching the supply of repair reference information.

[0042] Furthermore, the damaged training dataset is used as the core input of the network, along with multiple corresponding reference image training datasets, which are then input into the initial mural restoration network without parameter optimization. The initial network restores each damaged training image according to the inherent reasoning logic of feature extraction, feature matching, feature fusion, and image reconstruction, and outputs the corresponding initial restoration training mural. This yields the restoration output in the unoptimized state of the network, which serves as the direct basis for subsequent loss calculation and network parameter adjustment.

[0043] Furthermore, using the complete images in the initial training dataset as the ground truth, we compare them with the initial restoration training murals at the feature and pixel levels, constructing a composite loss function to accurately calculate the loss value between the restoration result and the real image from multiple dimensions.

[0044] The composite loss function includes reconstruction loss, perceptual loss, global style loss, reference consistency contrast loss, and style consistency loss of damaged areas. This composite loss function accurately calculates the loss value between the repair result and the real image from five dimensions: pixel precision, semantic perception, global artistic style, reference matching reliability, and local style fusion.

[0045] In one embodiment, the style consistency loss of the damaged region is calculated through the following steps: separating the damaged region and the intact region of each damaged image in the damaged training dataset; extracting the regional features of the damaged region and the regional features of the intact region; determining the calculation range of the style constraint in the style consistency loss of the damaged region based on the regional features of the damaged region; using the regional features of the intact region as the style reference for the style consistency loss of the damaged region, and constructing the style consistency loss of the damaged region based on the regional features of the damaged region and the regional features of the intact region.

[0046] In this embodiment, the style consistency loss of the damaged area is one of the core components of the composite loss function used in training the mural restoration network. It is specifically designed to address the problem of local style inconsistency and visual disjointness that easily occurs between the restored area and the intact area of ​​the original image in Thangka mural restoration. The core is to accurately focus the style constraint on the damaged and restored area, using the intact area of ​​the image itself as a natural style reference, so as to achieve style unity between the restored area and the overall mural.

[0047] Specifically, for each damaged image in the damaged training dataset, the damaged area and the intact area are accurately separated by a preset binary damage mask. In this damage mask, the pixel area with a value of 1 corresponds to the damaged area of ​​the image, and the pixel area with a value of 0 corresponds to the intact area of ​​the image. By matching the mask with the image element by element, the two regions are clearly divided, laying the foundation for subsequent feature processing of the region.

[0048] Furthermore, the separated damaged images are input into the encoder of the mural restoration network. Through a multi-scale feature extraction mechanism, the regional features of the damaged area and the regional features of the intact area are extracted respectively. The extracted regional features contain core information related to the style of Thangka murals, such as texture details, color parameters, brushstroke style, and structural features of the corresponding area, which are the key basis for determining style consistency.

[0049] Furthermore, based on the extracted regional features of the damaged area, the calculation range of style constraint for style consistency loss in the damaged area is strictly limited to the damaged area. This ensures that the optimization of style loss is only for the area that needs to be repaired, making the style constraint more targeted and avoiding interference from the features of the intact area with the style learning of the repaired area.

[0050] Furthermore, using the intact region features of the damaged image itself as the most suitable style reference, the style consistency loss of the damaged region is constructed by calculating the difference between the regional features of the damaged region and the regional features of the intact region at the feature correlation matrix level. During the training process, the restoration network continuously adjusts its parameters with the goal of reducing this loss, so that the restoration style of the damaged region is highly consistent with that of the intact region at the feature statistics level, and finally achieves style fusion between the restored region and the mural as a whole.

[0051] Furthermore, based on the loss values ​​of the restoration results and the real images, the network parameters of each layer of the initial mural restoration network are iteratively adjusted and optimized through the backpropagation algorithm, continuously reducing the loss values ​​of each dimension until the network converges to the preset accuracy, and finally obtaining a mural restoration network that has been trained and has efficient Thangka mural restoration capabilities.

[0052] In some embodiments, obtaining a reference image dataset from the initial image dataset in step S120 includes the following steps S210 to S220: Step S210: Calculate the overall similarity between each image in the initial image dataset and the mural to be restored based on a multi-level similarity evaluation model. The multi-level similarity evaluation model includes a low-level structure evaluation layer, a mid-level perception evaluation layer, and a high-level semantic evaluation layer. The low-level structure evaluation layer is used to evaluate the degree of alignment between the geometric structure and the details of the edge texture of the image. The mid-level perception evaluation layer is used to measure the consistency of texture and color style at the visual perception level. The high-level semantic evaluation layer is used to evaluate the overall semantic consistency of the image. Step S220: Use images with a comprehensive similarity greater than a preset similarity threshold as reference images to form a reference image dataset.

[0053] In this embodiment, the multi-level similarity evaluation model is an evaluation model used to quantify the overall matching degree between two images. It integrates the evaluation results of three dimensions: low-level structure, mid-level perception, and high-level semantics, and achieves a comprehensive determination of the image matching degree through hierarchical evaluation and weighted synthesis.

[0054] The multi-level similarity evaluation model consists of three layers: a low-level structure evaluation layer, a foundational layer specifically designed to assess the alignment and similarity of basic structural details such as geometric contours and edge textures between images; a mid-level perception evaluation layer, a visual perception evaluation layer that measures the consistency between images in terms of texture features, color systems, and artistic styles from the perspective of human visual perception; a high-level semantic evaluation layer, the core content evaluation layer that focuses on the overall semantic expression of images and assesses the matching of images in terms of main content, compositional logic, and semantic connotation; and a comprehensive similarity score, which is the overall matching score between images obtained by weighting and synthesizing the evaluation results of the three layers according to preset weights. The higher the score, the higher the matching degree between images.

[0055] Specifically, the preset similarity threshold is a comprehensive similarity score standard set for screening high-matching reference images. It is a quantitative basis for determining whether an initial image can be included in the reference image dataset. Images below this threshold are excluded because they have a low match with the mural to be restored, thus ensuring the effectiveness of the reference image dataset.

[0056] In this embodiment, a multi-level similarity evaluation model that has been pre-trained is first used to perform similarity quantification calculation on each image in the initial image dataset and the mural to be restored. Specifically, multi-dimensional matching degree analysis is achieved through hierarchical evaluation of the low-level structure evaluation layer, the middle-level perception evaluation layer, and the high-level semantic evaluation layer.

[0057] The low-level structural evaluation layer focuses on evaluating the alignment of basic visual details such as geometric structure and edge texture between images. The mid-level perceptual evaluation layer measures the consistency of stylistic features such as texture distribution and color matching between images from the perspective of human visual perception. The high-level semantic evaluation layer evaluates the matching of core information such as the overall semantic content and subject composition of the image. The model weights and fuses the evaluation results of the three layers according to preset weights to finally obtain the comprehensive similarity between each image in the initial image dataset and the mural to be restored.

[0058] Furthermore, based on the actual needs of Thangka mural restoration and the model training rules, a preset similarity threshold is set. All images in the initial image dataset with a comprehensive similarity value greater than the threshold are selected and integrated into a reference image dataset. This ensures that the selected reference images and the murals to be restored have a high degree of correlation in structure, style, and semantics, providing reliable and effective reference guidance information for feature matching and image restoration of the subsequent mural restoration network.

[0059] The method provided in this embodiment has at least the following beneficial effects: This method obtains a reference image dataset by filtering from an initial image dataset; extracts the key and value features of each reference image in the reference image dataset, as well as the damaged area features of the mural to be restored; calculates a target attention weight matrix based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored; fuses the target attention weight matrix and the value features of each reference image in the reference image dataset to obtain matching features; and performs image restoration on the mural to be restored based on the damaged area features and matching features to obtain the target restored mural. This method can solve the problems of insufficient semantic information, inability to extract effective matching features in large-area damage and complex semantic environments, reference mismatch, and style inconsistency caused by restoration without reference image guidance. Through the synergistic effect between various dimensions, it achieves a comprehensive improvement in restoration effect.

[0060] For ease of understanding, this application also provides an embodiment of a method for restoring Thangka murals, including the following: In this embodiment, a training dataset for training the mural restoration network is first constructed. Specifically, a multi-strategy reference image construction method is adopted to construct data augmentation reference images, multi-position overlapping cropping reference images, best matching reference images, and threshold filtering reference images based on the original Thangka mural image set, so as to form a training dataset and increase the diversity of the Thangka mural image dataset.

[0061] The data-enhanced reference image is obtained through the following steps: First, the original 720×720 pixel Thangka mural image is standardized and preprocessed to be scaled to 400×400 pixels. Then, a variety of data enhancement methods are used to generate the reference image, including cropping, contrast adjustment, resolution variation, noise addition, and complex composite transformation.

[0062] Specifically, cropping can be used to cut out a 380×380 pixel region from the center of a 400×400 pixel image to cover different content of the image and improve sample diversity. Subsequent processing is based on the cropped image. Contrast adjustment can increase the image contrast by 1.2 times to simulate well-lit scenes with clear light and shadow, thereby enhancing the visual diversity of the dataset. Resolution change can reduce the image resolution to 90% of the original to simulate long-distance shooting, image blurring, or equipment performance limitations, increasing the model's adaptability to different visual conditions. Brightness adjustment can reduce the image brightness by 20% to simulate the visual effect in low-light or dim environments, enriching the lighting variations of the data samples. Noise addition can introduce 10% Gaussian noise into the image to simulate imaging equipment interference or signal transmission errors, improving the model's robustness in noisy environments. Complex composite transformation can be used to perform multiple transformations on the image, including contrast enhancement, brightness reduction, noise addition, and resolution reduction, to simulate image changes under complex environments and multiple factors, further expanding the diversity of the data.

[0063] Furthermore, after completing the above six types of processing, the original image and the processed reference image are uniformly scaled to 256×256 pixels to form a standardized image reference pair, providing diverse inputs for subsequent Thangka mural restoration tasks based on the reference image.

[0064] The multi-position overlapping cropping reference image is obtained through the following steps: The original 720x720 Thangka mural image is cropped into a 400x400 Thangka mural image. Specifically, the image is cropped along the upper left, lower left, upper right, lower right, and center to obtain 5 sub-images with overlapping edges, in order to increase the diversity of samples and cover different image content. Then, the 720x720 Thangka mural image (original image) and the 5 400x400 sub-images are uniformly processed to a size of 256x256 and used to form image reference pairs, thereby extracting features with better local continuity, which is beneficial for aligning image blocks around the masked area.

[0065] The optimal matching reference image is obtained through the following steps: A pre-constructed multi-level similarity index system is used to select the optimal matching reference image from the original Thangka mural image set. This multi-level similarity index system includes three levels of similarity assessment: low-level structure, mid-level perception, and high-level semantics. The structural, perceptual, and semantic similarities are then calculated based on these three levels of indices, and a weighted composite score is synthesized. The image with the highest composite score is selected as the optimal matching reference image pair with the target image.

[0066] Specifically, the low-level structural layers focus on the alignment of details such as the geometric structure and edge texture of the image, using three metrics to evaluate structural similarity: Feature Similarity Index Measure (FSIM), Multi-Scale Structural Similarity Index (MS-SSIM), and Gradient Magnitude Similarity Deviation (GMSD). The weights are designed so that FSIM and MS-SSIM have higher weights (0.4 each), while GMSD has a lower weight (0.2), highlighting the importance of structural alignment. The mid-level perceptual layers focus on the consistency of texture and color style perceived by the human eye, using Learned Perceptual Image Patch Similarity (LPIPS) and Deep Image Structure and Texture Similarity (DISTS). Similarity measures visual differences and style matching. Its weight design is that LPIPS has a higher weight (0.6) and DISTS has a lower weight (0.4) to ensure consistency of visual style and perceptual differences. The high-level semantic layer evaluates the overall semantic consistency. It uses the image embedding feature similarity of the cross-modal image-text contrast pre-training model (CLIP, Contrastive Language–Image Pretraining), the label-free self-distillation visual representation model (DINOv2, Distillation with No Labels v2), and the 50-layer residual neural network (ResNet50) to ensure that the reference image is semantically consistent with the original image. Its weight design is that CLIP and DINOv2 have equal weights (0.4 each), and ResNet50 has a lower weight (0.2) to ensure semantic consistency.

[0067] The threshold-filtered reference images are obtained through the following steps: Using the aforementioned multi-level similarity index system, structural, perceptual, and semantic similarities are calculated based on the three levels of indices. These are then weighted and synthesized into a comprehensive score. A threshold calculation is then applied for threshold filtering. By setting a score threshold, it is ensured that the reference images meet a certain similarity standard, avoiding the introduction of low-quality reference images that could lead to structural mismatches, stylistic inconsistencies, and a decline in visual quality in the repaired area. After threshold filtering, multiple matching images may appear, forming multiple pairs of threshold-filtered reference images with the target image.

[0068] Furthermore, a joint sparse reference method is innovatively introduced into the attention mechanism. This aims to incorporate feature priors into traditional attention computation, focusing on the most relevant reference features, enhancing the network's feature focusing ability in high-confidence matching regions, reducing global dilution, and enabling the network to accurately locate reference information when facing matching features, thereby improving the overall stability of the restoration. Specifically, given the target damaged region feature Q (Query) and reference image features K (Key) and V (Value), a Top-k sparse selection is first performed. For each Query vector, only the top k most relevant matches with the reference features are retained, and the relevance of the remaining positions is set to negative infinity to automatically zero in Softmax, significantly reducing interference from low-confidence matching and focusing attention on the most relevant regions. The expression is as follows: ; in, Indicates the characteristics of the damaged area of ​​the target. This represents the transpose matrix of the reference image features after sparsification. The transpose matrix representing the features of the reference image. This represents the position index in the reference feature space. Represents the feature vector of the reference image. This indicates the preset matching threshold (i.e., the threshold for retaining the first match). The most relevant item).

[0069] Furthermore, a temperature scaling mechanism is implemented, introducing a temperature parameter τ>1 based on the retained top k matches to smoothly control the attention distribution: ; in, This represents a sparse and temperature-controlled attention weight matrix, where a larger τ value makes the attention distribution smoother and reduces sharpness, thus avoiding the model's over-reliance on a single matching point.

[0070] Furthermore, the sparse and temperature-tuned attention weights are then fused with the reference feature V to obtain the output: ; in, This represents a sparse and temperature-controlled attention weight matrix. This indicates the output characteristics of the fused reference guidance information.

[0071] Furthermore, to simultaneously ensure the structural correctness, semantic rationality, and consistency of overall and local visual style of the repaired content, a reference-guided composite loss function is constructed as a joint optimization objective during the network training phase. This function constrains the repair process from multiple dimensions, including pixel level, feature semantic level, style statistics level, and reference matching reliability. The composite loss function includes L1 reconstruction loss, perceptual loss, global style loss, reference consistency contrast loss, and style consistency loss of damaged areas.

[0072] The L1 reconstruction loss (L1) is calculated at the final output of the network and is used to generate the restored image. With real complete image The absolute error at the pixel level, also known as the mean absolute error (MAE), constrains the accuracy of the repair results in terms of overall brightness, color intensity, and basic structure. Its definition is as follows: ; in, This represents the repaired image generated by the repair network. This represents the corresponding real, complete image. The total number of pixels in the image. Represents the spatial index of a pixel (i.e., the first pixel) (pixel position) This represents the mean absolute error between the generated restored image and the original complete image (i.e., (Loss value). L1 loss can effectively suppress overall color shift and brightness drift in the restoration results, and is a fundamental constraint to ensure the authenticity and stability of the restoration results.

[0073] Perceived loss ( The L1 loss is used to measure the difference between the restored image and the original image in the high-level semantic feature space, thereby improving the quality of the restored result at the visual perception level. To compensate for the image blurring caused by L1 loss, this loss utilizes a pre-trained deep convolutional neural network as a feature extractor to specifically extract features from the restored image. and real complete image The high-level semantic features are obtained, and the Euclidean distance between the two in the feature space is calculated. Its expression is: ; in, This represents the perceived loss value. Indicates the pre-trained network's... Feature extraction function of layer, This represents the repaired image generated by the repair network. Represents a true and complete image. , , They represent the first The number of channels, height, and width of the layer feature map. Indicates the selected feature layer index. This represents the selected set of perceptual feature layers. Perceptual loss can effectively constrain the performance of the restored image in terms of structural integrity and semantic consistency, making the restored result more consistent with human visual perception.

[0074] Global style loss ( The aim is to ensure that the restored image maintains consistency with the original mural in terms of overall texture distribution and color combination, and this also applies to the restored image. With real complete image This is achieved by comparing the differences between the two Gram matrices (feature correlation matrices) in the feature space. For feature mappings... Its Gram matrix is ​​defined as: ; in, Indicates the first In the layer feature map, channels With channel Gram matrix elements between The feature layer index used to calculate style loss. Indicates the index of the feature map channel dimension. Indicates the index of the feature map channel dimension. Indicates the first The number of channels in the layer feature map. Indicates the first The height of the layer feature map, Indicates the first The width of the layer feature map, The coordinate index representing the spatial height of the feature map. The coordinate index representing the width of the feature map space. Indicates the first In the layer feature map, in the channel ,Location( The activation value (feature response) at ).

[0075] Based on this, the global style loss is defined as: ; in, This represents the global style loss value. The feature layer index used to calculate style loss. This represents the repaired image generated by the repair network. Represents a true and complete image. Indicates the first The Gram matrix corresponding to the layer feature map This is the set of feature layers used to calculate style loss. This represents the Frobenius norm. This loss ensures the harmony and consistency of the restoration result in terms of overall artistic style, color relationships, and textural statistical properties.

[0076] Furthermore, in reference image-based inpainting tasks, the selection and matching accuracy of reference features have a significant impact on the inpainting quality. In matching scenarios, mismatched references are easily introduced, leading to misaligned results and structural confusion. Therefore, a reference consistency contrast loss is proposed. This loss is applied to the feature matching stage. It enhances the model's discriminativeness in reference feature selection through feature comparison learning. Unlike the loss mentioned above, which applies to both the repaired image and the real complete image, this loss is specifically applied to the matching stage of the network.

[0077] Specifically, the matching confidence between the target feature Q and the reference feature K is calculated. Through a contrastive learning mechanism, the distance between high-confidence reference features (positive samples) and the target feature is narrowed, while irrelevant noise (negative samples) is pushed away, thereby ensuring that the network can accurately utilize the information of the reference image. Specifically, the reference consistency contrastive loss selects the Top-k high-confidence reference positions for each Q feature during the attention matching stage. This contrastive loss narrows the distance between relevant matches (positive samples) while pushing away irrelevant matches (negative samples), thus enhancing the model's ability to discriminate against reference features.

[0078] Among them, the target feature is defined as The reference feature is The matching weight is The selected Top-k high-confidence matching set is ( For positive sample sets, (for negative sample sets) ; in, Indicates that for the first The first Q-vector with the highest correlation is selected. Location index, This represents the matching weight (or similarity score) between the target feature and the reference feature. Indicates that for the first The set of negative sample indices for each target location (i.e., non-Top-k matches). This represents the position index in the reference feature space.

[0079] Furthermore, the reference consistency comparison loss is defined as: ; in, This represents the reference consistency comparison loss value. This represents the index of the feature point Q in the target image. This represents the position index in the reference feature space. Indicates the first Feature vectors of each target region Indicates the first Feature vectors of a reference image Indicates that for the first A set of positive sample indices for each target location. Indicates that for the first The set of negative sample indices for each target location (i.e., non-Top-k matches). τ represents the cosine similarity, and τ is a temperature parameter used to adjust the smoothness of the distribution.

[0080] This loss is applied directly to the attention matching stage. Combined with the Top-k sparse reference method, it can dynamically adjust the reliability of reference matching during training. Features of high-confidence matching are enhanced, while low-confidence matching is suppressed, thereby reducing mismatches.

[0081] Loss of style consistency in damaged areas ( This method focuses style constraints on the damaged area, using a damaged area mask to separate the repaired and undamaged areas. The calculation of style constraints is then limited to the features of the damaged area, using the features of the undamaged area as a "style teacher." Local Gram matrix calculations guide the style of the repaired area to adapt to the realistic texture. This loss specifically addresses the problem of inconsistency between the local style and the overall style of the repaired area. It does not rely on an external real image; instead, it uses the undamaged areas of the repaired image itself to guide the damaged area. Through mask separation, it calculates the Gram matrix difference between the features of the repaired area and the features of the undamaged areas in the same image, ensuring that the repaired traces blend perfectly with the surrounding environment.

[0082] set up For the first Layer feature mapping, Given a mask for the damaged area (a value of 1 indicates a repaired area, and 0 indicates an undamaged area), the style consistency loss for the damaged area is defined as follows: ; in, This represents the loss value in terms of style consistency in the damaged area. Indicates the first Layer feature mapping, This represents the mask for the damaged area. The term represents Gram matrix operations, and ⊙ represents element-wise multiplication. This loss method compares the statistical differences in features between the repaired and undamaged areas, ensuring that the features of the damaged area remain consistent with the undamaged part in terms of channel covariance. Simultaneously, the introduction of this loss maintains the stylistic characteristics of the local displacement of the repaired area consistent with the overall image, thereby significantly improving the visual harmony and artistic integrity of the repaired result.

[0083] In summary, these five loss functions evaluate and optimize the restoration quality from different perspectives, including pixel accuracy, semantic awareness, global artistic style, reference matching reliability, and local style fusion, and work together to achieve a comprehensive improvement in image restoration results.

[0084] Therefore, in this embodiment, the mural restoration network is constructed based on an encoder-decoder architecture and integrates the aforementioned modules. The mural restoration network is then trained using the original Thangka mural image set, training damage masks, and a reference image set obtained through a multi-strategy construction method, resulting in a trained mural restoration network. The training damage masks are binary damage masks generated for each image in the original Thangka mural image set through training with manual annotation or by using edge detection algorithms combined with morphological operations. This allows the original Thangka mural image set to be processed with varying degrees of damage using the binary damage masks of each image, resulting in a damaged Thangka mural image set.

[0085] Furthermore, the damaged Thangka mural image set and the reference image set were each processed through an encoder network to extract multi-scale features. The encoder contained three structurally identical hierarchical embedding modules, with residual connections between adjacent modules to perform residual learning. This configuration mitigates structural defects during network propagation and compensates for errors caused by the baseline model's inability to learn the differences between the embedding results of each module during training. The hierarchical embedding module consists of a global alignment module and a local alignment module. The local alignment module performs coarse alignment of local damaged areas. In the global alignment module, a joint sparse reference method is introduced, including Top-k sparse selection and a temperature scaling mechanism, to select the best matching features and avoid global dilution effects, ultimately achieving global alignment and obtaining finely aligned features.

[0086] Furthermore, reference-guided feature fusion is performed, fusing the obtained finely aligned features with the target features in the backbone feature processing module. This provides explicit guidance for the fusion of mural features in the next layer, effectively improving the restoration quality. Next, decoding is performed. The decoder consists of a Transformer decoding module and a convolutional tail. The Transformer decoding module receives the fused features from the backbone feature processing module as input and outputs decoded features aligned with the feature space of the backbone feature processing module at each scale. The convolutional tail is responsible for fusing the multi-scale hierarchical features from the encoding stage with the alignment features from the decoding stage, ultimately generating a complete restored image.

[0087] Furthermore, based on the complete restored image and the original Thangka mural image set, the network parameters are optimized using a composite loss function to obtain the trained mural restoration network. During training, external image information is introduced, and the reference image provides reliable texture, structure, and style support for restoration. To comprehensively and accurately select features from the damaged image and the reference image, a reference feature-guided optimization mechanism is proposed using a joint sparse reference method.

[0088] In one implementation of this embodiment, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are used as the main evaluation metrics to quantify performance improvements during training and to continuously monitor the model's repair effectiveness. During iteration, these metrics provide real-time feedback, ensuring the correctness of the model's optimization direction.

[0089] Specifically, by comparing the method of this embodiment with several existing image inpainting methods, the performance of each method in the training and inpainting process was systematically evaluated. Specifically, DeepFill v2 (Free-Form Image Inpainting with GatedConvolution), EdgeConnect (Generative Image Inpainting with Adversarial Edge Learning), and RFR-Net (Recurrent Feature Reasoning Network) were selected as comparison objects. The inpainting results were quantitatively analyzed using three key evaluation metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). The comparison results are shown in Table 1 below. Table 1

[0090] Comparative analysis of the experimental data shows that the method in this embodiment achieves better results in all performance evaluation indicators, and its overall performance is significantly better than the comparison method. This indicates that the proposed image restoration method can effectively ensure the visual consistency and aesthetic quality of the restoration results while improving the reconstruction accuracy.

[0091] like Figure 2 As shown, the complete process of this embodiment includes three core modules: dataset construction, composite loss function, and region matching and inpainting. The dataset construction module includes: constructing a Thangka mural reference image pair dataset for each original image in the collected original Thangka mural image set, thus obtaining the Thangka mural reference image pair dataset.

[0092] Specifically, this section combines four construction methods to build data-augmented reference images, multi-position overlap cropping reference images, best-matching reference images, and threshold-filtered reference images to form a dataset of Thangka mural reference image pairs. The data-augmented reference images are generated by augmenting the original images to simulate various situations in real-world photography, such as different contrasts, resolutions, brightness, and complex transformations, forming a reference image pair with the original image. The multi-position overlap cropping reference images are created by cropping the original image into five sub-images: top left, bottom left, top right, bottom right, and center. These sub-images will have overlapping areas at the edges, and each of these five sub-images forms a reference image pair with the original image. The best-matching reference image constructs a multi-level similarity index system, inputting the original image and candidate images (from the original Thangka mural image set) into the dataset. The similarity assessment is conducted at three levels: low-level structure, mid-level perception, and high-level semantics. The low-level structure layer focuses on evaluating geometric structure and detail alignment, the mid-level perception layer focuses on visual style consistency, and the high-level semantic layer ensures semantic matching. The similarity scores at each level are then weighted and synthesized to obtain a comprehensive score. Finally, the reference image with the highest comprehensive score is selected to form a reference image pair with the original image. After obtaining the comprehensive score from the multi-level similarity index system, a score threshold (0.6) is introduced to avoid the introduction of low-quality reference images. After thresholding, multiple reference images that meet the conditions are obtained, which together with the original image form multiple reference image pairs.

[0093] The evaluation process for the multi-level similarity index system is as follows: Figure 3As shown, the original image and the candidate image are input into a multi-level similarity index system, the features of the original image and the candidate image are extracted, and the three-level similarity score between the original image and the candidate image is calculated. Specifically, the low-level structural layers focus on the alignment of details such as geometric structure and edge texture of the image, using three metrics—FSIM, MS-SSIM, and GMSD—to evaluate structural similarity. FSIM and MS-SSIM have higher weights (0.4 each), while GMSD has a lower weight (0.2), highlighting the importance of structural alignment. The mid-level perceptual layers focus on the consistency of texture and color style perceived by the human eye, using LPIPS and DISTS to measure visual differences and style matching. LPIPS has a higher weight (0.6), while DISTS has a lower weight (0.4), ensuring consistency in visual style and perceptual differences. The high-level semantic layers evaluate overall semantic consistency, using CLIP, DINOv2, and ResNet50 image embedding feature similarity to ensure semantic consistency between the reference image and the original image. CLIP and DINOv2 have equal weights (0.4 each), while ResNet50 has a lower weight (0.2), ensuring semantic consistency. Finally, the three-level similarity scores are weighted and synthesized to obtain a comprehensive similarity score, with the highest score being the best matching reference image. Then set a threshold for the overall similarity score (0.6), and those exceeding the threshold will be used as reference images for threshold filtering.

[0094] The region matching and restoration module includes the following steps: First, the reference image of the Thangka mural is input into the dataset. The original image, along with the corresponding binary mask of the damaged area and the reference image, are input into the network. Features of the damaged image (obtained by processing the original image through the corresponding binary mask of the damaged area) and the reference image are extracted respectively. After overlapping embedding, these features are input into the local alignment module for coarse alignment of the features of the local damaged areas, resulting in local alignment features. Next, the local alignment features and the features of the reference image undergo further micro-embedding processing and are input into the global alignment module, which incorporates a joint sparse reference method (including Top-k sparse selection and temperature scaling mechanisms), for global fine alignment, outputting fine alignment features. The fine alignment features are then fused with the features from the backbone feature processing module. The fused features are input into the next layer's hierarchical embedding module (the encoder consists of three structurally identical hierarchical embedding modules). Finally, the Transformer decoding module decodes the fused features from the backbone feature processing module. The convolutional tail receives and fuses the multi-scale hierarchical features transmitted from the encoder and the alignment decoding features generated by the decoder. By fusing these two parts of information, the complete restored image is finally reconstructed and output.

[0095] Among them, the joint sparse reference method is as follows: Figure 4As shown, the method involves extracting feature points from the target image (the damaged image obtained by processing the original image through a binary mask of the corresponding damaged area) and the reference image. By using a joint sparse reference method (Top-k sparse selection and temperature scaling mechanism), the method selects the reference image and k matching feature points and performs smoothing control, which improves the feature focusing ability, reduces the global dilution phenomenon, and can accurately locate reference information when facing matching features, thereby improving the stability of the overall restoration.

[0096] The composite loss function module is applied to the region matching and inpainting stage. The composite loss function jointly optimizes the entire network to generate a higher quality inpainted image.

[0097] In one implementation of this embodiment, the original image and a binary damage mask are first collected. This can be achieved through methods such as collaborating with relevant institutions, utilizing online resources, or acquiring on-site images, or by collaborating with cultural heritage preservation institutions to acquire high-resolution digital images of the Thangka murals in the grottoes. Then, the original image is processed using a binary damage mask to obtain an image to be restored. This image contains varying degrees of peeling, cracks, and stains. Simultaneously, all mural images from areas other than the original image region (including images taken from different locations and angles) are included in a candidate reference library to screen and construct a reference image database.

[0098] Furthermore, a multi-level similarity index system is used to extract features between the image to be restored and the mural image in the candidate reference library. Then, a three-level similarity score is calculated between the image to be restored and the mural image. The three-level similarity scores are weighted and synthesized to obtain a comprehensive similarity score. The image with the highest comprehensive similarity score is then selected as the best matching reference image. The best matching reference image and the image to be restored are then combined to form a reference pair and saved.

[0099] Furthermore, the original image, the corresponding binary mask of the damaged area, and the best-matching reference image are input into the mural restoration network. Specifically, the mural restoration network obtains the restored image through the following steps: First, multi-scale feature extraction is performed. The damaged image (obtained by performing binary masking on the original image to destroy the corresponding damaged areas) and the best matching reference image are each passed through an encoder network to extract multi-level feature representations. Then, a hierarchical alignment mechanism is executed, i.e., the input is fed into a hierarchical embedding module. Each hierarchical embedding module includes a local alignment module and a global alignment module. The local alignment module is responsible for preliminary, coarse alignment of features in the local damaged areas; the global alignment module, based on the local alignment, introduces a joint sparse reference method (including Top-k sparse selection and temperature scaling mechanisms) to precisely select the optimal matching features from the reference features, avoiding attention distraction, and ultimately achieving fine-grained global feature alignment, outputting finely aligned reference features.

[0100] Furthermore, the fine alignment reference features output by the global alignment module are fused with the target features in the backbone feature processing module, providing clear guiding signals for the further fusion of mural features in subsequent levels, thereby effectively improving the accuracy of feature fusion and the quality of restoration results.

[0101] Furthermore, the image is reconstructed based on the fused features. The decoder of the mural restoration network consists of a Transformer decoding module and a convolutional tail. The Transformer decoding module receives fused features from the backbone feature processing module as input, and its output is spatially semantically aligned with the features extracted by the backbone feature processing module at each corresponding scale, generating decoded features. The convolutional tail receives and fuses information from two sources: first, the multi-scale hierarchical features transmitted by the encoder; and second, the aligned decoded features generated by the Transformer decoding module. By fusing these two pieces of information, a complete restored image is finally reconstructed and output. The damaged areas are effectively reconstructed in terms of structural integrity and texture details, and the visual naturalness and realism are significantly improved compared to the pre-restoration state.

[0102] like Figure 5 As shown in one embodiment of this application, a Thangka mural restoration system is provided, the system comprising: The acquisition module 510 is used to acquire the mural to be restored and the initial image dataset; the mural to be restored and the initial image dataset come from the same mural area. The filtering module 520 is used to filter the initial image dataset to obtain a reference image dataset; Repair module 530 is used to input the mural to be repaired and the reference image dataset into the mural repair network to obtain the target mural to be repaired output by the mural repair network; The process of restoring murals, as output by the mural restoration network, is as follows: Extract the key features and value features of each reference image in the reference image dataset, as well as the features of the damaged areas of the mural to be restored; Based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored, the target attention weight matrix is ​​calculated; The matching features are obtained by fusing the target attention weight matrix and the value features of each reference image in the reference image dataset; Based on the damaged area features and matching features of the mural to be restored, image restoration is performed on the mural to be restored to obtain the target mural to be restored.

[0103] like Figure 6One embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for restoring Thangka murals. The electronic device includes: At least one battery; At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the above-described method for restoring Thangka murals.

[0104] Electronic devices can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0105] The electronic devices according to embodiments of this application will now be described in detail.

[0106] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to perform a Thangka mural restoration method according to an embodiment of this disclosure.

[0107] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0108] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described method for restoring Thangka murals.

[0109] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0110] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0111] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0113] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0114] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0115] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0116] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0117] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0119] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0120] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A method for restoring Thangka murals, characterized in that, The method includes: Obtain the mural to be restored and the initial image dataset; the mural to be restored and the initial image dataset originate from the same mural area; A reference image dataset is obtained by filtering from the initial image dataset; The mural to be restored and the reference image dataset are input into the mural restoration network to obtain the target mural to be restored output by the mural restoration network. The process by which the mural restoration network outputs the target mural for restoration is as follows: Extract the key features and value features of each reference image in the reference image dataset, as well as the features of the damaged area of ​​the mural to be restored; Based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored, the target attention weight matrix is ​​calculated; The matching features are obtained by fusing the target attention weight matrix and the value features of each reference image in the reference image dataset; Based on the damaged area features of the mural to be repaired and the matching features, image restoration is performed on the mural to be repaired to obtain the target mural to be repaired.

2. The method for restoring Thangka murals according to claim 1, characterized in that, The step of filtering the initial image dataset to obtain the reference image dataset includes: The overall similarity between each image in the initial image dataset and the mural to be restored is calculated based on a multi-level similarity evaluation model. The multi-level similarity evaluation model includes a low-level structure evaluation layer, a mid-level perception evaluation layer, and a high-level semantic evaluation layer. The low-level structure evaluation layer is used to evaluate the degree of alignment between the geometric structure and the details of the edge texture of the image. The mid-level perception evaluation layer is used to measure the consistency of texture and color style at the visual perception level. The high-level semantic evaluation layer is used to evaluate the overall semantic consistency of the image. Images with a comprehensive similarity greater than a preset similarity threshold are used as reference images to form the reference image dataset.

3. The method for restoring Thangka murals according to claim 1, characterized in that, The mural restoration network was trained through the following steps: Obtain the initial training dataset; The training data in the initial training dataset are damaged to generate a damaged training dataset. Based on the initial training dataset and the damaged training dataset, a variety of reference image training datasets are constructed. The generation methods of each reference image dataset in the multiple reference image training datasets are different; There is a correspondence between the training images in the reference image training dataset and the damaged training dataset; The damaged training dataset and the multiple reference image training datasets are input into the initial mural restoration network to obtain the initial restoration training mural output by the initial mural restoration network. Based on the initial training dataset and the initial mural restoration training, a composite loss function is constructed to optimize the initial mural restoration network, thereby obtaining the mural restoration network.

4. The method for restoring Thangka murals according to claim 3, characterized in that, The multiple reference image datasets include a first reference image dataset, a second reference image dataset, a third reference image dataset, and a fourth reference image dataset. The construction of multiple reference image training datasets based on the initial training dataset and the damaged training dataset includes: The initial training dataset is subjected to data augmentation processing on each training image to obtain the first reference image dataset; the data augmentation processing includes at least one of image cropping, image contrast adjustment, image resolution adjustment, image brightness adjustment, noise addition, and complex composite transformation; The training images in the initial training dataset are overlapped and cropped to obtain the second reference image dataset; the overlapped cropping process includes cropping along the upper left, lower left, upper right, lower right and center positions of the training images; Calculate the overall training similarity between each training image in the initial image dataset and the corresponding training images in the damaged training dataset; The training image corresponding to the maximum value in the training comprehensive similarity is used as the training reference image to form the third reference image dataset; The training images whose overall training similarity is greater than a preset similarity training threshold are used as the training reference images to form the fourth reference image dataset.

5. The method for restoring Thangka murals according to claim 3, characterized in that, The composite loss function includes reconstruction loss, perception loss, global style loss, reference consistency contrast loss, and style consistency loss for damaged regions.

6. The method for restoring Thangka murals according to claim 5, characterized in that, The loss of style consistency in the damaged area is calculated through the following steps: Separate the damaged and intact regions of each damaged image in the damaged training dataset; Extract the regional features of the damaged area and the regional features of the intact area; Based on the regional characteristics of the damaged area, the calculation range of style constraints in the style consistency loss of the damaged area is determined; The regional characteristics of the intact region are used as the style reference for the style consistency loss of the damaged region. The style consistency loss of the damaged region is constructed based on the regional characteristics of the damaged region and the regional characteristics of the intact region.

7. The method for restoring Thangka murals according to claim 1, characterized in that, The target attention weight matrix is ​​calculated using the following formula: ； ； in, Let be the target attention weight matrix. The characteristics of the damaged area of ​​the mural to be repaired are as follows: This represents the transpose matrix corresponding to the key features after sparsification. For temperature parameters, The transpose matrix corresponding to the key features of each reference image in the reference image dataset. The key features of each reference image in the reference image dataset correspond to the position index in the feature space. The key features of each reference image in the reference image dataset, Set the preset threshold for filtering matching items.

8. A Thangka mural restoration system, characterized in that, The system includes: The acquisition module is used to acquire the mural to be restored and the initial image dataset; the mural to be restored and the initial image dataset originate from the same mural area; A filtering module is used to filter a reference image dataset from the initial image dataset; The restoration module is used to input the mural to be restored and the reference image dataset into the mural restoration network to obtain the target mural to be restored output by the mural restoration network; The process by which the mural restoration network outputs the target mural for restoration is as follows: Extract the key features and value features of each reference image in the reference image dataset, as well as the features of the damaged area of ​​the mural to be restored; Based on the key features of each reference image in the reference image dataset and the damaged area features of the mural to be restored, the target attention weight matrix is ​​calculated; The matching features are obtained by fusing the target attention weight matrix and the value features of each reference image in the reference image dataset; Based on the damaged area features of the mural to be repaired and the matching features, image restoration is performed on the mural to be repaired to obtain the target mural to be repaired.

9. An electronic device, characterized in that, It includes at least one controller and a memory for communicatively connecting with the controller; the memory stores instructions executable by the at least one controller, which, when executed by the at least one controller, causes the at least one controller to perform a Thangka mural restoration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a method for restoring Thangka murals as described in any one of claims 1 to 7.

Images