Real-time Thangka image semantic segmentation method and system based on improved PIDNet

Abstract

The invention proposes a real-time Thangka image semantic segmentation method and system based on improved PIDNet, and the method comprises the steps: carrying out the optimization of three branches of PIDNet for solving the problems of the fine texture of Thangka, and the wrong division, missing division and poor real-time performance caused by the interweaving of a main body and a background: sequentially embedding an ECA-Pag channel attention module and an LGFM local-global fusion module at the tail end of a P branch, details of the main esterm and the summator are strengthened; pConv3 mask partial convolution is introduced into a D branch, background noise is suppressed, and boundary definition is improved; the I branch keeps multi-scale pooling and provides global semantics. During reasoning, three-branch optimization features are fused after scale alignment and channel unification, and a pixel-level segmentation result is output through Softmax. According to the method, while the light weight of the network is kept, the confusion and edge fuzzy defects of the Thangka similarity summator are remarkably improved, and the method can be operated in real time, is suitable for Thangka digital repair, interactive labeling and culture research, and has the capability of migrating to other semantic segmentation tasks.

Description

Technical Field

[0001] This invention relates to the field of image restoration technology, and in particular to a real-time semantic segmentation method and system for Thangka images based on an improved PIDNet. Background Technology

[0002] Thangka, a treasure of traditional Tibetan painting art, contains rich cultural symbols and historical information. In digital preservation, restoration, and academic research, real-time semantic segmentation technology is the core support for rapid analysis of Thangka images. Semantic segmentation can accurately identify elements in Thangkas, such as the main Buddha image, ritual implements (e.g., vajra, dharma wheel, lotus), and background patterns, providing pixel-level references for the digital restoration of damaged Thangkas and laying a technological foundation for the digital transmission and research of Thangka culture.

[0003] In the current field of computer vision, real-time semantic segmentation technology has been widely applied in scenarios such as autonomous driving and medical image analysis, giving rise to mainstream models such as DDRNet, STDC, and PIDNet. These models improve inference speed while maintaining a certain level of segmentation accuracy by optimizing network structure (such as multi-branch design and dilated convolution), thus meeting real-time requirements. However, the unique characteristics of Thangka images themselves pose significant challenges to existing technologies:

[0004] ① High complexity of details and background: The main deity of the Thangka is intertwined with the background patterns and ritual implements, and the boundaries are blurred (such as the folds of the clothing of the main deity overlapping with the auspicious cloud patterns in the background). Moreover, the sizes of the ritual implements vary greatly (such as a small vajra bell and a large ritual umbrella). Existing models are prone to loss of details or missegmentation of the background.

[0005] ② Insufficient feature extraction capability: When dealing with the complex textures of Thangka (such as dense scripture patterns and fine carvings on ritual implements), traditional models have difficulty balancing local details with global semantics, leading to confusion in the categories of ritual implements (such as misjudging vajra and vajra bell).

[0006] ③ It is difficult to balance real-time performance and accuracy: Some models (such as DDRNet) are lightweight, but their segmentation accuracy for low-contrast areas in Thangka is insufficient; while high-precision models (such as some Transformer-based methods) have high computational complexity and cannot meet the needs of real-time interaction in Thangka digital restoration.

[0007] Furthermore, existing Thangka semantic segmentation research suffers from dataset deficiencies: publicly available datasets mostly focus on the overall style classification of Thangkas, lacking refined annotations for "main deity + ritual implements," and the small sample size leads to poor model generalization ability. To address these issues, this invention proposes a real-time Thangka image semantic segmentation method based on an improved PIDNet. By designing a novel feature fusion module and optimizing the network branch structure, it achieves high-precision, real-time segmentation of Thangka elements. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a real-time semantic segmentation method for Thangka images based on an improved PIDNet, comprising the following steps:

[0009] S1: Construct an improved PIDNet semantic segmentation model. Embed the ECA-Pag module and LGFM module sequentially at the end of the P branch of the original PIDNet, and embed the PConv3 module before feature fusion in the D branch to form a deep learning network for semantic segmentation of Thangka images.

[0010] S2: Input the Thangka image to be segmented into the improved PIDNet model, extract local detail features, multi-scale semantic features and boundary features through the three branches P, I and D respectively, and then pass them through the ECA-Pag module, LGFM module and PConv3 module for feature enhancement and optimization to obtain optimized multi-branch features;

[0011] S3: Perform scale alignment and channel unification on the optimized multi-branch features, and fuse the features by element-wise addition to generate a fused feature map;

[0012] S4: Input the fused feature map into the Softmax classifier and output the pixel-level semantic segmentation result map to complete the real-time semantic segmentation of the Thangka image.

[0013] Furthermore, the feature enhancement of the P branch is achieved through the following steps:

[0014] Local detail map extracted from P branch Semantic feature map F extracted from branch I I Input the ECA-Pag module;

[0015] In the ECA-Pag module, firstly, a 1×1 convolution pair is used... Perform channel dimensionality reduction to make it compatible with The number of channels is consistent, thus completing channel alignment;

[0016] By utilizing the ECA attention mechanism, channel weights W are adaptively generated through global average pooling and one-dimensional convolution operations. ECA ;

[0017] Channel weight W ECA With Alignment Multiply, then multiply Perform residual connections to output enhanced local feature maps. ;

[0018] Will The LGFM module is input, and through a two-branch structure of local and global branches, it further fuses local details and global semantic information, outputting an optimized P-branch feature map. .

[0019] Furthermore, the feature optimization of the I branch is achieved through the following steps:

[0020] Branch I performs multi-scale pooling operations on the input image through a network structure that includes a PPM pooling module, extracting multi-scale semantic feature maps. ;

[0021] The PPM pooling module employs average pooling operations of four different scales: 1×1, 2×2, 3×3, and 6×6, to capture multi-scale information from details to the whole in Thangka images.

[0022] Upsample the feature maps after pooling at different scales to make their size similar to the original feature maps. Figure 1 To;

[0023] The upsampled feature maps are fused using 1×1 convolutions to generate the final multi-scale semantic feature maps. It is used to assist in feature extraction and optimization of the P and D branches.

[0024] Furthermore, the feature optimization of the D branch is achieved through the following steps:

[0025] The D branch extracts boundary feature maps of the input image through a sub-network containing two convolutional blocks. ;

[0026] Will Inputting the PConv3 module first generates a mask M through 3×3 convolution and the Sigmoid activation function. The mask M is used to distinguish the effective area (main deity, ritual implements) from the background area in the Thangka image.

[0027] In the PConv3 module, partial convolution operations are performed only on regions where the mask M is "1". Feature extraction is performed using 3×3 convolution kernels, and a normalization strategy is adopted to avoid interference from invalid background regions on feature extraction.

[0028] The number of channels is adjusted using 1×1 convolution, and the optimized boundary feature map is output. This allows it to more accurately capture boundary information such as the outline of the main deity and the edges of ritual implements in the Thangka image during subsequent feature fusion.

[0029] Furthermore, the multi-branch feature fusion step includes:

[0030] For the optimized P-branch feature map Perform an upsampling operation to make its spatial resolution match that of the D-branch feature map. Consistent;

[0031] I-branch feature map Perform upsampling and adjust the number of channels using a 1×1 convolution to match the desired channel count. and The number of channels is uniform;

[0032] After upsampling and channel adjustment , , The three feature maps are fused element-wise to generate a fused feature map F. fuse ;

[0033] The fused feature map F_ fuse By integrating the local detail information of the P branch, the multi-scale semantic information of the I branch, and the boundary information of the D branch, rich feature representations are provided for subsequent semantic segmentation.

[0034] Furthermore, during model training, a joint loss function of cross-entropy loss and Dice loss is used with a weight ratio of 0.7:0.3, and the parameters are optimized using the Adam optimizer and cosine annealing learning rate strategy.

[0035] A real-time Thangka image semantic segmentation system based on an improved PIDNet includes:

[0036] The image preprocessing module is used to perform size unification, normalization, and format conversion on the input Thangka images;

[0037] An improved PIDNet model was developed, including three branches (P, I, and D) and modules (ECA-Pag, LGFM, and PConv3) for extracting and optimizing image features.

[0038] The feature fusion module is used to perform scale alignment and fusion on the optimized multi-branch features;

[0039] The semantic segmentation output module is used to input the fused feature map into the Softmax classifier and output a pixel-level semantic segmentation result map. Compared with the prior art, the beneficial effects of the above technical solution in this invention are as follows:

[0040] (1) Improved accuracy: Through the collaboration of ECA-Pag and LGFM modules, the segmentation accuracy of key elements (main deity, ritual implements) of Thangka is significantly improved, with mIoU reaching 73.28%, which solves the problems of detail loss and category confusion in traditional models;

[0041] (2) Real-time performance guarantee: The PConv3 module reduces the amount of computation, and the model FPS reaches 109.03, which meets the real-time interaction requirements in the digital restoration of Thangka (such as real-time annotation and restoration preview).

[0042] (3) Strong generalization: It performs well on the Cityscapes dataset and can be transferred to other semantic segmentation scenarios, with a wide range of applications;

[0043] (4) High practicality: It provides technical support for the digital protection of Thangka, and can help restorers accurately locate damaged areas (such as missing parts of ritual implements), reduce manual labeling costs, and improve the efficiency of Thangka cultural heritage. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the overall framework of the improved PIDNet based on the present invention;

[0045] Figure 2 This is a schematic diagram of the LGFM module structure of the present invention;

[0046] Figure 3 This is a schematic diagram of the PConv3 module structure of the present invention;

[0047] Figure 4 This is a visualization of the semantic segmentation results of Thangka images according to the present invention;

[0048] Figure 5 This is a chart showing the changes in the ablation experiment evaluation indicators of the present invention;

[0049] Figure 6 This is a visualization of the Cityscapes dataset results from the present invention;

[0050] Figure 7 A comparison chart of the loss functions between the baseline model of this invention and this patent;

[0051] Figure 8 This is a visualization of the ablation experiment results of the present invention. Detailed Implementation

[0052] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of application of the present invention.

[0053] Reference Figure 1 As shown, a real-time semantic segmentation method and system for Thangka images based on an improved PIDNet specifically includes the following steps:

[0054] Step 1: Improve PIDNet network initialization

[0055] ① Load the PIDNet-m pre-trained weights (pre-trained based on the ImageNet dataset), and initialize the parameters of the convolutional layers and normalization layers (BN layers) of the P, I, and D branches;

[0056] ② The parameters of the newly added ECA-Pag, LGFM, and PConv3 modules are randomly initialized. The convolution kernel weights are initialized using the He normal distribution, and the bias terms are initialized with constants (the bias value is set to 0.01).

[0057] Step 2: Feature Extraction and Optimization of Thangka Images

[0058] ① Input layer processing: Convert the Thangka image (resolution is uniformly 512×512) into RGB three-channel format, normalize it to the [0,1] interval (pixel value divided by 255), and use it as the input of the improved PIDNet;

[0059] ② P-branch feature processing:

[0060] The LGFM module not only extracts more accurate local features but also achieves a deeper fusion of local and global features by introducing global features, thereby enhancing the model's feature representation capabilities. The specific model structure is as follows: Figure 2 As shown.

[0061] The input image is processed through a P-branch (containing 3 convolutional blocks, each consisting of a 3×3 convolution + BN + ReLU) to extract initial local feature maps. (Dimensions: 128×128×64)

[0062] Input to the ECA-Pag module: First, it is compared with the semantic feature map output from the I branch. (Size 128×128×128) Channel alignment (through 1×1 convolution) The number of channels is reduced to 64, denoted as Then, channel weights are calculated using ECA attention. The fusion features are weighted, and the output is shown in formula (1):

[0063] (1)

[0064] in, The channel weight vector generated for the ECA attention module has dimensions and The number of channels is consistent (64 dimensions), which is used to dynamically enhance the channels of key features such as the main deity and ritual implements.

[0065] Input the LGFM module and perform local-global feature fusion using the following formula:

[0066] Local branch: Generate Q, K, and V vectors through linear transformation, as shown in formula (2):

[0067] (2)

[0068] in, , The weights are 1×1 convolutional matrices, each with a dimension of 64×64, which achieve linear transformation of the feature channels.

[0069] After K is convolved with 3×3 (padding=1) to keep its size unchanged, it is concatenated with Q, as shown in formula (3):

[0070] (3)

[0071] Here, Concat represents the concatenation operation at the channel level. This represents a 3×3 convolution operation;

[0072] Z is convolved by 1×1 θ (weight matrix) ) and δ (weight matrix) The transformation is shown in formula (4):

[0073] (4)

[0074] in, , All dimensions are 64×128 (because the number of channels is 128 after Z-stitching), achieving feature dimension compression and information integration;

[0075] The local branch output is the sum of the results of B and V after a 1×1 convolution, as shown in formula (5):

[0076] (5)

[0077] in, This represents a 1×1 convolution used to align the number of channels in V with the number of channels in B (both are 64).

[0078] Global branch: First, the number of channels is reduced to 32 using a 1×1 convolution, and then the receptive field is expanded using a 3×3 dilated convolution (dilation=2, padding=2) to generate... As shown in formula (6):

[0079] (6)

[0080] in, This represents a 3×3 dilated convolution, with dilation=2 ensuring that the receptive field is expanded without losing pixels;

[0081] Calculate the global attention map S, as shown in Equation (7):

[0082] (7)

[0083] in, =32( (channel dimension) The function is used to normalize the attention weights to the [0,1] interval;

[0084] The global branch output is the attention-weighted value. After 1×1 convolution, and with the original Add them together, as shown in formula (8):

[0085] (8)

[0086] Where * denotes matrix multiplication, and attention graph pairing is implemented. Weighted average;

[0087] The final output of the LGFM module is the sum of the elements output by the local branch and the global branch, as shown in formula (9):

[0088] (9)

[0089] ③ D-branch feature processing:

[0090] In PIDNet, the D branch is primarily responsible for capturing fine details in the image. PConv3 is a local convolution operation that preserves more effective information when processing complex textures and details in images. Therefore, the PConv3 local convolution operation is introduced, and its structure is as follows: Figure 3 As shown.

[0091] The input image is processed through a D-branch (containing two convolutional blocks) to extract the initial boundary feature map. (Dimensions: 256×256×32);

[0092] Input the PConv3 module: Generate a mask M, and apply the Sigmoid activation function to... Binarization is performed as shown in formula (10):

[0093] (10)

[0094] in, It is a 3×3 convolution weight matrix (dimension 32×32). The bias term (dimension 32) is σ, which represents the Sigmoid function. and All dimensions are the same (256×256×32), "1" corresponds to the valid area of ​​the Thangka (main deity, ritual implements), and "0" corresponds to the invalid background;

[0095] Perform partial convolution operation, convolving only the region where M is "1", as shown in formula (11):

[0096] (11)

[0097] in, The weight matrix is ​​a 3×3 convolutional matrix (32×32 dimensions), where ⊙ represents element-wise multiplication. Used to avoid a denominator of 0;

[0098] Output after 1×1 convolution As shown in formula (12):

[0099] (12)

[0100] in, It is a 1×1 convolution weight matrix (dimension 32×64). For the bias term (dimension 64), the number of channels is increased to 64, which prepares for subsequent feature fusion.

[0101] ④ I-branch feature processing: The input image is processed by the I-branch (including the PPM pooling module, with pooling scales of 1×1, 2×2, 3×3, and 6×6) to extract multi-scale semantic features. (Size 128×128×128), used to assist in P-branch feature optimization.

[0102] Step 3: Multi-branch feature fusion and semantic segmentation

[0103] 1. Feature scale alignment:

[0104] right Upsample (128×128×64) to 256×256×64 using bilinear interpolation, denoted as ;

[0105] right (128×128×128) is upsampled to 256×256×128, then reduced to 64 channels by 1×1 convolution, denoted as ;

[0106] Keep it at 256×256×64, no scale adjustment required, and denote it as... ;

[0107] Three-branch feature fusion: The features after the three branches are aligned are added element-wise, as shown in formula (13):

[0108] (13)

[0109] in, The fused feature map (size 256×256×64) integrates local details, global semantics, and boundary features;

[0110] Semantic classification: Input a Softmax classification head (containing one 3×3 convolution + one 1×1 convolution, the number of output channels of the 1×1 convolution is equal to the number of semantic categories, 12), and output a semantic segmentation result image of the Thangka image (size 512×512). The category probability of each pixel is shown in formula (14):

[0111] (14)

[0112] Where c represents the semantic category (c=0 for background, c=1 for main character, c=2 to c=12 for various magical artifacts), and P(c) is the probability that a pixel belongs to category c.

[0113] Step 4: Model Training and Optimization

[0114] ①Loss function: The cross-entropy loss (CE Loss) and Dice loss are jointly optimized, as shown in formula (15):

[0115] (15)

[0116] Among them, CE Loss is used to measure the difference between the predicted probability of the class and the true label, and Dice Loss is used to alleviate class imbalance (such as the small sample size of the ritual implement "karma pestle"). The weight ratio of 0.7:0.3 has been verified by experiments as the optimal one.

[0117] ② Optimizer and learning rate: The Adam optimizer is used, with an initial learning rate of 0.01 and a weight decay coefficient of 1e-4; a cosine annealing learning rate strategy is adopted, as shown in formula (16):

[0118] (16)

[0119] in, =0.01 is the initial learning rate. This represents the current iteration number. =45000 represents the total number of iterations;

[0120] ③ Training iterations: The total number of iterations is 45,000, and the batch size is set to 4; the model weight with the highest mIoU is saved every 500 iterations.

[0121] The abbreviations above are explained as follows:

[0122] PIDNet: The full name of the English word is "Parallel, Interleaved, and Dilated Network". It is a multi-branch deep learning model for real-time semantic segmentation. It contains three branches: P (proportional), I (integral), and D (differential), which are responsible for extracting local details, fusing multi-scale semantics, and capturing high-frequency boundary features, respectively.

[0123] The ECA-Pag module, short for "Efficient Channel Attention-Path AggregationModule," is an improved module that introduces the ECA channel attention mechanism and residual connections on the basis of the original Pag module of PIDNet. It is used to enhance the P branch's ability to perceive features of key areas of Thangka (such as the face of the main deity and the outline of ritual implements).

[0124] LGFM module: The full English name is "Local-Global Feature Fusion Module". It achieves deep fusion of local details (such as the texture of ritual implements) and global semantics (such as the spatial positional relationship between the main deity and the ritual implements) of Thangka through the collaboration of local branches (Transformer-like structure) and global branches (dilated convolution + attention).

[0125] PConv3: The full English name is "Partial Convolution 3". It is a convolution operation that uses a masking mechanism to filter effective features. It can ignore invalid background areas in Thangka images and focus on extracting detailed features of the main deity and ritual implements, while reducing computational complexity.

[0126] The specific experimental procedures are as follows:

[0127] 1. Experimental Environment

[0128] Table 1 Environment Configuration Table

[0129]

[0130] 2. Evaluation Indicators

[0131] ①mIoU (mean intersection-union ratio): a core indicator that measures the degree of overlap between the predicted area and the actual area, as shown in formula (17):

[0132] (17)

[0133] Where N is the number of categories (12 categories). For the number of correctly predicted pixels of class i, Let i be the number of pixels predicted in the i-th type of error. This represents the number of missed prediction pixels for the i-th class.

[0134] ②mB-Fscore (mean boundary F-score): measures the accuracy of boundary segmentation. First, the predicted and true boundaries are extracted through dilation, and the boundary IoU is calculated (as shown in Equation 18). Then, the boundary F-score is calculated (as shown in Equation 19), and finally, the average value is taken (as shown in Equation 20).

[0135] (18)

[0136] (19)

[0137] (20)

[0138] in, Let be the number of pixels where the i-th predicted boundary intersects with the true boundary. The number of pixels in the union. Avoid having a denominator of 0;

[0139] ③ FPS (Frames Per Second): Measures real-time performance. It calculates the inference time of the model for a single 512×512 image. FPS = 1 / average inference time.

[0140] ④ Parameters: Measures the lightweight nature of the model, and counts the total number of trainable parameters in the network (unit: M).

[0141] 3. Comparison Algorithm

[0142] Five mainstream real-time semantic segmentation models were selected for comparison to ensure experimental fairness (uniform training iteration count, batch size, and learning rate strategy):

[0143] STDC1 and STDC2: Lightweight models that use stepwise dilated convolution to optimize the receptive field;

[0144] DDRNet-s and DDRNet: Dual-resolution networks that balance accuracy and speed through high and low resolution branches;

[0145] PIDNet-s and PIDNet-m: Original PIDNet series models, with a multi-branch design adapted for real-time segmentation.

[0146] The experimental results obtained from the above are as follows:

[0147] 1. Experimental results of the Thangka dataset

[0148] Quantitative Results: The performance metrics of the improved model on the Thangka test set are shown in the table below. Compared with the control model, the mIoU reaches 73.28% (an improvement of 3.14% over PIDNet-m), the mB-Fscore reaches 40.01% (an improvement of 4.07% over PIDNet-m), the FPS reaches 109.03 (meeting real-time requirements), and the number of parameters is 29.418M (only an increase of 0.657M over PIDNet-m).

[0149] Table 2. Results of the model comparison experiment on the Thangka dataset.

[0150]

[0151] Category-level results: In key ritual implement categories, the present invention demonstrates significant performance advantages: Vajra IoU reaches 58.91% (an improvement of 11.12% compared to PIDNet-m), Lotus IoU reaches 70.05% (an improvement of 6.60% compared to PIDNet-m), and Sutra IoU reaches 81.82% (an improvement of 11.81% compared to PIDNet-m), solving the problem of missegmentation of similar ritual implements and fine textures by traditional models.

[0152] Qualitative results: Visual results show that the present invention can accurately separate the main deity from various ritual implements: for example, the boundaries between the vajra and the vase held by the main deity are clear without background misjudgment, there is no omission or repetition in dense scripture areas, and the small vajra bell and the large ritual umbrella can be accurately distinguished without category confusion.

[0153] like Figure 4 As shown in the visualization results, the improved model based on the proposed PIDNet demonstrates significant advantages over existing models such as STDC2, DDRNet, and PIDNet in terms of segmentation accuracy and detail capture in the semantic segmentation task of Thangka images. The STDC2 model exhibits numerous false negatives and missed detections when handling complex textures and backgrounds. While the DDRNet model shows improvement in overall segmentation, its detail accuracy remains insufficient. The PIDNet model performs well in capturing central regions and some details, but still has limitations in handling local details and complex backgrounds. In contrast, the improved model accurately captures detailed information in Thangka images, exhibiting clear segmentation boundaries and superior overall performance. This effectively addresses the challenges of complex backgrounds, diverse textures, and rich details in Thangka images, providing strong technical support for Thangka image research.

[0154] 2. Ablation experiment results (verifying the effectiveness of the core module)

[0155] Using PIDNet-m as the baseline model, ablation experiments were conducted with ECA-Pag, LGFM, and PConv3 modules added respectively. The results are shown in the table below:

[0156] Table 3 Comparison of ablation experiment results for Tangka dataset.

[0157]

[0158] The results show that all three modules can improve model performance. LGFM has the most significant improvement on mIoU (+4.0%), PConv3 can improve FPS (+12.41), and ECA-Pag can optimize boundary segmentation (mB-Fscore +0.76%). When the three work together, the mB-Fscore is improved the most significantly (+4.83%), which verifies the rationality of the module combination.

[0159] Figure 5 The paper presents the loU and BF scores for different element categories in ablation experiments. By adding different modules to the baseline model, the model's performance is significantly improved. In particular, the proposed model not only performs well in terms of mloU but also achieves state-of-the-art results in boundary prediction. This finding indicates that each proposed module has a positive impact on the overall model performance, especially in enhancing boundary prediction capabilities.

[0160] 3. Generalization Experiment Results (Cityscapes Dataset)

[0161] Tested on the publicly available Cityscapes dataset (an urban scene semantic segmentation benchmark), this invention achieved an mIoU of 75.40%, a 1.02% improvement over PIDNet-m (74.38%), indicating that the model is not only applicable to Thangka images but can also be transferred to other scenes, demonstrating strong generalization ability.

[0162] Table 4 Comparison Experiment of Cityscapes Dataset

[0163]

[0164] As can be seen from the above description, the present invention has the following improvements:

[0165] ① Improved accuracy: Through the collaboration of ECA-Pag and LGFM modules, the segmentation accuracy of key elements (main deity, ritual implements) of Thangka is significantly improved, with mIoU reaching 73.28%, solving the problems of detail loss and category confusion in traditional models;

[0166] ② Real-time performance guarantee: The PConv3 module reduces the amount of computation, and the model FPS reaches 109.03, which meets the real-time interaction requirements in the digital restoration of Thangka (such as real-time annotation and restoration preview).

[0167] ③ Strong generalization ability: It performs well on the Cityscapes dataset and can be transferred to other semantic segmentation scenarios, making it widely applicable;

[0168] ④ High practicality: It provides technical support for the digital preservation of Thangka paintings, and can help restorers accurately locate damaged areas (such as missing parts of ritual implements), reduce manual labeling costs, and improve the efficiency of Thangka cultural heritage preservation.

[0169] like Figure 6 As shown in the visualization comparison, our model significantly outperforms DDRNet and PIDNet in segmentation accuracy in key regions. For example, in the first column of images, DDRNet incorrectly identifies the background as a vehicle, while PIDNet also produces misclassifications in large areas; in contrast, our model perfectly preserves the truck's outline with minimal background interference. In the second column of images, PIDNet's recognition of distant pedestrians is blurred, while our model clearly distinguishes pedestrians from the background. In the third column of images, the edges of our proposed model are smoother, resulting in clearer boundaries for each object and reducing mislabeling. Overall, our model reduces misclassifications and blurred boundaries.

[0170] Comparison of loss functions, such as Figure 7 As shown, the loss function comparison between the baseline model and the proposed model is illustrated. Overall, the proposed model achieves lower loss throughout the training process, indicating that it outperforms the PIDNet model under the same training steps, thus validating the effectiveness of the experiments.

[0171] The "visualization" results of the ablation experiment are as follows Figure 8 As shown in the figure, through analysis of the visualization results, we can see that the semantic segmentation model's prediction performance on Thangka images gradually improves with the individual introduction of different modules. Each column represents the individual introduction of a module, while the last column shows the final effect after comprehensively applying all modules to the baseline model. When using PIDNet-m as the baseline model (first column), the model is able to capture the outline of the main object relatively well. However, there is still room for improvement in detail and background separation. For example, the separation of image edges and segmentation boundaries from the background is not clear enough, with some blurring and omissions.

[0172] All the above experiments were conducted under the following conditions:

[0173] ① Hardware environment: It adopts an Intel Xeon Gold 6240 processor (16 cores and 32 threads), 256GB DDR4 memory, and an NVIDIA RTX 3090 graphics card (24GB video memory) to ensure hardware support for model training and inference;

[0174] ② Software environment: Install Ubuntu 20.04 LTS operating system, configure Python 3.8, create a virtual environment through Anaconda, and install PyTorch 1.12.0 (CUDA version 11.6), OpenCV-Python 4.6.0 and other dependent libraries;

[0175] ③ Pre-trained weight preparation: Download the PIDNet-m pre-trained weights (pidnet_m_80k_cityscapes.pth) from the mmsegmentation official repository.

[0176] (2) Model training implementation steps

[0177] Configuration file writing: Create a configuration file named "pidnet_m_thangka.py" in the "configs / pidnet / " directory of mmsegmentation, and set the following:

[0178] Dataset path: Specifies the path to the Thangka training / test set images and annotation files;

[0179] Model structure: Configure the P, I, and D branch parameters of the improved PIDNet, and add configurations for ECA-Pag, LGFM, and PConv3 modules (such as kernel size, number of channels, and attention parameters).

[0180] Training parameters: The optimizer is set to Adam, the initial learning rate is 0.01, the weight decay is 1e-4, the batch size is 4, the total number of iterations is 45000, and the learning rate strategy is cosine annealing as shown in formula (16).

[0181] Loss function: Set the weight ratio of CE Loss to Dice Loss as shown in formula (15) to 0.7:0.3.

[0182] Model training startup: Execute the command "python tools / train.py configs / pidnet / pidnet_m_thangka.py --work-dir work_dirs / pidnet_m_thangka --gpu-id 0" in the terminal to start training;

[0183] During training, every 1000 iterations, mIoU and mB-Fscore are evaluated on the validation set (calculated as in formula 17-20), and the optimal model weights are saved to "work_dirs / pidnet_m_thangka / best_mIoU.pth";

[0184] The training log is saved to "work_dirs / pidnet_m_thangka / train.log", recording the changes in loss value (as in Equation 15), learning rate (as in Equation 16), and evaluation metrics.

[0185] Inference script writing: Create the script "infer_thangka.py" to implement the following functions:

[0186] Load the optimal model weights (best_mIoU.pth) and configuration file;

[0187] Read the Thangka image to be segmented (supports JPG / PNG format) and preprocess it (resize to 512×512, normalize).

[0188] Call the model to perform inference, output the semantic segmentation result map (colored by category) according to formula (14), and save it to the "results / " directory;

[0189] Calculate the inference time for a single image and calculate the FPS.

[0190] Inference execution: Execute the command "python infer_thangka.py --img-path data / thangka / test / --save-path results / --gpu-id 0" in the terminal to perform batch inference on the test set images;

[0191] After inference is complete, view the segmentation results in the "results / " directory and verify the real-time performance using FPS statistics (FPS ≥ 100 is required).

[0192] Implementation effect verification

[0193] ① Quantitative Validation: Run the script "eval_thangka.py" to calculate the mIoU, mB-Fscore, and FPS of the test set according to formulas (17-20). Compare the results with those of the comparison models (such as PIDNet-m and DDRNet) to verify the accuracy and speed advantages of the present invention. In specific operation, the script will automatically traverse the 922 images in the test set, calculate the IoU of each category for each image and take the average (corresponding to formula 17), obtain the predicted and true boundaries through the boundary extraction algorithm and calculate the mB-Fscore (corresponding to formulas 18-20), and convert the inference time of a single image into FPS. Finally, output a complete quantitative index comparison table to intuitively reflect the performance improvement of the present invention relative to the comparison models.

[0194] ② Qualitative Verification: Twenty test images were randomly selected (covering different styles of Thangka, such as painted Thangka and embroidered Thangka, and including scenes with the main deity and various ritual implements). Based on the semantic segmentation requirements of Thangka images, evaluation criteria were established to compare the segmentation results of this invention with those of PIDNet-m. The evaluation criteria included: ① Accuracy of main deity segmentation (whether the outline of the main deity is complete and whether there are no background areas misidentified as the main subject); ② Accuracy of ritual implement categories (whether various ritual implements are correctly classified, without confusion between similar categories such as vajra and bell, dharma wheel and lotus); ③ Boundary clarity (whether the boundaries between the main deity and the background, and between the ritual implements and the main deity are smooth, without obvious jagged edges or missing segments). Each image meeting the above three criteria was deemed "qualified." The qualitative evaluation pass rate of this invention must be ≥95% to ensure that the segmentation results meet practical application requirements.

[0195] ③ Stability verification: Repeat the training 3 times, each time using a different random seed (set to 123, 456, 789 respectively), keeping other training parameters (such as learning rate, batch size, number of iterations) completely consistent. After training, extract the highest mIoU value of the validation set from the 3 training sessions, calculate the mIoU standard deviation according to the standard deviation calculation formula, which should be ≤0.5%, to verify the stability of the model training.

[0196] The following points should be noted during the experiment:

[0197] ① When preprocessing Thangka images, it is necessary to maintain the original aspect ratio of the image and avoid element deformation caused by stretching (bilinear interpolation is used during resizing to maintain a consistent aspect ratio).

[0198] ② If overfitting occurs during training (high mIoU on the training set and low mIoU on the validation set), data augmentation strategies (such as random rotation or horizontal flipping) can be added, or the learning rate can be reduced.

[0199] ③ During inference, if the image resolution exceeds 512×512, it is necessary to perform block processing first (such as dividing a 1024×1024 image into 4 512×512 sub-images), and then stitch them together after inference to avoid insufficient video memory.

[0200] ④ When deploying the model, it can be accelerated through TensorRT to further improve FPS (after acceleration, FPS can be increased to over 120) and adapt to embedded devices (such as Thangka digital restoration workstation).

[0201] The above description is only a preferred embodiment of this disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features can be replaced with technical features with similar functions disclosed in the embodiments of this disclosure (but not limited to those with similar functions).

Claims

1. A real-time Thangka image semantic segmentation method based on improved PIDNet, characterized in that, Comprising the following steps: S1: constructing an improved PIDNet semantic segmentation model, embedding an ECA-Pag module and an LGFM module in turn at the end of the P branch of the original PIDNet, and embedding a PConv3 module before the feature fusion of the D branch, to form a deep learning network for Thangka image semantic segmentation; S2: inputting the Thangka image to be segmented into the improved PIDNet model, extracting local detail features, multi-scale semantic features and boundary features through the P, I and D branches respectively, and performing feature enhancement and optimization through the ECA-Pag module, the LGFM module and the PConv3 module in turn to obtain optimized multi-branch features; S3: performing scale alignment and channel unification processing on the optimized multi-branch features, and performing feature fusion through element addition to generate a fused feature map; S4: inputting the fused feature map into a Softmax classifier to output a pixel-level semantic segmentation result map, and completing real-time semantic segmentation of the Thangka image.

2. The method of claim 1, wherein, The feature enhancement of the P branch is realized by the following steps: local detail feature maps extracted by the p branch semantic feature maps F extracted by the i branch I input the ECA-Pag module; In the ECA-Pag module, first, the channel dimension is reduced by 1x1 convolution to make it consistent with the channel number of , and channel alignment is completed; By using the ECA attention mechanism, the channel weight W is adaptively generated through global average pooling and one-dimensional convolution operation ECA ; The channel weight W ECA is multiplied by the aligned , and the output enhanced local feature map is obtained through residual connection ; Will The input LGFM module further fuses the local details and the global semantic information through a double-branch structure of a local branch and a global branch, and outputs an optimized P-branch feature map .

3. The method of claim 1, wherein, The feature optimization of the I branch is realized by the following steps: The I branch extracts a multi-scale semantic feature map by performing a multi-scale pooling operation on the input image through a network structure comprising a PPM pooling module ; The PPM pooling module adopts 1×1, 2×2, 3×3 and 6×6 average pooling operations of four different scales to capture multi-scale information from details to the whole in the Thangka image; The feature maps after different scale pooling are upsampled to make their sizes consistent with the original feature maps; The up-sampled feature map is fused in channel by 1x1 convolution to generate a final multi-scale semantic feature map , for assisting feature extraction and optimization of P branch and D branch.

4. The method of claim 1, wherein, The feature optimization of the D branch is realized by the following steps: The D branch extracts the boundary feature map of the input image through a subnetwork comprising two convolution blocks ; Will The PConv3 module is input, first through a 3x3 convolution and a sigmoid activation function, to generate a mask M, which is used to distinguish the effective area (main Buddha, ritual implements) and the background area in the Thangka image. In the PConv3 module, only the region with mask M as "1" is executed for partial convolution operation, the feature is extracted through a 3×3 convolution kernel, and a normalization strategy is adopted to avoid the interference of invalid background region on feature extraction; The channel number is adjusted by 1x1 convolution, and an optimized boundary feature map is output So that it can more accurately capture the boundary information such as the main worship outline and the edge of the ritual instrument in the Thangka image in the subsequent feature fusion process.

5. The method of claim 1, wherein, The multi-branch feature fusion step includes: the optimized p-branch feature map is up-sampled so that its spatial resolution is consistent with the d-branch feature map ; The I branch feature map An up-sampling operation is performed, and the number of channels is adjusted by a 1x1 convolution to be uniform with and the number of channels of the I branch feature map The up-sampling and channel-adjusted , , The three feature maps are fused in an element-wise addition manner to generate a fused feature map F fuse ; Fused feature map F fuse The local detail information of the P branch, the multi-scale semantic information of the I branch, and the boundary information of the D branch are integrated, so that rich feature representation is provided for subsequent semantic segmentation.

6. The method of claim 1, wherein, During the model training process, a joint loss function of cross-entropy loss and Dice loss is adopted, the weight ratio is 0.7:0.3, and the Adam optimizer and the cosine annealing learning rate strategy are used for parameter optimization.

7. A real-time Thangka image semantic segmentation system based on improved PIDNet, characterized in that, Comprise: An image preprocessing module for uniformizing the size, normalizing and format converting of the input Thangka image; An improved PIDNet model comprising P, I and D branches and ECA-Pag, LGFM and PConv3 modules for extracting and optimizing image features; A feature fusion module for scale alignment and fusion of the optimized multi-branch features; A semantic segmentation output module for inputting the fused feature map into a Softmax classifier to output a pixel-level semantic segmentation result map.

Images