Thangka image segmentation method based on edge feature guidance and detail feature denoising

Abstract

The present invention provides a thangka image segmentation method based on edge feature guidance and detail feature denoising, which relates to the technical field of computer vision and image processing. The present invention introduces a detail feature enhancement attention module based on a U-Net network, and trains the detail feature enhancement attention module by adding random noise and performing reverse denoising; secondly, the trained module is applied to the feature map of the thangka image, and Gaussian noise is added for iterative denoising to generate a real sample; then, a window multi-head self-attention pooling pyramid is constructed by combining multi-head self-attention and global average pooling of different window sizes to capture multi-scale feature information in the global context; finally, by introducing the fusion interaction of edge features and semantic information, DW convolution is used to optimize the feature layer information output of different scales, and the edge feature extraction effect is improved by the edge loss function. The method effectively improves the denoising and segmentation effect of thangka images.

Description

Technical Field

[0001] The invention relates to the technical field of computer vision and image processing, and in particular to a thangka image segmentation method based on edge feature guidance and detail feature denoising. Background Art

[0002] Thangka image segmentation technology, as an important innovation in the field of cultural protection, records the structural layout and complex patterns of Thangka works through meticulous image segmentation processing. Image noise removal and detail feature enhancement are key technologies to improve image quality and resolution. In recent years, research on Thangka images has mainly focused on damaged area detection, and only a few scholars have focused on semantic segmentation of Thangka elements. There are also various problems with semantic segmentation, including high image blur in the dataset, which leads to unclear segmentation of the headdress part of the Thangka image, better extraction of the Buddha part in the Thangka image, but the number of categories in the segmentation result is small, resulting in failure to capture other fine-grained elements in the Thangka image, and the use of multi-scale fusion and attention mechanism to achieve accurate extraction of Thangka elements, but the division of each element is not accurate enough.

[0003] To solve the above problems, a thangka image segmentation method based on edge feature guidance and detail feature denoising is proposed. This method is different from the traditional single processing mode that only relies on segmentation labels. By introducing a carefully crafted edge feature map as a benchmark for edge guidance, it can accurately capture and segment thangka image information.

[0004] In the prior art, publication number CN116229104A discloses a salient target detection method guided by edge features, which extracts edge data of salient objects from a true value map of salient objects, and uses the edge data as an additional preliminary training set; uses a basic network to extract image features, and based on the image features, obtains edge features and multi-scale position features of salient objects respectively; based on a multi-layer perceptron, dynamically learns the relationship between edge features and multi-scale position features to obtain edge enhancement fusion features; sends the edge enhancement fusion features to a convolutional network for refinement processing, and obtains the final segmentation result based on the refined edge enhancement fusion features.

[0005] The main problems with the above methods are: the dynamic learning of the relationship between edge features and multi-scale position features by the multi-layer perceptron may introduce too much model complexity and easily lead to overfitting problems on small sample data sets; the above methods mainly focus on the extraction of edge and position features, ignoring the contextual information in the image, which may lead to information loss or feature redundancy, and lack the ability to understand complex backgrounds and object interactions, affecting the final segmentation results.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not constitute the prior art that is already known to one of ordinary skill in the art. Summary of the invention

[0007] The purpose of the present invention is to provide a thangka image segmentation method based on edge feature guidance and detail feature denoising to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A thangka image segmentation method based on edge feature guidance and detail feature denoising, the specific steps include:

[0010] Step 1: Collect thangka images according to the set time step, and perform detail feature enhancement and denoising of thangka images based on the U-Net network. Introduce a detail feature enhancement attention module into the U-Net network structure, use the collected thangka images and encoder as the input of the detail feature enhancement attention module, add random noise, and train the detail feature enhancement attention module;

[0011] Step 2: Input the feature map of the thangka image into the trained detail feature enhancement attention module. In the forward process, add Gaussian noise to the feature map to obtain the noisy feature map. In the reverse process, iterative denoising is performed on the noisy feature map and the target result to generate a real sample.

[0012] Step 3: Input the processed feature map, combine the window multi-head self-attention of different window sizes with the global average pooling, and generate a window multi-head self-attention pooling pyramid to capture feature information of different scales in the global context information;

[0013] Step 4: Fuse the edge features and semantic information of features of different scales, introduce feature layer information of different scales, perform average pooling and maximum pooling on the feature layer information respectively, and splice them to generate the output of feature layer information of different scales. Fuse the output of feature layer information through DW convolution to generate edge features, and optimize the extraction of edge features through edge loss function.

[0014] Furthermore, the principle for training the detail feature enhanced attention module is:

[0015]

[0016] I out =Conv(Softmax(I c ))

[0017] Among them, I 1 ∈RC×H×W , indicating I 1 is a feature map with C channels, H height, and W width, and I′ 1 ∈R C×H×W , is I 1 ∈R C×H×W The feature map obtained by 3×3 full-dimensional dynamic convolution operation, Resize(I′ 1 ) represents the feature map I′ 1 Resize, Permute(Resize(I′ 1 )) indicates that the dimension of the resized feature map is rearranged, including exchanging the order of the number of channels, height and width of the image. Represents matrix multiplication, I a Represents the feature map of enhanced detail features obtained by matrix multiplication, I 2 ∈R 2C×H×W , represents a feature map with 2C channels, H height, and W width, I′ 2 Indicates that I 2 and I 1 After concat, the feature map is obtained by 3×3 full-dimensional dynamic convolution operation. Indicates element addition, I b It represents the feature map integrating different feature information obtained by element addition, I c represents the feature map combining detail features and semantic information obtained by matrix multiplication, Softmax represents the normalization of the feature map value, Conv represents the convolution operation on the normalized feature map, I out Represents the feature map of the final output after detail feature enhancement and convolution processing.

[0018] Furthermore, the principle of generating real samples by enhancing the attention module through detail features is based on:

[0019] The forward process can be expressed as:

[0020]

[0021] Among them, q represents the conditional probability distribution of the forward diffusion process, x t represents the feature map after noise addition at time step t, x t-1 represents the feature map at time step t-1, t represents the index of the time step, and t = 1, 2, ..., T, T represents the maximum value of the time step, β t represents the variance of the noise at time step t, that is, the noise intensity added at time step t, N represents the Gaussian distribution, and the mean of the Gaussian distribution is The variance is β tI, I represents the covariance matrix of the noise, q(x t |x t-1 ) represents the result of the forward process;

[0022] α t =1-β t

[0023]

[0024] Among them, α t represents the diffusion probability corresponding to the step length t, represents the cumulative product of all diffusion probabilities from time step 1 to t, x 0 represents the original thangka image without noise, and ε represents Gaussian noise;

[0025] According to the feature map after adding noise and the target result, iterative denoising is performed to generate real samples, so p is used θ (x t-1 |x t ) as q(x t-1 |x t ) is used for denoising, which can be expressed as:

[0026] p θ (x t-1 |x t )~N(x t-1 ;z θ (x t ,t),ε θ (x t ,t))

[0027] Among them, p θ represents the conditional probability distribution of the reverse denoising process, θ represents the learning parameters of the model, and z θ (x t ,t) represents the reverse diffusion variable, ε θ (x t ,t) represents the noise variance of the model parameterization;

[0028] The output after detail feature enhancement is generated based on the formula:

[0029]

[0030] in, Represents the predicted value of the original thangka image;

[0031] The loss function is used to calculate the module loss based on the following principle:

[0032]

[0033] Among them, Lmse represents the mean square error loss function, Represents the joint expectation of three variables: time step, original thangka image and random noise.

[0034] Furthermore, the principle for generating the window multi-head self-attention pooling pyramid is:

[0035] The feature map is divided into multiple windows, and the feature representation is performed on each window. The formula is:

[0036] X i ∈R M×M×C

[0037] Among them, X i represents the feature representation of the i-th window, R M×M×C represents a window of size M×M and number of channels C;

[0038] The self-attention of the generated window is based on the formula:

[0039] Q=X i ·W Q

[0040] K=X i ·W K

[0041] V=X i ·W V

[0042]

[0043] Attention(X i )=A·V

[0044] Where Q represents the query vector, W Q represents the query weight matrix, K represents the key vector, and W K represents the key weight matrix, V represents the value vector, and W V represents the value weight matrix, A represents the attention weight, K T represents the transpose of the key vector, d represents the dimension of the key, and d = C, Attention(X i ) represents the self-attention output of the i-th window;

[0045] The calculation of each attention head is:

[0046] head h =Attention(X i W Qh ,X i W Kh ,X i WVh )

[0047] The output of multi-head self-attention is:

[0048] MultiHead(X i )=concat(head 1 ,head 2 ,……,head H )·W O

[0049] Among them, head h represents the output of the h-th attention head, W Qh , W Kh , W Vh They represent the query weight matrix, key weight matrix, and value weight matrix corresponding to the h-th attention head, respectively. MultiHead(X i ) indicates window X i The multi-head self-attention output, W O The learned weight matrix representing the output;

[0050] For the multi-head self-attention output features of each window, global average pooling is performed based on the formula:

[0051]

[0052] Among them, Y i represents the feature map of the i-th window after multi-head self-attention output, GAP(Y i ) represents the result of global average pooling of the feature map output by the multi-head self-attention of the i-th window, Y i [j,k] represents the vector at position (j,k) of the ith window after self-attention processing, where j and k represent the row and column indices respectively;

[0053] The input feature map is processed at multiple scales by different scaling factors. The above calculations of MultiHead and GAP are repeated for each scale to generate feature vectors for each scale. The feature vectors of multiple scales are concatenated to obtain the final multi-scale feature representation. The formula is:

[0054] g final =concat(g s1 ,g s2 ,……,g sn )

[0055] Among them, g final represents multi-scale feature representation, g sn represents the feature vector generated from the scale sn, where sn represents the scaling factor.

[0056] Furthermore, the principle of generating edge features and optimizing the extraction of edge features through edge loss function is as follows:

[0057] Assume that the input feature layer information of different scales is F 4 、F 5 , the formula for splicing and generating the output of feature layer information of different scales is based on:

[0058]

[0059] F′ 5 =Concat(F 5 ,avgpool 1 (F 5 ),avgpool 2 (F 5 ),avgpool 3 (F 5 ))

[0060] Among them, F′ 4 Indicates that the original feature layer information F 4 and F 4 Output after multiple maximum pooling, maxpool 1 (F 4 ) represents the first maximum pooling operation, maxpool 2 (maxpool 1 (F 4 )) indicates that the second maximum pooling is performed based on the first maximum pooling, maxpool 3 (maxpool 2 (maxpool 1 (F 4 ))) indicates that the third maximum pooling is performed on the basis of the first two maximum poolings, F′ 5 Indicates that the original feature layer information F 5 Output after multiple average pooling, avgpool 1 (F 5 ) represents the first average pooling operation, avgpool 2 (F 5 ) represents the second average pooling operation, avgpool 3 (F 5 ) represents the third average pooling operation;

[0061] The Dice loss function is selected as the edge loss function, and the formula is:

[0062] L edge_x =Dice(Upsample(Ex ),E GT )

[0063] Among them, L edge_x represents the edge loss of the input image x, E x Represents the edge feature map of the model output, Upsample(E x ) represents upsampling the edge feature map output by the model, E GT Represents the edge features of the real image.

[0064] Compared with the prior art, the present invention has the following beneficial effects:

[0065] The present invention introduces a detail feature enhancement attention module to more effectively focus on and strengthen the key detail features in the thangka image. By adding Gaussian noise in the forward process and performing iterative denoising in the reverse process, more refined noise suppression is achieved, so that the model can maintain good performance when facing images of different noise levels and types; based on the introduction of detail feature enhancement and denoising modules, the model effectively reduces the noise in the feature map while retaining the image detail information as much as possible, which helps to more accurately identify the structural boundaries of the thangka, thereby improving the segmentation accuracy of the thangka image. In addition, the detail feature enhancement module and the denoising module have adaptive adjustment capabilities, so that the model can flexibly respond to diverse and complex scenes, and can adapt to thangka images of different clarity during segmentation, further enhancing the robustness and accuracy of segmentation.

[0066] The present invention also combines the window multi-head self-attention mechanism with the global average pooling to generate a window multi-head self-attention pooling pyramid, captures different scale features in the global context information, and flexibly captures feature information of different scales in the thangka image by selecting a variety of window sizes. This self-attention mechanism enables the model to automatically focus on key areas in the thangka image, such as important religious symbols and characters, thereby optimizing the expression of features and improving the accuracy of segmentation. The detail information provided by the small window is combined with the context information provided by the large window, so that the model can pay attention to details and grasp the overall situation, effectively extract multi-scale context features in the image, and retain spatial information, ensuring the accurate extraction of complex patterns and details in the thangka image, thereby further improving the segmentation performance of the algorithm.

[0067] The present invention also introduces feature layer information of different scales and fuses them, and performs average pooling and maximum pooling on the feature layer information respectively, so as to effectively ensure the uniformity and importance of features, improve the expressiveness of features, and form more comprehensive and comprehensive features and edge features based on DW convolution, so that when processing thangka images, the segmentation process is faster and more efficient, while maintaining high-quality segmentation effects, further enhancing the effectiveness of feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a schematic diagram of a method flow of an embodiment of the present invention;

[0069] Figure 2 This is a flow chart of a module for training detail features to enhance attention according to an embodiment of the present invention;

[0070] Figure 3 A schematic diagram of a detail feature enhancement attention module according to an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of a window multi-head self-attention pooling pyramid according to an embodiment of the present invention;

[0072] Figure 5 Schematic diagram of a multi-scale edge feature guidance module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0074] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0075] Example:

[0076] See also Figures 1 to 5 , the present invention provides a technical solution:

[0077] A thangka image segmentation method based on edge feature guidance and detail feature denoising, the specific steps include:

[0078] Step 1: Collect thangka images according to the set time step, and perform detail feature enhancement and denoising of thangka images based on the U-Net network. Introduce a detail feature enhancement attention module into the U-Net network structure, use the collected thangka images and encoder as the input of the detail feature enhancement attention module, add random noise, and train the detail feature enhancement attention module;

[0079] In this embodiment, high-definition thangka images are collected, and the thangka images are annotated at the pixel level using Labelme software, and the images are scaled to a size of 512×512 to establish a high-definition thangka dataset;

[0080] In this embodiment, the principle of training the detail feature enhanced attention module is:

[0081]

[0082] I out =Conv(Softmax(I c ))

[0083] Among them, I 1 ∈R C×H×W , indicating I 1 is a feature map with C channels, H height, and W width, and I′ 1 ∈R C×H×W , is I 1 ∈R C×H×W The feature map obtained by 3×3 full-dimensional dynamic convolution operation, Resize(I′ 1 ) represents the feature map I′ 1 Resize, Permute(Resize(I′ 1 )) indicates that the dimension of the resized feature map is rearranged, including exchanging the order of the number of channels, height and width of the image. Represents matrix multiplication, I a Represents the feature map of enhanced detail features obtained by matrix multiplication, I 2 ∈R 2C×H×W , represents a feature map with 2C channels, H height, and W width, I′ 2 Indicates that I 2 and I 1 After concat, the feature map is obtained by 3×3 full-dimensional dynamic convolution operation. Indicates element addition, I b It represents the feature map integrating different feature information obtained by element addition, I c It represents the feature map combining detail features and semantic information obtained by matrix multiplication. The collected image and the result after the image is processed by the encoder are used as input. Softmax represents the normalization of the value of the feature map. Conv represents the convolution operation on the normalized feature map. out Represents the feature map of the final output after detail feature enhancement and convolution processing.

[0084] Generate I′ 1and I′ 2 The formula is:

[0085] I′ 1 =Conv 3×3 (I 1 )

[0086] I′ 2 =Conv 3×3 (Concat(I 2 ,I 1 ))

[0087] Among them, Conv 3×3 (I 1 ) indicates that I 1 Apply a 3×3 convolution operation, Conv 3×3 (Concat(I 2 ,I 1 )) indicates the spliced ​​I 2 ,I 1 Apply a 3×3 convolution operation;

[0088] to I′ 1 Resize to ensure that the size is consistent when the feature map is concatenated. The Permute operation is to change the dimension order of the feature map, changing the dimension from R C×H×W Transform to R H×W×C To facilitate subsequent operations.

[0089] Step 2: Input the feature map of the thangka image into the trained detail feature enhancement attention module. In the forward process, add Gaussian noise to the feature map to obtain the noisy feature map. In the reverse process, iterative denoising is performed on the noisy feature map and the target result to generate a real sample.

[0090] In this embodiment, the principle for generating real samples by enhancing the attention module through detail features is:

[0091] The forward process can be expressed as:

[0092]

[0093] Among them, q represents the conditional probability distribution of the forward diffusion process, x t represents the feature map after noise addition at time step t, x t-1 represents the feature map at time step t-1, t represents the index of the time step, and t = 1, 2, ..., T, T represents the maximum value of the time step. When T is set to the maximum number, the reverse process can better approximate the Gaussian distribution. trepresents the variance of the noise at time step t, that is, the noise intensity added at time step t, N represents the Gaussian distribution, and the mean of the Gaussian distribution is The variance is β t I, I represents the covariance matrix of the noise, q(x t |x t-1 ) represents the result of the forward process;

[0094] According to the input noise-free original thangka image and the known diffusion probability, the above forward process can be expressed as:

[0095] α t =1-β t

[0096]

[0097] Among them, α t represents the diffusion probability corresponding to the step length t, represents the cumulative product of all diffusion probabilities from time step 1 to t, x 0 represents the original thangka image without noise, and ε represents Gaussian noise;

[0098] According to the feature map after adding noise and the target result, iterative denoising is performed to generate real samples, so p is used θ (x t-1 |x t ) as q(x t-1 |x t ) is used for denoising, which can be expressed as:

[0099] p θ (x t-1 |x t )~N(x t-1 ;z θ (x t ,t),ε θ (x t ,t))

[0100] Among them, p θ represents the conditional probability distribution of the reverse denoising process, θ represents the learning parameters of the model, and z θ (x t ,t) represents the reverse diffusion variable, ε θ (x t ,t) represents the noise variance of the model parameterization;

[0101] In each training process, θ (x t ,t) Parameterize the denoising model to obtain the predicted x 0 ;

[0102] The output after detail feature enhancement is generated based on the formula:

[0103]

[0104] in, Represents the predicted value of the original thangka image;

[0105] The loss function is used to calculate the module loss based on the following principle:

[0106]

[0107] Among them, L mse represents the mean square error loss function, Represents the joint expectation of three variables: time step, original thangka image and random noise.

[0108] The diffusion model is a generative model based on the Markov chain, which gradually transforms the Gaussian noise distribution into a more complex distribution, making it suitable for fields such as semantic segmentation and text-to-image generation. The diffusion model includes a forward process and a reverse process, namely the learning denoising stage and the implementation denoising stage. In the forward process, Gaussian noise is added to the feature map of the model to obtain a noisy feature map, and denoising is performed in the reverse process.

[0109] Step 3: Input the processed feature map, combine the window multi-head self-attention of different window sizes with the global average pooling, and generate a window multi-head self-attention pooling pyramid to capture feature information of different scales in the global context information;

[0110] In this embodiment, the multi-head self-attention mechanism calculates multiple self-attention mechanisms in parallel, so that the model can capture the relationship between elements in the input sequence in different subspaces. CNN is used as the backbone network, and the window self-attention mechanisms of different window sizes are combined to generate a window multi-head self-attention pooling pyramid to achieve the capture of global context information of feature information of different scales.

[0111] In this embodiment, the principle for generating the window multi-head self-attention pooling pyramid is:

[0112] The feature map is divided into multiple windows, and the feature representation is performed on each window. The formula is:

[0113] X i ∈R M×M×C

[0114] Among them, X i represents the feature representation of the i-th window, R M×M×C represents a window of size M×M and number of channels C;

[0115] The self-attention of the generated window is based on the formula:

[0116] Q=X i ·W Q

[0117] K=X i ·W K

[0118] V=X i ·W V

[0119]

[0120] Attention(X i )=A·V

[0121] Where Q represents the query vector, W Q represents the query weight matrix, K represents the key vector, and W K represents the key weight matrix, V represents the value vector, and W V represents the value weight matrix, A represents the attention weight, K T represents the transpose of the key vector, d represents the dimension of the key, Attention(X i ) represents the self-attention output of the i-th window;

[0122] Weight matrix W Q , W K , W V The dimensions are all (C, d), and the weight matrix is ​​updated in each training cycle through the ADAM optimization algorithm;

[0123] The calculation of each attention head is:

[0124] head h =Attention(X i W Qh ,X i W Kh ,X i W Vh )

[0125] The output of multi-head self-attention is:

[0126] MultiHead(X i )=concat(head 1 ,head 2 ,……,head H )·W O

[0127] Among them, head h represents the output of the h-th attention head, WQh , W Kh , W Vh They represent the query weight matrix, key weight matrix, and value weight matrix corresponding to the h-th attention head, respectively. MultiHead(X i ) indicates window X i The multi-head self-attention output, W O The learned weight matrix representing the output;

[0128] For the multi-head self-attention output features of each window, global average pooling is performed based on the formula:

[0129]

[0130] Among them, Y i represents the feature map of the i-th window after multi-head self-attention output, GAP(Y i ) represents the result of global average pooling of the feature map output by the multi-head self-attention of the i-th window, Y i [j,k] represents the vector at position (j,k) of the ith window after self-attention processing, where j and k represent the row and column indices respectively;

[0131] The input feature map is processed at multiple scales by different scaling factors. The above calculations of MultiHead and GAP are repeated for each scale to generate feature vectors for each scale. The feature vectors of multiple scales are concatenated to obtain the final multi-scale feature representation. The formula is:

[0132] g final =concat(g s1 ,g s2 ,……,g sn )

[0133] Among them, g final represents multi-scale feature representation, g sn represents the feature vector generated from the scale sn, where sn represents the scaling factor.

[0134] After the feature layer, the window self-attention mechanism (WMSA) with window sizes of 2, 4, 6, and 8 and global average pooling are used. WMSA with different window sizes can focus on local and global features at the same time. Small windows capture fine-grained local spatial information, and large windows effectively capture global contextual information. The information interaction between WMSA with different window sizes can achieve a complementary effect, combining the detail information provided by small windows with the contextual information provided by large windows, so that the model can focus on details and grasp the overall situation. The output of WMSA with different window sizes is spliced ​​and fused with the global average pooling result, and the fused feature layer is reduced to the target number of channels in combination with a 1×1 convolutional layer, so that the WMSA with different window sizes forms a complementary effect, so that the model can focus on details and grasp the overall situation.

[0135] Step 4: Fuse the edge features and semantic information of features of different scales, introduce feature layer information of different scales, perform average pooling and maximum pooling on the feature layer information respectively, and splice them to generate the output of feature layer information of different scales. Fuse the output of feature layer information through DW convolution to generate edge features, and optimize the extraction of edge features through edge loss function.

[0136] In this embodiment, the principle of generating edge features and optimizing the extraction of edge features through the edge loss function is:

[0137] Assume that the input feature layer information of different scales is F 4 、F 5 , feature layer Indicates the number of channels is 2C 0 , height is H 0 , width is W 0 The feature map contains semantic information. By applying the average pooling of different scale pooling kernels in parallel, F is extracted. 5 The diversified feature representation in the feature layer effectively reduces the information loss that may occur during the pooling process, ensuring that important features are fully preserved. On this basis, the most significant features in the region are retained through maximum pooling. Indicates the number of channels is C 0 , height is 2H 0 , width is 2W 0 The feature map contains spatial information. By serially cascading and applying pooling kernels of different sizes, the maximum pooling is performed on the feature layer F. 4 The edge features are deeply extracted. The serial maximum pooling uses pooling kernels of different sizes in each layer and extracts features in a layer-by-layer progressive manner, thereby effectively capturing edge information from subtle to global ranges. The formula for splicing and generating the output of feature layer information of different scales is based on:

[0138]

[0139] F′ 5 =Concat(F 5 ,avgpool 1 (F 5 ),avgpool 2 (F 5 ),avgpool 3 (F 5 ))

[0140] Among them, F′ 4 Indicates that the original feature layer information F 4 Output after multiple maximum pooling, maxpool 1 (F 4 ) represents the first maximum pooling operation, maxpool 2 (maxpool 1 (F 4 )) indicates that the second maximum pooling is performed based on the first maximum pooling, maxpool 3 (maxpool 2 (maxpool 1 (F 4 ))) indicates that the third maximum pooling is performed on the basis of the first two maximum poolings, F′ 5 Indicates that the original feature layer information F 5 Output after multiple average pooling, avgpool 1 (F 5 ) represents the first average pooling operation, avgpool 2 (F 5 ) represents the second average pooling operation, avgpool 3 (F 5 ) represents the third average pooling operation;

[0141] Through DW convolution, F′ 5 and F′ after downsampling 4 Perform feature fusion to generate a more comprehensive and comprehensive feature representation f 5 and edge feature E 5 , F′ 5 Incorporated F' 4 The spatial information in the image is extracted, and the fine edge features and high-level semantic information in the image are extracted, and the edge features E are predicted and generated. 5 As edge guidance information, the effectiveness of features is further enhanced, and the edge loss mechanism is introduced. 5 Guided by , the extraction of edge features is directly optimized.

[0142] In each channel, a separate convolution kernel is used for convolution. The size of each convolution kernel is K×K. For each channel of the input feature map,

[0143] For unbalanced data sets, the Dice loss function is selected as the marginal loss function, and the formula is:

[0144] L edge_x =Dice(Upsample(E x ),E GT )

[0145] Among them, L edge_x represents the edge loss of the input image x, E x Represents the edge feature map of the model output, Upsample(E x ) represents upsampling the edge feature map output by the model, E GT Represents the edge features of the real image.

[0146] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0147] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0148] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A thangka image segmentation method based on edge feature guidance and detail feature denoising, characterized in that: The specific steps include: Step 1: Collect thangka images according to the set time step, and perform detail feature enhancement and denoising of thangka images based on the U-Net network. Introduce a detail feature enhancement attention module into the U-Net network structure, use the collected thangka images and encoder as the input of the detail feature enhancement attention module, add random noise, and train the detail feature enhancement attention module; Step 2: Input the feature map of the thangka image into the trained detail feature enhancement attention module. In the forward process, add Gaussian noise to the feature map to obtain the noisy feature map. In the reverse process, iterative denoising is performed on the noisy feature map and the target result to generate a real sample. Step 3: Input the processed feature map, combine the window multi-head self-attention of different window sizes with the global average pooling, and generate a window multi-head self-attention pooling pyramid to capture feature information of different scales in the global context information; Step 4: The features of different scales are fused and interacted with the edge features and semantic information, and the feature layer information of different scales is introduced. The feature layer information is average pooled and maximum pooled respectively, and the output of the feature layer information of different scales is generated by splicing. The output of the feature layer information is fused by DW convolution to generate edge features, and the edge feature extraction is optimized by the edge loss function. The principle behind generating a windowed multi-head self-attention pooling pyramid is: The feature map is divided into multiple windows, and the feature representation is performed on each window. The formula is: X j ∈R M×M×C Among them, X i represents the feature representation of the i-th window, R M×M×C represents a window of size M×M and number of channels C; The self-attention of the generated window is based on the formula: Q=X i ·W Q K=X i ·W K V=X i ·W V Attention(X i )=A·V Where Q represents the query vector, W Q represents the query weight matrix, K represents the key vector, and W K represents the key weight matrix, V represents the value vector, and W V represents the value weight matrix, A represents the attention weight, K T represents the transpose of the key vector, d represents the dimension of the key, and d = C, Attention(X i ) represents the self-attention output of the i-th window; The calculation of each attention head is: head h =Attention(X i W Qh ,X i W Kh ,X i W Vh ) The output of multi-head self-attention is: MultiHead(X i )=concat(head1,head2,......,head H )·W O Among them, head h represents the output of the h-th attention head, W Qh , W Kh , W Vh They represent the query weight matrix, key weight matrix, and value weight matrix corresponding to the h-th attention head, respectively. MultiHead(X i ) indicates window X i The multi-head self-attention output, W O The learned weight matrix representing the output; For the multi-head self-attention output features of each window, global average pooling is performed based on the formula: Among them, Y i represents the feature map of the i-th window after multi-head self-attention output, GAP(Y i ) represents the result of global average pooling of the feature map output by the multi-head self-attention of the i-th window, Y i [j, k] represents the vector at position (j, k) of the ith window after self-attention processing, where j and k represent the row and column indices respectively; The input feature map is processed at multiple scales by different scaling factors. The above calculations of MultiHead and GAP are repeated for each scale to generate feature vectors for each scale. The feature vectors of multiple scales are concatenated to obtain the final multi-scale feature representation. The formula is: g final =concat(g s1 ,g s2 ,......,g sn ) Among them, g final represents multi-scale feature representation, g sn represents the feature vector generated from the scale sn, where sn represents the scaling factor.

2. The method for thangka image segmentation based on edge feature guidance and detail feature denoising according to claim 1, characterized in that: The principle of training the detail feature enhanced attention module in step 1 is: I out =Conv(Softmax(I c )) Where I1∈R C×H×W , indicating that I1 is a feature map with C channels, H height, and W width, and I′1∈R C×H×W , is given by I1∈R C×H×W The feature map is obtained through 3×3 full-dimensional dynamic convolution operation. Resize(I′1) means resizing the feature map I′1. Permute(Resize(I′1)) means rearranging the dimensions of the resized feature map, including exchanging the order of the number of channels, height, and width of the image. Represents matrix multiplication, I a Represents the feature map of enhanced detail features obtained by matrix multiplication, I2∈R 2C×H×W , represents a feature map with 2C channels, H height, and W width. I′2 represents the feature map obtained by concat I2 and I1 through a 3×3 full-dimensional dynamic convolution operation. Indicates element addition, I b It represents the feature map integrating different feature information obtained by element addition, I c represents the feature map combining detail features and semantic information obtained by matrix multiplication, Softmax represents the normalization of the feature map value, Conv represents the convolution operation on the normalized feature map, I out Represents the feature map of the final output after detail feature enhancement and convolution processing.

3. The method for thangka image segmentation based on edge feature guidance and detail feature denoising according to claim 1, characterized in that: The principle of generating real samples by enhancing the attention module through detail features in step 2 is: The forward process is expressed as: Among them, q represents the conditional probability distribution of the forward diffusion process, x t represents the feature map after noise addition at time step t, x t-1 represents the feature map at time step t-1, t represents the index of the time step, and t = 1, 2, ..., T, T represents the maximum value of the time step, β t represents the variance of the noise at time step t, that is, the noise intensity added at time step t, N represents the Gaussian distribution, and the mean of the Gaussian distribution is The variance is β t I, I represents the covariance matrix of the noise, q(x t |x t-1 ) represents the result of the forward process; α t ＝1-β t Among them, α t represents the diffusion probability corresponding to the step length t, represents the cumulative product of all diffusion probabilities from time step 1 to t, x0 represents the original thangka image without noise, and ε represents Gaussian noise; According to the feature map after adding noise and the target result, iterative denoising is performed to generate real samples, so p is used θ (x t-1 |x t ) as q(x t-1 |x t ) is used for denoising, which can be expressed as: p θ (x t-1 |x t )~N(x t-1 ;z θ (x t ,t),ε θ (x t ,t)) Among them, p θ represents the conditional probability distribution of the reverse denoising process, θ represents the learning parameters of the model, and z θ (x t ,t) represents the reverse diffusion variable, ε θ (x t ,t) represents the noise variance of the model parameterization; The output after detail feature enhancement is generated based on the formula: in, Represents the predicted value of the original thangka image; The loss function is used to calculate the module loss based on the following principle: Among them, L mse represents the mean square error loss function, Represents the joint expectation of three variables: time step, original thangka image and random noise.

4. The method for thangka image segmentation based on edge feature guidance and detail feature denoising according to claim 1, characterized in that: The principle of generating edge features in step 4 and optimizing the extraction of edge features through edge loss function is: Assume that the input feature layer information of different scales are F4 and F5 respectively, and the formula for generating the output of feature layer information of different scales by splicing is: F′5=Concat(F5, avgpool1(F5), avgpool2(F5), avgpool3(F5)) Among them, F′4 represents the output after performing multiple maximum pooling on the original feature layer information F4, maxpool1(F4) represents the first maximum pooling operation, maxpool2(maxpool1(F4)) represents the second maximum pooling based on the first maximum pooling, maxpool3(maxpool2(maxpool1(F4))) represents the third maximum pooling based on the first two maximum pooling, F′5 represents the output after performing multiple average pooling on the original feature layer information F5, avgpool1(F5) represents the first average pooling operation, avgpool2(F5) represents the second average pooling operation, and avgpool3(F5) represents the third average pooling operation; The Dice loss function is selected as the edge loss function, and the formula is: THE edge_x =Dice(Upsample(E x ),AND GT ) Among them, L edge_x represents the edge loss of the input image x, E x Represents the edge feature map of the model output, Upsample(E x ) represents upsampling the edge feature map output by the model, E GT Represents the edge features of the real image.

Images