A Weakly Supervised Histopathological Image Segmentation Method Based on Fuzzy Enhancement Transformer

Abstract

This invention provides a weakly supervised histopathological image segmentation method based on fuzzy-enhanced Transformer, belonging to the field of intelligent medical image processing technology. It solves the technical problem that the high cost of acquiring medical image data labels hinders practical application. The technical solution is as follows: First, images are read from the histopathological dataset and preprocessed. Next, the images are input into a Transformer to obtain the attention matrix and the final layer output. Then, a fuzzy-enhanced attention map is generated using a fuzzy-enhanced attention module and used to guide pooling operations to train the Transformer model. Finally, the fuzzy-enhanced attention map of the model is extracted and post-processed to generate the pathological image segmentation result. The beneficial effects of this invention are: achieving accurate histopathological image segmentation using easily obtainable image-level labels, promoting the practical application of deep learning in the field of medical diagnosis.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical image processing technology, and in particular to a weakly supervised tissue pathology image segmentation method based on fuzzy enhancement Transformer. Background Technology

[0002] Segmentation of histopathological images plays a crucial role in medical image analysis, particularly in the diagnosis of tumors, inflammation, and other pathological changes. Accurate segmentation of different tissue regions allows physicians to better identify the type, extent, and invasiveness of tumors, providing vital information for clinical treatment and prognostic assessment. Especially in the study of the tumor microenvironment, segmenting different tissue types such as tumor epithelium, tumor-infiltrating lymphocytes, and tumor-associated stroma is significant for revealing tumor biological behavior, predicting treatment response, and assessing patient prognosis. With the continuous development of deep learning technology, image segmentation methods based on convolutional neural networks (CNNs) and Transformers have become mainstream. Deep learning models can automatically learn complex feature representations from data, greatly improving the efficiency and accuracy of medical image segmentation, thereby assisting physicians in clinical decision-making.

[0003] For example, Van et al. proposed HookNet, a multi-branch segmentation framework based on convolutional neural networks, in their paper "HookNet: Multi-resolution convolutional neural networks for semantic segmentation in histopathology whole-slide images." This framework generates fine-grained segmentation maps for histopathological images by combining contextual information and high-resolution details. However, the collection of medical image annotation data is often expensive and time-consuming, especially for high-resolution histopathological images such as whole-slide images (WSI), whose gigapixel resolution makes manual annotation extremely difficult and labor-intensive. Furthermore, the high heterogeneity of tumor regions leads to diverse morphologies and tissue structures, posing even greater challenges to image segmentation tasks. There is an urgent need for a new segmentation method that can utilize simple and readily available image-level labels for pixel-level histopathological image segmentation, thereby reducing the reliance on large amounts of high-quality annotation data. Simultaneously, by incorporating theoretical methods such as fuzzy logic, this method can effectively address the problem of blurred tissue category boundaries in histopathological images, improving the accuracy and robustness of weakly supervised segmentation. This is significant for building reliable and low-cost medical decision support systems. Summary of the Invention

[0004] The purpose of this invention is to provide a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer, which solves the problems of existing histopathological image segmentation methods such as dependence on high-quality data annotation and blurred tissue category boundaries, and effectively improves the accuracy of weakly supervised segmentation.

[0005] To achieve the above-mentioned objectives, this invention is implemented through the following measures: a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer, comprising the following steps:

[0006] S1: Read images from the histopathology dataset Perform data preprocessing operations, and then cut the image into 196 non-overlapping image blocks of size 16×16;

[0007] S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token;

[0008] S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map.

[0009] S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is dimensionality reduced to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model.

[0010] S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and finally generate the segmentation result after the argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

[0011] Furthermore, the specific steps of step S2 are as follows:

[0012] Step S2.1: Input tensor T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output

[0013]

[0014] T output =AV (2)

[0015] Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L.

[0016] Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers.

[0017]

[0018] in Let i be the aggregate attention matrix;

[0019] Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final layer token output Extract the final layer image block token output and category token output

[0020] Furthermore, the specific steps of step S3 are as follows:

[0021] Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows:

[0022]

[0023] Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. d Let μ represent the d-th feature map. d,k (Td ) represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (4).

[0024] Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows:

[0025]

[0026] Where T d ' ,k This represents the feature maps obtained through sampling, where the pixel values ​​follow a mean of c. d,k The standard deviation is Gaussian distribution;

[0027] Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation:

[0028]

[0029] Where τ is the scaling temperature parameter controlling the smoothness, and the second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation:

[0030]

[0031] The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation:

[0032]

[0033] in For blurring results The result of taking the mean along the second dimension;

[0034] Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them:

[0035]

[0036] Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated.

[0037] Furthermore, the specific steps of step S4 are as follows:

[0038] Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y :

[0039]

[0040] in This represents the feature point in the c-th channel of the blurred enhanced attention map, where c = 1, ..., C.

[0041] Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores.

[0042]

[0043] Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction.

[0044] y class =Linear(T) class (15)

[0045] Where Linear(·) represents a linear layer;

[0046] Step S4.4: Image block token prediction score With category token prediction score Different from real labels

[0047] Calculate the multi-label soft edge loss:

[0048]

[0049] Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

[0050] Furthermore, the specific steps of step S5 are as follows:

[0051] Step S5.1: After the model training is complete, extract the blur-enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization:

[0052]

[0053] in The fuzzy enhanced attention map corresponding to category c, where max(·) represents the maximum value operation;

[0054] Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map:

[0055]

[0056] in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape;

[0057] Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred:

[0058]

[0059] Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized.

[0060] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0061] 1. This invention proposes a weakly supervised learning method based on image-level labels, which can effectively alleviate the high cost and high labor intensity problems in the medical image annotation process. Unlike traditional methods that require a large amount of pixel-level labeled data, this invention can achieve high-precision pixel-level segmentation through image-level labels, greatly reducing the need for data annotation and lowering annotation costs and time consumption.

[0062] 2. This invention proposes a fuzzy-enhanced class-aware attention map generation method. Multiple Gaussian membership functions and fuzzy rules are set on the output feature map to extract fuzzy high-matching semantic features, fuzzy low-matching semantic features, and fuzzy general semantic features. The fuzzy-enhanced attention map generated by fusing these fuzzy features can provide accurate object position information. At the same time, the fuzzy-enhanced attention map can guide the pooling operation to reasonably expand the attention region.

[0063] 3. This invention combines the fuzzy enhanced attention map in the trained model with the self-attention map in the sub-attention module of the Transformer model to generate the final attention map result as the segmentation result of the histopathological image. The segmentation result can accurately identify key tissue types such as tumor regions, tumor-infiltrating lymphocytes, and tumor-related stroma, providing strong support for the segmentation of histopathological images. Attached Figure Description

[0064] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used to explain the invention but do not constitute a limitation thereof.

[0065] Figure 1 This is a diagram illustrating the overall framework of the weakly supervised tissue pathology image segmentation method based on fuzzy enhancement Transformer of the present invention.

[0066] Figure 2 This is a flowchart of the weakly supervised tissue pathology image segmentation method based on fuzzy enhancement Transformer of the present invention.

[0067] Figure 3 This is a flowchart of the fuzzy enhanced attention module of the present invention;

[0068] Figure 4 This is an example of the weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Of course, the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0070] Example 1

[0071] See Figures 1 to 4 This embodiment provides a technical solution: a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer. Taking a histopathological image dataset as an example, the method includes the following steps from loading the data to obtaining the segmentation result:

[0072] S1: Read images from the histopathology dataset Perform data preprocessing operations, and then cut the image into 196 non-overlapping image blocks of size 16×16;

[0073]

[0074] S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token;

[0075] S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map.

[0076] S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is dimensionality reduced to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model.

[0077] S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and finally generate the segmentation result after the argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

[0078] Furthermore, the specific steps of step S2 are as follows:

[0079] Step S2.1: Input tensor T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output

[0080]

[0081] T output =AV (3)

[0082] Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L.

[0083] Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers.

[0084]

[0085] in Let i be the aggregate attention matrix;

[0086] Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final level token output Extract the final layer image block token output and category token output A affinity T patch and T class As shown in formulas (5), (6) and (7):

[0087]

[0088] T class =(0.36540.0239...-1.3109) 1×384 (7)

[0089] Furthermore, the specific steps of step S3 are as follows:

[0090] Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows:

[0091]

[0092] Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. d Let μ represent the d-th feature map. d,k (T d ) represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (8). Where the membership matrix μ 0,0 As shown in formula (9):

[0093]

[0094] Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows:

[0095]

[0096] Where T d ' ,k This represents the feature maps obtained through sampling, where the pixel values ​​follow a mean of c. d,k The standard deviation is A Gaussian distribution, wherein the sampled feature map T0' ,0 As shown in formula (11):

[0097]

[0098] Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation:

[0099]

[0100]

[0101] Where τ is the scaling temperature parameter that controls the smoothness, and the high-match semantic fuzzy features. As shown in formula (14):

[0102]

[0103] The second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation:

[0104]

[0105] low-match semantic ambiguity features As shown in formula (17):

[0106]

[0107] The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation:

[0108]

[0109] in For blurring results The result of averaging along the second dimension shows low-match semantic ambiguity. As shown in formula (20):

[0110]

[0111] Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them:

[0112]

[0113] Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated. As shown in formula (22):

[0114]

[0115] Furthermore, the specific steps of step S4 are as follows:

[0116] Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y :

[0117]

[0118] in This represents the feature point in the c-th channel of the blurred enhanced attention map, where c = 1, ..., C. α c,x,y As shown in formula (24):

[0119]

[0120] Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores.

[0121]

[0122] y patch As shown in formula (26):

[0123] y patch = (7.7413 - 5.4955 - 8.54437.2036) 1×4 (26)

[0124] Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction.

[0125] y class =Linear(T) class (27)

[0126] Where Linear(·) represents a linear layer, y class As shown in formula (28):

[0127] y class = (8.2471 -4.5638 -8.5215 6.5667) 1×4 (28)

[0128] Step S4.4: Image block token prediction score With category token prediction score Different from real labels Calculate the multi-label soft edge loss:

[0129]

[0130] Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

[0131] Furthermore, the specific steps of step S5 are as follows:

[0132] Step S5.1: After the model training is complete, extract the blur-enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization:

[0133]

[0134] in The fuzzy enhanced attention map corresponding to category c, where max(·) represents the maximum value operation. As shown in formula (32):

[0135]

[0136] Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map:

[0137]

[0138] in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape, A final As shown in formula (34):

[0139]

[0140] Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred:

[0141]

[0142] Where Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized. The segmentation result pred is shown in formula (36):

[0143]

[0144] All images in the dataset were fed into the model to obtain segmentation results. The performance of the segmentation results was evaluated using the real segmentation labels. The accuracy was 84.51% and the average intersection-over-union ratio was 75.65%, which is significantly better than traditional weakly supervised segmentation methods.

[0145]

[0146] Example 2

[0147] See Figures 1 to 4 This embodiment provides a technical solution: a weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer. Taking a histopathological image dataset as an example, the method includes the following steps from loading the data to obtaining the segmentation result:

[0148] S1: Read images from the histopathology dataset Perform data preprocessing operations, and then cut the image into 196 non-overlapping image blocks of size 16×16;

[0149]

[0150] S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token;

[0151] S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map.

[0152] S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is dimensionality reduced to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model.

[0153] S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and finally generate the segmentation result after the argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

[0154] Furthermore, the specific steps of step S2 are as follows:

[0155] Step S2.1: Input tensor T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output

[0156]

[0157] T output =AV (3)

[0158] Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L.

[0159] Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers.

[0160]

[0161] in Let i be the aggregate attention matrix;

[0162] Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final level token output Extract the final layer image block token output and category token output A affinity T patch and T class As shown in formulas (5), (6) and (7):

[0163]

[0164] T class =(-10.1635-0.3858...18.8144) 1×384 (7)

[0165] Furthermore, the specific steps of step S3 are as follows:

[0166] Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows:

[0167]

[0168] Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. d Let μ represent the d-th feature map. d,k (T d ) represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (8). Where the membership matrix μ 0,0 As shown in formula (9):

[0169]

[0170] Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows:

[0171]

[0172] Where T d ' ,k This represents the feature maps obtained through sampling, where the pixel values ​​follow a mean of c. d,k The standard deviation is A Gaussian distribution, wherein the sampled feature map T0' ,0 As shown in formula (11):

[0173]

[0174] Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation:

[0175]

[0176] Where τ is the scaling temperature parameter that controls the smoothness, and the high-match semantic fuzzy features. As shown in formula (14):

[0177]

[0178] The second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation:

[0179]

[0180] low-match semantic ambiguity features As shown in formula (17):

[0181]

[0182] The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation:

[0183]

[0184] in For blurring results The result of averaging along the second dimension shows low-match semantic ambiguity. As shown in formula (20):

[0185]

[0186] Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them:

[0187]

[0188] Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated. As shown in formula (22):

[0189]

[0190] Furthermore, the specific steps of step S4 are as follows:

[0191] Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y :

[0192]

[0193] in This represents the feature point in the c-th channel of the blurred enhanced attention map, where c = 1, ..., C. α c,x,y As shown in formula (24):

[0194]

[0195] Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores.

[0196]

[0197] y patch As shown in formula (26):

[0198] y patch = (7.7469 - 5.9008 - 8.12457.8962) 1×4 (26)

[0199] Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction.

[0200] y class =Linear(T) class (27)

[0201] Where Linear(·) represents a linear layer, y class As shown in formula (28):

[0202] y class = (6.8711 - 7.5792 - 8.94048.5667) 1×4 (28)

[0203] Step S4.4: Image block token prediction score With category token prediction score Different from real labels Calculate the multi-label soft edge loss:

[0204]

[0205] Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

[0206] Furthermore, the specific steps of step S5 are as follows:

[0207] Step S5.1: After the model training is complete, extract the blurred and enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization:

[0208]

[0209] in The fuzzy enhanced attention map corresponding to category c, where max(·) represents the maximum value operation. As shown in formula (32):

[0210]

[0211] Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map:

[0212]

[0213] in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape, A final As shown in formula (34):

[0214]

[0215] Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred:

[0216]

[0217] Where Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized. The segmentation result pred is shown in formula (36):

[0218]

[0219] All images in the dataset were fed into the model to obtain segmentation results. The performance of the segmentation results was evaluated using the real segmentation labels. The accuracy was 83.68% and the average intersection-over-union ratio was 67.68%, which is significantly better than traditional weakly supervised segmentation methods.

[0220]

[0221] The above description is merely a preferred embodiment of this practice and is not intended to limit the scope of this practice. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this practice should be included within the protection scope of this practice.

Claims

1. A weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer, characterized in that, Includes the following steps: S1: Read images from the histopathology dataset H and W represent the height and width of the image, respectively. After data preprocessing, the image is cut into N non-overlapping image blocks of size P×P, where N is the number of image blocks and P is the side length of the image block. S2: Construct input tokens from preprocessed histopathological images. The input is fed into the Transformer encoder layer to obtain the attention matrix and extract the affinity matrix. Final layer image block token output and category token output Where D is the dimension of the token; S3: Output the image block token The fuzzy enhanced attention module constructed from the reshaped input extracts fuzzy features from the image block token output based on fuzzy rules. The fuzzy features are then concatenated with the image block token output and fused through a convolution operation to generate a fuzzy enhanced attention map. S4: The generated blurred enhanced attention map is processed and used as weights to guide the pooling operation, generating image block token prediction scores. The category token output is reduced in dimensionality to generate category token prediction scores. The two scores are compared with the image's true labels to calculate the multi-label soft edge loss and train the Transformer model. S5: Extract the blurred enhanced attention map from the trained model, apply the ReLU activation function and maximum normalization operation, then perform post-processing using the affinity matrix, and generate the segmentation result after upsampling and argmax operation, where argmax is the operation to obtain the index of the position of the maximum value.

2. The weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer according to claim 1, characterized in that, The specific steps of step S2 are as follows: Step S2.1: Input token T input The input Transformer encoder layer undergoes L iterations, during which a query matrix is ​​generated through projection transformation. Key matrix Sum matrix Extracting the attention matrix and final layer token output T output =OFF (2) Where d k Let be the dimension of the key matrix, and softmax(·) be the activation function. The aggregated attention matrix is ​​obtained by aggregating the attention matrices of layers L. Step S2.2: Apply the aggregated attention matrix A all The average attention matrix is ​​obtained by averaging across the layers. in Let i be the aggregate attention matrix; Step S2.3: From the average attention matrix A mean Extracting the affinity matrix Output T from the final level token output Extract the final layer image block token output and category token output 3. The weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S3.1: Output the extracted image block tokens Shape reshaping Represented as D units of size The feature maps are then analyzed, and fuzzy membership functions are used to characterize the association between elements in the feature maps and fuzzy concepts, effectively capturing the uncertainty and fuzziness in the features. K Gaussian membership functions are assigned to each feature map, and the formula for calculating the Gaussian membership function is as follows: Where c d,k and σ d,k Tables 1 and 2 show the center and width of the k-th Gaussian membership function on the d-th feature map, where d = 1, ..., D, k = 1, ..., K, exp(·) is the exponential function, and T is the width of the Gaussian membership function. d Let μ represent the d-th feature map. d,k This represents the fuzzy membership matrix obtained by assigning the k-th Gaussian membership function on the d-th feature map. The final fuzzification result is calculated using formula (4). Step S3.2: Sample the fuzzy Gaussian kernel to synthesize fuzzy features with semantic information. The sampling formula is as follows: Where T′ d,k This represents the feature map obtained through sampling; Step S3.3: Different membership values ​​represent the membership degrees of different locations in the image to a specific fuzzy set. Three sets of fuzzy rules are established to extract fuzzy weights, which are then multiplied with the sampled feature maps to extract fuzzy features. The first set of fuzzy logic is used to extract highly matched semantic fuzzy features. The maximum membership value of each feature point on the feature map is selected as the output weight. To obtain highly matching semantically ambiguous features Implementing fuzzy logic using scaling softmax operation: Where τ is the scaling temperature parameter controlling the smoothness, and the second set of fuzzy logic is used to extract semantically fuzzy features with low matching, selecting the minimum membership value of each feature point on the feature map as the output weight. To obtain semantically ambiguous features with low matching Implementing fuzzy logic using scaling softmax operation: The third set of fuzzy logic is used to extract general semantic fuzzy features, selecting the average membership value of each feature point on the feature map as the output weight. To obtain general semantic fuzzy features Implementing fuzzy logic using scaling softmax operation: in For blurring results The result of taking the mean along the second dimension; Step S3.4: After concatenating the three extracted fuzzy features with the original features, perform a convolution operation to fuse them: Concat(·) is a concatenation operation, and Conv(·) is a 3×3 convolution operation. The concatenated feature channels are adjusted to C, where C is the number of classes in the dataset. Finally, a blurred and enhanced attention map is generated.

4. The weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer according to claim 1, characterized in that, The specific steps of step S4 are as follows: Step S4.1: The blurred attention map contains positional information to guide the pooling operation. First, softmax normalization is performed on the channel dimension to generate pooling weights α. c,x,y : in This represents the feature point on the c-th channel of the blurred enhanced attention map. Step S4.2: With α c,x,y For weighted pairs A fuzzy Weighted pooling is applied to increase the weight of the correct category at the corresponding position, thereby guiding the attention map to expand within a reasonable range and generating image patch token prediction scores. Step S4.3: Output the category token obtained in step S2.3 Category token prediction scores are generated through linear layer dimensionality reduction. y class =Linear(T class ) (15) Where Linear(·) represents a linear layer; Step S4.4: Image block token prediction score With category token prediction score Different from real labels Calculate the multi-label soft edge loss: Where σ(·) represents the sigmoid activation function, and finally L patch and L class The sum is used as the final loss function, and the model parameters are optimized by minimizing the loss function through backpropagation.

5. The weakly supervised histopathological image segmentation method based on fuzzy enhancement Transformer according to claim 1, characterized in that, The specific steps of step S5 are as follows: Step S5.1: After the model training is complete, extract the blurred and enhanced attention map after feeding the image back into the model. The fuzzy enhanced attention map corresponding to each category is then processed using the ReLU activation function and maximum normalization: in This represents the fuzzy enhanced attention map corresponding to category c, and max(·) represents the maximum value operation; Step S5.2: Use the affinity matrix extracted in step S2.3 Fine-grained post-processing of the blurred enhanced attention map: in R represents matrix multiplication operations. N×C (·)and These represent reshaping the matrix into N×C and respectively. The shape; Step S5.3: Finally, place A final Upsample to the input image size and apply the argmax operation to generate the final segmentation result, pred. Upsample(·) represents the upsampling operation. This indicates that the maximum index is taken in the category dimension, and the segmentation result is visualized.

Images