Text-guided multi-stage cross-perception medical image segmentation method and system
By employing a multi-stage cross-sensory medical image segmentation method, which utilizes Swing Transformer and BERT for multi-stage cross-modal interaction and alignment, this method addresses the issues of weak semantic expression and feature offset in existing methods. It achieves high-precision and stable lesion segmentation and is applicable to various medical imaging modalities and disease types.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIHANG UNIV
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-15
AI Technical Summary
Existing medical image segmentation methods suffer from weak semantic expression and insufficient generalization in low-contrast and blurred-boundary lesion segmentation tasks. Furthermore, existing text-guided methods have limited cross-modal interaction, resulting in insufficient propagation of text semantics and feature shift.
A multi-stage cross-sensory medical image segmentation method is adopted. Through multi-stage cross-attention interaction using the Swin Transformer visual encoder and the BERT text encoder, combined with multi-stage alignment loss, deep interaction and alignment of image and text features are achieved, generating cross-modal enhanced visual features.
It significantly improves the segmentation accuracy and stability of lesion areas, especially performing excellently in the segmentation of lesions with low contrast and blurred boundaries, with a Dice coefficient of 84.65%~88.09%. It supports unified segmentation of multiple medical imaging modalities and disease types, and has good clinical applicability and interpretability.
Smart Images
Figure CN122049352A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of medical image processing and computer vision technology, specifically to a text-guided multi-stage cross-sensory medical image segmentation method and system. Background Technology
[0002] Medical image segmentation is a key technology for clinical diagnosis, treatment planning, and disease monitoring. Its core is to accurately segment lesion regions or anatomical structures of interest from medical images. Traditional segmentation methods, such as UNet and its variants, mainly rely on the visual features of the image itself for segmentation. However, this purely visual architecture suffers from weak semantic representation and insufficient generalization, making it difficult to handle lesion segmentation tasks with low contrast, blurred boundaries, or heterogeneous morphology.
[0003] To address these issues, researchers have proposed introducing text-guided information, using semantic cues provided by text descriptions to assist in lesion localization. Existing text-guided medical image segmentation methods, such as LViT and SimTxtSeg models, while achieving some progress, still have significant drawbacks: First, cross-modal interaction is limited, with most methods performing feature fusion only once at the encoder bottleneck layer or output layer, making it difficult for text semantics to fully propagate to visual features at all levels. Second, there is a lack of explicit cross-modal alignment mechanisms, making intermediate features prone to semantic shifts during evolution, especially when lesion boundaries are blurred or text descriptions are abstract, thus affecting segmentation accuracy. Therefore, a segmentation method capable of achieving multi-stage deep cross-modal interaction and feature alignment is urgently needed to improve the accuracy and robustness of medical image segmentation. Summary of the Invention
[0004] The purpose of this invention is to provide a text-guided multi-stage cross-sensory medical image segmentation method and system. By using multi-stage cross-attention interaction and cross-modal alignment mechanisms, it solves the problems of insufficient cross-modal interaction and inadequate feature representation in existing methods, thereby improving the accuracy and generalization ability of medical image segmentation.
[0005] Specific technical solutions:
[0006] A text-guided multi-stage cross-sensory medical image segmentation method, characterized by the following steps:
[0007] (1) Data preparation steps: Obtain the medical image to be segmented and the corresponding text description, the text description includes semantic information such as lesion location, shape, and range; perform preprocessing on the medical image, including size normalization and grayscale normalization, and perform word segmentation, truncation or filling on the text description;
[0008] (2) Feature extraction steps: Input the preprocessed medical image into the visual encoder based on Swin Transformer to output a multi-stage visual feature map; input the preprocessed text description into the text encoder based on BERT to output a text feature sequence, and obtain the stage-specific text features corresponding to each visual feature stage through linear projection.
[0009] (3) Multi-stage cross-modal interaction steps: Multi-stage cross-attention modules (MCM) are introduced in stages 2-4 of the visual encoder. Based on stage-specific text features, the visual features of the corresponding stages are dynamically modulated to realize bidirectional cross-attention interaction between image and text features and generate cross-modal enhanced visual features.
[0010] (4) Cross-modal alignment step: Before the cross-attention interaction at each stage, the visual features and text features are linearly projected and mapped to the same feature space. The cosine similarity between the global visual embedding and the text embedding is calculated. The cross-modal feature consistency is optimized by multi-stage alignment loss (MA Loss).
[0011] (5) Segmentation result generation steps: Input the cross-modal enhanced visual features of each stage into the dual-path U-shaped decoder, fuse multi-scale features through encoder downsampling, decoder upsampling and jump connection, and output the lesion region segmentation mask of the medical image through 1×1 convolution and sigmoid activation.
[0012] Specifically:
[0013] (1) Data preparation: Collect medical images and corresponding radiological text descriptions. The text should contain key semantic information such as the location, shape, and extent of the lesion. Normalize the image size to 224×224 and the grayscale to [0,1]. After word segmentation, the text is truncated or padded to a fixed length of 10.
[0014] The medical images include chest X-ray images, chest CT images, and breast DCE-MRI images, with corresponding text descriptions that are radiological report text fragments containing the anatomical location, morphological characteristics, and extent of the lesion.
[0015] (2) Feature extraction:
[0016] Visual encoder: The Swin Transformer is used as the backbone network to encode the preprocessed medical images into visual feature maps in four stages, with channel dimensions of 96, 192, 384 and 768 for each stage.
[0017] Text encoder: The BERT-base model is used to encode the preprocessed text description into a 768-dimensional text feature sequence. The stage-specific text features corresponding to each stage of the visual features are generated through linear projection to ensure the consistency of cross-modal feature dimensions.
[0018] (3) Multi-stage cross-modal interaction: Multi-stage cross-attention modules (MCMs) are embedded in stages 2-4 of the visual encoder to achieve bidirectional dynamic interaction between image and text features:
[0019] (3.1) Visual features V of stage i i As a query, the text feature L corresponding to the stage i As keys and values, visual features are updated through multi-head cross-attention computation to obtain preliminary enhanced visual features;
[0020] (3.2) Using the updated visual features as keys and values, the text features L of stage i i As a query, the text features are updated again through multi-head cross-attention computation;
[0021] (3.3) The updated visual features and text features are processed by a position-aware feedforward network (MLP) to obtain the final cross-modal enhanced visual features. and updated text features .
[0022] (4) Cross-modal alignment optimization: To address the cross-modal semantic offset problem, a multi-stage alignment loss (MALoss) is designed:
[0023] (4.1) Perform global average pooling on the visual features Vi of stage i to obtain the global visual embedding. ;
[0024] (4.2) Extracting the [CLS] token feature from the text features in stage i as the text embedding ;
[0025] (4.3) Calculation and cosine similarity ;
[0026] (4.4) The single-stage alignment loss is calculated using the logistic contrast loss function, and the formula is as follows:
[0027] in, Let i be the learnable temperature parameter for stage i. It is the sigmoid function;
[0028] (4.5) The total multi-stage alignment loss is obtained by averaging the alignment losses of all participating stages (stages 2-4). .
[0029] (5) Dual-path segmentation and decoding: A CNN+ViT dual-path U-shaped decoder is adopted, which integrates multi-scale cross-modal enhancement features:
[0030] (5.1) Encoder stage i: Enhance visual features across modalities Visual features X of the CNN path respectively i Visual features of the ViT path Y i Channel splicing is performed, and the next stage features are obtained through the downsampling module;
[0031] (5.2) Decoder stage i: Combine the decoder features from the previous stage with the cross-modal enhanced visual features of the corresponding stage. Encoder CNN path features X i Channel splicing is performed, and the decoder features at the current stage are obtained through the upsampling module;
[0032] (5.3) The final decoder output features are adjusted by 1×1 convolution and then activated by sigmoid to obtain a pixel-level segmentation probability map.
[0033] The model training in this invention uses the Adam optimizer, and the initial learning rate is set to 1×10⁻³ or 3×10⁻³ depending on the characteristics of the dataset. 4 The learning rate decays by 0.1 when the validation set performance does not improve for 10 consecutive epochs; the total training loss is the weighted sum of the binary classification cross-entropy loss and the multi-stage alignment loss (weight factor λ=0.1), the maximum training epoch is 200, and an early stopping strategy is adopted (training stops if there is no improvement after 20 epochs).
[0034] The present invention has the following beneficial effects:
[0035] 1. Multi-stage cross-modal interaction design: By enabling bidirectional interaction between images and text in multiple stages of the encoder, text semantics can be fully propagated to visual features at all levels, significantly improving the ability to locate fine-grained lesions, especially suitable for segmenting lesions with low contrast and blurred boundaries;
[0036] 2. Cross-modal alignment mechanism: By using multi-stage alignment loss, visual and textual features are forced to be semantically consistent at each level, which effectively alleviates the semantic shift problem in the feature evolution process and improves the stability and accuracy of the segmentation results;
[0037] 3. Excellent segmentation performance: On three major public datasets, QaTa-COV19 (chest X-ray), MosMedData (chest CT), and Duke-Breast-Cancer-MRI, the Dice coefficients reached 84.65%, 78.39%, and 88.09%, respectively, which are significantly better than the UNet series and existing text-guided segmentation methods.
[0038] 4. Strong clinical applicability: It supports natural language interaction, and clinicians can guide the segmentation process through radiological terminology without the need for additional pixel-level annotation. The segmentation results have good interpretability, which helps to improve clinical trust.
[0039] 5. Excellent generalization ability: It is compatible with multiple medical image modalities and disease types, providing a unified and efficient solution for medical image segmentation in different clinical scenarios. Attached Figure Description
[0040] Figure 1 is a diagram of the overall architecture of the text-guided multi-stage cross-sensory network (TMC) of this invention;
[0041] Figure 2 is a qualitative comparison of the segmentation performance of the present invention and existing methods on three major datasets;
[0042] Figure 3 is a visualization of the attention heatmap of the multi-stage cross-attention module of the present invention. Detailed Implementation
[0043] The specific technical solutions of the present invention will be described with reference to the embodiments.
[0044] Figure 1 This is a diagram of the overall architecture of the text-guided multi-stage cross-sensory network (TMC) of this invention.
[0045] Example 1: Model Training Process
[0046] 1. Dataset configuration:
[0047] QaTa-COV19 dataset: contains 9258 positive chest X-ray images of COVID19, which are divided into training set (6480 images), validation set (926 images) and test set (1852 images) in a 7:1:2 ratio.
[0048] The MosMedData dataset contains 1000 chest CT scans (500 normal and 500 COVID-19 positive), stratified into a training set (700 cases), a validation set (100 cases), and a test set (200 cases) in a 7:1:2 ratio.
[0049] The Duke-Breast-Cancer-MRI dataset contains 922 breast DCE-MRI scans, divided into a training set (645 cases), a validation set (92 cases), and a test set (185 cases) in a 7:1:2 ratio.
[0050] 2. Model parameter settings:
[0051] Visual encoder: Swin Transformer, number of layers = 12, number of heads = 3 / 6 / 12 / 24 (corresponding to 4 stages);
[0052] Text encoder: BERT-base-uncased, number of layers=12, number of headers=12, hidden layer dimension=768;
[0053] Multi-stage cross-attention module: Number of heads = 8, Key dimension = 64;
[0054] Optimizer: Adam, weight decay = 1e5;
[0055] Learning rate: 3e-4 for QaTa-COV19 and Duke-Breast datasets, and 1e-3 for MosMedData dataset;
[0056] Batch size = 8, number of training rounds = 200, early stopping patience = 20.
[0057] 3. Training process:
[0058] Initialize the parameters of the visual encoder, text encoder, multi-stage cross-attention module, and decoder;
[0059] Load the training data, preprocess the images and text, and then input them into the model;
[0060] Extract multi-stage visual and textual features, and achieve cross-modal interaction through MCM;
[0061] Calculate the multi-stage alignment loss and segmentation loss, and jointly optimize the model parameters;
[0062] After each training round, the performance is evaluated on the validation set, and the optimal model weights are saved.
[0063] Example 2: Model Reasoning Process
[0064] 1. Input: Medical image to be segmented (e.g., chest CT slice) and corresponding text description (e.g., "subpleural micronodule in the upper lobe of the left lung");
[0065] 2. Preprocessing: Scale the image to 224×224 and normalize it to [0,1]; pad the text to a length of 10 after word segmentation;
[0066] 3. Feature extraction: Multi-stage features are extracted using pre-trained visual encoders and text encoders respectively;
[0067] 4. Cross-modal interaction and alignment: Multi-stage bidirectional interaction is achieved through the MCM module to generate cross-modal enhanced features;
[0068] 5. Segmentation and Decoding: The dual-path decoder fuses multi-scale features and outputs a segmentation probability map;
[0069] 6. Post-processing: The probability map is binarized with a threshold of 0.5 to obtain the final lesion area segmentation mask.
[0070] Experimental verification results
[0071] The performance of this invention on three major public datasets is shown in Table 1 below:
[0072] Table 1 Performance on three major public datasets
[0073] Dataset Dice coefficient (%) mIoU (%) QaTa-COV19 84.65±0.18 76.14±0.42 MosMedData 78.39±0.14 67.48±0.26 Duke-Breast-Cancer-MRI 88.09±0.68 82.74±0.73
[0074] Figure 2 This is a qualitative comparison chart of the segmentation performance of the present invention and existing methods on three major datasets; Figure 3 This is a visualization of the attention heatmap of the multi-stage cross-attention module of the present invention.
[0075] Experimental results show that the segmentation performance of this invention is significantly better than pure vision methods such as UNet and TransUNet, as well as existing text-guided methods such as LViT and CLIP, on all datasets. Furthermore, ablation experiments verify that the multi-stage cross-attention module (MCM) and multi-stage alignment loss (MA Loss) can independently improve segmentation performance, and their synergistic effect can achieve the best results.
[0076] The model of this invention can be implemented based on the PyTorch framework and deployed on computing devices equipped with GPUs such as the NVIDIA RTX 4090, supporting batch processing of medical image segmentation tasks. In clinical applications, it can be integrated into medical imaging workstations, allowing doctors to quickly obtain accurate lesion region segmentation results by inputting key text fragments from radiology reports, thus assisting in clinical diagnosis and treatment decisions.
Claims
1. A text-guided multi-stage cross-sensory medical image segmentation method, characterized in that, Includes the following steps: (1) Data preparation steps: Obtain the medical image to be segmented and the corresponding text description, wherein the text description contains semantic information such as the location, shape and extent of the lesion; Preprocessing of medical images includes size normalization and grayscale normalization; and word segmentation, truncation, or padding of text descriptions. (2) Feature extraction steps: Input the preprocessed medical image into the visual encoder based on Swin Transformer to output a multi-stage visual feature map; input the preprocessed text description into the text encoder based on BERT to output a text feature sequence, and obtain the stage-specific text features corresponding to each visual feature stage through linear projection. (3) Multi-stage cross-modal interaction steps: Multi-stage cross-attention modules (MCMs) are introduced in stages 2-4 of the visual encoder. Based on stage-specific text features, the visual features of the corresponding stages are dynamically modulated to achieve bidirectional cross-attention interaction between image and text features and generate cross-modal enhanced visual features. (4) Cross-modal alignment step: Before the cross-attention interaction at each stage, the visual features and text features are linearly projected and mapped to the same feature space. The cosine similarity between the global visual embedding and the text embedding is calculated. The cross-modal feature consistency is optimized by multi-stage alignment loss MA Loss. (5) Segmentation result generation steps: Input the cross-modal enhanced visual features of each stage into the dual-path U-shaped decoder, fuse multi-scale features through encoder downsampling, decoder upsampling and jump connection, and output the lesion region segmentation mask of the medical image through 1×1 convolution and sigmoid activation.
2. The text-guided multi-stage cross-sensory medical image segmentation method according to claim 1, characterized in that, The multi-stage cross-attention module (MCM) interaction process includes: (2.1) Visual features V of stage i i As a query, the text feature L corresponding to the stage i As keys and values, visual features are updated through multi-head cross-attention computation to obtain preliminary enhanced visual features; (2.2) Using the updated visual features as keys and values, the text features L of stage i i As a query, the text features are updated again through multi-head cross-attention computation; (2.3) The updated visual features and text features are processed by a position-aware feedforward network (MLP) to obtain the final cross-modal enhanced visual features. and updated text features .
3. The text-guided multi-stage cross-sensory medical image segmentation method according to claim 1, characterized in that, The calculation process of the multi-stage alignment loss (MA Loss) is as follows: (3.1) Perform global average pooling on the visual features Vi of stage i to obtain the global visual embedding. ; (3.2) Extracting the [CLS] token feature from the text features in stage i as the text embedding ; (3.3) Calculation and cosine similarity ; (3.4) The single-stage alignment loss is calculated using the logistic contrast loss function, and the formula is as follows: ； in, Let i be the learnable temperature parameter for stage i. It is the sigmoid function; (3.5) The total multi-stage alignment loss is obtained by averaging the alignment losses of all participating stages (stages 2-4). .
4. The text-guided multi-stage cross-sensory medical image segmentation method according to claim 1, characterized in that, The dual-path U-shaped decoder includes a CNN path and a ViT path, and the fusion process is as follows: (4.1) Encoder stage i: Enhance visual features across modalities Visual features X of the CNN path respectively i Visual features of the ViT path Y i Channel splicing is performed, and the next stage features are obtained through the downsampling module; (4.2) Decoder stage i: Combine the decoder features from the previous stage with the cross-modal enhanced visual features of the corresponding stage. Encoder CNN path features X i Channel splicing is performed, and the decoder features at the current stage are obtained through the upsampling module; (4.3) The final decoder output features are adjusted by 1×1 convolution and then activated by sigmoid to obtain a pixel-level segmentation probability map.
5. The text-guided multi-stage cross-sensory medical image segmentation method according to claim 1, characterized in that, The total loss function used in the training process is a weighted sum of the segmentation loss and the multi-stage alignment loss, as shown in the formula: ; in, For pixel-level binary classification cross-entropy loss, This is the weighting factor, with a value of 0.
1.
6. A text-guided multi-stage cross-sensory medical image segmentation system, characterized in that, A system for implementing a text-guided multi-stage cross-sensory medical image segmentation method according to any one of claims 1 to 5; the system comprises: Data preprocessing module: used to receive medical images and text descriptions, and perform image size normalization, grayscale normalization, and text segmentation, truncation / padding processing; Feature extraction module: includes a visual encoder and a text encoder. The visual encoder is based on SwinTransformer and is used to extract multi-stage visual features. The text encoder is based on BERT and is used to extract text features and generate stage-specific text features. Multi-stage cross-attention module: used to realize bidirectional dynamic interaction between visual features and text features at each stage, and generate cross-modal enhanced visual features; Cross-modal alignment module: used to calculate multi-stage alignment loss and optimize cross-modal feature consistency; The segmentation and decoding module adopts a dual-path U-shaped architecture to fuse multi-scale cross-modal features and generate the final lesion region segmentation mask. Model training module: used to define the total loss function, use the Adam optimizer to update model parameters, and implement model training.
7. A text-guided multi-stage cross-sensory medical image segmentation system according to claim 6, characterized in that, The visual encoder uses a Swin-Tiny configuration, with the channel dimensions of each stage set to C1=96, C2=192, C3=384, and C4=768; the text encoder uses a BERT-base model, with a hidden layer dimension of 768 and a fixed length of 10 for the text feature sequence.