VMUNet ultrasonic medical image segmentation method fused with text prompt
By integrating text prompt information into the VMUNet network and utilizing FiLM conditional modulation and cross-modal attention mechanism, the problems of blurred boundaries and noise interference in ultrasound medical image segmentation are solved, achieving lesion segmentation with higher accuracy and stronger generalization ability.
Patent Information
- Application Number
- CN202510744134.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-05
- Publication Date
- 2025-09-23
AI Technical Summary
When processing ultrasound medical images, the existing VMUNet model has low segmentation accuracy when dealing with fuzzy boundaries, complex lesion morphology, or low-contrast lesions. The model also has insufficient generalization ability, severe noise interference, and lacks effective conditional feature modulation, resulting in inaccurate and unstable segmentation results.
A VMUNet network integrating textual cues was constructed. Through FiLM conditional modulation and cross-modal attention mechanism, visual and textual features were fused at different stages of the network to optimize segmentation mask generation. This included introducing FiLM layers in the input embedding layer and feature encoder, CAFM modules in the feature decoder, and FICAM modules after the decoder output, and combining clinical diagnosis text information for feature modulation and mask optimization.
It significantly improves the segmentation accuracy and generalization ability of ultrasound medical images, can more accurately identify lesion boundaries, suppress noise interference, enhance the interpretability and stability of segmentation results, and adapt to the needs of different clinical scenarios.
Smart Images
Figure CN120689352A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a VMUNet ultrasonic medical image segmentation method integrating text prompts. Background Art
[0002] Ultrasound medical imaging is widely used in clinical diagnosis due to its advantages such as non-invasiveness and real-time performance; however, ultrasound imaging often has defects such as poor image quality, blurred lesion boundaries, and susceptibility to noise.
[0003] Although the existing VMUNet model has achieved certain results in capturing long-range context information and controlling computational complexity, it still has the following shortcomings:
[0004] 1. When processing ultrasound images with blurred boundaries, complex lesion morphology, or low-contrast lesions, the segmentation accuracy is significantly reduced, which cannot fully meet the clinical needs of refined segmentation;
[0005] 2. The accuracy of lesion edge characterization is insufficient, and there is a lack of means to finely modulate the features of the lesion area edge. This leads to significant deviations in the segmentation results of medical images with fuzzy boundaries and complex lesion details, manifested as unclear boundaries or obvious over-segmentation and under-segmentation.
[0006] 3. The model's generalization ability is poor, and it is unable to dynamically optimize the network segmentation strategy based on specific clinical diagnostic prompts. As a result, it has poor generalization ability and insufficient adaptability to complex and diverse clinical application scenarios, which restricts the model's potential for application in actual clinical diagnosis.
[0007] 4. Failure to take effective measures to suppress ultrasound image noise artifacts will significantly interfere with segmentation accuracy and affect the stability and reliability of segmentation;
[0008] 5. The lack of effective conditional feature modulation methods leads to insufficient interpretability of segmentation results. Summary of the Invention
[0009] To address the shortcomings of existing methods, the present invention constructs a VMUNet network that integrates textual prompt information. In the network's final prediction stage, the network innovatively combines FiLM conditional modulation with a cross-modal attention mechanism to generate and optimize segmentation masks. This ensures that the segmentation results accurately respond to different textual prompts, achieving precise segmentation of lesion areas and improving the model's controllable generalization capabilities. The technical solution adopted by the present invention is: The VMUNet method for ultrasound medical image segmentation integrating textual prompts includes the following steps:
[0010] Step 1: Acquire ultrasound images;
[0011] Step 2: Build the FC-VMUNET network and introduce the FiLM layer between the input embedding layer of the VMUNET network and the first VSS Block of the feature encoder to fuse the feature vectors of the visual feature map and the textual prompt;
[0012] As a preferred embodiment of the present invention, constructing the FC-VMUNET network further includes: introducing a CAFM module between the second VSS Block of the feature decoder of the VMUNET network and the first patch expansion operation of the feature decoder to integrate the feature vector of the text prompt with the encoder output feature vector.
[0013] As a preferred embodiment of the present invention, the method further includes: introducing a FICAM module after the output features of the decoder of the FC-VMUNET network.
[0014] As a preferred embodiment of the present invention, the FICAM module includes: the input feature vector is divided into two branches after being input into the FiLM layer after 1x1 convolution, one branch serves as the input of the CAFM module, and the other branch introduces the feature vector of the text prompt as another input of the CAFM model; the output of the CAFM is divided into two branches, one branch inputs bilinear interpolation, and the other branch introduces the feature vector of the text prompt as another input of the bilinear interpolation; the output of the bilinear interpolation is the segmentation result.
[0015] As a preferred embodiment of the present invention, acquiring the feature vector of the text prompt includes:
[0016] Enter clinical diagnosis text;
[0017] Construct medical text professional vocabulary and perform Vocab Token sequence mapping;
[0018] The Vocab Token sequence is input into the Embedding layer, Transformer encoder and pooling layer.
[0019] As a preferred embodiment of the present invention, the clinical diagnosis text is preprocessed.
[0020] As a preferred embodiment of the present invention, the clinical diagnosis text is segmented.
[0021] As a preferred embodiment of the present invention, a VMUNet ultrasonic medical image segmentation system integrating text prompts includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement a VMUNet ultrasonic medical image segmentation method integrating text prompts.
[0022] As a preferred embodiment of the present invention, a computer-readable medium stores computer program code, and when the computer program code is executed by a processor, the computer program code implements a VMUNet ultrasonic medical image segmentation method integrating text prompts.
[0023] Beneficial effects of the present invention:
[0024] 1. By innovatively introducing a text prompt module, FiLM conditional modulation mechanism, and cross-modal attention mechanism, clinical semantic information is effectively integrated with visual features to achieve more precise and accurate segmentation of lesion area boundaries, thereby significantly improving the accuracy and generalization ability of medical image segmentation, effectively meeting the requirements of clinical applications for segmentation accuracy and model robustness;
[0025] 2. The present invention integrates clinical text prompts at the most superficial level, allowing the subsequent network to establish responses to key areas with low contrast and fuzzy boundaries in ultrasound images earlier, effectively suppressing speckle noise interference and improving the signal-to-noise ratio of initial features;
[0026] 3. This paper introduces a cross-modal attention mechanism between the VSS Block and patch expansion of the decoder, which can use clinical text priors to perform directionally weighted downsampled features, finely enhance lesion edges, suppress ultrasound noise, and compensate for detail loss before upsampling, thereby ensuring that the high-resolution segmentation contours are highly consistent with the clinical description in shape and position;
[0027] 4. The present invention constructs a FICAM module based on the segmentation mask of text-conditional modulation and cross-modal attention optimization. It not only relies on visual features, but also can adjust the segmentation mask in combination with text prompt information, thereby dynamically adapting to changes in the lesion area under different text prompt conditions. Compared with the traditional segmentation method based only on visual features, it can more accurately adapt to different clinical text descriptions, improve the model's ability to perceive fuzzy boundaries, and enhance the generalization ability and controllability of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 This is the VMUNet ultrasonic medical image segmentation model diagram of the present invention;
[0029] Figure 2 This is the detailed structure diagram of the Transformer text encoder;
[0030] Figure 3 This is a detailed diagram of the fused text prompts within VSSBlock;
[0031] Figure 4 It is a schematic diagram of the FICAM module structure;
[0032] Figure 5 This is the segmentation effect diagram. DETAILED DESCRIPTION
[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore only shows the components related to the present invention.
[0034] The existing VMUNet network is an asymmetric U-shaped network, which includes an input embedding layer (PatchEmbedding), a feature encoder (Encoder), a feature decoder (Decoder), and cross-layer skip connections (SkipConnections).
[0035] like Figure 1 As shown in Figure 1, the VMUNet ultrasound medical image segmentation method integrating text prompts includes the following steps:
[0036] Step 1: Acquire ultrasound images;
[0037] Step 2: Construct the FiLM-CAFM-FICAM-VMUNET network (FCF-VMUNET for short). The construction of the FiLM-CAFM-VMUNET network (FC-VMUNET for short) includes:
[0038] First, the feature-wise linear modulation layer (FiLM) is used between the input embedding layer of VMUNET and the first VSS Block of the feature encoder to incorporate textual cues.
[0039] The present invention integrates clinical text prompts at the shallowest level, allowing the subsequent network to respond earlier to key areas with low contrast and fuzzy boundaries in ultrasound images, effectively suppressing speckle noise interference and improving the signal-to-noise ratio of initial features.
[0040] Secondly, a cross-modal attention fusion module (CAFM) is used between the second VSS Block of the feature decoder and the first patch expansion operation of the feature decoder to incorporate textual cues;
[0041] The present invention introduces a cross-modal attention mechanism between the VSS Block and patch expansion of the decoder, which can perform directionally weighted downsampled features with the help of clinical text priors, finely enhance the lesion edges, suppress ultrasound noise and compensate for detail loss before upsampling, thereby ensuring that the high-resolution segmentation contour is highly consistent with the clinical description in shape and position.
[0042] The FCF-VMUNET network is constructed by introducing the final segmentation prediction module (FICAM) after the last VSS Block of the feature decoder of the FC-VMUNET network;
[0043] The ultrasound image is input into the FCF-VMUNET network and the segmentation result is output.
[0044] The specific process of the FCF-VMUNET network is as follows:
[0045] First, the input embedding layer divides the input ultrasound image X (H × W × 3) into several non-overlapping 4 × 4 blocks, where H = W = 256 in this embodiment; a feature map is obtained by linear mapping to the number of channels C and layer normalization (LayerNorm), and the parameter γ of the FiLM layer is used. expand , β expand Fusing textual hints to generate initial encoding features
[0046] Secondly, the feature encoder consists of four stages, each stage contains 2 VSS blocks (including text prompts); the initial encoding feature Input encoder, set feature merging operation after each VSS Block, and change the feature map size from Downsample to The four stages are equipped with 2, 2, 2, and 2 VSS Blocks, and the number of channels in each stage is C, 2C, 4C, and 8C respectively;
[0047] Secondly, the feature decoder is also divided into four stages, and the text prompts and the CAFM module generate encoding features before entering the feature decoder. The features are then upsampled by a patch expansion operation, reducing the size of the encoded feature map from Upsample to And input the VSS Block containing the text prompt; the four stages are equipped with 2, 2, 2 and 1 VSS Blocks, and the corresponding number of channels is 8C, 4C, 2C and C respectively; the output encoding feature is upsampled by three layers to obtain the encoding feature F init ;
[0048] Finally, F init The FICAM module is used as input to fuse the textual hints with the decoded features again and map them into segmentation category predictions.
[0049] Among them, the skip connection (Skip Connections) uses cross-layer feature addition between the corresponding stages of the encoder and decoder The fusion is performed while keeping the number of channels unchanged and adding no new parameters, thus achieving a lightweight design.
[0050] like Figure 2 For text prompts, the Transformer encoder is used to extract high-dimensional features from clinical text and the global features are obtained through the pooling layer. The acquisition of text prompts includes:
[0051] S1. Input: clinical diagnosis text, such as “thyroid nodule located in the middle of the left lobe with irregular borders”;
[0052] S2, text preprocessing;
[0053] S3: Construct medical text professional vocabulary and perform vocab token sequence mapping; for example, use BPE / WordPiece / SentencePiece for word segmentation; build token sequence based on medical special vocabulary; tokenization processing: use BPE / WordPiece / SentencePiece for word segmentation;
[0054] S4. Input the Vocab Token sequence into the Embedding layer, that is, the Token sequence is mapped to the word vector representation;
[0055] S5, then input the Transformer encoder, that is, using a multi-layer self-attention mechanism to obtain contextual semantics;
[0056] S6, Pooling layer, that is, obtaining global features through MeanPooling or MaxPooling;
[0057] S7. Output feature vector of text prompt
[0058] Multi-stage text feature fusion is performed at various stages of the network, such as post-Patch Embedding fusion, VSS Block internal fusion, and pre-Patch Expanding fusion. FiLM layer and CAFM module are used for deep fusion, thereby effectively improving segmentation accuracy.
[0059] Among them, Patch Embedding is fused after that, and the FiLM layer is used to globally modulate the visual features. The text prompt information is used to modulate the visual features to ensure that the text prompt information can be effectively utilized during the encoder learning process.
[0060] Patch Embedding post-fusion includes:
[0061] First, the FiLM layer projects the features of the textual prompts to the channel dimension, as follows:
[0062] γ=Linearγ (F text );
[0063] β=Linear β (F text );
[0064] And expanded to the visual feature space dimension:
[0065] Performing element-wise modulation (Element-wiseModulation) yields:
[0066] The parameters are defined as follows:
[0067] B is the batch size (mini-batch size), that is, the number of samples in one forward propagation; C is the number of visual feature channels; H is the height of the visual feature map; W is the width of the visual feature map; d txt is the text feature dimension; The global text feature vector output by the Transformer text encoder; is a learnable linear mapping (fully connected layer), and its parameters are weight matrices and bias is a learnable linear mapping (fully connected layer), and its parameters are weight matrices and bias is the conditional modulation gain vector; is the conditional modulation offset vector; is the tensor after γ and β are replicated in batch and spatial dimensions; The original visual feature map obtained after Patch Embedding; ⊙ is element-by-element multiplication; Fusion of image and text feature encoding.
[0068] The internal fusion of VSS Block is performed before the SS2D module, using the FiLM layer to perform channel modulation to ensure that the number of channels remains unchanged after SS2D; the role of the internal fusion of VSS Block is fine-grained feature modulation, optimizing the SS2D module input, so that it can accurately model the lesion area.
[0069] like Figure 3 For internal fusion within the VSS Block, text features modulate the visual features through the FiLM layer, including:
[0070] 1. Visual feature normalization:
[0071] 2. Text feature projection: γ = Linear γ (F text ),β=Linear β (F text );
[0072] 3. Expand γ and β to the visual feature space:
[0073] 4. Channel modulation (keep the number of channels unchanged after SS2D):
[0074]
[0075] 5. Send the modulated features to SS2D calculation (the number of channels remains unchanged):
[0076] 6. Residual Connection:
[0077] 7. The residual fusion result is normalized layer by layer and output through the fully connected layer to obtain the final features of the VSS Block:
[0078] The parameters are defined as follows:
[0079] It is the input visual feature map in VSS Block; For F vss The mean and standard deviation calculated separately in the channel dimension; is the normalized visual feature map; SS2D(·) is the spatial state space (or convolution) module in VSS Block, keeping the number of channels unchanged; It is the feature map fed into SS2D after FiLM conditional modulation fusion; It is the feature map output by the SS2D module; is another output of depth-wise separable convolution; Output features for the residual connection in the VSS Block; It is the final output of the VSS block.
[0080] Patch Expanding pre-fusion uses the CAFM module to deeply fuse text features and visual features;
[0081] Patch Expanding pre-fusion includes:
[0082] First, the initial segmentation mask is generated by 1×1 convolution;
[0083] Secondly, before the final mask is upsampled to the original image size, a cross-modal attention mechanism is introduced to optimize the mask feature expression, using text features to guide the mask to accurately depict the lesion edge;
[0084] Take the FiLM modulated features as the query and the expanded text features as the key / value. The query / key / value settings are:
[0085] Q = Conv q (F dec )
[0086]
[0087] Calculate attention distribution:
[0088] Fusion features:
[0089] Finally, bilinear interpolation is used to upsample the mask features optimized by the cross-modal attention mechanism to the original image size;
[0090] like Figure 4 ,FICAM module, in the final stage of the network, the present invention ensures the accuracy of the segmentation mask and the high consistency of the text prompt through the following two innovative mechanisms;
[0091] The FICAM module includes: the input feature vector is input into the FiLM layer after 1x1 convolution and then divided into two branches, one as the input of the CAFM module, and the other as the other input of the CAFM model after introducing text prompts; the output of CAFM is divided into two branches, one inputs bilinear interpolation, and the other as the other input of bilinear interpolation after introducing text prompts; the output of bilinear interpolation is the segmentation result.
[0092] The specific process of the FICAM module:
[0093] Input F after decoder init After generating a preliminary segmentation mask through 1×1 convolution, the mask is further modulated by FiLM using text conditional features, so that the final segmentation mask can flexibly adapt to different lesion areas according to the text description:
[0094] F final =Conv 1×1 (F init );
[0095]
[0096] Cross-modal attention mechanism optimizes the mask upsampling process:
[0097] Before bilinear interpolation upsampling to the original image size, a cross-modal attention mechanism is introduced to optimize mask features. The query-key-value mechanism is used to fuse text features to guide the mask to finely express the edge details of the lesion:
[0098] Take the FiLM modulated features as the query and the expanded text features as the key / value:
[0099]
[0100] Finally, the features are upsampled to the original image size through bilinear interpolation:
[0101]
[0102] The parameters are defined as follows:
[0103] is the visual feature map output by the decoder; d is the number of channels of the query (Q) and key (K) vectors, used for the dimension of the attention inner product; Conv q ,Conv k ,Conv v There are three 1×1 convolutional layers, which are used to generate query, key, and value tensors, respectively, each with learnable weights and biases; Q = Conv q (F dec ) is the query tensor; A copy / expanded tensor of the text feature vector in the channel and spatial dimensions; is the key tensor; is a value tensor; Softmax(·) normalizes the attention similarity so that the sum of the weights is 1; The value tensor after weighting the cross-modal attention; is the residual fusion of decoder features and attention features; F init is the input feature map of the FICAM module; Conv 1×1 is a 1×1 convolutional layer used to transform F init Mapped to the number of segmentation mask channels; M is the number of segmentation mask channels (e.g., the number of categories, 1 or 2 for binary classification); F final =Conv 1×1 (F init ) is the initial segmentation mask logits; is the final segmentation mask after FiLM conditional modulation; It is the fusion of the final mask and cross-modal attention features.
[0104] FICAM generates segmentation masks based on text-conditioned modulation (FiLM) and cross-modal attention optimization. It not only relies on visual features, but also adjusts the segmentation mask in combination with text prompt information, thereby dynamically adapting to changes in the lesion area under different text prompt conditions. Compared with traditional segmentation methods based only on visual features, it can more accurately adapt to different clinical text descriptions, improve the model's ability to perceive fuzzy boundaries, and enhance the generalization and controllability of the segmentation results.
[0105] The method of the present invention enables the final segmentation prediction to not only rely on visual features, but also to accurately integrate text prompts, significantly improving the segmentation accuracy and generalization ability of the lesion area, especially the fuzzy boundaries.
[0106] Comparative analysis of experimental results
[0107] like Figure 5 As shown, GroundTruth (real annotation) Figure 5 a shows the real labeled image of the patient's thyroid nodule, which serves as the standard reference for the segmentation results. Figure 5 b is the corresponding mask; Figure 5 The VMUNET model is not effective in processing lesions with blurred boundaries. The lesion boundaries are blurred, the segmentation results are unclear, and the accuracy is low. Figure 5 d is the segmentation result of the method of the present invention. Combining conditional modulation with cross-modal attention mechanism, the segmentation accuracy is significantly improved. Especially in the processing of lesions with blurred boundaries, the segmentation results are more consistent with the true annotations, and the text prompts help improve the accuracy and generalization ability of segmentation.
[0108] The present invention performs conditional modulation, modulating visual features through text prompts, and significantly improves the segmentation accuracy of lesions with blurred boundaries;
[0109] The present invention uses a cross-modal attention mechanism to deeply integrate text and visual features, more accurately identify lesion edges, and improve the segmentation effect of complex lesions;
[0110] The final segmentation prediction module of the present invention optimizes mask generation and upsampling through FiLM modulation and cross-modal attention mechanism to ensure accurate segmentation of lesion boundaries with richer details.
[0111] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.
Claims
1. The VMUNet ultrasound medical image segmentation method integrating text prompts is characterized by: The following steps are involved: Step 1: Acquire ultrasound images; Step 2: Build the FC-VMUNET network and introduce the FiLM layer between the input embedding layer of the VMUNET network and the first VSSBlock of the feature encoder to fuse the feature vectors of the visual feature map and the text prompt.
2. The VMUNet ultrasonic medical image segmentation method integrating text prompts according to claim 1 is characterized in that: Building the FC-VMUNET network also includes: introducing a CAFM module between the second VSS Block of the feature decoder of the VMUNET network and the first patch expansion operation of the feature decoder to integrate the feature vector of the text prompt with the encoder output feature vector.
3. The VMUNet ultrasonic medical image segmentation method integrating text prompts according to claim 1 is characterized in that: Also includes: The FICAM module is introduced after the output features of the decoder of the FC-VMUNET network.
4. The VMUNet ultrasonic medical image segmentation method integrating text prompts according to claim 3 is characterized in that: The FICAM module includes: the input feature vector is divided into two branches after 1x1 convolution and then input into the FiLM layer. One branch serves as the input of the CAFM module, and the other branch introduces the feature vector of the text prompt as the other input of the CAFM model; the output of CAFM is divided into two branches, one inputs bilinear interpolation, and the other introduces the feature vector of the text prompt as the other input of bilinear interpolation; the output of bilinear interpolation is the segmentation result.
5. The VMUNet ultrasonic medical image segmentation method integrating text prompts according to any one of claims 1 to 4, characterized in that: The feature vector acquisition of text prompts includes: Enter clinical diagnosis text; Construct medical text professional vocabulary and perform Vocab Token sequence mapping; The Vocab Token sequence is input into the Embedding layer, Transformer encoder and pooling layer.
6. The VMUNet ultrasonic medical image segmentation method integrating text prompts according to claim 5 is characterized in that: Preprocess clinical diagnosis text.
7. The VMUNet ultrasonic medical image segmentation method integrating text prompts according to claim 5 is characterized in that: Segment the clinical diagnosis text.
8. A VMUNet ultrasonic medical image segmentation system integrated with text prompts, comprising: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the VMUNet ultrasonic medical image segmentation method integrated with text prompts as described in any one of claims 1 to 7.
9. A computer-readable medium storing computer program code, characterized in that When the computer program code is executed by a processor, the computer program code implements the VMUNet ultrasonic medical image segmentation method integrated with text prompts according to any one of claims 1 to 7.