A lesion segmentation method of adaptive dynamic text prompt

Abstract

The present application relates to the technical field of image processing, and more particularly to a lesion segmentation method of adaptive dynamic text prompt, comprising: acquiring a lesion image, and generating an adaptive dynamic text prompt based on the lesion image; constructing a multi-modal enhanced fusion text prompt adapter based on a FiLM global channel recalibration module, a spatial cross-attention interaction module and a zero initialization gate nonlinear integration module cascade; the global channel recalibration module of FiLM utilizes a feature linear modulation mechanism to perform channel-level weighting on a visual feature map; before the feature enters spatial interaction, according to the text semantic enhancement band response, the activation of the background noise channel is inhibited. The present application overcomes the problems of insufficient utilization of multi-modal prompts, dependence on artificial static labeling of text prompts, serious dependence on prior information in the reasoning stage and difficulty in multi-modal feature fusion in a low-contrast environment in the existing medical image segmentation technology.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for lesion segmentation using adaptive dynamic text prompts. Background Technology

[0002] With the rapid development of deep learning technology in the field of medical image analysis, automated lesion segmentation has become a core part of computer-aided diagnosis (CAD). In recent years, multimodal "vision-language" models with text prompts have provided new ideas for solving the above problems. However, existing medical image segmentation technologies that integrate text prompts still have significant limitations in practical applications.

[0003] 1. The generation of text prompts mainly relies on manual intervention and lacks adaptability and dynamism. Most text prompts in existing technologies come from doctors' manual static annotations or predefined simple templates (such as coarse orientations containing only "left / right"). Such static prompts cannot capture the individual differences of lesions in different patients (such as specific fine-grained positional shifts and varying shapes and sizes). The lack of refined description of the geometric shape of lesions means that the model cannot obtain explicit spatial and morphological prior guidance when facing complex anatomical structures, which limits the upper limit of segmentation accuracy.

[0004] 2. The reasoning stage suffers from the "prior dependence paradox" and poor generalization ability. Current visual-language segmentation models usually require high-quality text prompts during both the training and inference stages. In actual clinical reasoning scenarios, doctors often cannot provide accurate text descriptions for each image to be tested in real time. If text prompts are missing or only general prompts are provided during reasoning, the performance of models that heavily rely on specific prior information will drop sharply.

[0005] 3. The multimodal feature fusion mechanism is crude and difficult to cope with noise interference in medical images. Existing multimodal fusion methods mostly use simple feature splicing or element-by-element addition. This shallow fusion method lacks a fine-grained screening mechanism for feature channels. When the background noise of medical images is large, simple fusion will introduce irrelevant text information (or noise) into the visual feature flow, which will interfere with the extraction of lesion edges.

[0006] 4. Lack of semantic-level supervision consistency; traditional segmentation loss functions (such as Dice Loss, IoU Loss) only focus on pixel-level overlap and ignore semantic alignment in the feature space; the model is prone to "accurate segmentation (shape overlap)" but "incorrect segmentation (semantic error)", often resulting in misdetection of non-lesion areas with similar shapes.

[0007] In summary, existing technologies lack a medical image segmentation method that can automatically generate high-precision text prompts, possess noise-resistant deep feature fusion capabilities, and exhibit strong inference robustness. Summary of the Invention

[0008] To address the shortcomings of existing methods, the present invention aims to overcome the problems in existing medical image segmentation technology, such as insufficient utilization of multimodal prompts, reliance on manual static annotation for text prompts, heavy reliance on prior information during the inference stage, and difficulty in multimodal feature fusion in low-contrast environments.

[0009] The technical solution adopted in this invention is: an adaptive dynamic text prompt lesion segmentation method comprising the following steps: Step 1: Acquire lesion images and generate adaptive dynamic text prompts based on the lesion images; In a preferred embodiment of the present invention, the lesion image is preprocessed by randomly flipping horizontally or vertically.

[0010] As a preferred embodiment of the present invention, step one specifically includes: Step 11: Calculate the zeroth and first moments of the binary mask of the lesion and construct a grid coordinate system; Step 12: Calculate the area ratio of the lesion region pixels to the whole image, assign multi-scale morphological text descriptions, and synthesize text sequence features T.

[0011] Step 2: Construct a multimodal enhanced fusion text prompt adapter based on the cascaded FiLM global channel recalibration module, spatial cross-attention interaction module, and zero-initialization gated nonlinear integration module; FiLM's global channel recalibration module uses a feature linear modulation mechanism to perform channel-level weighting on visual feature maps; before features enter spatial interaction, it enhances the frequency band response according to text semantics and suppresses the activation of background noise channels.

[0012] In a preferred embodiment of the present invention, the global channel recalibration module of FiLM includes: Assuming the input visual features are The global features of the text are The modulated characteristics are: ; in, and These are the scaling and translation parameters for the generated channels, respectively. and The features are filtered and modulated using the weights and biases of the linear projection layer, with subscripts... Indicates visual features Broadcast operations along the channel dimension. This indicates element-wise multiplication.

[0013] In a preferred embodiment of the present invention, the spatial cross-attention interaction module includes: right Flatten out, and let the visual characteristics after flattening be... The text sequence features are ; generated by linear projection And calculate cross-modal attention: ; in, , , For spatial cross-attention interaction projection matrix, This means mapping the image to a query vector. This indicates that the text is mapped to index keys. This indicates that the text is mapped to a content value; Scaling factor Represents the normalized exponential function; matrix This is the correlation matrix.

[0014] In a preferred embodiment of the present invention, the zero-initialization gated nonlinear integration module includes:

[0015] in, For GELU activation function, and This indicates that the features are being upgraded and expanded. and This indicates that the features are reduced in dimensionality and fused. For learnable gated scalars, This represents a linear mapping.

[0016] In a preferred embodiment of the present invention, the alignment loss function of the multimodal enhanced fusion text prompt adapter is: ; in, Represents visual feature map of high-rise buildings. Represents the binary true mask. l represents the mask average pooling operation. This indicates a global average pooling operation. Represents the cosine similarity function. Features of text sequences.

[0017] As a preferred embodiment of the present invention, the adaptive dynamic text prompt lesion segmentation system includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the adaptive dynamic text prompt lesion segmentation method.

[0018] As a preferred embodiment of the present invention, a computer-readable medium storing computer program code implements an adaptive dynamic text prompting lesion segmentation method when executed by a processor.

[0019] The beneficial effects of this invention are: 1. This invention proposes an automated dynamic prompt generation mechanism based on geometric morphology, which can generate semantically rich prompts containing fine orientation and scale information in real time based on lesion characteristics without manual intervention, greatly improving the accuracy and flexibility of the prompts. 2. This invention innovatively designs a training-inference random decoupling strategy; by using rich semantic prompts and general prompts in combination during the training phase, the model is forced to learn deep mapping relationships across modalities, so that the model can activate high-precision segmentation capabilities with only simple general task instructions (without specific lesion information) during the inference phase, successfully avoiding dependence on prior information for inference.

[0020] 3. This invention constructs a cascaded multimodal enhanced fusion text prompt adapter, uses the FiLM mechanism for channel-level recalibration, and combines it with sequence-aware spatial cross-attention; this deep interaction mechanism can actively suppress background noise channels and accurately locate potential lesion areas consistent with the text description in low-contrast feature maps, which is significantly better than traditional shallow fusion methods. 4. This invention introduces a foreground-aware semantic consistency constraint, which explicitly constrains the segmented region to maintain semantic consistency with the text description at the high-dimensional feature manifold level, thereby effectively eliminating interference items that are similar in shape but contradictory in meaning; Attached Figure Description

[0021] Figure 1 This is a flowchart of the adaptive dynamic text prompt lesion segmentation method of the present invention; Figure 2 This invention relates to a FiLM-based global channel recalibration module; Figure 3 This is the spatial cross-attention interaction module for sequence perception of the present invention; Figure 4 This invention relates to a zero-initialization gated nonlinear integration module. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments. The drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0023] like Figure 1 As shown, an adaptive dynamic text prompt lesion segmentation method includes the following steps: To address the problem that existing medical image segmentation relies excessively on manual static annotation and struggles to capture individual differences in lesions, this invention proposes an automated dynamic text prompt generation strategy that requires no manual intervention: Step 1: Construct an adaptive dynamic text prompt generation mechanism based on geometric morphology; To address individual differences in lesions, an automated, dynamic text prompt generation strategy was developed that requires no manual intervention. Step 11, Refined Spatial Discrete Localization: Calculate the zeroth and first moments of the binary mask of the lesion to solve for the centroid; construct a 5×5 fine-grained grid coordinate system, and discretize the anatomical space into five-level scales in the horizontal and vertical dimensions respectively (such as {Left, Left-Center...} and {Top, Upper...}); automatically generate a high-precision orientation descriptor (such as "upper left-center") based on the centroid's landing point.

[0024] Step 12, Scale-aware adaptive morphological description: Calculate the area ratio of the lesion region pixels to the whole image, and adaptively assign multi-scale morphological text descriptions of "small", "medium" or "large" to the target through a dynamic threshold mapping function, and synthesize text sequence features T with structured semantics.

[0025] Step 13, Training-Inference Random Decoupling Strategy: During the training phase, a 7:3 hybrid cue sampling strategy (70% semantically rich cue, 30% general cue) is adopted to learn deep mapping relationships; during the inference phase, general cue (such as "breast tumor lesion segmentation task") is used to activate the segmentation task by leveraging cross-modal feature extraction capabilities, thus avoiding dependence on prior information during the inference phase.

[0026] Step 2: Construct a cascaded multimodal enhanced fusion text prompt adapter. Design a three-stage cascaded feature fusion text prompt module, from coarse to fine and from channel dimension to spatial dimension, to address edge blurring and low contrast issues in medical images. like Figure 2 Step 21: Based on the FiLM global channel recalibration module; Global average pooling is performed on the text sequence features T to extract the global features of the text embedding. Feature scaling and translation factors are generated through linear mapping. Channel-level affine transformation is performed on the visual feature map. The feature-wise linear modulation (FiLM) mechanism is used to weight the visual feature map at the channel level. Before the features enter the spatial interaction, the relevant frequency band response is enhanced according to the text semantics (such as "small"), and the activation of background noise channels is effectively suppressed. Assume the input visual features are: The global features of the text are The modulated features The calculation is as follows: ; in, and These are the scaling and translation parameters for the generated channels, respectively. and The features are filtered and modulated using the weights and biases of the linear projection layer, with subscripts... Indicates visual features Broadcast operations along the channel dimension. This indicates element-wise multiplication.

[0027] The formula is passed The residual form enables the enhancement of key feature channels and the suppression of irrelevant channels.

[0028] The formula for global text features is: ; in, L Indicates the sequence length.

[0029] like Figure 3 Step 22: Sequence-aware spatial cross-attention interaction module; Channel-modulated visual features Flattening is used as the query vector, and the text sequence features T containing fine directional semantics are used as the key and value. The correlation matrix between the text directional words and the visual spatial region is dynamically calculated using a multi-head attention mechanism, so as to accurately locate the potential lesion region consistent with the text description in the low contrast feature map.

[0030] Modulated visual features of the input Flatten the image, and let the visual characteristics after flattening be... The text sequence features are ; L This indicates a question asking for the length of a text sequence. C Represents the embedding vector dimension; generated through linear projection. And calculate cross-modal attention: ; in, , , For spatial cross-attention interaction projection matrix, This means mapping the image to a query vector. This indicates that the text is mapped to index keys. This indicates that the text is mapped to a content value; Scaling factor Represents the normalized exponential function; matrix This is the correlation matrix, which is used to achieve weighted activation of potential lesion areas, allowing the network to focus on the image features described in the text prompts.

[0031] like Figure 4 Step 23: Zero-initialization gated nonlinear integration module; A feedforward neural network (FFN) is cascaded after cross-attention, and learnable zero-initialized gating parameters are introduced; Let the output features of the cross-attention layer be This module performs nonlinear transformation and gating fusion on it: ; in, For GELU activation function, , For the interlayer projection weights of FFN, and This indicates that the features are being upgraded and expanded. and This indicates feature reduction and fusion; key parameters For learnable gated scalars, The condition represents a linear mapping; this condition makes the gradient flow in the initial training phase approximately an identity mapping, thus avoiding abrupt interference from multimodal features on the feature distribution of the pre-trained backbone network. This design not only enhances the expressive power of multimodal features through nonlinear transformation, but also achieves a smooth transition of the residual manifold through a gating mechanism. This ensures that the network retains the pre-trained distribution of the visual backbone in the early stages of training, avoids interference from multimodal noise, and ensures the stability of model convergence.

[0032] Step 3: Introduce foreground-aware semantic consistency constraints. During model training, text and image feature alignment loss is introduced. To address the shortcomings of traditional pixel-level loss functions (such as IoU Loss) in constraining high-level semantic consistency, this invention introduces text and image feature alignment loss, constructing an alignment bridge mapping from pixel space to semantic feature space: Step 31: Mask-guided aggregation of region of interest features: Using the real segmentation mask as spatial attention weights M Visual feature map of high-rise buildings Perform masked average pooling; this operation achieves the extraction of clean foreground features, mathematically eliminating the pollution of feature vectors by background pixels, and ensuring that the extracted visual vectors only represent the texture and semantic information of the lesion area.

[0033] High-rise visual feature map It is obtained by extracting visual features X using a backbone network, such as ResNet. Step 32: Consistency optimization in high-dimensional semantic space: The visual foreground vector, obtained by aggregating the features of the region of interest guided by the mask, is projected onto the text feature space, and the cosine similarity between it and the input text prompt vector is calculated. The optimization objective is to maximize this similarity and explicitly constrain the segmented region to maintain consistency with the text description on the feature manifold. This mechanism forces the network to focus not only on "accurate segmentation (shape)" but also on "correct segmentation (semantics)," effectively suppressing false positives caused by similar shapes but contradictory semantics (such as misclassifying blood vessels as nodules), and enabling the network to learn the image features described by the text prompt. ; in, Represents visual feature map of high-rise buildings. Represents the binary true mask. l represents the mask average pooling operation. This indicates a global average pooling operation. Represents the cosine similarity function. Features of text sequences.

[0034] Experimental setup: Hardware and software environment; Hardware: Ubuntu 20.04 LTS; Intel Core i9-10900K; 64 GB DDR4; NVIDIA RTX 3080Ti (12 GB); Software: Python 3.8.10; PyTorch 1.13.0; CUDA 12.8; cuDNN 8.x.

[0035] Data loading: During training, data is loaded in parallel using 4 threads; during validation / testing, it is not loaded in parallel.

[0036] Optimization strategies and hyperparameters

[0037] Optimizer: AdamW (β1 = 0.9, β2 = 0.999, ε = 1×10) -8 Weight Decay = 1 × 10 - ²), initial learning rate 1×10 - ³.

[0038] Learning rate scheduling: Cosine Annealing (T max = 800, η min = 1×10 -5The training process consists of 800 rounds, with the learning rate updated after each round.

[0039] Batch size: Batch = 4 for training phase; Batch = 1 for validation / testing phase.

[0040] Mixed Precision: Automatic Mixed Precision (AMP) is used throughout the training process, and gradient scaling is employed.

[0041] Dataset: To verify the effectiveness and robustness of the method of the present invention, this embodiment conducted evaluation experiments on two independent ultrasound datasets: the Private Thyroid Ultrasound Dataset and the publicly available BUSI_WHU breast tumor ultrasound dataset.

[0042] This private thyroid ultrasound dataset contains 1,426 thyroid ultrasound images acquired from the Second People's Hospital of Changzhou, affiliated with Nanjing Medical University. The images were acquired using various commercial ultrasound scanners from Philips, Siemens, and GE, ensuring data diversity. In addition to the original images, corresponding text annotation files were also created during the dataset construction process to mark the specific spatial location information of lesions.

[0043] The BUSI_WHU public dataset was collected from the Department of Radiology at Wuhan University People's Hospital and contains 927 breast ultrasound images, including both benign and malignant lesions. The collection of this dataset was approved by the Ethics Committee of Wuhan University People's Hospital.

[0044] Data partitioning and augmentation strategies: To ensure the rigor of the experiment, the datasets were randomly divided into training, validation, and test sets in a strict 6:2:2 ratio. During the data preprocessing stage, in order to enhance the robustness of the model to geometric transformations, this embodiment performed online data augmentation operations on the input data during training: random flipping: the input image was randomly flipped horizontally or vertically with a probability of 0.5; random rotation: the image was randomly rotated within an angle range of 20° to 80°.

[0045] The experimental results are compared and analyzed as shown in Table 1. Table 1. Results of ablation experiments on private thyroid ultrasound images.

[0046] Table 2. Ablation and segmentation results of BUSI_WHU breast ultrasound images.

[0047] The experimental results in Tables 1 and 2 show that the fusion method proposed in this invention outperforms the pure visual benchmark model on both the private thyroid dataset and the public BUSI_WHU breast tumor dataset. On the private thyroid dataset, the mIoU is improved by 5.62% and the sensitivity by 6.89%, while on BUSI_WHU, the mIoU is improved by 4.07% and the sensitivity by 4.90%, fully demonstrating the effectiveness of the method. The layer-by-layer ablation experiment further reveals the unique contributions of each module: the addition of adaptive text prompts provides key spatial and morphological priors; the addition of a sequence-aware spatial cross-attention interaction mechanism enhances the accurate localization of lesions; the addition of FiLM global channel recalibration effectively suppresses background noise interference; and the addition of a zero-initialization gated nonlinear integration mechanism ensures the stability of multimodal feature fusion. The synergistic effect of each module successfully overcomes the technical challenges of variable lesion scale, blurred edges, and low contrast in medical ultrasound images.

[0048] Based on the above-described preferred embodiments of the present invention, and through the foregoing description, those skilled in the art can make various changes and modifications without departing from the inventive concept. The technical scope of this invention is not limited to the contents of the specification, but must be determined according to the scope of the claims.

Claims

1. A lesion segmentation method with adaptive dynamic text prompts, characterized in that, Includes the following steps: Step 1: Acquire lesion images and generate adaptive dynamic text prompts based on the lesion images; Step 2: Construct a multimodal enhanced fusion text prompt adapter based on the cascaded FiLM global channel recalibration module, spatial cross-attention interaction module, and zero-initialization gated nonlinear integration module; FiLM's global channel recalibration module uses a feature linear modulation mechanism to perform channel-level weighting on visual feature maps; before features enter spatial interaction, it enhances the frequency band response according to text semantics and suppresses the activation of background noise channels.

2. The lesion segmentation method with adaptive dynamic text prompts according to claim 1, characterized in that, The FiLM global channel recalibration module includes: Assuming the input visual features are The global features of the text are The modulated characteristics are: ； in, and These are the scaling and translation parameters for the generated channels, respectively. and Weights and biases of the linear projection layer, subscripts Indicates visual features Broadcast operations along the channel dimension. This indicates element-wise multiplication.

3. The lesion segmentation method with adaptive dynamic text prompts according to claim 2, characterized in that, The spatial cross-attention interaction module includes: right Flatten out, and let the visual characteristics after flattening be... The text sequence features are ; generated by linear projection And calculate cross-modal attention: ； in, , , For spatial cross-attention interaction projection matrix, This means mapping the image to a query vector. This indicates that the text is mapped to an index key. This indicates that the text is mapped to a content value; Scaling factor Represents the normalized exponential function; matrix This is the correlation matrix.

4. The lesion segmentation method with adaptive dynamic text prompts according to claim 3, characterized in that, The zero-initialization gated nonlinear integration module includes: ； in, For GELU activation function, and This indicates that the features are being upgraded and expanded. and This indicates that the features are reduced in dimensionality and fused. For learnable gated scalars, This represents a linear mapping.

5. The lesion segmentation method with adaptive dynamic text prompts according to claim 1, characterized in that, The alignment loss function for the multimodal enhanced fusion text prompt adapter is: ； in, Represents visual feature map of high-rise buildings. Represents the binary true mask. l represents the mask average pooling operation. This indicates a global average pooling operation. Represents the cosine similarity function. Features of text sequences.

6. The lesion segmentation method with adaptive dynamic text prompts according to claim 1, characterized in that, Step one specifically includes: Step 11: Calculate the zeroth and first moments of the binary mask of the lesion and construct a grid coordinate system; Step 12: Calculate the area ratio of the lesion region pixels to the whole image, assign multi-scale morphological text descriptions, and synthesize text sequence features T.

7. The lesion segmentation method with adaptive dynamic text prompts according to claim 1, characterized in that, The lesion images are preprocessed by randomly flipping them horizontally or vertically.

8. A lesion segmentation system with adaptive dynamic text prompts, characterized in that, include: Memory is used to store instructions that can be executed by the processor; A processor for executing instructions to implement the lesion segmentation method with adaptive dynamic text prompts as described in any one of claims 1-7.

9. A computer-readable medium storing computer program code, characterized in that, The computer program code, when executed by a processor, implements the adaptive dynamic text prompting lesion segmentation method as described in any one of claims 1-7.

Images