A deep learning-based text-assisted medical image segmentation method

Abstract

The application discloses a kind of text auxiliary medical image segmentation methods based on deep learning, comprising: the multi-modal segmentation network of constructing including visual encoder, text encoder and cascaded text guide decoder;Through visual-linguistic coordinator, the feature alignment of each hidden layer of text encoder and each convolution layer of visual encoder is sequentially carried out from deep layer to shallow layer;Each decoding stage of text guide decoder sequentially executes channel attention modulation, spatial attention modulation and receptive field modulation, and generates segmentation prediction feature map;After the training sample is augmented, it is input into network training with corresponding text description, freezes text encoder, and reversely updates network parameters according to segmentation error, obtains prediction model;The image to be segmented and its diagnosis report text are input into prediction model, and segmentation mask chart is output after activation decision.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically, to a text-assisted medical image segmentation method based on deep learning. Background Technology

[0002] In the fields of medical imaging and clinical diagnosis, automated medical image segmentation technology is a core foundation for computer-aided diagnosis, lesion quantification analysis, and radiotherapy planning. Efficient and accurate segmentation algorithms help clinicians quickly locate lesion regions, calculate organ volume, and monitor disease progression. However, medical images often present inherent challenges such as high imaging noise, blurred lesion boundaries, complex anatomical structures, and significant individual differences. Traditional segmentation methods relying solely on visual information are prone to missed or false detections in complex clinical scenarios. Furthermore, because medical image annotation requires highly specialized knowledge and is time-consuming and labor-intensive, high-quality labeled datasets are extremely scarce, further limiting the generalization ability of deep learning models.

[0003] To overcome the aforementioned difficulties, existing technologies have begun to explore using textual information such as diagnostic reports as auxiliary guidance to enhance segmentation performance. However, existing solutions still have the following shortcomings:

[0004] 1. Existing methods fail to achieve cross-modal collaboration at the deep abstraction level, resulting in a semantic gap between visual and linguistic features that is difficult to eliminate, thus limiting the guiding effectiveness of prior textual information.

[0005] 2. Standard dot product attention mainly focuses on spatial dimension association modeling, but lacks effective utilization of channel dimension semantic information, resulting in insufficient differentiation of key tissues and lesion boundaries when facing cross-modal complex contexts.

[0006] 3. There is a significant scale difference between the global wide receptive field of the pre-trained visual encoder and the local high-resolution features of the decoder. The multi-scale features of the two are misaligned during layer-by-layer upsampling and fusion, causing the prior information of the text to lose accurate contextual anchors. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a text-assisted medical image segmentation method based on deep learning.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] A deep learning-based text-assisted medical image segmentation method includes the following steps:

[0010] Step S1: Construct a multimodal medical image segmentation neural network. The network includes a visual encoder, a text encoder, and a cascaded text-guided decoder. The hidden layers of the text encoder and the convolutional layers of the visual encoder are aligned sequentially from deep to shallow through a visual-language coordinator to obtain aligned visual feature maps and text features.

[0011] In step S2, the text-guided decoder sequentially performs channel attention modulation, spatial attention modulation, and receptive field modulation in each decoding stage to generate a segmentation prediction feature map with the same resolution as the input image.

[0012] Step S3: Augment the training samples, input the augmented images and corresponding text descriptions into the multimodal medical image segmentation neural network for forward propagation, freeze the text encoder, and update the parameters of the visual encoder and text-guided decoder in reverse according to the segmentation error to obtain the prediction model.

[0013] Step S4: Input the medical image to be segmented and its corresponding diagnostic report text into the prediction model, perform activation processing and pixel-level category decision on the output segmentation prediction feature map, and output the final segmentation mask map.

[0014] Furthermore, in step S1, the visual-language coordinator performs feature alignment including:

[0015] Extract the global descriptor of visual features of the current layer. and text feature global descriptor ;

[0016] Global descriptor of visual features and text feature global descriptor Projecting onto a latent manifold space of the same dimension to perform cross-modal interactions, and mapping back to their respective original spaces, yields aligned semantic representations:

[0017]

[0018] In the formula, This represents a hierarchy- and mode-specific dimensionality-reduced projection matrix; This represents the hierarchical and modality-specific upscaling projection matrix; Indicates the activation function; For hierarchical-specific cross-modal manifold shared interaction matrices, , For the first Laminar manifold dimension For layer indexes in visual encoders or text encoders. It is the set of real numbers; For the aligned semantic representation; Represents the global descriptor. For modal indexing, , Representing visual modality, Represents the text modality.

[0019] Furthermore, in step S1, the aligned semantics are inversely injected back into the corresponding encoder features to update the visual feature map and text features:

[0020]

[0021]

[0022] In the formula, This represents the updated visual feature map; Represents the original visual feature map; This indicates a broadcast copy operation along a spatial dimension; This represents the complete text feature sequence after text feature replacement and updating; For the semantic representation of aligned visual features; For the semantics of the aligned text; This represents the sequence of text lexical features in the current layer, excluding the categorical lexical features. The length of the text sequence.

[0023] Furthermore, in step S2, when the text-guided decoder performs channel attention modulation during the decoding phase, it adjusts the channel attention of the first channel. Layer input features Global average pooling and global max pooling are performed separately to obtain the visual descriptor. and ;

[0024] The categorical lexical features output by the text encoder are projected onto the visual channel dimension via a multilayer perceptron to obtain the text descriptor. And calculate the channel saliency map. :

[0025]

[0026] In the formula, The kernel size is One-dimensional convolution operation; Use the Sigmoid activation function;

[0027] Channel saliency map and input features Element-wise multiplication yields the features modulated by channel attention. :

[0028]

[0029] In the formula, For element-wise multiplication; For the decoding stage index, .

[0030] Furthermore, in step S2, when the text-guided decoder performs spatial attention modulation during the decoding phase, it calculates the features after channel attention modulation. Text features output by the text encoder The affinity between them, and through the text length Aggregate dimensions to generate spatial saliency maps :

[0031]

[0032] In the formula, and The projection matrix is ​​learnable; Features The number of feature channels; This is the matrix transpose. and Representing the length along the text Perform average pooling and max pooling operations on the dimension; This represents the normalized exponential function;

[0033] Spatial saliency map Features after channel attention modulation Element-wise multiplication yields the features modulated by spatial attention. :

[0034]

[0035] In the formula, This is for element-wise multiplication.

[0036] Furthermore, in step S2, when the text-guided decoder performs receptive field modulation during the decoding phase, it modulates the features after spatial attention modulation. The feature maps are divided into four groups along the channel dimension. Different convolution operations are applied to each of the four groups of feature maps. The four results are then concatenated along the channel dimension and convolved. Finally, the convolution is performed with the spatially attention-modulated feature maps. The expression for addition is:

[0037]

[0038] In the formula, Indicates the kernel size as Convolution; The preset bar convolution kernel size parameters; Indicates the kernel size as Depth-separable convolution; Indicates the kernel size as The horizontal bar depth can be separable convolution; Indicates the kernel size as Vertical stripe depth can be separable convolution; , , and For feature submaps, feature submaps Preserve the identity mapping; This provides the output features for the text-guided decoder.

[0039] Furthermore, the preset strip convolution kernel size parameters The value is the feature after spatial attention modulation of distance. The nearest odd number that is half the size of the spatial dimension.

[0040] Furthermore, the features output by each decoding stage of the text-guided decoder are upsampled and then fused with the skip connection features output by the corresponding feature extraction stage of the visual encoder. The fused features are then used as the input for the next decoding stage.

[0041] Furthermore, in step S3, training sample augmentation includes: simultaneously applying random rotation, random cropping, and scaling operations to the training image and its corresponding segmentation mask; and randomly applying Gaussian noise perturbation and Gibbs ringing artifact simulation to the training image.

[0042] Furthermore, in step S4, for single-class tasks, the activation process uses the Sigmoid activation function and compares the probability value of each pixel position with a preset threshold to determine the pixel-level class; for multi-class tasks, the activation process uses the Softmax function and selects the class with the highest probability value at each pixel position as the pixel class.

[0043] The beneficial effects of this invention are:

[0044] 1. This invention uses a vision-language coordinator to align the features of each convolutional layer of the visual encoder and each hidden layer of the text encoder from deep to shallow. It utilizes modality-specific dimensionality reduction and dimensionality increase projection matrices to map the global descriptors of vision and text to a unified latent manifold space for cross-modal interaction, effectively reducing the semantic gap between visual features and language features, and enabling prior text knowledge to accurately guide the expression of visual features.

[0045] 2. This invention constructs semantic-aware channel attention modulation and spatial attention modulation in the text-guided decoder, which fuse text information from the channel dimension and spatial dimension respectively, enhancing the network's feature response capability to key tissues and lesion boundaries, and solving the problem that existing standard dot product attention only focuses on spatial dimension correlation and ignores channel semantic information.

[0046] 3. This invention uses a receptive field modulation module to group features along the channel dimension and apply depthwise separable convolutions with different kernel sizes to each group. This enables parallel capture of multi-scale contextual information, eliminates scale misalignment between the global receptive field of the visual encoder and the local features of the decoder, and provides accurate contextual anchors for text prior information.

[0047] 4. Each module of this invention adopts an efficient and lightweight design, introducing only a small number of parameters and achieving high inference efficiency compared to existing methods, which meets the requirements of actual clinical use; at the same time, text-assisted training effectively reduces the dependence on high-quality pixel-level labeled data, alleviating the problem of high cost of medical image labeling. Attached Figure Description

[0048] Figure 1 This is a flowchart of a text-assisted medical image segmentation method based on deep learning in this embodiment;

[0049] Figure 2 This is a flowchart of a vision-language coordinator in this embodiment;

[0050] Figure 3 This is a flowchart of a text-guided decoder in this embodiment. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Example: A deep learning-based text-assisted medical image segmentation method, comprising three parts: construction of a multimodal medical image segmentation neural network, end-to-end text-assisted network training, and text-assisted medical image segmentation. The multimodal medical image segmentation neural network is a fusion improvement based on Transformer and CNN, consisting of a BERT network as a text encoder and a ConvNeXtV2 network as a visual encoder. A visual-language coordinator sequentially aligns the hidden layers of the text encoder with the convolutional layers of the visual encoder from deep to shallow layers. A text-guided decoder, incorporating channel attention modulation, spatial attention modulation, and receptive field modulation, generates segmentation prediction feature maps. The end-to-end training process is completed iteratively through data augmentation, forward propagation, error calculation, and backpropagation, during which the text encoder is frozen to maintain its pre-trained prior knowledge. In the inference phase, the image to be segmented and its diagnostic report text are input into the trained model, and after activation processing and pixel-level class decision, the final segmentation mask map is output.

[0053] like Figures 1-3 As shown, the method includes the following steps:

[0054] Step S1: Construct a multimodal medical image segmentation neural network. The network includes a visual encoder, a text encoder, and a cascaded text-guided decoder. The hidden layers of the text encoder and the convolutional layers of the visual encoder are aligned sequentially from deep to shallow through a visual-language coordinator to obtain aligned visual feature maps and text features.

[0055] This step aims to establish a multimodal segmentation network architecture capable of simultaneously processing medical images and corresponding text descriptions. By using a vision-language coordinator, cross-modal feature alignment between the visual encoder and the text encoder at a deep abstraction level is achieved, providing an aligned multimodal feature foundation for text-guided fusion in the subsequent decoding process.

[0056] Specifically, the multimodal medical image segmentation neural network is a multimodal image segmentation network that integrates and improves upon Transformer and CNN. Its network structure consists of a Bert network as a text encoder and a ConvNeXtV2 network as a visual encoder.

[0057] The total number of layers in the Bert network is: , its first Output features of hidden layers for:

[0058]

[0059] In the formula, For layer index, ; Indicates the first Classification (CLS) lexical features output from the hidden layer; Indicates the first The output of the hidden layer, excluding the classification terms. Text lexical features, The length of the text sequence.

[0060] The total number of layers in the ConvNeXtV2 network is: , its first The visual feature map output by the layer is , , For layer index, , , and They represent the first Height, width, and number of channels of the layer visual feature map; It is the set of real numbers.

[0061] The construction of a multimodal medical image segmentation neural network includes:

[0062] Step S11: Align the features of each hidden layer of the BERT network with each convolutional layer of the ConvNeXtV2 network, sequentially from deep to shallow layers, using a vision-language coordinator. In this embodiment, all 12 hidden layers of the BERT network are aligned with the last 12 convolutional layers of the ConvNeXtV2 network.

[0063] Furthermore, such as Figure 2 As shown, the visual-language coordinator performs feature alignment through the following steps:

[0064] Step S111: Extract the global descriptor of the visual features of the current layer. Global descriptor of text features For visual features, global average pooling is used to obtain global descriptors for the visual features. For text features, the corresponding categorical lexical features of this layer are directly extracted to obtain the global descriptor of the text features. .

[0065] Step S112, global descriptor ( For modal indexing, , Representing visual modality, The text modalities are projected onto a unified latent manifold for cross-modal interaction, and then mapped back to their respective original spaces to obtain aligned semantic representations. The calculation formula is as follows:

[0066]

[0067] In the formula, This represents a hierarchy- and mode-specific dimensionality-reduced projection matrix; This represents the hierarchical and modality-specific upscaling projection matrix; Indicates the activation function; For hierarchical-specific cross-modal manifold shared interaction matrices, , The dimension of the manifold is set to half the dimension of the visual feature channels in this embodiment. .

[0068] Step S113, align the semantic representation The features are then inverted back into the corresponding encoder features to update the visual feature map and text features, respectively. The calculation formula is as follows:

[0069]

[0070]

[0071] In the formula, This represents the updated visual feature map; Represents the original visual feature map; This indicates a broadcast copy operation along a spatial dimension; This represents the complete text feature sequence after text feature replacement and updating; For the semantic representation of aligned visual features; For the semantics of the aligned text; This represents the sequence of text lexical features in the current layer, excluding the categorical lexical features. The length of the text sequence.

[0072] Step S2: Construct a text-guided decoder. Each decoding stage of the text-guided decoder sequentially performs channel attention modulation, spatial attention modulation, and receptive field modulation to generate a segmentation prediction feature map with the same resolution as the input image.

[0073] Step S2 aims to gradually restore the multi-scale features extracted by the encoder to the original input image resolution by using a cascaded attention mechanism that integrates textual information and a receptive field alignment strategy. During the decoding process, prior textual knowledge is used to enhance the channel-level and spatial-level responses of visual features to key tissues and lesion boundaries. Furthermore, a multi-branch convolutional structure is used to alleviate the scale misalignment between the encoder and decoder.

[0074] The text-guided decoder comprises multiple cascaded decoding stages. In this embodiment, such as... Figure 3As shown, the text-guided decoder consists of three cascaded decoding stages, each of which includes channel attention modulation, spatial attention modulation, and receptive field modulation. Let... For the decoding stage index, , No. The input features of the layer decoder are the decoding features output by the previous stage decoder. and the text features ultimately output by the BERT network. ,for Its input features Directly use the feature map output from the deepest layer of the visual encoder .

[0075] The steps for building a text-guided decoder are as follows:

[0076] Step S21: Construct a semantic perception channel attention modulation module. The function of this module is to generate a saliency map of the channel dimension by fusing global statistical information of visual features with classification semantic information output by the text encoder, enhance the feature channels that contribute more, and suppress irrelevant channels, thereby improving the network's ability to distinguish key tissues and lesions.

[0077] Specifically, regarding the first Layer input features Global average pooling and global max pooling are performed separately to obtain the visual descriptor. and Simultaneously, extract text features. The categorical lexical features are then projected onto the visual channel dimension via a multilayer perceptron to obtain the text descriptor. Calculate the channel saliency plot. The calculation formula is as follows:

[0078]

[0079] In the formula, The kernel size is One-dimensional convolution operation, , Based on input features Number of feature channels Sure, This indicates taking the nearest odd number; This is the Sigmoid activation function.

[0080] Next, channel attention is applied element-wise multiplication along the visual feature channel dimension to obtain the channel attention-modulated features. :

[0081]

[0082] In the formula, This is for element-wise multiplication.

[0083] Step S22: Construct a semantic perception spatial attention modulation module. The function of this module is to generate a spatial saliency map by calculating the affinity between each word in the visual features and text features, so that the network can accurately locate the target region in the image space based on the position and shape information in the text description.

[0084] Specifically, calculate the features after channel attention modulation. Text features The affinity between them, and through the text length Aggregate dimensions to generate spatial saliency maps The calculation formula is:

[0085]

[0086] In the formula, and The projection matrix is ​​learnable; Features The number of feature channels; This is the matrix transpose. and Representing the length along the text Perform average pooling and max pooling operations on the dimension; This represents the normalized exponential function.

[0087] Next, spatial attention is applied using element-wise multiplication in the visual feature space dimension to obtain the features modulated by spatial attention. :

[0088]

[0089] In the formula, This is for element-wise multiplication.

[0090] Step S23: Construct a receptive field alignment module. The function of this module is to group feature maps along the channel dimension and apply convolution operations of different receptive fields to them respectively, so that the network can capture contextual information at different scales at the same time, eliminate the scale difference between the global receptive field of the visual encoder and the local features of the decoder, and provide accurate contextual anchors for text prior information.

[0091] Specifically, the features modulated by spatial attention The feature maps are divided into four groups along the channel dimension. Different convolution operations are applied to each of the four groups of feature maps. The four results are then concatenated along the channel dimension and convolved. Finally, the convolution is performed with the spatially attention-modulated feature maps. The expression for addition is:

[0092]

[0093] In the formula, Indicates the kernel size as Convolution; The preset bar convolution kernel size parameters; Indicates the kernel size as Depth-separable convolution, i.e., feature submap Apply convolution kernel size is Depth-separable convolution; Indicates the kernel size as The horizontal bar depth can be separable convolution, i.e., feature submap. Apply convolution kernel size is The horizontal bar depth can be separable convolution; Indicates the kernel size as The vertical bar depth can be used to separate the convolution, i.e., the feature submap. Apply convolution kernel size is Vertical bar depthwise separable convolution; feature submap Preserve the identity mapping; This guides the output features of the text-guided decoder. The four branches are concatenated and then sequentially processed... Convolution, normalization, GELU activation and Convolution completes channel mapping, which is then combined with the features modulated by spatial attention. The residuals are summed. In this embodiment, the strip convolution kernel size parameters are added in descending order of the decoding stage. They were set to 7, 11, and 21 in sequence.

[0094] Step S24: Construct the decoder output module.

[0095] Specifically, the visual encoder (ConvNeXtV2) comprises four feature extraction stages, and the output of each stage is denoted as... , , , (From shallow to deep). For the decoding stage index , Take the first Output of each feature extraction stage As a skip connection feature, the skip connection feature is combined with the upsampled output feature of the current-level text-guided decoder. To integrate, through The convolution is used as the output feature of this level of decoder. The calculation formula is as follows:

[0096]

[0097] In the formula, This indicates a doubling of the upsampling value; express Convolution; For the first The skip connection characteristics of the stage output.

[0098] Features output by the deepest encoder stage It does not participate in skip connection fusion, but directly serves as a third-level decoder. ) input The shallowest layer ( Decoder output After upsampling and linear mapping, a segmentation prediction feature map with the same resolution as the input image is obtained. , , , and These represent the image's height, width, and number of categories, respectively.

[0099] Step S3: Augment the training samples, input the augmented images and corresponding text descriptions into the multimodal medical image segmentation neural network for forward propagation, freeze the text encoder, update the parameters of the visual encoder and text-guided decoder in reverse according to the segmentation error, iterate to a preset number of times to obtain and save the prediction model.

[0100] Step S3 aims to enable the network to learn the mapping relationship from multimodal inputs of images and text to accurate segmentation masks through an end-to-end training strategy, while freezing the text encoder to preserve its pre-trained prior knowledge and reduce the model's dependence on high-quality labeled data.

[0101] Specifically, the end-to-end text-assisted network training method includes the following steps:

[0102] Step S31, randomly read from the dataset Group training samples :

[0103]

[0104] In the formula, Represents medical images; This represents the corresponding text description; A segmentation mask representing a medical image; Indicates the number of sample groups; This represents the sample index within the training batch.

[0105] Step S32: Simultaneously apply random rotation, random cropping, and scaling operations to the training image and its corresponding segmentation mask. Randomly apply Gaussian noise perturbation and Gibbs ringing artifact simulation to the training image to obtain the augmented training samples. .

[0106] The execution probability (first predetermined probability) of the Gaussian noise perturbation method is set to 0.2, and the standard deviation of the generated independent and identically distributed Gaussian noise matrix is... Set to 0.1; the execution probability (second preset probability) of the Gibbs ringing artifact simulation method is set to 0.2, and the cutoff strength parameter is... The sampling interval is set to [0.8, 1.0].

[0107] Step S33, augment the training samples A batch of these images is fed into a multimodal medical image segmentation neural network for forward propagation, resulting in a single-channel segmentation prediction feature map. When calculating the training loss, the segmentation prediction feature map will be used. Directly input step S34; when calculating the training monitoring index, analyze the segmentation prediction feature map. Perform sigmoid activation to obtain a pixel-by-pixel object category probability map.

[0108] Step S34, segmentation prediction feature map The segmentation ground truth mask is input into the Dice loss function, which internally performs segmentation prediction on the feature map. Perform sigmoid activation and calculate the overlap error between the predicted and labeled regions on a batch basis. Freeze the BERT text encoder backbone network, perform backpropagation based on the error, and update the parameters of the visual encoder, multimodal coordinator, and text-guided decoder.

[0109] Step S35: Repeat steps S31 to S34 until the iteration is complete. The calculation formula is:

[0110]

[0111] In the formula, This represents the number of samples in the dataset. For pre-set rounds; This represents the total number of iterations during the training process. In actual calculations, the rounding is rounded up. During iteration, the optimal model parameters are recorded using the validation set. After training, the optimal model is obtained, which is the prediction model used for prediction.

[0112] Step S4: Input the medical image to be segmented and its corresponding diagnostic report text into the prediction model, perform activation processing and pixel-level category decision on the output segmentation prediction feature map, and output the final segmentation mask map.

[0113] Step S4 aims to apply the trained model to real-world clinical scenarios, using the image to be segmented and the corresponding diagnostic report text to jointly infer accurate segmentation results, thereby achieving text-assisted automated medical image segmentation.

[0114] Specifically, text-assisted medical image segmentation includes the following steps:

[0115] Step S41: Read the network weight file and configuration file that have been trained end-to-end, and initialize the multimodal medical image segmentation neural network.

[0116] Step S42: Read in the unlabeled medical image to be diagnosed and segmented in the current clinical environment, as well as the corresponding standard electronic medical record or diagnostic report text description.

[0117] Step S43: Simultaneously input the image and text description into the multimodal medical image segmentation neural network for forward propagation to obtain the segmentation prediction feature map output by the model. For segmentation prediction feature maps Activate the image using either the Sigmoid function (for single-class tasks) or the Softmax function (for multi-class tasks) to obtain the class-specific prediction probability for each pixel. .

[0118] Step S44, predict the probability The final mask image is calculated based on the task type.

[0119] For single-class tasks, a threshold method is used to determine the segmentation mask based on the predicted probabilities. The calculation formula is as follows:

[0120]

[0121] In the formula, This indicates the final segmentation mask at the pixel level. The value can be 1 to represent the target category and 0 to represent the background. This represents the pre-set segmentation decision threshold, which is usually 0.5.

[0122] If it is a multi-class task, then probability prediction is needed. Each pixel is assigned the category with the highest probability, calculated using the following formula:

[0123]

[0124] In the formula, Represents pixels In the Predicted probability values ​​for each target category; The target category channel index to be segmented; This represents the category index function for finding the independent variable that maximizes the objective function.

[0125] The invention will be further explained below in conjunction with specific application scenarios of chest X-ray infection area segmentation.

[0126] This embodiment applies the present invention to lung infection region segmentation in chest X-ray images. The training samples consist of a chest X-ray image, a pixel-level infection region mask with the same name as the image, and short English text describing the side of infection, the number of infection regions, and the location of the lung field. The network outputs a single-channel infection region segmentation feature map.

[0127] In a reproducible data configuration, the QaTa dataset is used. The dataset is fixedly divided into a training set of 6430 cases, a validation set of 715 cases, and a test set of 2113 cases. Each sample includes a chest X-ray image, an infection area mask, and a corresponding text description. An alternative data configuration is MosMedData+, whose partitioning file contains 2183 cases for the training set, 273 cases for the validation set, and 273 cases for the test set. These partitions remain unchanged during training, model selection, and independent testing to avoid reusing the same sample across sets.

[0128] A typical input text could be "Bilateral pulmonary infection, two infected areas, middle left lung and lower right lung.", which indicates bilateral pulmonary infection and two infected areas, located in the middle of the left lung and the lower part of the right lung, respectively. Batch encoding is performed using the bert-base-uncased tokenizer, with padding and truncation implemented. The maximum text sequence length is set to 512.

[0129] The training image and mask are read by PIL and converted to a channel-first format. If the chest X-ray image is single-channel, it is copied along the channel dimension to a three-channel image. During training, Gaussian noise with a standard deviation of 0.1 is added to the image with a probability of 0.2, Gibbs ringing artifacts with a truncation intensity parameter in the range [0.8, 1.0] are applied with a probability of 0.2, random rotation of 0.3 radians is performed synchronously on the image and mask with a probability of 0.5, and random scaling with a scaling factor in the range [0.8, 1.2] is performed with a probability of 0.5. Subsequently, a 384×384 region is randomly cropped and scaled to 224×224, and the image is normalized channel by channel. Bicubic interpolation is used for the image, and nearest neighbor interpolation is used for the mask to prevent the mask class value from being changed by interpolation. During the validation and testing phases, no random noise, artifacts, rotation, or random cropping is applied; the image and mask are simply scaled to 224×224, and the image is normalized channel by channel.

[0130] The visual encoder uses the convnextv2-tiny-22k-224 pre-trained model, and the text encoder uses the bert-base-uncased pre-trained model. The ConvNeXtV2 visual encoder includes four feature extraction stages, with default output channel numbers of 96, 192, 384, and 768 respectively. The BERT text encoder includes 12 Transformer hidden layers, with a text hidden dimension of 768. Twelve vision-language coordinators are set up, corresponding one-to-one between the 7th to 18th convolutional blocks of the visual encoder and the 1st to 12th hidden layers of the text encoder. Each coordinator calculates the average spatial dimension of the current visual features to obtain a visual global descriptor and extracts the classification terms from the current text hidden layer to obtain a text global descriptor; both are then subjected to modality-specific dimensionality reduction projection, shared linear transformation, and modality-specific dimensionality increase projection, respectively. The latent manifold dimension is half the number of channels of the current visual features. The visual output is broadcast and added to the original visual features, while the text output replaces the classification terms in the current text sequence. The parameters of the shared linear transformation are used by both the visual and textual paths and are updated together during training to constrain the two modalities to form consistent semantics within the same latent representation space.

[0131] All pre-trained backbone parameters of the text encoder are frozen; the normalized parameters of the final layer of the visual encoder are frozen; the remaining trainable parameters of the visual encoder, the multimodal coordinator parameters, and the text-guided decoder parameters participate in backpropagation. This setup preserves BERT's pre-trained language priors while allowing visual features and cross-modal mappings to be adapted to the chest X-ray infection region segmentation task.

[0132] The text-guided decoder consists of three decoding stages. The deepest visual features enter the first decoding stage first. Each stage sequentially performs block embedding and positional encoding, channel attention modulation, spatial attention modulation, and receptive field modulation, and then fuses them with the skip connection features from the corresponding visual encoding stage through a 2x upsampling. The strip convolution kernel sizes R for the three stages, from deep to shallow, are set to 7, 11, and 21, respectively; the corresponding channel expansion ratios are set to 3, 4, and 4, respectively. Finally, subpixel upsampling is used to restore the features to 224×224, and a 1×224×224 single-channel model output is obtained through output convolution.

[0133] Training employed the AdamW optimizer and hierarchical learning rates. A base learning rate of 6 × 10⁻⁶ was used. -4 For example, the learning rate of the non-coordinator parameters of the visual encoder is 6×10. -5 The learning rate for the coordinator and decoder parameters is 6×10. -4 The weight decay coefficient is 1×10 -2 Training is conducted for 100 epochs, with a total batch size of 64 and a random seed of 3396. If the Dice metric on the validation set fails to improve for three consecutive epochs, the learning rate is halved, with a minimum learning rate of 1×10⁻⁶. -6 Early stopping is triggered if there is no improvement after 30 consecutive rounds. The loss function used is Dice loss. Each training batch completes forward propagation, loss calculation, gradient backpropagation, and AdamW parameter update. After each training round, Dice, pixel accuracy, and intersection-union ratio are calculated on the validation set, and Dice is used as the model selection metric; when the validation set Dice improves, the current round, model parameters, and best metric are saved.

[0134] During testing, the optimal model weights from the validation set are loaded. The same deterministic preprocessing as in the validation phase is performed on the test images and corresponding text. The network output is converted into pixel-wise infection probabilities using a Sigmoid function. A binary infection region mask is obtained with a threshold of 0.5. The testing program calculates Dice, pixel accuracy, intersection-over-union ratio, recall, and precision.

[0135] By following the above configuration and the module connection relationships shown in the attached diagram, text-assisted automatic segmentation of the infected area on chest X-ray can be implemented.

[0136] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A text-assisted medical image segmentation method based on deep learning, characterized in that, Includes the following steps: Step S1: Construct a multimodal medical image segmentation neural network. The network includes a visual encoder, a text encoder, and a cascaded text-guided decoder. The hidden layers of the text encoder and the convolutional layers of the visual encoder are aligned sequentially from deep to shallow through a visual-language coordinator to obtain aligned visual feature maps and text features. In step S2, the text-guided decoder sequentially performs channel attention modulation, spatial attention modulation, and receptive field modulation in each decoding stage to generate a segmentation prediction feature map with the same resolution as the input image. Step S3: Augment the training samples, input the augmented images and corresponding text descriptions into the multimodal medical image segmentation neural network for forward propagation, freeze the text encoder, and update the parameters of the visual encoder and text-guided decoder in reverse according to the segmentation error to obtain the prediction model. Step S4: Input the medical image to be segmented and its corresponding diagnostic report text into the prediction model, perform activation processing and pixel-level category decision on the output segmentation prediction feature map, and output the final segmentation mask map. In step S2, when the text-guided decoder performs channel attention modulation during the decoding phase, it adjusts the channel attention of the first channel. Layer input features Global average pooling and global max pooling are performed separately to obtain the visual descriptor. and ; The categorical lexical features output by the text encoder are projected onto the visual channel dimension via a multilayer perceptron to obtain the text descriptor. And calculate the channel saliency map. : In the formula, The kernel size is One-dimensional convolution operation; Use the Sigmoid activation function; Channel saliency map and input features Element-wise multiplication yields the features modulated by channel attention. : In the formula, For element-wise multiplication; For the decoding stage index, ; In step S2, when the text-guided decoder performs spatial attention modulation during the decoding phase, it calculates the features after channel attention modulation. Text features output by the text encoder The affinity between them, and through the text length Aggregate dimensions to generate spatial saliency maps : In the formula, and The projection matrix is ​​learnable; Features The number of feature channels; This is the matrix transpose. and Representing the length along the text Perform average pooling and max pooling operations on the dimension; This represents the normalized exponential function; Spatial saliency map Features after channel attention modulation Element-wise multiplication yields the features modulated by spatial attention. : In the formula, For element-wise multiplication; In step S2, when the text-guided decoder performs receptive field modulation during the decoding phase, it modulates the features after spatial attention modulation. The feature maps are divided into four groups along the channel dimension. Different convolution operations are applied to each of the four groups of feature maps. The four results are then concatenated along the channel dimension and convolved. Finally, the convolution is performed with the spatially attention-modulated feature maps. The expression for addition is: In the formula, Indicates the kernel size as Convolution; The preset bar convolution kernel size parameters; Indicates the kernel size as Depth-separable convolution; Indicates the kernel size as The horizontal bar depth can be separable convolution; Indicates the kernel size as Vertical stripe depth can be separable convolution; , , and For feature submaps, feature submaps Preserve the identity mapping; This provides the output features for the text-guided decoder.

2. The text-assisted medical image segmentation method based on deep learning according to claim 1, characterized in that, In step S1, the vision-language coordinator performs feature alignment, including: Extract the global descriptor of visual features of the current layer. and text feature global descriptor ; Global descriptor of visual features and text feature global descriptor Projecting onto a latent manifold space of the same dimension to perform cross-modal interactions, and mapping back to their respective original spaces, yields aligned semantic representations: In the formula, This represents a hierarchy- and mode-specific dimensionality-reduced projection matrix; This represents the hierarchical and modality-specific upscaling projection matrix; Indicates the activation function; For hierarchical-specific cross-modal manifold shared interaction matrices, , For manifold dimension, It is the set of real numbers; For the aligned semantic representation; Represents the global descriptor. For modal indexing, , Representing visual modality, Represents the text modality.

3. The deep learning-based text-assisted medical image segmentation method according to claim 2, characterized in that, In step S1, the aligned semantics are inversely injected back into the corresponding encoder features to update the visual feature map and text features: In the formula, This represents the updated visual feature map; Represents the original visual feature map; This indicates a broadcast copy operation along a spatial dimension; This represents the complete text feature sequence after text feature replacement and updating; For the semantic representation of aligned visual features; For the semantics of the aligned text; This represents the sequence of text lexical features in the current layer, excluding the categorical lexical features. The length of the text sequence.

4. The text-assisted medical image segmentation method based on deep learning according to claim 1, characterized in that, Preset bar convolution kernel size parameters The value is the feature after spatial attention modulation of distance. The nearest odd number that is half the size of the spatial dimension.

5. The text-assisted medical image segmentation method based on deep learning according to claim 1, characterized in that, The features output from each decoding stage of the text-guided decoder are upsampled and then fused with the skip connection features output from the corresponding feature extraction stage of the visual encoder. The fused features are then used as the input for the next decoding stage.

6. The text-assisted medical image segmentation method based on deep learning according to claim 1, characterized in that, In step S3, training sample augmentation includes: simultaneously applying random rotation, random cropping, and scaling operations to the training image and its corresponding segmentation mask; and randomly applying Gaussian noise perturbation and Gibbs ringing artifact simulation to the training image.

7. The deep learning-based text-assisted medical image segmentation method according to claim 1, characterized in that, In step S4, for single-class tasks, the activation process uses the Sigmoid activation function and compares the probability value of each pixel position with a preset threshold to determine the pixel-level class. For multi-class tasks, the activation process uses the Softmax function and selects the class with the highest probability value at each pixel position as the pixel class.

Images