Medical image segmentation method fusing multi-modal graphic and text information

Abstract

The invention discloses a medical image segmentation method fusing multi-modal graphic and text information, and the method comprises the steps: carrying out the preprocessing and coding of a metadata text of a patient through a pre-trained BERT model, extracting the feature representation, and carrying out the attention-guided data enhancement operation of a pathological image. And finally, constructing a medical image segmentation network composed of an image encoder, a text encoder and a decoder, carrying out pathological image segmentation based on the feature representation extracted in the step 1 and the pathological image after data enhancement, and visualizing a segmentation result. In the segmentation network, a multi-modal feature fusion module is designed to fuse information of two modals, the module adopts a Transform architecture, a cross-modal attention mechanism and a self-attention mechanism are introduced, and the effectiveness of information fusion is ensured. According to the method, a more accurate segmentation result is obtained, the generalization ability on new data is improved, and the common problem of data imbalance in the medical image is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image segmentation, and in particular relates to a medical image segmentation method integrating multi-modal graphic and text information. Background Art

[0002] With the continuous advancement of multimodal medical imaging technology, research on multimodal medical image segmentation methods has also received widespread attention. Multimodal medical image segmentation refers to the fusion of multimodal image information to improve segmentation performance. Compared with single-modal medical images, multimodal medical images can provide doctors with rich complementary information. Studies have shown that the patient's basic information metadata text can also assist in the segmentation of medical images to a certain extent. Therefore, the application of multimodal medical images for segmentation can greatly reduce the uncertainty of information and improve the accuracy of clinical diagnosis and segmentation.

[0003] With the development of deep learning, the international academic community has generally adopted methods such as convolutional neural networks (CNNs) and achieved remarkable results in medical image segmentation tasks. However, commonly used convolutional neural networks lack the ability to capture global context. In current medical image segmentation research, more and more attention is focused on methods that integrate multimodal information. Recently, as a series of studies in the field of natural image segmentation have shown that information in the text modality can be used to improve the segmentation ability of the model, inspired by this, some studies have begun to use patient metadata text information to assist in medical image segmentation. The field of medical image segmentation is gradually moving towards the research direction of deep learning and multimodal integration to analyze medical image information more comprehensively and accurately.

[0004] Although multimodal medical image segmentation methods based on deep learning have made significant progress, there are still some key issues that need to be addressed. First, although the research on multimodal medical image segmentation methods based on text and image fusion has made some progress, the generalization ability of these methods is still insufficient and cannot cover all lesion areas to be segmented in pathological images. In practical applications, the segmentation effect of existing multimodal fusion methods has not reached the ideal level and there is still significant room for improvement. In addition, since data of different modalities have different characteristics and properties, how to effectively utilize and fuse multimodal information to achieve accurate segmentation remains a challenge.

[0005] In summary, how to design a medical image segmentation method that integrates multimodal image and text information to improve the segmentation effect of the segmentation network on different data sets. At the same time, effectively integrating pathological images and patient metadata text information to extract more comprehensive and rich features, thereby assisting pathologists in providing more comprehensive and accurate tumor diagnosis results, is an urgent problem to be solved in the field of medical image segmentation. Summary of the invention

[0006] The present invention proposes a medical image segmentation method that integrates multimodal image and text information. Through deep learning, the present invention uses a backbone network to extract pathological image features and corresponding patient metadata text features. On the basis of the initial medical image segmentation network, a multimodal feature fusion module (MFFM) is introduced to replace the channel stacking fusion mechanism in the traditional method. The MFFM module innovatively uses the Transformer architecture to process multimodal data. By introducing a cross-modal attention mechanism, useful feature information is captured and fused in multimodal data. With the help of this module, the network can better focus on specific lesion areas and understand the content of pathological images, thereby improving the accuracy of segmentation results.

[0007] In order to solve the above technical problems, the present invention provides a medical image segmentation method integrating multimodal image and text information. The main steps of the method are as follows:

[0008] Step 1: Preprocess and encode the patient metadata text. Use the pre-trained BERT model to preprocess and encode the patient metadata text and extract the text feature vector for subsequent segmentation tasks.

[0009] Step 2: Perform attention-guided data augmentation operations on pathological images.

[0010] Step three, construct a medical image segmentation network consisting of an image encoder, a text encoder and a decoder, perform pathological image segmentation based on the text feature representation and data enhanced pathological images, and obtain the segmentation results.

[0011] Preferably, the step 2, performing an attention-guiding data enhancement operation on the pathological image, comprises the following steps:

[0012] Attention-guided data augmentation techniques aim to guide the model to focus on the key parts of pathological images and generate more representative training samples by introducing an attention mechanism.

[0013] S2.1. Generate an attention map from the input pathological image data by using the Transformer model with the attention mechanism. Then, calculate the attention score of each attention map. The commonly used calculation method is to normalize the attention map and then calculate the weight sum of each pixel.

[0014] S2.2. Select the three images with the highest attention scores from the generated attention maps and perform attention-guided cropping, CutMix, and Mixup enhancements respectively.

[0015] Preferably, the specific implementation process of the medical image segmentation network in step 3 is as follows:

[0016] S3.1. The medical image segmentation network model consists of three parts: image encoder, text encoder and decoder.

[0017] S3.2, the input of the image encoder is the pathological image, and the encoder is used to extract the characteristic information in the pathological image. The image encoder includes a total of 5 cascaded image encoding modules I i , where i = 1, 2, 3, 4, 5. Each image encoding module consists of two consecutive convolutional layers, a batch normalization operation, a ReLU activation function and downsampling operations.

[0018] S3.3, perform multimodal feature fusion. The network constructs a special multimodal feature fusion module (Multimodel Feature Fusion Module, MFFM) to fuse the feature information of image and text modalities, and uses the output features of image encoding modules I2, I3, and I4 as the three inputs of the MFFM module, and performs layer normalization on them.

[0019] S3.4, after layer normalization operation, three outputs are obtained, denoted as L1, L2, and L3. L1, L2, and L3 are concatenated to obtain a new vector L ∑ Since the multi-head cross attention mechanism in Transformer is used in the MFFM module, for subsequent calculations, the query vector Q is obtained by the following formula j , key vector K and value vector V (where j = 1, 2, 3):

[0020]

[0021] in, represents the query weight matrix, W K represents the key weight matrix, W V Represents the value weight matrix, which is gradually optimized through model training.

[0022] S3.5. In order to enhance the feature representation capability and better handle multimodal tasks, the model introduces a multi-head cross-attention module (MCA) after the middle-layer standardization of the MFFM module. Multiple attention heads are used to establish connections between different modalities and fuse features from different network branches. The MCA module contains 5 inputs, namely Q1, Q2, Q3, K and V. For each attention head, its attention weight CA can be obtained by the following formula j (where j = 1, 2, 3):

[0023]

[0024] Among them, σ(·) represents the softmax activation function, ψ(·) represents the instance normalization process, and d k Indicates the dimension of the key.

[0025] S3.6. After calculating the attention of each attention head, the multi-head attention MCA is obtained by integrating the following formula j Here, head is set to h, j = 1, 2, 3:

[0026]

[0027] Among them, CA j h is the weight of the h-th attention head;

[0028] S3.7, get the output of MFFM. Finally, by combining the MLP layer and residual operation, the output M of the multimodal feature fusion module is obtained. j (where j = 1, 2, 3), the calculation formula is as follows:

[0029] M j =MCA j +MLP(Q j +MCA j )

[0030] S3.8. The input of the text encoder is the patient's metadata, which belongs to text modal information. The text encoder contains a text preprocessing block TPB and three text embedding blocks TEB. TPB encodes the patient's text metadata into a network feature vector and serves as the input of the next TEB1 module. TEB1 performs feature splicing and fusion on the text feature vector output by the TPB module and the image feature vector M1 of the corresponding scale. The output of the TEB1 module after the fusion operation and the image feature vector M2 serve as the input of the TEB2 module. The output of the TEB2 module and the image feature vector M3 serve as the input of the TEB3 module. After TEB3 completes the feature splicing and fusion operation, the output is fused with the bottom-level image encoding module I5 to the image-text fusion feature F. fusion , thereby more comprehensively extracting information from pathological images and patient metadata and improving the effect of feature fusion.

[0031] S3.9. Train the model. Set the training loss function to a custom combined loss function MixDiceBCE, which combines weighted binary cross entropy loss (BCE) and weighted Dice loss to optimize the binary segmentation task.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] First, textual data such as medical records or patient metadata are integrated into the medical image segmentation network to further explore the potential correlations and patterns between the data. The auxiliary network learns more comprehensive feature information under multiple modalities, improves the performance of the network, and obtains more accurate segmentation results.

[0034] Secondly, a multimodal feature fusion module (MFFM) is designed to more effectively extract and utilize text metadata and image information at different scales. By using the Transformer module instead of the traditional channel stacking fusion method, the effective feature information is more prominent, thereby further improving the fusion effect.

[0035] In addition, an attention-guided data augmentation method is used to guide the model to focus on key areas in the image, generate more representative training samples, increase data diversity, reduce the risk of model overfitting, and improve the model's generalization ability on new data.

[0036] Finally, a custom combined loss function MixDiceBCE was used. By combining weighted binary cross entropy loss and weighted Dice loss, global and local information are taken into account, the model's learning and generalization capabilities for different features are enhanced, and the common data imbalance problem in medical images is effectively handled. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution in the present invention, a brief description of the technology in the present invention is given below using the accompanying drawings, which will give people in the field a clearer understanding of the performance of the technology.

[0038] Figure 1 It is a schematic diagram of the model structure in the present invention. DETAILED DESCRIPTION

[0039] The present invention proposes a medical image segmentation method that integrates multimodal graphic information. The core idea of ​​the method is to introduce a multimodal feature fusion module (MFFM) to replace the channel stacking fusion mechanism in the traditional method. The MFFM module innovatively uses the Transformer architecture to process multimodal data. By introducing a cross-modal attention mechanism, useful feature information is captured and fused in multimodal data, thereby helping the network to better understand the content of medical images, guiding the segmentation network to focus on specific areas, and improving the accuracy of the segmentation results of the lesion area, thereby reducing the workload of medical staff and promoting scientific research progress in the field of medical image segmentation. In order to further explain the present invention in detail, the following is combined with specific examples and reference to the accompanying drawings.

[0040] refer to Figure 1 , a medical image segmentation method integrating multimodal image and text information, the main steps of the method are as follows:

[0041] Step 1: Preprocess and encode the patient metadata text. Use the pre-trained BERT model to preprocess and encode the patient metadata text, and extract the text feature vector for subsequent segmentation tasks. Among them, patient metadata usually includes the patient's personal information and medical records, including basic information such as gender, age, or degree of differentiation of tumor cells. For example, when performing gland segmentation experiments on the GlaS dataset, the metadata text used in the present invention is the benign, malignant, and differentiation degree of the gland to be segmented.

[0042] Step 2: Perform attention-guided data enhancement operations on the pathological images. Different from traditional data enhancement methods such as rotation, translation, and scaling, the present invention innovatively adopts an attention-guided data enhancement method. First, the attention map of the pathological image is generated through weakly supervised learning; secondly, the three images with the highest attention scores are selected from the generated attention map, and attention-guided cropping, CutMix, and Mixup enhancements are performed respectively, thereby increasing the diversity of the data and the generalization ability of the model.

[0043] Step three, construct a medical image segmentation network consisting of an image encoder, a text encoder and a decoder, perform pathological image segmentation based on the text feature representation and data enhanced pathological images, and obtain the segmentation results.

[0044] The present invention designs a multimodal feature fusion module (Multimodel Feature Fusion Module, MFFM) to process image and text information. This module innovatively adopts the Transformer architecture and realizes the effective fusion of different modal information by introducing the cross-modal attention mechanism. Specifically, the cross-modal attention mechanism first extracts pathological image and text features, uses the self-attention mechanism to enhance the internal feature representation of each modality, and then calculates the attention weights between different modalities to capture and fuse the correlation between them. The process includes performing cross-modal attention calculations on image features and text features as queries, keys, and values, respectively, and performing weighted summation on the features based on the calculated attention weights, and finally obtaining the fused feature representation.

[0045] Preferably, the step 1 of preprocessing and encoding the patient metadata text comprises the following steps:

[0046] S1.1. Select a pre-trained BERT model based on the characteristics of the medical image segmentation task. Since text data in the medical field often contains professional terms and specific medical expressions, it is necessary to select a BERT model that can effectively understand and process these terms. For example, the bert-base-uncased model is pre-trained on large-scale text data, has rich language representation capabilities, is suitable for processing medical text, and can effectively improve the accuracy and reliability of image segmentation.

[0047] S1.2. Load a pre-trained BERT model suitable for a specific task. Download the pre-trained BERT model from the model library and configure it to be usable so that it can receive input text and generate corresponding feature representations, preparing for subsequent text encoding and processing.

[0048] S1.3. Instantiate the Tokenizer object and convert the patient metadata text into a token sequence that can be processed by the BERT model, so that the BERT model can process and understand the input text data.

[0049] S1.4. Use the pre-trained BERT model to encode the token sequence and use the obtained text feature vector for downstream medical image segmentation tasks.

[0050] Preferably, the step 2, performing an attention-guiding data enhancement operation on the pathological image, comprises the following steps:

[0051] Attention-guided data augmentation techniques aim to guide the model to focus on the key parts of pathological images and generate more representative training samples by introducing an attention mechanism.

[0052] S2.1. Generate an attention map from the input pathological image data by using the Transformer model with the attention mechanism. Then, calculate the attention score of each attention map. The commonly used calculation method is to normalize the attention map and then calculate the weight sum of each pixel.

[0053] S2.2. Select the three images with the highest attention scores from the generated attention maps and perform attention-guided cropping, CutMix, and Mixup enhancements respectively:

[0054] Attention-guided cropping operation: the area with attention score higher than the set threshold a1 is cropped out and resized to the size of the pathological image;

[0055] CutMix operation: In the pathological image, the area with an attention score higher than the set threshold a2 is cut out and superimposed on the upper left corner of the pathological image;

[0056] Mixup operation: The area in the pathological image with an attention score higher than the set threshold a3 is intercepted and resized to the size of the pathological image, and then mixed with the pathological image in equal proportion.

[0057] Through the above three data enhancement methods, three new feature images can be generated as network input, thereby completing the expansion of the data set.

[0058] Preferably, the specific implementation process of the medical image segmentation network in step 3 is as follows:

[0059] S3.1. The medical image segmentation network model consists of three parts: image encoder, text encoder and decoder. The image encoder completes the feature extraction task at the image level through operations such as convolution and pooling. Correspondingly, the text encoder extracts text feature information from the patient metadata text corresponding to the image. Finally, the decoder generates the lesion area segmentation result of the corresponding image through upsampling and jump connection.

[0060] S3.2, the input of the image encoder is a pathological image of size 224×224, and the encoder is used to extract feature information from the pathological image. The image encoder contains a total of 5 cascaded image encoding modules I i , where i = 1, 2, 3, 4, 5. Each image encoding module includes two consecutive convolutional layers, a batch normalization operation, a ReLU activation function and downsampling operations, which compress image features and retain key feature information by gradually reducing the spatial dimensions of the feature map.

[0061] S3.3, perform multimodal feature fusion. The network fuses the feature information of image and text modalities by constructing a special multimodal feature fusion module (Multimodel Feature Fusion Module, MFFM). In order to effectively extract image feature information at different scales, the output features of image encoding modules I2, I3, and I4 are first used as three inputs of the MFFM module, and layer normalization is performed on them. Layer normalization aims to solve the problems of gradient vanishing and gradient exploding in deep neural network training, promote the stability of the training process, and accelerate the convergence speed. Unlike batch normalization, layer normalization normalizes the feature dimension of each training sample instead of on batch samples. The specific steps of layer normalization include:

[0062] Calculate the mean and variance of a given sample on the feature dimension;

[0063] Use the calculated mean and variance to normalize each feature of the sample. The specific calculation formula is as follows:

[0064]

[0065] Among them, x i represents the input feature, μ is the mean, σ 2 is the variance, and c is a small constant used to avoid the denominator being zero.

[0066] Perform linear transformation on the normalized features. The specific formula is as follows:

[0067]

[0068] Among them, γ and β are learnable parameters used to restore the sample distribution characteristics before normalization.

[0069] S3.4, after layer normalization operation, three outputs are obtained, denoted as L1, L2, and L3. L1, L2, and L3 are concatenated to obtain a new vector L ∑ Since the multi-head cross attention mechanism in Transformer is used in the MFFM module, for subsequent calculations, the query vector Q is obtained by the following formula j , key vector K and value vector V (where j = 1, 2, 3):

[0070]

[0071] in, represents the query weight matrix, W K represents the key weight matrix, W V Represents the value weight matrix, which is gradually optimized through model training.

[0072] S3.5. In order to enhance the feature representation capability and better handle multimodal tasks, the model introduces a multi-head cross-attention module (MCA) after the middle-layer standardization of the MFFM module. Multiple attention heads are used to establish connections between different modalities and fuse features from different network branches. The MCA module contains 5 inputs, namely Q1, Q2, Q3, K and V. For each attention head, its attention weight CA can be obtained by the following formula j (where j = 1, 2, 3):

[0073]

[0074] Among them, σ(·) represents the softmax activation function, ψ(·) represents the instance normalization process, and d k Indicates the dimension of the key.

[0075] S3.6. After calculating the attention of each attention head, the multi-head attention MCA is obtained by integrating the following formulaj Here, head is set to h, j = 1, 2, 3:

[0076]

[0077] Among them, CA j h is the weight of the h-th attention head;

[0078] S3.7, get the output of MFFM. Finally, by combining the MLP layer and residual operation, the output M of the multimodal feature fusion module is obtained. j (where j = 1, 2, 3), the calculation formula is as follows:

[0079] M j =MCA j +MLP(Q j +MCA j )

[0080] S3.8. The input of the text encoder is the patient's metadata, which belongs to text modal information. The text encoder contains a text preprocessing block TPB and three text embedding blocks TEB. TPB encodes the patient's text metadata into a network feature vector and serves as the input of the next TEB1 module. TEB1 performs feature splicing and fusion on the text feature vector output by the TPB module and the image feature vector M1 of the corresponding scale. The output of the TEB1 module after the fusion operation and the image feature vector M2 serve as the input of the TEB2 module. The output of the TEB2 module and the image feature vector M3 serve as the input of the TEB3 module. After TEB3 completes the feature splicing and fusion operation, the output is fused with the bottom-level image encoding module I5 to the image-text fusion feature F. fusion , thereby more comprehensively extracting information from pathological images and patient metadata and improving the effect of feature fusion.

[0081] S3.9, train the model. Set the training loss function to a custom combined loss function MixDiceBCE, which combines the weighted binary cross entropy loss (Weighted Binary Cross-Entropy, BCE) and the weighted Dice loss, to optimize the binary segmentation task. Among them, the Dice loss is mainly used to evaluate the overlap between two samples, and the binary cross entropy loss is used to evaluate the difference between the model prediction results and the actual labels. By combining these two losses, the new combined loss function MixDiceBCE can not only accurately divide the target area, but also reduce errors, thereby further improving the performance of the model. Taking a specific experiment as an example, the authenticity and effectiveness of the present invention are verified by analyzing the experimental results:

[0082] The present invention selects the GlaS dataset as the experimental dataset. The GlaS dataset is a dataset for the segmentation task of colorectal glandular tissue, consisting of 165 images, 85 training set images, 80% of which are used as training sets, 20% as validation sets, and 80 test set images. In the GlaS dataset, the patient's text metadata includes two parts: benign and malignant glands and the degree of glandular differentiation.

[0083] This paper takes the advanced medical image segmentation network ACC-UNet as the basis and adds a multimodal feature fusion module MFFM to improve the performance of the segmentation network. In the experiment, the commonly used Dice and IoU indicators were used to evaluate the performance of the model. These indicators are usually used to measure the similarity between two sets, indicating the similarity between the segmentation results predicted by the network and the results of manual annotation.

[0084] The experiment was conducted based on the open source framework PyTorch. The model was trained on an Nvidia 4090 GPU, and five cross-validations were used during the training process to ensure that the experimental results were more convincing. The Adam optimizer was used to train the model, the learning rate was set to 1e-3, the model was trained for 1000 epochs, and the batch size was set to 2. In addition, all images were resized to 224×224 before entering the network.

[0085] In order to better evaluate the model performance, the proposed method is compared with other state-of-the-art methods (TransUNet, UCTransNet and ACC-UNet). Table 1 below shows the Dice and IoU scores of all models on the GlaS dataset (the higher the score, the better the model performance):

[0086] Table 1

[0087]

[0088] The experimental results show that the performance of the method and model proposed in the present invention is better than other models and achieves better segmentation effect, which also confirms the authenticity and effectiveness of the present invention.

[0089] The above is only a basic implementation example of the present invention, and the above examples are only to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A medical image segmentation method integrating multimodal image and text information, characterized in that: The steps include: Step 1: Use the pre-trained BERT model to preprocess and encode the patient metadata text and extract the text feature vector; Step 2: Perform attention-guided data augmentation operations on pathological images; Step 3: Construct a medical image segmentation network consisting of an image encoder, a text encoder, and a decoder, and perform pathological image segmentation based on the text feature representation and data-enhanced pathological images to obtain segmentation results.

2. The medical image segmentation method integrating multimodal image and text information according to claim 1, characterized in that: The patient metadata includes the patient's personal information and medical records.

3. The medical image segmentation method integrating multimodal image and text information according to claim 2, characterized in that: The preprocessing and encoding of the patient metadata text specifically includes the following steps: S1.

1. Based on the medical image segmentation task, select the pre-trained BERT model; S1.

2. Convert patient metadata text into a token sequence that can be processed by the BERT model. S1.

3. Use the pre-trained BERT model to encode the token sequence and use the obtained text feature vector for the medical image segmentation task.

4. The medical image segmentation method integrating multimodal image and text information according to claim 3, characterized in that: The data enhancement operation of performing attention guidance on the pathological image specifically includes the following steps: S2.

1. Generate attention maps from the input pathological image data by using the Transformer model with attention mechanism, and calculate the attention score of each attention map. S2.

2. Select the three images with the highest attention scores from the generated attention maps and perform attention-guided cropping, CutMix, and Mixup enhancements respectively.

5. The medical image segmentation method for fusing multimodal image and text information according to claim 4, characterized in that: The calculation of the attention score of each attention map is specifically as follows: normalizing the attention map and then calculating the weight sum of each pixel.

6. The medical image segmentation method for fusing multimodal image and text information according to claim 5, characterized in that: The attention-guided cropping, CutMix and Mixup enhancements described in step S2.2 are as follows: Attention-guided cropping operation: the area with attention score higher than the set threshold a1 is cropped out and resized to the size of the pathological image; CutMix operation: In the pathological image, the area with an attention score higher than the set threshold a2 is cut out and superimposed on the upper left corner of the pathological image; Mixup operation: The area in the pathological image with an attention score higher than the set threshold a3 is intercepted and resized to the size of the pathological image, and then mixed with the pathological image in equal proportion.

7. The medical image segmentation method for fusing multimodal image and text information according to claim 6, characterized in that: The specific implementation process of the medical image segmentation network is as follows: S3.

1. Construct a medical image segmentation network model consisting of three parts: image encoder, text encoder and decoder. S3.2, the input of the image encoder is the pathological image, and feature information in the pathological image is extracted; The image encoder consists of five cascaded image encoding modules. i , where i = 1, 2, 3, 4, 5; each image encoding module includes two consecutive convolutional layers, a batch normalization operation, a ReLU activation function and a downsampling operation; S3.3, construct a multimodal feature fusion module MFFM to fuse the feature information of image and text modalities, use the output features of image encoding modules I2, I3, and I4 as three inputs of the MFFM module, and perform layer normalization on them; S3.4, after layer standardization operation, three outputs are obtained, denoted as L1, L2, and L3; L1, L2, and L3 are concatenated to obtain a new vector L ∑ ; The query vector Q is obtained by the following formula j , key vector K and value vector V, where j = 1, 2, 3: in, represents the query weight matrix, W K represents the key weight matrix, W V Represents the value weight matrix, which is gradually optimized through training; S3.

5. After the MFFM module layer is standardized, a multi-head cross attention module MCA is introduced. The MCA module contains 5 inputs, namely Q1, Q2, Q3, K and V. For each attention head, its attention weight CA is obtained by the following formula j , where j = 1, 2, 3: where σ(·) is the softmax activation function, ψ(·) is the instance normalization process, and d k is the dimension of the key; S3.

6. After calculating the attention of each attention head, the multi-head attention MCA is obtained by integrating the following formula j , where head is set to h, j = 1, 2, 3: Among them, CA j h is the weight of the h-th attention head; S3.7, through the MLP layer and residual operation, the output M of the multimodal feature fusion module is obtained j , where j = 1, 2, 3, the calculation formula is as follows: M j =MCA j +MLP(Q j +MCA j ) S3.

8. The input of the text encoder is the patient's metadata, which belongs to text modal information. The text encoder contains a text preprocessing block TPB and three text embedding blocks TEB. TPB encodes the patient's text metadata into a network feature vector and serves as the input of the next TEB1 module. TEB1 performs feature concatenation and fusion on the text feature vector output by the TPB module and the image feature vector M1 of the corresponding scale. The output of the TEB1 module after the fusion operation and the image feature vector M2 serve as the input of the TEB2 module, and the output of the TEB2 module and the image feature vector M3 serve as the input of the TEB3 module. After TEB3 completes the feature concatenation and fusion operation, the output is fused with the bottom-level image encoding module I5 to the image-text fusion feature F. fusion ; S3.

9. Set the training loss function to a custom combined loss function that combines weighted binary cross entropy loss BCE and weighted Dice loss.

8. The medical image segmentation method for fusing multimodal image and text information according to claim 7, characterized in that: The specific steps of layer standardization include: Calculate the mean and variance of samples in feature dimensions; Normalize each feature of the sample using the mean and variance; After normalization, the features Perform linear transformation, the specific formula is as follows: Among them, γ and β are learnable parameters.

Images