Medical image segmentation and labeling method and system based on multi-modal information fusion

Abstract

The invention relates to a medical image processing technology, and discloses a medical image segmentation and labeling method and system based on multi-modal information fusion, and the method comprises the steps: constructing a model structure of a medical image segmentation and labeling model based on multi-modal information fusion; preprocessing the data set, constructing the data set by using a random sampling method, and dividing the data set into a training set, a verification set and a test set according to a proportion; training and convergence of the model: training the model through a loss function and an optimizer until the model is converged; and S3, cutting an organ to be segmented from the original image, inputting the cut organ to the model trained in the step 3, inputting a text prompt word for semantic segmentation, and taking a mask predicted by the model as a model segmentation labeling result. According to the method, the model structure is constructed, then the medical image data set is subjected to necessary preprocessing, and then the medical image data is segmented in combination with the guiding text semantic information, so that the efficiency of segmenting and marking the medical image from 0 to 1 is effectively improved.

Description

Technical Field

[0001] The present invention relates to medical image processing technology, and in particular to a medical image segmentation and annotation method and system based on multimodal information fusion. Background Art

[0002] With the development of artificial intelligence (AI) technology, its application and significance in the field of medical imaging are becoming increasingly significant. By extracting and learning features from a large amount of case data, AI models can quickly locate lesions in tissues and organs, accurately segment lesions, and classify disease types, thereby improving doctors' diagnostic efficiency and reducing their workload.

[0003] A large amount of high-standard labeled data is a necessary condition for training an excellent AI model. The current method of segmenting and labeling three-dimensional medical images mainly relies on manual painting or outlining of the area of ​​interest layer by layer to further obtain the labeling results. This labeling method is time-consuming and labor-intensive in actual application, and the labeling is difficult and has high professional requirements for the labelers.

[0004] For example, prior art 1: CN117893738A, an image segmentation method and device based on adaptive weighted mean filtering, may have certain advantages in certain specific applications, but its adaptability and robustness are poor, and it cannot adapt to the segmentation tasks of complex scenes. Moreover, this method relies on the designer's prior knowledge and experience, and it is difficult to use large-scale data for optimization and accuracy improvement.

[0005] Prior art 2: CN118314161A, a small sample medical image segmentation method based on text semantics guidance introduces text description in interactive segmentation to realize the segmentation and annotation of small sample medical images. During the implementation process, this technology adopts a method of prompting segmentation and annotation layer by layer using two-dimensional slices, and finally combines the segmentation results of each layer as the final annotation. Although this method improves the annotation speed, for three-dimensional medical images, it ignores the spatial information of the image and may cause fault phenomena. Summary of the invention

[0006] The present invention aims at the problem that segmentation and annotation of three-dimensional medical images in the prior art is time-consuming and labor-intensive, difficult to annotate, and requires high professional skills of annotators. A medical image segmentation and annotation method and system based on multimodal information fusion are provided.

[0007] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0008] A medical image segmentation and annotation method based on multimodal information fusion, which includes:

[0009] Step 1: constructing a model structure, constructing a model structure of a medical image segmentation and annotation model with multimodal information fusion. The model structure of the segmentation and annotation model includes an image encoder, a text encoder, and a mask decoder.

[0010] Step 2, data set processing, preprocessing the data set and constructing the data set using random sampling method, dividing the data set into training set, validation set and test set according to the proportion;

[0011] Step 3: Model training and convergence: Train the model through loss function and optimizer until the model converges;

[0012] In step 4, the organ to be segmented is cropped from the original image and input into the model trained in step 3. At the same time, text prompt words are input for semantic segmentation, and the mask predicted by the model is used as the model segmentation annotation result.

[0013] As a preferred method, the image encoder uses 3D Vision Transformer as the backbone network to extract features from the input image cube and obtain feature maps of different depths;

[0014] The text encoder uses a 3D Transformer architecture to extract text semantic information and linearly aligns the extracted text semantic information with the image semantic information extracted by the image encoder.

[0015] The mask decoder obtains the final segmentation annotation by integrating the multi-scale image feature map output by the image encoder and the text hint embedding vector output by the text encoder.

[0016] Preferably, the preprocessing of the data set includes:

[0017] Crop the organ and the corresponding mask according to the maximum bounding rectangle of the single organ mask;

[0018] Binarize all mask pixels, set the background pixel value to 0, and the foreground pixel value to 1, and unify the cube size of the organ and mask.

[0019] Preprocess the radiological examination report to remove extra spaces and punctuation marks in the text of the radiological examination report.

[0020] Preferably, in step 3, an appropriate loss function and optimizer are used to optimize the model, a joint loss function is used to calculate the loss between the predicted mask and the true value mask, and an AdamW optimizer is used to update and optimize the model parameters. The joint loss calculation formula is:

[0021]

[0022] L=αL DL +βL CE ;

[0023] Among them, L is the joint loss function, L DL It is Dice Loss, L CE is the cross entropy loss, α and β represent the weight parameters of DiceLoss and cross entropy loss respectively, N is the total number of pixels of the mask, s represents the predicted mask, g represents the true value mask, s i represents the confidence of the i-th pixel of the predicted mask, g i Represents the confidence of the i-th pixel of the true value mask.

[0024] Preferably, the model is trained by fine-tuning the image encoder and obtaining the image embedding vector, specifically including:

[0025] The first step is to set the appropriate window width and window position according to the different types of input cubes before inputting the image encoder, and normalize the image to [0, 1].

[0026] In the second step, in the image encoder, the cube is discretized into a vector through the convolution operation of the convolution kernel, and the vector is flattened and input into the cascaded multi-layer 3DVIT unit. The features after the last and three different depth 3DVIT units are selected as the intermediate feature vectors, and the features are post-processed. Each feature vector is projected, transposed and resized, and multi-scale feature maps are generated through convolution and convolution transposition operations, namely feature_1, feature_2, feature_3 and feature_4, each feature map represents features at different depths.

[0027] As a preferred method, during the model training phase, fine-tune the text encoder to obtain the text prompt embedding vector including

[0028] The text prompt data is a radiological examination report, which includes information such as diseased tissue, lesion location, and lesion signs. Before entering the text encoder, it needs to be preprocessed and entity recognition and relationship extraction are performed to form a triple format. The triple is represented as (s, r, o), where s represents the head entity, r represents the relationship, and o represents the tail entity. Jieba word segmentation is used to perform named entity recognition on the text and the CasRel model is used to extract the relationship between entities. The extracted relationships are manually verified to filter out incorrect relationships and form triples.

[0029] The 3D Transformer architecture is used to encode the triples. Its structure includes a multi-head attention mechanism layer, LayerNorm, a multi-layer perceptron, and a LayerNorm. The number of heads in the attention mechanism layer is 8, and the structure of the multi-layer perceptron is a linear layer, a Quick GELU activation function, and a linear layer. The formula of Quick GELU is:

[0030]

[0031] Among them, x represents the tensor obtained after the text entity relationship triples are processed;

[0032] The text tensor x undergoes repeated self-attention calculations and residual calculations, and then passes through a multi-layer perceptron layer to obtain the final prompt embedding vector.

[0033] Preferably, in the model training stage, a mask decoder is fine-tuned and trained to obtain a predicted mask by fusing multi-scale image features and text features;

[0034] Fuse multi-scale image features with text features to generate the final prediction mask;

[0035] The multi-scale image input features feature_1, feature_2, feature_3 and feature_4 are first processed by a convolution layer with a convolution kernel size of 3 to obtain the corresponding feature maps. The feature fusion module is used to gradually fuse the multi-scale image features and perform feature dimensionality reduction to obtain the final fused image features. After that, the image features and text features are normalized to obtain the corresponding unit feature vectors.

[0036] The inner product operation is performed on the two unit feature vectors to obtain the matching score between the two. The final output feature map is upsampled through convolution transposition and linear interpolation method to generate a prediction mask.

[0037] In order to solve the above technical problems, the present invention also provides a medical image segmentation and annotation system based on multimodal information fusion, and a medical image segmentation and annotation method based on multimodal information fusion, which method includes:

[0038] The model structure building module builds the model structure of the medical image segmentation and annotation model with multimodal information fusion. The model structure of the segmentation and annotation model includes an image encoder, a text encoder, and a mask decoder.

[0039] The data set processing module preprocesses the data set and constructs the data set using the random sampling method, and divides the data set into training set, validation set and test set according to the proportion;

[0040] The model training and convergence module trains the model through loss function and optimizer until the model converges;

[0041] The model segmentation and annotation result acquisition module crops the organ to be segmented from the original image and inputs it into the trained model. At the same time, it inputs text prompt words for semantic segmentation and uses the mask predicted by the model as the model segmentation and annotation result.

[0042] The present invention has significant technical effects due to the adoption of the above technical solution:

[0043] The present invention is based on a medical image segmentation and annotation method based on multimodal information fusion, constructs a model structure, then performs necessary preprocessing on the medical image data set, and then segments the medical image data in combination with guiding text semantic information, effectively improving the efficiency of medical image segmentation and annotation from 0 to 1. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart of the present invention;

[0045] Figure 2 is a system diagram of the present invention;

[0046] Figure 3-1 A 3D abdominal CT image of the kidney segmented and labeled by the present invention;

[0047] Figure 3-2 A 3D abdominal CT image of the stomach segmented and annotated by the present invention;

[0048] Figure 3-3 This is a 3D abdominal CT image of the liver segmented and labeled by the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0050] Example 1

[0051] A medical image segmentation and annotation method based on multimodal information fusion, the method comprising:

[0052] Step 1: Constructing the model structure. Constructing the model structure of the medical image segmentation and annotation model with multimodal information fusion. The model structure of the segmentation and annotation model includes an image encoder, a text encoder, and a mask decoder.

[0053] Step 2, data set processing, preprocessing the data set and constructing the data set using random sampling method, dividing the data set into training set, validation set and test set according to the proportion;

[0054] Step 3: Model training and convergence: Train the model through loss function and optimizer until the model converges;

[0055] In step 4, the organ to be segmented is cropped from the original image and input into the model trained in step 3. At the same time, text prompt words are input for semantic segmentation, and the mask predicted by the model is used as the model segmentation annotation result.

[0056] The image encoder uses 3D Vision Transformer as the backbone network to extract features from the input image cube and obtain feature maps of different depths;

[0057] The text encoder uses a 3D Transformer architecture to extract text semantic information and linearly aligns the extracted text semantic information with the image semantic information extracted by the image encoder.

[0058] The mask decoder obtains the final segmentation annotation by integrating the multi-scale image feature map output by the image encoder and the text hint embedding vector output by the text encoder.

[0059] The preprocessing of the dataset includes:

[0060] Crop the organ and the corresponding mask according to the maximum bounding rectangle of the single organ mask;

[0061] Binarize all mask pixels, set the background pixel value to 0, and the foreground pixel value to 1, and unify the cube size of the organ and mask.

[0062] Preprocess the radiological examination report to remove extra spaces and punctuation marks in the text of the radiological examination report.

[0063] Step 3: Use appropriate loss functions and optimizers to optimize the model. Use the joint loss function to calculate the loss between the predicted mask and the true mask. Use the AdamW optimizer to update and optimize the model parameters. The joint loss calculation formula is:

[0064]

[0065] L=αL DL +βL CE ;

[0066] Among them, L is the joint loss function, L DL It is Dice Loss, L CE is the cross entropy loss, α and β represent the weight parameters of DiceLoss and cross entropy loss respectively, N is the total number of pixels of the mask, s represents the predicted mask, g represents the true value mask, s i represents the confidence of the i-th pixel of the predicted mask, g i Represents the confidence of the i-th pixel of the true value mask.

[0067] The model is trained by fine-tuning the image encoder and obtaining the image embedding vector, including:

[0068] The first step is to set the appropriate window width and window position according to the different types of input cubes before inputting the image encoder, and normalize the image to [0, 1].

[0069] In the second step, in the image encoder, the cube is discretized into a vector through the convolution operation of the convolution kernel, and the vector is flattened and input into the cascaded multi-layer 3DVIT unit. The features after the last and three different depth 3DVIT units are selected as the intermediate feature vectors, and the features are post-processed. Each feature vector is projected, transposed and resized, and multi-scale feature maps are generated through convolution and convolution transposition operations, namely feature_1, feature_2, feature_3 and feature_4, each feature map represents features at different depths.

[0070] During the model training phase, fine-tune the text encoder to obtain the text prompt embedding vector, including:

[0071] The text prompt data is a radiological examination report, which includes information such as diseased tissue, lesion location, and lesion signs. Before entering the text encoder, it needs to be preprocessed and entity recognition and relationship extraction are performed to form a triple format. The triple is represented as (s, r, o), where s represents the head entity, r represents the relationship, and o represents the tail entity. Jieba word segmentation is used to perform named entity recognition on the text and the CasRel model is used to extract the relationship between entities. The extracted relationships are manually verified to filter out incorrect relationships and form triples.

[0072] The 3D Transformer architecture is used to encode the triples. Its structure includes a multi-head attention mechanism layer, LayerNorm, a multi-layer perceptron, and a LayerNorm. The number of heads in the attention mechanism layer is 8, and the structure of the multi-layer perceptron is a linear layer, a Quick GELU activation function, and a linear layer. The formula of Quick GELU is:

[0073]

[0074] Among them, x represents the tensor obtained after the text entity relationship triples are processed;

[0075] The text tensor x undergoes repeated self-attention calculations and residual calculations, and then passes through a multi-layer perceptron layer to obtain the final prompt embedding vector.

[0076] In the model training phase, the mask decoder is fine-tuned and trained to obtain the predicted mask by fusing multi-scale image features and text features.

[0077] Fuse multi-scale image features with text features to generate the final prediction mask;

[0078] The multi-scale image input features feature_1, feature_2, feature_3 and feature_4 are first processed by a convolution layer with a convolution kernel size of 3 to obtain the corresponding feature maps. The feature fusion module is used to gradually fuse the multi-scale image features and perform feature dimensionality reduction to obtain the final fused image features. After that, the image features and text features are normalized to obtain the corresponding unit feature vectors.

[0079] The inner product operation is performed on the two unit feature vectors to obtain the matching score between the two. The final output feature map is upsampled through convolution transposition and linear interpolation method to generate a prediction mask.

[0080] Example 2

[0081] Based on Example 1, this example is a medical image segmentation and annotation system based on multimodal information fusion, and a medical image segmentation and annotation method based on multimodal information fusion, and the method includes:

[0082] The model structure building module builds the model structure of the medical image segmentation and annotation model with multimodal information fusion. The model structure of the segmentation and annotation model includes an image encoder, a text encoder, and a mask decoder.

[0083] The data set processing module preprocesses the data set and constructs the data set using the random sampling method, and divides the data set into training set, validation set and test set according to the proportion;

[0084] The model training and convergence module trains the model through loss function and optimizer until the model converges;

[0085] The model segmentation and annotation result acquisition module crops the organ to be segmented from the original image and inputs it into the trained model. At the same time, it inputs text prompt words for semantic segmentation and uses the mask predicted by the model as the model segmentation and annotation result.

[0086] Example 3

[0087] Based on the above embodiments, this embodiment provides a method for segmenting and labeling three-dimensional medical images, including but not limited to common medical image types such as CT, MRI, X-ray, and ultrasound. Figure 1 As shown, the medical image segmentation and annotation method and system based on multimodal information fusion includes the following steps:

[0088] Construct a medical image segmentation and annotation model with multimodal information fusion. The constructed model architecture consists of three parts: image encoder, text encoder and mask decoder. Figure 2 As shown in Figure 2, image encoder and text encoder are both based on Transformer structures, which are used to extract input image features and text features respectively. Mask decoder is used to integrate features and generate the final segmentation mask.

[0089] Data preprocessing, obtain medical image data sets, including three-dimensional medical images and corresponding masks and text prompt data corresponding to the images, and randomly sample the data sets, and divide them into training sets, validation sets, and test sets in a ratio of 7:2:1 to ensure the effectiveness and generalization ability of the model.

[0090] The dataset used in this implementation is a 3D CT image of multiple human organs. There are 2284 CT image sequences, including 54 organs such as heart, liver, spleen, lung, kidney, muscle, and brain. The preprocessing method is as follows:

[0091] First, the organ and the corresponding mask are cropped according to the maximum circumscribed rectangular frame of the single organ mask, and the pixels of all masks are binarized, where 0 represents the background point pixel value and 1 represents the foreground point pixel value. Then the size of the organ and mask cube is uniformly set to (128, 128, 128). If one of the dimensions of the cube's length, width, and depth is greater than 128, a resizing operation is performed, otherwise a constant filling operation is performed. The processed data set contains 79,942 cubes and 79,942 corresponding masks.

[0092] In the model training phase, the image encoder is fine-tuned and trained to obtain the image embedding vector. The three-dimensional Vision Transformer is used as the backbone network of the image encoder. Its structure consists of 12 cascaded 3DVIT units. The structure of each VIT unit is LayerNorm, multi-head attention mechanism layer, LayerNorm, and multi-layer perceptron layer. Before inputting the image encoder, the appropriate window width and window position are set according to the different types of the input cube, and the image is normalized to [0,1]. In the image encoder, the cube is discretized into a vector with a width of 16, a height of 16, a depth of 16, and a channel number of 768 through a convolution operation with a convolution kernel size and a step size of 16. The vector is flattened and input into the cascaded multi-layer 3DVIT unit, and the features after the last and three different depth 3DVIT units are selected as the intermediate feature vectors for feature post-processing. Each feature vector is subjected to feature projection, transposition and size adjustment, and finally a multi-scale feature map is generated through convolution and convolution transposition operations, namely feature_1, feature_2, feature_3 and feature_4, each of which represents features at different depths.

[0093] In the model training phase, the text encoder is fine-tuned to obtain the text prompt embedding vector. The text prompt data is a radiology examination report, including information such as diseased tissue, lesion location, and lesion signs. Before entering the text encoder, it needs to be preprocessed and entity recognition and relationship extraction are performed to form a triple format. The triple is represented as (s, r, o), where s represents the head entity, r represents the relationship, and o represents the tail entity. Jieba word segmentation is used to perform named entity recognition on the text and the CasRel model is used to extract the relationship between entities. The extracted relationships are manually verified to filter out erroneous relationships and form triples. For example, if the radiology examination report mentions "a tumor was found in the caudate lobe of the liver", then the final relationship between the head entity "liver" and the tail entity "tumor" is "located in".

[0094] After that, the triples are encoded. The text encoder uses a 3D Transformer architecture, which includes a multi-head attention mechanism layer, LayerNorm, a multi-layer perceptron, and LayerNorm. The number of heads in the attention mechanism layer is 8, and the structure of the multi-layer perceptron is a linear layer, a Quick GELU activation function, and a linear layer. The formula of Quick GELU is as follows:

[0095]

[0096] Among them, x represents the tensor obtained after processing the text entity relationship triples.

[0097] The text tensor x undergoes repeated self-attention calculations and residual calculations, and then passes through a multi-layer perceptron layer to obtain the final text prompt vector.

[0098] During the model training phase, the mask decoder is fine-tuned and trained to obtain the predicted mask by fusing multi-scale image features and text features.

[0099] The structure of the mask decoder includes multiple convolutional layers and a feature fusion module, which fuses multi-scale image features with text features to generate the final prediction mask. The multi-scale image input features feature_1, feature_2, feature_3 and feature_4 are first processed by a convolutional layer with a convolution kernel size of 3 to obtain the corresponding feature map. The feature fusion module is used to gradually fuse the multi-scale image features. It adopts a dual residual convolution unit structure, in which each residual convolution unit includes two convolutional layers with a convolution kernel size of 3, combined with the ReLU activation function and the BatchNorm layer, and finally, the convolutional layer with a convolution kernel size of 1 is used to reduce the feature dimension to obtain the final fused image features. After that, the image features and text features are normalized to obtain the corresponding unit feature vector. Then, the inner product operation is performed on the two unit feature vectors to obtain the matching score of the two. The final output feature map is upsampled by the convolution transpose and linear interpolation method to generate the final prediction mask.

[0100] During the model training phase, appropriate loss functions and optimizers are used to optimize the model. A joint loss function is used to calculate the loss between the predicted mask and the true value mask, and the AdamW optimizer is used to update and optimize the model parameters. The formula for calculating the joint loss is as follows:

[0101] L=αL DL +βL CE

[0102] Among them, L is the joint loss function, L DL It is Dice Loss, L CE is the cross entropy loss, α and β represent the weight parameters of DiceLoss and cross entropy loss respectively, N is the total number of pixels of the mask, s represents the predicted mask, g represents the true value mask, s i represents the confidence of the i-th pixel of the predicted mask, g i Represents the confidence of the i-th pixel of the true value mask.

[0103] In the inference stage, the user cuts out the organ cube to be segmented from the original image and inputs it into the network, and then inputs the text prompt word to obtain the segmentation result.

[0104] like Figure 3-1 to Figure 3-3, is an example diagram of medical image segmentation with multimodal information fusion in actual use. A 3D abdominal CT image is used for demonstration. When performing intelligent image segmentation, the user must first draw a frame to circle the organ he wants to segment and set the corresponding window width and window position. The data preprocessing part of the model will crop the circled organ and scale it before inputting it into the model. The user then enters the corresponding prompt information, and finally the network will obtain the predicted mask based on the image information and text prompt information.

Claims

1. A medical image segmentation and annotation method based on multimodal information fusion, the method comprising: Step 1: constructing a model structure, constructing a model structure of a medical image segmentation and annotation model with multimodal information fusion. The model structure of the segmentation and annotation model includes an image encoder, a text encoder, and a mask decoder. Step 2, data set processing, preprocessing the data set and constructing the data set using random sampling method, dividing the data set into training set, validation set and test set according to the proportion; Step 3: Model training and convergence: Train the model through loss function and optimizer until the model converges; In step 4, the organ to be segmented is cropped from the original image and input into the model trained in step 3. At the same time, text prompt words are input for semantic segmentation, and the mask predicted by the model is used as the model segmentation annotation result.

2. The medical image segmentation and annotation method based on multimodal information fusion according to claim 1, characterized in that: The image encoder uses 3D Vision Transformer as the backbone network to extract features from the input image cube and obtain feature maps of different depths; The text encoder uses a 3D Transformer architecture to extract text semantic information and linearly aligns the extracted text semantic information with the image semantic information extracted by the image encoder. The mask decoder obtains the final segmentation annotation by integrating the multi-scale image feature map output by the image encoder and the text hint embedding vector output by the text encoder.

3. The medical image segmentation and annotation method based on multimodal information fusion according to claim 1, characterized in that: The preprocessing of the dataset includes: Crop the organ and the corresponding mask according to the maximum bounding rectangle of the single organ mask; Binarize all mask pixels, set the background pixel value to 0, and the foreground pixel value to 1, and unify the cube size of the organ and mask. Preprocess the radiological examination report to remove extra spaces and punctuation marks in the text of the radiological examination report.

4. The medical image segmentation and annotation method based on multimodal information fusion according to claim 1, characterized in that: Step 3: Use appropriate loss functions and optimizers to optimize the model. Use the joint loss function to calculate the loss between the predicted mask and the true mask. Use the AdamW optimizer to update and optimize the model parameters. The joint loss calculation formula is: L=αL DL +βL CE ; Among them, L is the joint loss function, L DL It is Dice Loss, L CE is the cross entropy loss, α and β represent the weight parameters of Dice Loss and cross entropy loss respectively, N is the total number of pixels of the mask, s represents the predicted mask, g represents the true value mask, s i represents the confidence of the i-th pixel of the predicted mask, g i Represents the confidence of the i-th pixel of the true value mask.

5. The medical image segmentation and annotation method based on multimodal information fusion according to claim 1, characterized in that: The model is trained by fine-tuning the image encoder and obtaining the image embedding vector, including: The first step is to set the appropriate window width and window position according to the different types of input cubes before inputting the image encoder, and normalize the image to [0, 1]. In the second step, in the image encoder, the cube is discretized into a vector through the convolution operation of the convolution kernel, and the vector is flattened and input into the cascaded multi-layer 3DVIT unit, and the features after the last and three different depth 3DVIT units are selected as the intermediate feature vectors, and the features are post-processed. Each feature vector is projected, transposed and resized, and multi-scale feature maps are generated through convolution and convolution transposition operations, namely feature_1, feature_2, feature_3 and feature_4, each feature map represents features at different depths.

6. The medical image segmentation and annotation method based on multimodal information fusion according to claim 1, characterized in that: In the model training phase, the text encoder is fine-tuned to obtain the text prompt embedding vector. include The text prompt data is a radiological examination report, which includes information such as diseased tissue, lesion location, and lesion signs. Before entering the text encoder, it needs to be preprocessed and entity recognition and relationship extraction are performed to form a triple format. The triple is represented as (s, r, o), where s represents the head entity, r represents the relationship, and o represents the tail entity. Jieba word segmentation is used to perform named entity recognition on the text and the CasRel model is used to extract the relationship between entities. The extracted relationships are manually verified to filter out incorrect relationships and form triples. The 3D Transformer architecture is used to encode the triples. Its structure includes a multi-head attention mechanism layer, LayerNorm, a multi-layer perceptron, and a LayerNorm. The number of heads in the attention mechanism layer is 8, and the structure of the multi-layer perceptron is a linear layer, a Quick GELU activation function, and a linear layer. The formula of Quick GELU is: Among them, x represents the tensor obtained after the text entity relationship triples are processed; The text tensor x undergoes repeated self-attention calculations and residual calculations, and then passes through a multi-layer perceptron layer to obtain the final prompt embedding vector.

7. The medical image segmentation and annotation method based on multimodal information fusion according to claim 1, characterized in that: In the model training phase, the mask decoder is fine-tuned and trained to obtain the predicted mask by fusing multi-scale image features and text features. Fuse multi-scale image features with text features to generate the final prediction mask; The multi-scale image input features feature_1, feature_2, feature_3 and feature_4 are first processed by a convolution layer with a convolution kernel size of 3 to obtain the corresponding feature maps. The feature fusion module is used to gradually fuse the multi-scale image features and perform feature dimensionality reduction to obtain the final fused image features. After that, the image features and text features are normalized to obtain the corresponding unit feature vectors. The inner product operation is performed on the two unit feature vectors to obtain the matching score between the two. The final output feature map is upsampled through convolution transposition and linear interpolation method to generate a prediction mask.

8. A medical image segmentation and annotation system based on multimodal information fusion, characterized in that: A medical image segmentation and annotation method based on multimodal information fusion, the method comprising: The model structure building module builds the model structure of the medical image segmentation and annotation model with multimodal information fusion. The model structure of the segmentation and annotation model includes an image encoder, a text encoder, and a mask decoder. The data set processing module preprocesses the data set and constructs the data set using the random sampling method, and divides the data set into training set, validation set and test set according to the proportion; The model training and convergence module trains the model through loss function and optimizer until the model converges; The model segmentation and annotation result acquisition module crops the organ to be segmented from the original image and inputs it into the trained model. At the same time, it inputs text prompt words for semantic segmentation and uses the mask predicted by the model as the model segmentation and annotation result.

Images