Medical Image Segmentation Methods

Abstract

The present invention relates to a medical image segmentation method, comprising: collecting magnetic resonance imaging (MRI) image data of tumor patients as a dataset; performing data processing on the image data in the dataset; using the multimodal images in the dataset that meet the requirements after data processing as model input; designing a separate Transformer for each modality to extract features; designing a modality fusion Transformer to fuse the data of multiple modalities; gradually reshaping encoder outputs of different scales to the input size to obtain a segmentation result that matches the original image; constructing weakly enhanced images and strongly enhanced images for unlabeled data in the dataset; selecting positive and negative examples based on the encoder outputs for the different enhanced images and calculating contrast loss; calculating DICE loss on the labels and segmentation results; training the model to obtain and save the final model. The present invention can better locate tumors and improve segmentation results.

Description

Technical Field

[0001] The invention relates to a medical image segmentation method. Background Art

[0002] Medical image segmentation is fundamental to various medical image applications. Its clinical value is increasing in clinical auxiliary diagnosis, image-guided surgery, and radiotherapy. Traditionally, medical image segmentation relies on manual segmentation performed by experienced physicians. This purely manual approach is often time-consuming and labor-intensive, and is significantly influenced by the physician's subjective judgment. With the rapid development of deep learning technology, fully automated image segmentation based on deep learning has rapidly developed. However, deep learning often relies on massive amounts of high-quality labeled data, which is relatively scarce, making acquisition of high-quality labeled data difficult.

[0003] Semi-supervised learning frameworks can directly learn from limited labeled data and a large amount of unlabeled data to obtain high-quality segmentation results. Current semi-supervised medical image segmentation methods can be divided into three categories: adversarial learning methods, consistency regularization methods, and collaborative training methods. Adversarial learning methods use discriminators to align the distribution of labeled and unlabeled data in the embedding space. This requires the data to meet the distribution assumption, and many adversarial learning models are difficult to train. The basic idea of the consistency regularization method is to regularize the model prediction, that is, a robust model should have similar outputs for similar inputs. The difference between each method lies in how to inject noise and how to calculate consistency, but the consistency regularization method relies on appropriate data augmentation strategies, and incorrect pseudo-labels will continue to be reinforced during training. The collaborative training method is based on the assumption of low-density separation of data. The disadvantage of this method is that if the generated pseudo-labels are inaccurate, they will lead to self-reinforcement of classification errors.

[0004] To fully utilize unlabeled data, semi-supervised segmentation is typically performed using adversarial learning, consistency regularization, and collaborative training methods. Overall, these methods all exploit consistency in the output space and lack constraints within the feature space. Consequently, in many cases, the model fails to recognize incorrect features, causing these errors to accumulate during training. Summary of the Invention

[0005] In view of this, it is necessary to provide a medical image segmentation method.

[0006] The present invention provides a medical image segmentation method, which includes the following steps: a. collecting magnetic resonance imaging (MRI) image data of tumor patients as a data set; b. performing data processing on the image data in the data set, wherein the data processing includes format conversion, resampling, registration, and standardization of the image data in the data set; c. using multimodal images that meet the requirements in the data set after data processing as input of a model; d. establishing a multi-branch Transformer neural network as an encoder, and designing a separate Transformer for each modality to extract features; e. designing a modality fusion Transformer to fuse data of multiple modalities; f. establishing a decoder, and gradually reshaping encoder outputs of different scales to input sizes to obtain a segmentation result that matches the original image; g. constructing weakly enhanced images and strongly enhanced images for unlabeled data in the data set; h. selecting positive examples and negative examples based on the encoder outputs for differently enhanced images, and calculating contrast loss; i. calculating dice loss for labels and segmentation results; and j. training the model, selecting the result with better effect as the final model, and saving the result.

[0007] Specifically, the patient's MRI image data is a multimodal MRI image; the MRI image data of each patient includes four commonly used modalities; the four commonly used modalities are T1, T2, T1C, and Flair modalities.

[0008] Specifically, the step b specifically includes:

[0009] First, the DICOM format is converted to NIFTI format; then the image is resampled; then the image is registered, and the points corresponding to the same spatial position at multiple time points are matched one by one. The rigid registration mode is used during registration, and mutual information is used as the image similarity measure; the image data in the dataset is standardized using grayscale normalization and histogram equalization methods.

[0010] Specifically, the step c specifically includes:

[0011] The multimodal images in the dataset that meet the requirements are used as the input of the model, and the dataset is divided into a training set and a test set. First, magnetic resonance image data with modality missing, registration failure, or no tumor are excluded to avoid affecting the generalization performance of the model. Then, the dataset is divided into a training set and a test set in a ratio of 4:1. For the training set, labeled data and unlabeled data are divided as needed and processed separately.

[0012] Specifically, the step d specifically includes:

[0013] A separate Transformer is designed for each modality to extract features. For input with four modalities, a multi-branch Transformer is proposed with the same number of branches as the input modalities in order to simultaneously extract independent features from multiple modalities. The three-dimensional whole-brain image is divided into K fixed-size three-dimensional image blocks, mapped into a one-dimensional vector of fixed length D, and position encoding is added to retain position information before input into the visual Transformer model.

[0014] Specifically, the step e specifically includes:

[0015] A separate fusion Transformer based on the cross-attention mechanism is designed: the fusion Transformer based on the cross-attention mechanism is divided into two parts, namely the partial fusion Transformer and the overall fusion Transformer; the partial fusion Transformer uses a single one-dimensional vector of each branch as a query to exchange information with other branches, and inputs the partial fusion result into the overall fusion Transformer. The self-attention mechanism therein is used to more thoroughly fuse the multimodal information together, thereby utilizing the global context information at the overall semantic structure level of the data.

[0016] Specifically, the step f specifically includes:

[0017] The decoder gradually reshapes the encoder outputs at different scales to the input size to obtain a segmentation result that matches the original image. The decoder takes the encoder output as its input, with five channels. It fuses the encoder outputs layer by layer through convolution and deconvolution operations, restores the image to the specified size, and applies the sigmoid function to obtain the final segmentation result.

[0018] Specifically, the step g specifically includes:

[0019] Two enhancement methods are designed for a single unlabeled image. In each training step, a transformation is randomly selected for each sample in the batch from a predefined range: the first enhancement method is weak enhancement, which is the result of a random flipping, moving and random scaling strategy with a probability of 50%; the other enhancement method is strong enhancement, which adds a grayscale transformation to the weakly enhanced image.

[0020] Specifically, the step h specifically includes:

[0021] The unlabeled data loss is divided into two parts, including output space consistency loss and contrastive learning loss. The contrastive learning loss is calculated by the encoder generating features based on weakly enhanced images and strongly enhanced images respectively. Features at the same position are regarded as positive examples, and features at different positions are regarded as negative examples. The negative examples are sampled using the gumbel sampling strategy, selecting k pixels with the smallest cosine similarity to form negative examples, or selecting pixels with a larger distance as negative examples based on anatomical prior knowledge. The pixel contrast loss is obtained by combining the InfoNCE loss with the cosine similarity.

[0022] Specifically, the step i specifically includes:

[0023] For the segmentation results obtained with labeled data, the dice loss is calculated with the label as the supervised learning loss; for unlabeled data, the consistency loss is calculated between the results of the weakly enhanced image and the strongly enhanced image.

[0024] Specifically, the step j specifically includes:

[0025] Use stochastic gradient descent as the optimizer for training, and use weight decay to prevent overfitting; after the model training is completed, save the model that is more accurate under supervised data of various proportions.

[0026] This application not only takes into account the consistency of the output space, but also solves to a certain extent the problem of error accumulation caused by the inability to filter out erroneous features in current methods. It also uses Transformer as the main feature extraction network and utilizes the attention mechanism and global receptive field advantages in Transformer to locate tumors faster and more accurately, thereby improving the accuracy compared to convolutional neural network methods with only local receptive fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the medical image segmentation method of the present invention;

[0028] Figure 2 A schematic diagram of a Transformer neural network provided by an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of the Transformer neural network segmentation process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] See Figure 1 FIG. 1 is a flowchart of a preferred embodiment of the medical image segmentation method of the present invention.

[0032] Please also refer to Figure 2-3 , step S1, collects magnetic resonance imaging data of tumor patients as a data set. Specifically:

[0033] In this embodiment, MRI data of tumor patients are collected. The MRI data of the patients are multimodal MRI images. The MRI data of each patient includes four common modalities: T1, T2, T1C, and Flair.

[0034] The patient images obtained in this step come from the patient image dataset jointly collected by the hospital, TCIA (The Cancer Imaging Archive) and TCGA (The Cancer Genome Atlas).

[0035] This embodiment does not limit the size of the data set; the larger the data set, the stronger the generalization capability.

[0036] Step S2, performing data processing on the image data in the dataset, wherein the data processing includes: format conversion, resampling, registration and standardization of the image data in the dataset. Specifically:

[0037] Perform format conversion, resampling, registration, and standardization on the image data in the dataset. DICOM (Digital Imaging and Communications in Medicine) refers to the Digital Imaging and Communications in Medicine (DICOM) protocol, a set of universal standard protocols for medical image processing, storage, printing, and transmission.

[0038] The data obtained from medical devices is in DICOM format. We first converted the DICOM format to NIFTI (NeuroImaging Informatics Technology Initiative). The images were then resampled to improve resolution. The images were then registered, associating points corresponding to the same spatial location at multiple time points. Rigid registration was used, with mutual information as the image similarity metric. After registration and resampling, the spatial resolution of the images was 1 mm. Grayscale normalization and histogram equalization were used to standardize the image data in the dataset.

[0039] Step S3: After the data is processed, the multimodal images that meet the requirements in the dataset are used as the input of the model, and the dataset is divided into a training set and a test set. The training set is divided into labeled data and unlabeled data as needed and processed separately. Specifically:

[0040] The model used multimodal images from the dataset that met the requirements as input, and the dataset was divided into training and test sets. MRI data with missing modalities, registration failures, or tumor-free images were first excluded to avoid affecting the model's generalization performance. The dataset was then divided into training and test sets in a 4:1 ratio. The training set was then divided into labeled and unlabeled data as needed and processed separately. In semi-supervised tasks, the proportion of labeled data significantly affects segmentation results. Therefore, the amount of labeled data in the training set was gradually reduced by 10%, and experiments were conducted separately.

[0041] Step S4: Establish a multi-branch Transformer neural network as an encoder, and design a separate Transformer for each modality to extract features. Specifically:

[0042] A multi-branch Transformer neural network was constructed. The intended segmentation model adopts an encoder-decoder structure, where the encoder extracts appropriate features, and the decoder restores the image to the input size. A separate Transformer was designed for each modality to extract features. For inputs with four modalities, a multi-branch Transformer with the same number of branches as the input modalities was proposed to simultaneously extract independent features from multiple modalities. A 3D whole-brain image was divided into K fixed-size 3D patches, mapped into a one-dimensional vector of fixed length D, and a positional encoding was added to preserve positional information before being fed into the visual Transformer model.

[0043] Step S5: Design a modality fusion Transformer to fuse data from multiple modalities. Specifically:

[0044] In order to fully fuse the features of each modality from multiple angles to produce stronger image features, this application separately designs a fusion Transformer based on the cross-attention mechanism. The fusion Transformer based on the cross-attention mechanism is divided into two parts, namely the partial fusion Transformer and the overall fusion Transformer. The partial fusion Transformer uses a single one-dimensional vector of each branch as a query to exchange information with other branches. The partial fusion result is input into the overall fusion Transformer, and the self-attention mechanism therein is used to more thoroughly fuse the multimodal information together, thereby utilizing the global context information at the overall semantic structure level of the data.

[0045] Step S6: Establish a decoder to gradually reshape the encoder outputs of different scales to the input size to obtain a segmentation result that matches the original image. Specifically:

[0046] The decoder gradually reshapes the encoder outputs at different scales to the input size to obtain a segmentation result that matches the original image. The decoder takes the encoder output as its input, with five channels. It fuses the encoder outputs layer by layer through convolution and deconvolution operations, restores the image to the specified size, and applies the sigmoid function to obtain the final segmentation result.

[0047] Step S7: For unlabeled data, construct weakly enhanced images and strongly enhanced images. Specifically:

[0048] Two augmentation methods are designed for a single unlabeled image. During each training step, a transformation is randomly selected for each sample in the batch from a predefined range. The first augmentation method, weak augmentation, is the result of a random flip, shift, and scale strategy with a 50% probability. The other augmentation method, strong augmentation, adds a grayscale transformation to the weakly enhanced image.

[0049] Step S8: Select positive and negative examples based on the encoder's output of different enhanced images and calculate the contrast loss. Specifically:

[0050] The unlabeled data loss consists of two parts: output space consistency loss and contrastive learning loss. The contrastive learning loss is calculated by having the encoder generate features based on the weakly enhanced image and the strongly enhanced image, respectively. Features at the same location are considered positive examples, while features at different locations are considered negative examples. Negative examples are sampled using a gumbel sampling strategy, selecting the k pixels with the smallest cosine similarity as negative examples. Alternatively, based on anatomical prior knowledge, distant pixels are selected as negative examples. The goal of the contrastive learning loss is to increase the similarity with positive pixels and decrease the similarity with the k negative pixels. To achieve this goal, the InfoNCE loss is combined with cosine similarity to obtain the pixel contrast loss. Specifically, the cross-entropy loss is calculated using all labels equal to 1 for positive examples and all labels equal to 0 for negative examples. The sum of these losses is the contrastive learning loss.

[0051] Step S9: Calculate the dice loss for the labels and segmentation results. Calculate the consistency loss for the outputs of the two branches of unlabeled data. The total loss is the supervised learning loss, contrastive learning loss, and consistency loss. Specifically:

[0052] Calculate the total loss. For segmentation results obtained from labeled data, calculate the Dice loss with the labels as the supervised learning loss. For unlabeled data, calculate the consistency loss between the weakly enhanced image and the strongly enhanced image; add the consistency loss and the contrastive learning loss to form the semi-supervised loss. The total loss is the sum of the semi-supervised loss and the supervised loss.

[0053] Step S10: train the model, select the best result as the final model and save it.

[0054] During training, data enhancement methods including but not limited to rotation, translation, scaling, and cropping are used to improve the generalization ability of the model;

[0055] Stochastic gradient descent is used as the optimizer for training, and weight decay is used to prevent overfitting. For the input image data, the network output is a binary segmentation result;

[0056] The results output by the network are mapped to the original image to assist doctors in diagnosing patients.

[0057] After the model training is completed, the model that is more accurate under supervised data of various proportions is selected and saved.

[0058] This application leverages the ability of contrastive learning to bring similar features closer together and dissimilar features further apart to constrain the feature space, further improving the effectiveness of semi-supervised learning. The model is constructed using a visual Transformer instead of a convolutional neural network, and the global receptive field provided by the attention mechanism is used to fuse multimodal information, better locating the tumor and thus improving segmentation.

[0059] Although the present invention has been described with reference to the current preferred embodiments, those skilled in the art should understand that the above-mentioned preferred embodiments are only used to illustrate the present invention and are not used to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A medical image segmentation method, characterized in that: The method comprises the following steps: a. Collect magnetic resonance imaging data of tumor patients as a dataset; b. performing data processing on the image data in the data set, the data processing comprising: format conversion, resampling, registration and standardization of the image data in the data set; c. After data processing, the multimodal images in the dataset that meet the requirements are used as the input of the model; d. Establish a multi-branch Transformer neural network as an encoder and design a separate Transformer for each modality to extract features; e. Design a modality fusion Transformer to fuse data from multiple modalities; f. Build a decoder that gradually reshapes the encoder outputs of different scales to the input size to obtain segmentation results that match the original image; g. For the unlabeled data in the dataset, construct weakly enhanced images and strongly enhanced images; h. Select positive and negative examples based on the encoder's output for different enhanced images and calculate the contrast loss; i. Calculate dice loss for labels and segmentation results; j. Train the model, select the best result as the final model and save it; The step f specifically includes: The decoder gradually reshapes the encoder outputs of different scales to the input size to obtain a segmentation result that matches the original image. The decoder takes the encoder output as five channel inputs, fuses the encoder outputs of each layer layer by layer through convolution and deconvolution operations, restores the image to the specified size, and applies the sigmoid function to obtain the final segmentation result; The step g specifically includes: Two augmentation methods are designed for a single unlabeled image. In each training step, a transformation is randomly selected for each sample in the batch from a predefined range: the first augmentation method is weak augmentation, which is the result of a random flip, shift, and random scaling strategy with a probability of 50%; the other augmentation method is strong augmentation, which adds a grayscale transformation to the weakly enhanced image. The step h specifically includes: The unlabeled data loss is divided into two parts, including output space consistency loss and contrastive learning loss. The contrastive learning loss is calculated by the encoder generating features based on weakly enhanced images and strongly enhanced images respectively. Features at the same position are regarded as positive examples, and features at different positions are regarded as negative examples. The negative examples are sampled using the gumbel sampling strategy, selecting k pixels with the smallest cosine similarity to form negative examples, or selecting pixels with a larger distance as negative examples based on anatomical prior knowledge. The pixel contrast loss is obtained by combining the InfoNCE loss with the cosine similarity.

2. The medical image segmentation method according to claim 1, wherein: The patient's MRI image data is a multimodal MRI image; the MRI image data of each patient includes four commonly used modalities; the four commonly used modalities are T1, T2, T1C, and Flair modalities.

3. The medical image segmentation method according to claim 2, wherein: The step b specifically includes: First, the DICOM format is converted to NIFTI format; then the image is resampled; then the image is registered, and the points corresponding to the same spatial position at multiple time points are matched one by one. The rigid registration mode is used during registration, and mutual information is used as the image similarity measure; the image data in the dataset is standardized using grayscale normalization and histogram equalization methods.

4. The medical image segmentation method according to claim 3, wherein: The step c specifically includes: The multimodal images in the dataset that meet the requirements are used as the input of the model, and the dataset is divided into a training set and a test set. First, magnetic resonance image data with modality missing, registration failure, or no tumor are excluded to avoid affecting the generalization performance of the model. Then, the dataset is divided into a training set and a test set in a ratio of 4:

1. For the training set, labeled data and unlabeled data are divided as needed and processed separately.

5. The medical image segmentation method according to claim 4, wherein: The step d specifically includes: A separate Transformer is designed for each modality to extract features. For input with four modalities, a multi-branch Transformer is proposed with the same number of branches as the input modalities in order to simultaneously extract independent features from multiple modalities. The three-dimensional whole-brain image is divided into K fixed-size three-dimensional image blocks, mapped into a one-dimensional vector of fixed length D, and position encoding is added to retain position information before input into the visual Transformer model.

6. The medical image segmentation method according to claim 5, wherein: The step e specifically includes: A separate fusion Transformer based on the cross-attention mechanism is designed: the fusion Transformer based on the cross-attention mechanism is divided into two parts, namely the partial fusion Transformer and the overall fusion Transformer; the partial fusion Transformer uses a single one-dimensional vector of each branch as a query to exchange information with other branches, and inputs the partial fusion result into the overall fusion Transformer. The self-attention mechanism therein is used to more thoroughly fuse the multimodal information together, thereby utilizing the global context information at the overall semantic structure level of the data.

7. The medical image segmentation method according to claim 6, wherein: The step i specifically includes: For the segmentation results obtained with labeled data, the dice loss is calculated with the label as the supervised learning loss; for unlabeled data, the consistency loss is calculated between the results of the weakly enhanced image and the strongly enhanced image.

8. The medical image segmentation method according to claim 7, wherein: The step j specifically includes: Use stochastic gradient descent as the optimizer for training, and use weight decay to prevent overfitting; after the model training is completed, save the model that is more accurate under supervised data of various proportions.

Images