Medical image segmentation method, system, terminal and storage medium

Abstract

The present application relates to a medical image segmentation method, system, terminal and storage medium. The method includes: obtaining medical image sample data, the medical image sample data includes multimodal medical images and clinical information of cases corresponding to the multimodal medical images; constructing a weak semi-supervised model, the weak semi-supervised model includes a segmentation branch for performing segmentation tasks and a survival prediction branch for performing survival prediction tasks, inputting the medical image sample data into the segmentation branch and the survival prediction branch respectively, fusing the features extracted by the segmentation branch and the survival prediction branch and iteratively training to obtain a trained image segmentation model; inputting the medical image to be segmented into the trained image segmentation model for image segmentation. The present application adopts a semi-supervised segmentation method, which does not rely on too much labeled data; combined with a weak supervision method, high-level semantics such as survival period are used as a weak supervision source to improve the accuracy of image segmentation.

Description

Technical Field

[0001] The present application belongs to the field of medical image processing technology, and in particular relates to a medical image segmentation method, system, terminal and storage medium. Background Art

[0002] Medical image segmentation is the basis of various medical image applications. In clinical auxiliary diagnosis, image-guided surgery and radiotherapy, medical image segmentation technology shows increasingly important clinical value. Traditional medical image segmentation is based on manual segmentation by experienced doctors, which is often time-consuming and laborious, and is greatly affected by the subjective influence of doctors. Even experienced doctors may make wrong segmentations when they are tired. In addition, the effect of segmentation by inexperienced doctors is often difficult to measure.

[0003] With the rapid development of deep learning technology, fully automatic image segmentation based on deep learning has developed rapidly, and has even surpassed humans in some areas. Therefore, fully automatic segmentation based on deep learning technology has become a research hotspot. However, deep learning often relies on a large amount of high-quality labeled data, while medical imaging data is often scarce, and it is usually difficult to obtain high-quality labeled data. In addition, the cost of manual labeling is also extremely high and is greatly affected by different labelers. Summary of the invention

[0004] The present application provides a medical image segmentation method, system, terminal and storage medium, aiming to solve at least one of the above-mentioned technical problems in the prior art to a certain extent.

[0005] In order to solve the above problems, this application provides the following technical solutions:

[0006] A medical image segmentation method, comprising:

[0007] Acquire medical image sample data, where the medical image sample data includes multimodal medical images and clinical information of cases corresponding to the multimodal medical images;

[0008] Constructing a weak-semi-supervised model, the weak-semi-supervised model includes a segmentation branch for performing a segmentation task and a life span prediction branch for performing a life span prediction task, inputting the medical image sample data into the segmentation branch and the life span prediction branch respectively, fusing the features extracted by the segmentation branch and the life span prediction branch and iteratively training them to obtain a trained image segmentation model;

[0009] The medical image to be segmented is input into the trained image segmentation model for image segmentation.

[0010] The technical solution adopted by the embodiment of the present application also includes: the obtaining of medical image sample data includes:

[0011] The multimodal medical images are four modality images of FLAIR, T1, T2 and T1c for each case;

[0012] The clinical information includes the survival period and survival status of the case.

[0013] The technical solution adopted by the embodiment of the present application also includes: the acquisition of medical image sample data specifically comprises:

[0014] Generating Mask data of the multimodal medical image sample data;

[0015] Preprocessing the multimodal medical image sample data and the Mask data to generate a medical image data set for model training;

[0016] The medical image dataset is grouped according to a set ratio to obtain a training set, a validation set, and a test set.

[0017] The technical solution adopted by the embodiment of the present application also includes: the preprocessing of the multimodal medical image sample data and the Mask data is specifically:

[0018] The multimodal medical image and the corresponding Mask data are cropped; the cropping method is specifically: obtaining the center point of each multimodal medical image, expanding an area of ​​a set size outward from the center point, and cropping the part outside the area to obtain the cropped medical image and Mask data;

[0019] Normalizing the cropped medical image using a min-max algorithm;

[0020] The normalized medical images of the four modalities and the cropped Mask data are spliced ​​respectively to obtain a preprocessed medical image data set.

[0021] The technical solution adopted by the embodiment of the present application also includes: the grouping of the medical image data set according to the set ratio is specifically:

[0022] A 10-fold cross-validation algorithm was used, and 10% of the training set data was taken as the validation set in each round.

[0023] The technical solution adopted by the embodiment of the present application also includes: the weak-semi-supervised model is constructed as a 3D U-Net network, and the training process of the 3D U-Net network includes:

[0024] The training set data is respectively input into the segmentation branch and the life span prediction branch for downsampling processing, and the features obtained by the downsampling of the segmentation branch are converted into one-dimensional features through a flatten operation and then input into the Transformer module;

[0025] The Transformer module adopts the idea based on residual connection, adds the input features to the data before input, and then reshapes it to the shape before input, and at the same time leads the features obtained by the Transformer module to the lifetime prediction branch; the lifetime prediction branch transforms the distribution of features through the Adapter module, and fuses the features output by the segmentation branch and the lifetime prediction branch through the information fusion module, and then passes through the fully connected layer to obtain the risk value of the lifetime prediction; after obtaining the reshaped feature map, the segmentation branch restores the feature map to the size of the initial input image through upsampling, and then undergoes binarization processing to obtain the output result of the segmentation task.

[0026] The technical solution adopted by the embodiment of the present application also includes: the training mode of the 3D U-Net network is specifically:

[0027] The 3D U-Net network is trained using the Teacher-Student training mode; pseudo labels generated from unlabeled medical images are added to the training set. In each round of training, if the current training effect is better than the previous round, the Student model is updated with the Teacher model, otherwise the training continues. If the Student model cannot be updated after the training times exceed the set number, the model is considered to have converged and the model training is terminated.

[0028] Another technical solution adopted by the embodiment of the present application is: a medical image segmentation system, comprising:

[0029] Data acquisition module: used to acquire medical image sample data, wherein the medical image sample data includes multimodal medical images and clinical information of cases corresponding to the multimodal medical images;

[0030] Model training module: used to construct a weak-semi-supervised model, the weak-semi-supervised model includes a segmentation branch for performing a segmentation task and a life span prediction branch for performing a life span prediction task, the medical image sample data is input into the segmentation branch and the life span prediction branch respectively, the features extracted by the segmentation branch and the life span prediction branch are fused and iteratively trained to obtain a trained image segmentation model;

[0031] Image segmentation module: used to input the medical image to be segmented into the trained image segmentation model for image segmentation.

[0032] Another technical solution adopted by the embodiment of the present application is: a terminal, the terminal includes a processor and a memory coupled to the processor, wherein:

[0033] The memory stores program instructions for implementing the medical image segmentation method;

[0034] The processor is used to execute the program instructions stored in the memory to control medical image segmentation.

[0035] Another technical solution adopted by the embodiment of the present application is: a storage medium storing program instructions executable by a processor, wherein the program instructions are used to execute the medical image segmentation method.

[0036] Compared with the prior art, the beneficial effects of the embodiments of the present application are as follows: the medical image segmentation method, system, terminal and storage medium of the embodiments of the present application adopt a semi-supervised segmentation method, which does not rely on too much labeled data; combined with a weak supervision method, high-level semantics such as survival period are used as weak supervision sources, and the Transformer module is used to mine the correlation between features, thereby focusing on the tumor area and further improving the segmentation accuracy. By combining the segmentation task with the survival period prediction task, feature sharing is achieved and mutual promotion is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart of a medical image segmentation method according to an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the 3D U-Net network structure of an embodiment of the present application;

[0039] Figure 3 This is a schematic diagram of the structure of a medical image segmentation system according to an embodiment of the present application;

[0040] Figure 4 This is a schematic diagram of the terminal structure of an embodiment of the present application;

[0041] Figure 5 A schematic diagram of the structure of a storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] See also Figure 1 , is a flow chart of the medical image segmentation method of the embodiment of the present application. The medical image segmentation method of the embodiment of the present application comprises the following steps:

[0044] S10: Acquire a certain amount of multimodal medical image sample data, and generate Mask (label) data of the multimodal medical image sample data;

[0045] In this step, the obtained multimodal medical image sample data includes FLAIR, T1, T2 and T1c four modality images of each case and clinical information of the case corresponding to the multimodal medical image. The clinical information of the case includes information such as survival period and survival status. The size of the four modality data is the same.

[0046] S20: preprocessing the multimodal medical image sample data and the Mask data to generate a medical image dataset for model training, and grouping the medical image dataset according to a set ratio to obtain a training set, a validation set, and a test set;

[0047] In this step, the preprocessing process of the multimodal medical image sample data specifically includes:

[0048] S21: cropping the original medical image and the corresponding Mask data according to the center point of each medical image;

[0049] Among them, since the original multimodal medical image is too large and the size is not uniform, there are a large number of background areas, and the tumor area is generally located in the middle area of ​​the medical image. Therefore, it is necessary to crop the central area of ​​the multimodal medical image so that all multimodal medical images can correspond one to one after cropping. The specific cropping method is: find the center point of each medical image, and expand the area of ​​a set size outward from this center point, cut off the part outside the area, and obtain the cropped medical image. In the embodiment of the present application, the size of the cropping area is set to 96*128*128, which can be set according to the actual operation.

[0050] S22: normalize the cropped medical images using the min-max algorithm to compress the pixel values ​​of all medical images to between 0 and 1;

[0051] Among them, due to the different imaging methods of the four modal data of FLAIR, T1, T2, and T1c, there are differences in the contrast of the image. Therefore, the embodiment of the present application uses a min-max algorithm to normalize the image pixel values ​​of different modalities to between 0 and 1, and the cropped Mask data does not need to be normalized.

[0052] S23: performing splicing operations on the normalized medical images of the four modalities and the cropped Mask data respectively to obtain a pre-processed medical image dataset;

[0053] Among them, the size of the spliced ​​medical image data is 96*128*128*4.

[0054] After the medical image data is preprocessed in the embodiment of the present application, the data set needs to be grouped according to the set ratio. Preferably, in the present embodiment, the test set is divided into 25% and the other 75% is used as the training set and the validation set, and a 10-fold cross-validation algorithm is used, that is, 10% of the training set data is taken as the validation set in each round, so that as much data as possible can be used for model training while evaluating the model training effect.

[0055] S30: construct a weak-semi-supervised model, and input the training set into the weak-semi-supervised model for iterative training to obtain a trained image segmentation model;

[0056] In this step, the embodiment of the present application uses the pytorch framework to build a weak-semi-supervised model. The weak-semi-supervised model is a 3D U-Net network transformed from a 2D U-Net network, that is, all 2D operations such as 2D convolution and 2D pooling in the 2D U-Net network are replaced by 3D operations. The 3D U-Net network structure is as follows Figure 2 As shown in the figure, the 3D U-Net network includes two branch structures: a segmentation branch for performing segmentation tasks and a survival prediction branch for performing survival prediction tasks. It also includes a residual module, a Transformer module, an Adapter module, an information fusion module, and a Survival Predict module. Specifically:

[0057] The training set is input into the segmentation branch and the survival prediction branch respectively, and the output of the segmentation branch is the features extracted from the medical image data; the survival prediction branch is implemented by a fully connected neural network, and the output value is the risk value corresponding to each case. Due to the differences in the distribution of features extracted by different segmentation tasks, if they are directly fused, there may be cancellation. In the embodiment of the present application, the features extracted by the segmentation branch are converted into data distribution through the Adapter module, so that the features obtained by the segmentation branch are integrated into the survival prediction branch, so that the survival prediction task can also use the features learned in the segmentation task. The data distribution conversion process of the Adapter module is specifically as follows: first, the mean and std of the features in the survival prediction task are calculated using the feature distribution of the survival prediction branch, and then data transformation is performed on the segmentation branch, and the mean is subtracted from the features in the segmentation task and then divided by the std, so that the segmentation task has the same mean and std as the survival prediction task. The features obtained by the segmentation branch are converted into the same distribution as the features obtained by the survival prediction branch through the Adapter module, which can effectively avoid information loss between features and provide information that other tasks cannot provide. Among them, the loss function of the segmentation branch is the Dice coefficient and BCE (Binary Cross Entropy) Loss, and the loss function of the survival prediction branch is Negative Log Likelihood (negative log likelihood).

[0058] The input and output dimensions of the Transformer module are 201 respectively, the number of internal network layers is 4, and n_head = 1. The features extracted by the segmentation branch are flattened to become one-dimensional features, and the one-dimensional features of each channel are concatenated and input into the Transformer module. The Transformer module adopts the idea based on residual connection. The residual module is used to add the input features to the data before input, and then reshape it to the shape before input, so as to further explore the internal connection between different features. Among them, each residual module is composed of 2 convolutional layers, that is, 2 convolution operations are performed and then residual connections are made. After each convolution, LeakyRelu is used for nonlinear mapping, and GroupNorm is used for normalization; the residual formula of the residual module is: x l+1 =x l +F(x l ).

[0059] The information fusion module is used to merge the features output by the segmentation branch and the lifetime prediction branch by means of convolution, and to perform a re-convolution operation on the merged features. The output result after convolution is the fused feature, thereby migrating the features learned in the lifetime prediction task to the segmentation task.

[0060] In order to fully learn the lifetime information and Mask information, the 3D U-Net network adopts a double downsampling method. The input data is input into two encoders (i.e., the segmentation branch and the lifetime prediction branch) respectively, and then the output results of the two encoders are fused. Finally, the fused image is introduced into the decoder for upsampling. Specifically, the 3D U-Net network training process of the embodiment of the present application includes: inputting the training set data into the segmentation branch and the life prediction branch for downsampling processing, performing a total of 3 downsamplings, and temporarily retaining the results of each downsampling; wherein the features obtained by the segmentation branch downsampling are first flattened to become one-dimensional features, and then input into the Transformer module; the Transformer module adopts the idea based on residual connection, adds the input features to the data before input, and then reshapes to the shape before input, and at the same time leads the features obtained by the Transformer module to the life prediction branch; the life prediction branch transforms the distribution of features through the Adapter module, and fuses the features output by the segmentation branch and the life prediction branch through the information fusion module, and then passes through the fully connected layer to obtain the risk value of the life prediction, binarizes the risk value, and then uses the binarization result as an influencing factor to calculate its impact on the life. After obtaining the reshaped feature map, the segmentation branch is upsampled 3 times to restore the feature map to the size of the initial input image, and then binarized to obtain the output result of the segmentation task.

[0061] The embodiment of the present application uses the Teacher-Student training mode to train the built 3D U-Net network. The Teacher-Student training mode is specifically: the pseudo labels generated by the unlabeled image data are added to the training set. In each round of training, if the current training effect is better than the previous round, the Student model is updated with the Teacher model, otherwise the training continues. If the Student model fails to be updated after more than 20 rounds of training, it is considered that the model has converged and the training can be terminated. The C-Index evaluation index and the Dice coefficient are used to evaluate the training effect of the current model.

[0062] Based on the above, the 3D U-Net network of the embodiment of the present application adopts a semi-supervised segmentation method, which can effectively reduce the required labeled data; combines the segmentation task with the survival period prediction task to achieve feature sharing and mutual promotion; combines with weak supervision methods, uses high-level semantics such as survival period as a weak supervision source, and uses the Transformer module to mine the correlation between features, thereby focusing on the tumor area and further improving the segmentation accuracy.

[0063] S40: input the validation data set into the trained image segmentation model for model evaluation;

[0064] In this step, after the model is built and trained, in order to further verify the segmentation effect of the image segmentation model, the P value and KM curve are calculated to evaluate the model performance. The results show that the image segmentation model of the embodiment of the present application has better segmentation effect than weak supervision or semi-supervision alone.

[0065] S50: input the test data set into the image segmentation model to perform model testing;

[0066] In this step, the test set data is input into the trained image segmentation model, the segmentation result is compared with the real manually annotated Mask, and the quality of the final model is judged by calculating the Dice loss.

[0067] S60: Input the medical image to be segmented into the trained image segmentation model, and output the segmentation result through the image segmentation model.

[0068] Based on the above, the medical image segmentation method of the embodiment of the present application adopts a semi-supervised segmentation method, which does not rely on too much labeled data; combined with a weak supervision method, it uses high-level semantics such as survival period as a weak supervision source, and uses the Transformer module to mine the correlation between features, thereby focusing on the tumor area and further improving the segmentation accuracy. By combining the segmentation task with the survival period prediction task, feature sharing is achieved and mutual promotion is achieved.

[0069] See also Figure 3 , is a schematic diagram of the structure of a medical image segmentation system according to an embodiment of the present application. The medical image segmentation system 40 according to an embodiment of the present application comprises:

[0070] Data acquisition module 41: used to acquire medical image sample data, where the medical image sample data includes multimodal medical images and clinical information of cases corresponding to the multimodal medical images;

[0071] Model training module 42: used to construct a weak-semi-supervised model, the weak-semi-supervised model includes a segmentation branch for performing a segmentation task and a survival prediction branch for performing a survival prediction task, the medical image sample data is input into the segmentation branch and the survival prediction branch respectively, the features extracted by the segmentation branch and the survival prediction branch are fused and iteratively trained to obtain a trained image segmentation model;

[0072] Image segmentation module 43: used for inputting the medical image to be segmented into the trained image segmentation model for image segmentation.

[0073] See also Figure 4, is a schematic diagram of the terminal structure of an embodiment of the present application. The terminal 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0074] The memory 52 stores program instructions for implementing the above-mentioned medical image segmentation method.

[0075] The processor 51 is used to execute program instructions stored in the memory 52 to control medical image segmentation.

[0076] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having the ability to process signals. The processor 51 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0077] See also Figure 5 , which is a schematic diagram of the structure of the storage medium of the embodiment of the present application. The storage medium of the embodiment of the present application stores a program file 61 that can implement all the above methods, wherein the program file 61 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or terminal devices such as a computer, a server, a mobile phone, and a tablet.

[0078] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present application. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined in this application can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown in the present application, but will conform to the widest range consistent with the principles and novel features disclosed in the present application.

Claims

1. A medical image segmentation method, It is characterized in that include: Acquire medical image sample data, where the medical image sample data includes multimodal medical images and clinical information of cases corresponding to the multimodal medical images; Constructing a weak-semi-supervised model, the weak-semi-supervised model includes a segmentation branch for performing a segmentation task and a life span prediction branch for performing a life span prediction task, inputting the medical image sample data into the segmentation branch and the life span prediction branch respectively, fusing the features extracted by the segmentation branch and the life span prediction branch and iteratively training them to obtain a trained image segmentation model; Input the medical image to be segmented into the trained image segmentation model for image segmentation; The weakly-semi-supervised model is constructed as a 3D U-Net network, and the training process of the 3D U-Net network includes: The training set data is respectively input into the segmentation branch and the life span prediction branch for downsampling processing, and the features obtained by the downsampling of the segmentation branch are converted into one-dimensional features through a flatten operation and then input into the Transformer module; The Transformer module adopts the idea based on residual connection, adds the input features to the data before input, and then reshapes it to the shape before input, and at the same time leads the features obtained by the Transformer module to the lifetime prediction branch; the lifetime prediction branch transforms the distribution of features through the Adapter module, and fuses the features output by the segmentation branch and the lifetime prediction branch through the information fusion module, and then passes through the fully connected layer to obtain the risk value of the lifetime prediction; after obtaining the reshaped feature map, the segmentation branch restores the feature map to the size of the initial input image through upsampling, and then undergoes binarization processing to obtain the output result of the segmentation task.

2. The medical image segmentation method according to claim 1, It is characterized in that The obtaining of medical image sample data comprises: The multimodal medical images are four modality images of FLAIR, T1, T2 and T1 c for each case; The clinical information includes the survival period and survival status of the case.

3. The medical image segmentation method according to claim 2, It is characterized in that The obtaining of medical image sample data specifically includes: Generating Mask data of the multimodal medical image sample data; Preprocessing the multimodal medical image sample data and the Mask data to generate a medical image data set for model training; The medical image dataset is grouped according to a set ratio to obtain a training set, a validation set, and a test set.

4. The medical image segmentation method according to claim 3, It is characterized in that The preprocessing of the multimodal medical image sample data and the Mask data is specifically as follows: The multimodal medical image and the corresponding Mask data are cropped; the cropping method is specifically: obtaining the center point of each multimodal medical image, expanding an area of ​​a set size outward from the center point, and cropping the part outside the area to obtain the cropped medical image and Mask data; Normalizing the cropped medical image using a min-max algorithm; The normalized medical images of the four modalities and the cropped Mask data are spliced ​​respectively to obtain a preprocessed medical image data set.

5. The medical image segmentation method according to claim 3, It is characterized in that The grouping of the medical image dataset according to the set ratio is specifically as follows: A 10-fold cross-validation algorithm was used, and 10% of the training set data was taken as the validation set in each round.

6. The medical image segmentation method according to claim 5, It is characterized in that The training mode of the 3D U-Net network is specifically: The 3D U-Net network is trained using the Teacher-Student training mode; pseudo labels generated from unlabeled medical images are added to the training set. In each round of training, if the current training effect is better than the previous round, the Student model is updated with the Teacher model, otherwise the training continues. If the Student model cannot be updated after the training times exceed the set number, the model is considered to have converged and the model training is terminated.

7. A medical image segmentation system using the medical image segmentation method according to claim 1, It is characterized in that include: Data acquisition module: used to acquire medical image sample data, wherein the medical image sample data includes multimodal medical images and clinical information of cases corresponding to the multimodal medical images; Model training module: used to construct a weak-semi-supervised model, the weak-semi-supervised model includes a segmentation branch for performing a segmentation task and a life span prediction branch for performing a life span prediction task, the medical image sample data is input into the segmentation branch and the life span prediction branch respectively, the features extracted by the segmentation branch and the life span prediction branch are fused and iteratively trained to obtain a trained image segmentation model; Image segmentation module: used to input the medical image to be segmented into the trained image segmentation model for image segmentation.

8. A terminal, It is characterized in that The terminal includes a processor and a memory coupled to the processor, wherein: The memory stores program instructions for implementing the medical image segmentation method according to any one of claims 1 to 6; The processor is used to execute the program instructions stored in the memory to control medical image segmentation.

9. A storage medium, It is characterized in that The device stores program instructions executable by a processor, wherein the program instructions are used to execute the medical image segmentation method according to any one of claims 1 to 6.

Images