An artificial intelligence-based self-supervised multimodal medical image segmentation method

Abstract

The present invention discloses a self-supervised multimodal medical image segmentation method based on artificial intelligence. First, all multimodal images are obtained as training sets, a self-supervised modal mask network is trained, and the network weights are used to initialize the segmentation network parameters to train the multimodal medical image segmentation network. The present invention uses a pre-training method of a self-supervised modal mask to fully utilize data to train the network without labels, strengthen the feature extraction capability of the network, transfer it to downstream tasks, and initialize the network parameters, thereby improving the model segmentation accuracy; fully utilizing unlabeled data, reducing the human resources consumed by annotating image target category labels; being applicable to a few sample segmentation scenarios, without the need for additional optimization during application, being able to balance the high mask rate of the mask image and the effective information retention rate in the medical image, being able to make the training more stable, and the obtained pre-trained weights performing better in downstream tasks.

Description

[0001] Field Technology

[0002] The present invention belongs to the field of artificial intelligence, and in particular relates to an artificial intelligence-based self-supervised multimodal medical image segmentation method, which is suitable for multimodal medical image segmentation. Background Art

[0003] Due to the differences in imaging technologies, medical devices can capture different organ and tissue characteristics, but can only obtain single-modality medical images, which cannot provide comprehensive and sufficient information. For example, computed tomography (CT) can capture dense structures and bones, while magnetic resonance imaging (MRI) can detect soft tissues. Even though CT and MRI are both anatomical images, and positron emission tomography (PET) and single photon emission computed tomography (SPECT) are both functional images, they display different information. CT is suitable for some bone tissue, lung, and coronary artery-related examinations, while MRI is suitable for some tissues with high water content and can detect subtle changes in tissues earlier. PET mainly displays tumor function and metabolic information, while SPECT mainly provides blood flow information for organs and tissues. Multimodal image segmentation allows the simultaneous extraction of complementary data from multiple modalities. However, direct feature extraction from multimodal images is inconvenient. Due to the limitations of single-modality medical images, multimodal medical image segmentation is a very necessary research field.

[0004] In recent years, the self-supervised deep learning method based on mask images has shown great potential in the feature extraction of unlabeled natural images. It does not require manual annotation, and the extracted features have strong generalization ability and are suitable for a variety of downstream tasks. The self-supervised learning algorithm based on mask images applies a high mask rate mask to the input image itself [Kaiming He, et al, CVPR 2022.], with the goal of restoring the pixels covered by the mask, and autonomously trains the encoder of the network. However, most of the existing self-supervised models based on mask images are applied to natural images [Hansen S, et al. MIA, 2022.], and no one has proposed it in medical image processing. Compared with natural images, the background information of medical images is single and accounts for a large proportion, and the self-supervised model based on masks can only work best at a high mask rate. This image mask self-supervised strategy is directly applied to medical images, which is easy to retain some invalid background information and cover up more foreground information that is easy to restore the image. Chen et al. deleted the decoder network module of the traditional mask image-based self-supervised network [Zekai Chen, WACV 2023.], allowing the encoder to learn features instead of the decoder, but there was no improvement in the mask strategy, and it still could not solve the problem of insufficient retention of effective information in medical images at high mask rates. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention proposes a self-supervised multimodal medical image segmentation method based on artificial intelligence.

[0006] The above-mentioned purpose of the present invention is achieved by the following technologies:

[0007] A self-supervised multimodal medical image segmentation method based on artificial intelligence comprises the following steps:

[0008] Step 1: Acquire multimodal images of multiple subjects, where images of the same subject in different imaging modes or 3D images of MRI under multiple sequences are taken as a group of multimodal images, i.e., a sample group. Select part of the sample group and annotate the target category for each modality image therein. The target category annotation of each modality image is used as the label corresponding to the target category of the corresponding modality image.

[0009] Step 2: The modal images of all sample groups excluding the labels of each modal image are used as the self-supervised training set;

[0010] Step 3: Construct a self-supervised modal mask network. The self-supervised modal mask network includes a mask network encoder, a mask network decoder and a self-supervised loss function L Self ,in,

[0011] The mask network encoder consists of 1 input layer and Transformer module;

[0012] Step 4: Input the multimodal images of the self-supervised training set into the self-supervised modality mask network for end-to-end pre-training, and save the pre-trained network parameters of the self-supervised modality mask network;

[0013] Step 5: Use the multimodal images with labels added in step 1 and the labels corresponding to the multimodal images as the segmentation training set;

[0014] Step 6: Construct a multimodal medical image segmentation network. The multimodal medical image segmentation network includes a segmentation network encoder and a segmentation network decoder, wherein:

[0015] The segmentation network encoder includes an input layer and a Transformer module, and the structure of the Transformer module in the segmentation network encoder is consistent with the structure of the Transformer module in the mask network encoder;

[0016] Step 7: Set the segmentation loss function L Seg ;

[0017] Step 8: Migrate the parameters of the Transformer module in the self-supervised modal mask network pre-trained in step 4 to the Transformer module of the segmentation network encoder for initialization;

[0018] Step 9: According to the segmentation loss function set in step 7, the multimodal medical image segmentation network with self-supervised modality mask pre-training initialization constructed in step 8 is end-to-end trained using the segmentation training set generated in step 5, and the parameters of the multimodal medical image segmentation network are saved;

[0019] Step 10: Input the unlabeled multimodal medical image to be segmented into the multimodal medical image segmentation network trained in step 9 to obtain the label of the target category segmentation.

[0020] As mentioned above, the input layer in the mask network encoder includes a random seed and a modality mask. The input layer randomly masks part of the modality images in the sample group, saves the pixels of the channel where the masked modality images are located, and saves an additional array named target, and replaces all the original positions of the masked modality images with 0.

[0021] As mentioned above, the Transformer modules of the mask network encoder and the segmentation network encoder both include an image embedding layer and multiple Transformer encoding layers, and the Transformer encoding layers include, in order, a first-level batch normalization layer, a multi-head attention layer, a first residual connection, a second-level batch normalization layer, a fully connected layer, and a second residual connection.

[0022] As mentioned above, in step 3, the mask network decoder has an output layer, which includes 1 fully connected layer and a matrix transformation function.

[0023] As mentioned above, the self-supervised loss function L in step 3 Self for:

[0024]

[0025] Among them, L 2 is the mean square error operation, X is the multimodal image, x is the image that is masked during the self-supervision process, and x ijk is the data of the jth row and kth column of the i-th slice of the masked image, S represents the self-supervised modal mask network, S(X) represents the output of the self-supervised modal mask network, S(X) ijk is the data of the jth row and kth column of the i-th slice output by the self-supervised modal mask network. L, M, and N represent the number of slices of the segmented image finally output by the self-supervised modal mask network, the number of rows and columns of each slice, and λ is a hyperparameter.

[0026] As described above, the input layer of the segmentation network encoder includes a batch normalization layer.

[0027] As described above, the segmentation network decoder consists of multiple upsampling layers and an output layer:

[0028] The number of upsampling layers is consistent with the number of Transformer encoding layers;

[0029] The first upsampling layer includes an upsampling module, a convolution layer, a batch normalization layer, and an activation layer; the other upsampling layers include a fusion module, an upsampling module, a convolution layer, a batch normalization layer, and an activation layer connected in sequence;

[0030] The input of the first upsampling layer in the segmentation network encoder includes the feature map output by the last Transformer coding layer of the Transformer module in the segmentation network encoder; the input of the kth upsampling layer, that is, the input of the fusion module in the kth upsampling layer, includes the feature map output by the kth last Transformer coding layer of the Transformer module in the segmentation network encoder and the feature map output by the k-1th upsampling layer, and the fusion module of the kth upsampling layer stacks the feature map output by the kth last Transformer coding layer of the Transformer module in the segmentation network encoder and the feature map output by the k-1 upsampling layers according to channels, and then reduces the number of channels by convolution; wherein k is an integer greater than 1 and less than or equal to the number of upsampling layers.

[0031] As described above, the output layer in the segmentation network decoder includes a fusion module, a convolution layer and an activation layer connected in sequence, and the input of the fusion module of the output layer in the segmentation network decoder includes the feature map output by the last upsampling layer of the segmentation network decoder and the feature map output by the input layer of the segmentation network encoder.

[0032] As mentioned above, the segmentation loss function L Seg for:

[0033]

[0034] Where N is the total number of sample groups in the segmentation training set, i represents the sample group number, i≤N, C is the total number of segmentation label categories, c represents the sample label category number, c≤C, is the c-th class label of the i-th sample group, It is the predicted result of the multimodal medical image segmentation network output corresponding to the c-th class label of the i-th sample group.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention uses a self-supervised modal mask pre-training method to fully utilize data to train the network without labels, strengthen the network's feature extraction capability, transfer it to downstream tasks, initialize the network parameters, and improve the model segmentation accuracy. Compared with the existing medical image segmentation method based on deep learning, the present invention can make full use of unlabeled data, reduce the human resources consumed by annotating image target category labels, and can be applied to few-sample segmentation scenarios without additional optimization, which has practical application value.

[0037] 2. Compared with the traditional self-supervised pre-training model, the method of the present invention can balance the high mask rate of mask images and the effective information retention rate in medical images, making the training more stable, and the obtained pre-training weights perform better in downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the process of the present invention;

[0039] Figure 2 A schematic diagram of the overall structure of the invented self-supervised modality mask network and multimodal medical image segmentation network;

[0040] Figure 3 Schematic diagrams of the input layer structure of the mask network encoder and the input layer structure of the segmentation network encoder of the present invention, wherein (a) is a schematic diagram of the input layer structure of the mask network encoder; (b) is a schematic diagram of the input layer structure of the segmentation network encoder;

[0041] Figure 4A schematic diagram of the Transformer module structure of the self-supervised modality mask network and the multimodal medical image segmentation network of the present invention;

[0042] Figure 5 Schematic diagram of the structure of the kth (k is greater than 1) upsampling layer in the multimodal medical image segmentation network of the present invention;

[0043] Figure 6 Schematic diagrams of the output layer structure of the mask network decoder and the output layer structure of the segmentation network decoder of the present invention, wherein (a) is a schematic diagram of the output layer structure of the mask network decoder; (b) is a schematic diagram of the output layer structure of the segmentation network decoder;

[0044] Figure 7 The performance comparison of the present invention and other methods on a multimodal brain target dataset is shown in Figure 1. Among them, GroundTruth is the label of the 3D image of MRI, and Layer Mask is the present invention.

[0045] Table 1 is a comparison table of the weights of the self-supervised training of the present invention and other methods. In the table: UNETR is a segmentation method that does not initialize the pre-training weights, that is, fills all 0s; 3D MAE is a self-supervised pre-training method widely used in natural images to obtain weights for initialization training of the segmentation network; SimMIM is a self-supervised pre-training method that removes the encoder in 3D MAE to obtain weights for initialization training of the segmentation network. DETAILED DESCRIPTION

[0046] In order to facilitate those skilled in the art to understand and implement the present invention, the following embodiments, Figure 1-Figure 3 , Table 1 further describes the present invention in detail. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0047] Example 1

[0048] An artificial intelligence-based self-supervised multimodal medical image segmentation method, such as Figure 1 As shown, the specific steps include:

[0049] Step 1, obtain multimodal images of multiple subjects, where images of the same subject in different imaging modes, or 3D images of MRI under multiple sequences are taken as a group of multimodal images, i.e., a sample group. Select part of the sample group and annotate the target category for each modality image therein, and the annotation of the target category of each modality image is used as the label corresponding to the target category of the corresponding modality image. In this embodiment, 3D images of four modalities of MRI of 498 subjects are obtained from the multimodal brain target segmentation public dataset BraTS2020, and the four sequences corresponding to the 3D images of the four modalities of MRI are FLAIR, T1, T1gd, and T2. The 3D images of the four modalities of MRI of 369 subjects are annotated with three target categories as the labels of the corresponding MRI 3D images, namely, the label of MRI 3D image target category 1, the label of MRI 3D image target category 2, and the label of MRI 3D image target category 3.

[0050] Step 2: Use the modal images of all sample groups in step 1 (excluding the labels of each modal image) as a self-supervised training set. In this embodiment, 3D images of four modal MRIs of 498 subjects (excluding the labels of 3D MRI images) are used as a self-supervised training set, and the image size of each 3D MRI image is 155×240×240.

[0051] Step 3: Construct a self-supervised modal mask network. The self-supervised modal mask network includes a mask network encoder and a mask network decoder, such as Figure 2 In this embodiment, the mask network encoder includes an input layer and a Transformer module, and the mask network decoder includes an output layer.

[0052] In this embodiment, the mask network encoder includes an input layer and a Transformer module in order, the input layer includes a random seed and a modality mask, the input layer uses a random mask to mask one of the 3D images of the four modalities of MRI, and the pixels of the channel where the 3D image of the masked MRI is located are saved in an additional array named target, and all the original positions of the masked 3D image of the MRI are replaced with 0. The Transformer module includes an image embedding layer and multiple Transformer encoding layers. In this embodiment, the Transformer module includes 1 image embedding layer and 4 sequentially connected Transformer encoding layers. The convolution kernel size of the image embedding layer is 16×16×16, the stride is 16, the input channels are 4, the output channels are 768, the input feature map size is 128×240×240, and the output feature map size is 1800×768. The four Transformer encoding layers all include a first-level batch normalization layer, a multi-head attention layer, a first residual connection, a second-level batch normalization layer, a fully connected layer, and a second residual connection in sequence; the input feature map size of each first-level batch normalization layer and second-level batch normalization layer is 1800×768, and the output feature map size is 1800×768.

[0053] The output layer in the mask network decoder includes a fully connected layer and a matrix transformation function. The fully connected layer input feature map size is 1800×768, and the output feature map size is 1800×4096. The matrix transformation function uses the view function in the pytorch library, the input feature map size is 1800×4096, and the output feature map size is 128×240×240.

[0054] The self-supervised loss function L Self Set to:

[0055]

[0056] Among them, L 2 is the mean square error operation, X is a multimodal image, which is four MRI 3D images in this embodiment, x is the modality image masked in the self-supervision process (i.e., the MRI 3D image in this embodiment), x ijk is the data of the i-th slice, j-th row, and k-th column of the masked modality image, S represents the self-supervised modality mask network, S(X) represents the output of the self-supervised modality mask network, and S(X) ijk is the data of the jth row and kth column of the i-th slice output by the self-supervised modal mask network, L, M, and N respectively represent the number of slices of the segmented image finally output by the self-supervised modal mask network, the number of rows and columns of each slice, and λ is a hyperparameter. In this embodiment, L=128, M=N=240, and λ=2.

[0057] Step 4. Input the multimodal images of the self-supervised training set generated in step 2 into the self-supervised modality mask network for end-to-end self-supervised training. In this embodiment, 3D images of four modal MRIs are input into the self-supervised modality mask network for end-to-end self-supervised training. The learning rate of the network is initialized to 0.01, the batch size is set to 32, and the self-supervised modality mask network is trained on the PyTorch platform based on minimizing the self-supervised loss function value. The training is stopped after the total number of iterations is 1000, and the pre-trained network parameters of the self-supervised modality mask network are saved.

[0058] Step 5, obtain multiple groups of multimodal images with labels added in step 1 and labels corresponding to the multimodal images (i.e., 3D images of the four modal MRIs in this embodiment and labels of the corresponding MRI 3D images), and use them as segmentation training sets, that is, the segmentation training set includes multiple sample groups and labels of the modal images in the sample groups. In this embodiment, the multimodal images are images under four magnetic resonance sequences of FLAIR, T1, T1gd, and T2 of the multimodal brain target segmentation data set, and three labels of the corresponding MRI 3D images. There are 369 subjects in total, and the image and label size of each MRI is 155×240×240.

[0059] Step 6: Construct a multimodal medical image segmentation network. The multimodal medical image segmentation network includes a segmentation network encoder and a segmentation network decoder, such as Figure 2 As shown in FIG. 1 , in this embodiment, the segmentation network encoder includes an input layer and a Transformer module in sequence. The segmentation network decoder includes 4 upsampling layers and 1 output layer in sequence.

[0060] In this embodiment, the segmentation network encoder includes an input layer and a Transformer module in order, and the input layer includes a batch normalization layer. Figure 2 As shown in the figure, the Transformer module includes 1 image embedding layer and 4 Transformer encoding layers. The image embedding layer is composed of a convolution kernel size of 16×16×16, a stride of 16, 4 input channels, 768 output channels, an input feature map size of 128×240×240, and an output feature map size of 1800×768. The 4 Transformer encoding layers include a first-level batch normalization layer, a multi-head attention layer, a first residual connection, a second-level batch normalization layer, a fully connected layer, and a second residual connection in order; the input feature map size of each first-level batch normalization layer and second-level batch normalization layer is 1800×768.

[0061] The segmentation network decoder includes multiple upsampling layers and an output layer, and the number of upsampling layers is consistent with the number of Transformer encoding layers. In this embodiment, the segmentation network decoder includes four upsampling layers and one output layer in order, where:

[0062] The first upsampling layer includes an upsampling module, a convolution layer, a batch normalization layer, and an activation layer. The feature map output by the last Transformer encoding layer of the segmentation network encoder is used as the input of the first upsampling layer of the segmentation network encoder. In the first upsampling layer: the upsampling method of the upsampling module is bilinear interpolation, the output feature map size of the upsampling module is 8 times the size of the input feature map, the input feature map size is 8×15×15, and the output feature map size is 16×30×30. The convolution kernel size in the convolution layer of the first upsampling layer is 3×3×3, the number of input channels is 768, and the number of output channels is 512. The input feature map size of the first upsampling layer is 8×15×15, and the output feature map size is 16×30×30.

[0063] The second upsampling layer includes a fusion module, an upsampling module, a convolution layer, a batch normalization layer, and an activation layer. The feature map output by the third Transformer encoding layer of the segmentation network encoder and the feature map output by the first upsampling layer of the segmentation network decoder are used as the input of the fusion module in the second upsampling layer of the segmentation network decoder. The fusion module stacks the feature map output by the third Transformer encoding layer and the feature map output by the first upsampling layer according to the channel, and then reduces the number of channels to half through 1×1×1 convolution. In the upsampling layer: the number of input channels of the fusion module is 512, and the number of output channels is 256. The upsampling method of the upsampling module is bilinear interpolation. The output feature map size of the upsampling module is 8 times the size of the input feature map. The input feature map size is 16×30×30, and the output feature map size is 32×60×60. The convolution kernel size in the convolution layer is 3×3×3, the number of input channels is 512, and the number of output channels is 256. The input feature map size of the second upsampling layer is 16×30×30, and the output feature map size is 32×60×60.

[0064] The third upsampling layer includes a fusion module, an upsampling module, a convolution layer, a batch normalization layer, and an activation layer. The feature map output by the second Transformer encoding layer of the segmentation network encoder and the feature map output by the second upsampling layer of the segmentation network decoder are used as the input of the fusion module in the third upsampling layer of the segmentation network decoder. The fusion module stacks the feature map output by the second Transformer encoding layer and the feature map output by the second upsampling layer according to the channel, and then reduces the number of channels to half through 1×1×1 convolution. The third upsampling layer Middle: The number of input channels of the fusion module is 256, and the number of output channels is 128. The upsampling method of the upsampling module is bilinear interpolation. The output feature map size of the upsampling module is 8 times the size of the input feature map. The input feature map size is 32×60×60, and the output feature map size is 64×120×120. The convolution kernel size in the convolution layer of the third upsampling layer is 3×3×3, the number of input channels is 256, and the number of output channels is 128. The input feature map size of the third upsampling layer is 32×60×60, and the output feature map size is 64×120×120.

[0065] The fourth upsampling layer includes a fusion module, an upsampling module, a convolution layer, a batch normalization layer, and an activation layer. The feature map output by the first Transformer encoding layer of the segmentation network encoder and the feature map output by the third upsampling layer of the segmentation network decoder are used as the input of the fusion module in the fourth upsampling layer of the segmentation network decoder. The fusion module stacks the feature map output by the first Transformer encoding layer and the feature map output by the third upsampling layer according to the channel, and then reduces the number of channels to half through 1×1×1 convolution. In the fourth upsampling layer: The number of input channels of the fusion module is 128, and the number of output channels is 64. The upsampling method of the upsampling module is bilinear interpolation. The output feature map size of the upsampling module is 8 times the size of the input feature map. The input feature map size is 64×120×120, and the output feature map size is 128×240×240. The convolution kernel size in the convolution layer of the fourth upsampling layer is 3×3×3, the number of input channels is 128, and the number of output channels is 64. The input feature map size of the fourth upsampling layer is 64×120×120, and the output feature map size is 128×240×240.

[0066] And so on, if there are more upsampling layers, such as Figure 2As shown in , the total number of upsampling layers of the segmentation network decoder is consistent with the total number of Transformer coding layers of the segmentation network encoder. For the kth upsampling layer of the segmentation network decoder, the feature map output by the kth last Transformer coding layer of the Transformer module in the segmentation network encoder and the feature map output by the k-1th upsampling layer of the segmentation network decoder are used as the input of the kth upsampling layer of the segmentation network decoder, that is, the input of the fusion module in the kth upsampling layer. The fusion module in the kth upsampling layer stacks the feature map output by the kth last Transformer coding layer of the Transformer module in the segmentation network encoder and the feature map output by the k-1th upsampling layer according to channels, and then reduces the number of channels by convolution. Where k is an integer greater than 1 and less than or equal to the total number of upsampling layers M.

[0067] The output layer in the segmentation network decoder includes a fusion module, a convolution layer and an activation layer connected in sequence. The input of the fusion module of the output layer of the segmentation network decoder includes the feature map output by the last upsampling layer of the segmentation network decoder and the feature map output by the input layer of the segmentation network encoder. In this embodiment, the feature map output by the input layer of the segmentation network encoder and the feature map output by the fourth upsampling layer are used as the input of the fusion module in the output layer of the segmentation network decoder. In the output layer of the segmentation network decoder: the fusion module stacks the feature map output by the input layer and the feature map output by the fourth upsampling layer according to channels, and then reduces the number of channels to half through 1×1 convolution. The number of input channels of the fusion module is 128, and the number of output channels is 64. The convolution kernel size in the convolution layer is 7×7×7, the number of input channels is 64, and the number of output channels is 3. The feature map size of the output layer input is 128×240×240, and the size of the output feature map is 128×240×240.

[0068] The output of the Transformer encoding layer is used as part of the input of the upsampling layer, which is conducive to the contraction constrained autoencoder denoising network to make full use of the image context information, better restore the image details, and accelerate the network convergence.

[0069] Except for the first upsampling layer, each upsampling layer has a fusion module; except for the output layer, each upsampling layer has a batch normalization layer after the convolution layer, and the batch normalization layer is followed by an activation layer. The activation function of the activation layer in each upsampling layer is the LeakReLU function; the activation function of the activation layer in the output layer is the tanh function.

[0070] Step 7: Set the segmentation loss function L Seg :

[0071]

[0072] Where N is the total number of sample groups in the segmentation training set, i represents the sample group number, i≤N, C is the total number of label categories, c represents the label category number, c≤C, and C=3 in this embodiment. y is the label of each modality image, that is, the label of the 3D image of MRI in this embodiment, is the label of the cth target category of the ith sample group, is the prediction of the output of the multimodal medical image segmentation network, It is the predicted result of the multimodal medical image segmentation network output corresponding to the c-th target category of the i-th sample group.

[0073] Step 8. Transfer the Transformer module parameters in the self-supervised modality mask network pre-trained in step 4 to the Transformer module of the segmentation network encoder of the multimodal medical image segmentation network for initialization. Since the Transformer module of the encoder of the self-supervised modality mask and the Transformer module of the multimodal medical image segmentation network have the same structure, they can be directly transferred.

[0074] Step 9. According to the segmentation loss function set in step 7, the multimodal medical image segmentation network with self-supervised modality mask pre-training initialization constructed in step 8 is end-to-end trained using the segmentation training set generated in step 5. The learning rate of the network is initialized to 0.001, the batch size is set to 8, and the network is trained on the PyTorch platform using the ADAM optimizer. After the total number of iterations reaches 1000, the training is stopped and the parameters of the multimodal medical image segmentation network are saved.

[0075] Step 10: Place the unlabeled multimodal medical image to be segmented into the multimodal medical image segmentation network trained in step 9 to obtain the label of the target category segmentation.

[0076] Compare the segmentation result obtained in step 10 with the label obtained in step 5, as shown in Figure 7 As shown in the figure, the first column is the FLAIR sequence image, the second column is the T1gd sequence imaging image, and the third column is the label of the 3D image of the label MRI. The medium gray area is the target category 1, the dark gray area is the target category 2, and the light gray area is the target category 3. The Dice coefficients of the three categories are calculated respectively, as shown in Table 1. It can be seen from the result figure that the self-supervised multimodal medical image segmentation method based on artificial intelligence provided by the present invention can effectively segment multiple target category areas, and the average segmentation accuracy of brain targets is improved from 0.8223 to 0.8587, thereby obtaining a high-quality segmented image of the brain target category.

[0077] The proposed method is compared with other methods, such as Figure 3As shown in Table 1, UNETR is a method that does not use self-supervised network weights for initialization. The target category 1 of this method is 0.8594, the target category 2 is 0.8054, the target category 3 is 0.8260, and the average is 0.8323; 3D MAE is a weight team segmentation network initialization obtained by a self-supervised pre-training method widely used in natural images; the target category 1 of this method is 0.8640, the target category 2 is 0.8095, the target category 3 is 0.8289, and the average is 0.8341; SimMIM is a 3D The weights obtained by the self-supervised training method without the encoder in MAE are used to initialize the segmentation network training. The target category 1 of this method is 0.8671, the target category 2 is 0.8283, the target category 3 is 0.8302, and the average value is 0.8419; the target category 1 of this method is 0.8921, the target category 2 is 0.8313, and the target category 3 is 0.8527. Compared with other methods, the self-supervised multimodal medical image segmentation method based on artificial intelligence provided by the present invention can better improve the segmentation accuracy.

[0078] Table 1 Comparison of the weights of the self-supervised training of the present invention and other methods

[0079] Target Category 1 Target Category 2 Target Category 3 Average UNETR 0.8594 0.8054 0.8260 0.8323 3D MAE 0.8640 0.8095 0.8289 0.8341 SimMIM 0.8671 0.8283 0.8302 0.8419 Ours 0.8921 0.8313 0.8527 0.8587

[0080] Example 2

[0081] An embodiment of the present invention provides an artificial intelligence-based self-supervised multimodal medical image segmentation device, which includes: an acquisition module, a self-supervised training set extraction module, a self-supervised modality mask network creation module, a self-supervised modality mask network training module, a segmentation training set extraction module, a multimodal medical image segmentation network creation module, a segmentation loss function construction module, a multimodal medical image segmentation network initialization module, a multimodal medical image segmentation network training module, and an output module connected in sequence.

[0082] The acquisition module is used to acquire multimodal images of multiple subjects, wherein images of the same subject in different imaging modes or 3D images of MRI under multiple sequences are taken as a group of multimodal images, i.e., a sample group. Part of the sample group is selected and target categories are annotated for each modality image therein. The target category annotation of each modality image is used as a label of the corresponding modality image.

[0083] A self-supervised training set extraction module, used for taking the modal images of all sample groups excluding the labels of each modal image as a self-supervised training set;

[0084] A self-supervised modal mask network creation module, used to construct a self-supervised modal mask network, the self-supervised modal mask network is the same as that in Example 1;

[0085] A self-supervised modality mask network training module is used to input the multimodal images of the self-supervised training set into the self-supervised modality mask network for end-to-end pre-training and save the pre-trained network parameters of the self-supervised modality mask network;

[0086] A segmentation training set extraction module is used to use the multimodal images with labels added in the acquisition module and the labels corresponding to the modal images as the segmentation training set;

[0087] A multimodal medical image segmentation network creation module, used to create a multimodal medical image segmentation network, the multimodal medical image segmentation network is the same as that in Example 1;

[0088] A segmentation loss function construction module, used to construct a segmentation loss function, the segmentation loss function is the same as that in Example 1;

[0089] The multimodal medical image segmentation network initialization module is used to migrate the parameters of the Transformer module in the pre-trained self-supervised modality mask network to the Transformer module of the segmentation network encoder for initialization;

[0090] A multimodal medical image segmentation network training module is used to perform end-to-end training on a multimodal medical image segmentation network with self-supervised modality mask pre-training initialization using a segmentation training set according to a set segmentation loss function, and save the parameters of the multimodal medical image segmentation network;

[0091] The output module is used to input the unlabeled multimodal medical image to be segmented into the trained multimodal medical image segmentation network to obtain the label of the target category segmentation.

[0092] The specific embodiments described herein are merely examples of the spirit of the present invention. A person skilled in the art of the present invention may make various modifications or additions to the specific embodiments described or replace them in a similar manner, but this will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A self-supervised multimodal medical image segmentation method based on artificial intelligence, characterized in that: The following steps are involved: Step 1, obtaining multimodal images of multiple subjects, wherein images of the same subject in different imaging modes, or 3D images of MRI under multiple sequences are taken as a group of multimodal images, i.e., a sample group, selecting part of the sample group and annotating the target category for each modality image therein, and the target category annotation of each modality image is used as the label corresponding to the target category of the modality image; Step 2: The modal images of all sample groups excluding the labels of each modal image are used as the self-supervised training set; Step 3: Construct a self-supervised modal mask network. The self-supervised modal mask network includes a mask network encoder, a mask network decoder and a self-supervised loss function L Self ,in, The mask network encoder consists of 1 input layer and Transformer module; Step 4: Input the multimodal images of the self-supervised training set into the self-supervised modality mask network for end-to-end pre-training, and save the pre-trained network parameters of the self-supervised modality mask network; Step 5: Use the multimodal images with labels added in step 1 and the labels corresponding to the multimodal images as the segmentation training set; Step 6: Construct a multimodal medical image segmentation network. The multimodal medical image segmentation network includes a segmentation network encoder and a segmentation network decoder, wherein: The segmentation network encoder includes an input layer and a Transformer module, and the structure of the Transformer module in the segmentation network encoder is consistent with the structure of the Transformer module in the mask network encoder; Step 7: Set the segmentation loss function L Seg ; Step 8: Migrate the parameters of the Transformer module in the self-supervised modal mask network pre-trained in step 4 to the Transformer module of the segmentation network encoder for initialization; Step 9: According to the segmentation loss function set in step 7, the multimodal medical image segmentation network with self-supervised modality mask pre-training initialization constructed in step 8 is end-to-end trained using the segmentation training set generated in step 5, and the parameters of the multimodal medical image segmentation network are saved; Step 10: input the unlabeled multimodal medical image to be segmented into the multimodal medical image segmentation network trained in step 9 to obtain the label of the target category segmentation; The input layer of the mask network encoder includes a random seed and a modality mask. The input layer randomly masks part of the modality images in the sample group, and saves the pixels of the channel where the masked modality images are located in an additional array named target, and replaces all the original positions of the masked modality images with 0; the self-supervised loss function L in step 3 Self for: ， Where L2 is the mean square error operation, X is the multimodal image, x is the image that is masked during the self-supervision process, and x ijk is the data of the jth row and kth column of the i-th slice of the masked image, S represents the self-supervised modal mask network, S(X) represents the output of the self-supervised modal mask network, S(X) ijk is the data of the jth row and kth column of the i-th slice output by the self-supervised modal mask network, L, M, and N represent the number of slices of the segmented image finally output by the self-supervised modal mask network, the number of rows and columns of each slice, and λ is a hyperparameter; The segmentation loss function L Seg for: ， Where N1 is the total number of sample groups in the segmentation training set, i1 represents the sample group number, i1≤N1, C is the total number of segmentation label categories, c represents the sample label category number, c≤C, is the label of the cth target category of the i1th sample group, It is the result predicted by the multimodal medical image segmentation network output corresponding to the c-th target category of the i1-th sample group.

2. The method for self-supervised multimodal medical image segmentation based on artificial intelligence according to claim 1, characterized in that: The Transformer modules of the mask network encoder and the segmentation network encoder both include an image embedding layer and multiple Transformer encoding layers, and the Transformer encoding layers include, in order, a primary batch normalization layer, a multi-head attention layer, a primary residual connection, a secondary batch normalization layer, a fully connected layer, and a secondary residual connection.

3. The method for self-supervised multimodal medical image segmentation based on artificial intelligence according to claim 1, characterized in that: In the step 3, the mask network decoder has an output layer, and the output layer includes 1 fully connected layer and a matrix transformation function.

4. The method for self-supervised multimodal medical image segmentation based on artificial intelligence according to claim 1, characterized in that: The input layer of the segmentation network encoder includes a batch normalization layer.

5. The method for self-supervised multimodal medical image segmentation based on artificial intelligence according to claim 2, characterized in that: The segmentation network decoder consists of multiple upsampling layers and an output layer: The number of upsampling layers is consistent with the number of Transformer encoding layers; The first upsampling layer includes an upsampling module, a convolution layer, a batch normalization layer, and an activation layer; the other upsampling layers include a fusion module, an upsampling module, a convolution layer, a batch normalization layer, and an activation layer connected in sequence; The input of the first upsampling layer in the segmentation network encoder includes the feature map output by the last Transformer coding layer of the Transformer module in the segmentation network encoder; the input of the kth upsampling layer, that is, the input of the fusion module in the kth upsampling layer, includes the feature map output by the kth last Transformer coding layer of the Transformer module in the segmentation network encoder and the feature map output by the k-1th upsampling layer, and the fusion module of the kth upsampling layer stacks the feature map output by the kth last Transformer coding layer of the Transformer module in the segmentation network encoder and the feature map output by the k-1 upsampling layers according to channels, and then reduces the number of channels by convolution; wherein k is an integer greater than 1 and less than or equal to the number of upsampling layers.

6. The method for self-supervised multimodal medical image segmentation based on artificial intelligence according to claim 5, characterized in that: The output layer in the segmentation network decoder includes a fusion module, a convolution layer and an activation layer connected in sequence, and the input of the fusion module of the output layer in the segmentation network decoder includes a feature map output by the last upsampling layer of the segmentation network decoder and a feature map output by the input layer of the segmentation network encoder.

Images