A Medical Image Segmentation Method Based on Self-Supervised Contrastive Learning

Abstract

The present invention relates to the field of medical image segmentation technology, and in particular to a medical image segmentation method based on self-supervised contrastive learning, comprising the following steps: step 1, obtaining diseased data without segmentation labels and performing data enhancement on each sample; step 2, performing feature extraction and mapping on the enhanced view using ViT; step 3, obtaining a low-pixel lesion activation map and a background activation map P <subgt;i< / subgt;<supgt;d< / supgt;与P<subgt;i< / subgt;<supgt;b< / supgt;;步骤4,得到低像素病灶特征向量V<subgt;i< / subgt;<supgt;d< / supgt;和背景特征向量V<subgt;i< / subgt;<supgt;b< / supgt;;步骤5,得到高像素病灶特征向量U<subgt;i< / subgt;<supgt;d< / supgt;和背景特征向量U<subgt;i< / subgt;<supgt;b< / supgt;,使用对比损失进行自监督学习;步骤6,得到最终的分割结果,本发明基于自监督对比学习的医学影像分割方法,解决了传统模型对有精确分割标签或分类信息的训练图像的依赖,以深度对比学习的方式充分利用无标签数据,达到自监督分割病灶区域的效果。

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and specifically to a medical image segmentation method based on self-supervised contrast learning. Background Art

[0002] Medical imaging techniques such as CT and MRI, as the most intuitive non-invasive methods for obtaining the structural information of the diseased parts of patients, have been widely used in the modern medical clinical diagnosis process. Their real-time and non-invasive characteristics play an important guiding role in the disease diagnosis, prediction, and subsequent follow-up treatment of patients without obvious symptoms. In the treatment stage, medical experts segment and annotate the diseased tissues and organs of patients based on the obtained medical images, thereby judging the stage of disease development and proposing targeted treatment plans. The accuracy of segmentation directly affects the doctor's judgment of the condition and is an important step in the diagnosis process.

[0003] Accurate segmentation and annotation rely on the decision-making judgment of professional physicians. When the number of patients is large and the imaging frequency of a single patient during the visit period is relatively high, this traditional manual segmentation method is inefficient and inevitably generates segmentation errors, failing to meet the requirements of rapid diagnosis. In recent years, the improvement of GPU computing power has elevated the processing ability of deep learning technology for natural images to a new level that can be combined with big data, making it possible for computers to perform rapid auxiliary diagnosis from the perspective of medical images. For a specific category of medical images, a deep learning model optimizes and trains a set of input data that meets a certain quantity through a constructed multi-layer neural network and the constraint of a loss function, enabling the model to ultimately possess the ability to recognize the unique features of medical images, and further achieving purposes such as the detection, classification, and segmentation of input data. While saving valuable human resources of professional physicians, the error is also quantified within an acceptable range, ensuring the accuracy of the next-step analysis and treatment.

[0004] Currently, most medical image segmentations based on deep learning are supervised methods, that is, the input of the neural network requires not only the original image of the diseased part but also the accurately labeled segmentation labels of the lesion area corresponding to the original image. The network makes a reasonable prediction using the original image, and then compares the prediction result with the segmentation label through the loss function to find the error in the network prediction. This error is backpropagated to the parameters of each layer of the neural network through methods such as gradient descent to readjust the network prediction to make it more similar to the true label. When the error of the loss function converges to an acceptable range, it is considered that the network model has achieved a certain accuracy of segmentation ability, and it can be used to segment the lesion area of new data of the same type.

[0005] Compared with the manual annotation segmentation by doctors, although the supervised deep learning segmentation model can give the segmentation guidance results within the second-level time range when dealing with new diseased data, it still requires a large amount of labeled data annotated by doctors as learning experience to continuously optimize its own segmentation ability during training, and these time-consuming and laborious data annotation processes also need to be completed by professional doctors. In the existing medical image resources, the phenomenon that the number of unannotated data is much larger than that of annotated data is widespread.

[0006] In recent years, some weakly supervised segmentation methods have also emerged to segment the lesion area and use unannotated data to train the weakly supervised deep learning model. In a weakly supervised manner, it improves the dependence of medical images on labeled data, and only extracts features from the two datasets of diseased and disease-free with segmentation annotations for comparison, so as to achieve the purpose of segmenting the lesion area. However, the existing weakly supervised methods still need to collect and organize the two types of datasets of diseased and disease-free, and these classification information also requires professional doctors to spend a lot of time and energy to organize.

[0007] Therefore, in view of the above situation, there is an urgent need to develop a medical image segmentation method based on self-supervised contrast learning to overcome the deficiencies in current practical applications. Summary of the Invention

[0008] The purpose of the present invention is to provide a medical image segmentation method based on self-supervised contrast learning to solve the problems raised in the above background technology.

[0009] To achieve the above purpose, the present invention provides the following technical solutions:

[0010] A medical image segmentation method based on self-supervised contrast learning includes the following steps:

[0011] Step 1, obtain diseased data without segmentation labels, and perform data augmentation on each sample;

[0012] Step 2, use ViT for feature extraction and mapping on the views after data augmentation, and then use a convolutional neural network with 2 channels and a softmax layer to obtain the initial lesion activation map and background activation map;

[0013] Step 3, calculate the mean M of the original image in three channels i , and respectively multiply the initial lesion activation map and background activation map obtained in the previous step by M i and 1 - M i , to obtain the low-pixel lesion activation map and background activation map P i d and P i b ;

[0014] Step 4: Perform matrix transformation on the feature map obtained by ViT in Step 2, and then multiply it with P i d Multiply with P i b to obtain the low-pixel lesion feature vector V i d and the background feature vector V i b , and use contrastive loss for self-supervised learning;

[0015] Step 5: Upsample and perform matrix transformation on the feature map obtained by ViT in Step 2, and perform upsampling on the low-pixel lesion activation map and background activation map P i d Multiply with P i b to obtain the high-pixel lesion activation map and background activation map Q i d Multiply with Q i b to obtain the high-pixel lesion feature vector U i d and the background feature vector U i b , and use contrastive loss for self-supervised learning;

[0016] Step 6: Output the high-pixel lesion activation map Q i d obtained by the trained model and the original image into the DenseCRF module to obtain the final segmentation result.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0018] In the process of processing medical images, for the problem of lesion area segmentation, it solves the dependence of traditional models on training images with accurate segmentation labels or classification information, makes full use of unlabeled data in the form of deep contrastive learning, and only performs self-supervised contrastive learning on the diseased dataset, so as to achieve the effect of self-supervised segmentation of the lesion area. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic flowchart of the medical image segmentation method based on self-supervised contrastive learning in the embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of the segmentation process in the embodiment of the present invention.

[0021] Figure 3 It is a schematic diagram of the network structure in the embodiment of the present invention.

[0022] Figure 43D glioma lesion area segmentation schematic diagram in an embodiment of the present invention.

[0023] Figure 5 Schematic diagram of skin lesion area segmentation in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0025] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0026] See also Figures 1 - 3 , an embodiment of the present invention provides a medical image segmentation method based on self-supervised contrastive learning, comprising the following steps:

[0027] Step 1: Obtain diseased data without segmentation labels and perform data augmentation on each sample;

[0028] Step 2: Use ViT to extract and map features from the data-augmented view. Then, use a convolutional neural network with 2 channels and a softmax layer to obtain the initial lesion activation map and background activation map.

[0029] Step 3: Calculate the mean M of the original image in three channels i , respectively use the initial lesion activation map and background activation map obtained in the previous step to multiply M i and 1-M i , get the low-pixel lesion activation map and background activation map P i d With P i b ;

[0030] Step 4: Perform matrix transformation on the feature map obtained by ViT in step 2, and then i d With P i b Multiply to get the low-pixel lesion feature vector V i d and background feature vector V i b , self-supervised learning using contrastive loss;

[0031] Step 5: Upsample and matrix transform the feature map obtained by ViT in step 2, and perform the low-pixel lesion activation map and background activation map P obtained in step 3. i d With P i b Upsampling is performed to obtain high-pixel lesion activation map and background activation map Q i d With Q i b , and then multiply to get the high-pixel lesion feature vector U i d and background feature vector U i b , self-supervised learning using contrastive loss;

[0032] Step 6: The high-pixel lesion activation map Q obtained from the trained model i d It is output together with the original image to the DenseCRF module to obtain the final segmentation result.

[0033] In step 1, it is characterized in that the data enhancement adopts the rotation, translation and scaling operations in the data enhancement tool provided by torchvision to eliminate the negative impact of lesion size, direction and position-independent features in the diseased data.

[0034] In step 5, the upsampling decoding network restores the feature map to its original size using a four-layer convolutional neural network, where the network parameters are trained using the reconstruction error.

[0035] For low-pixel lesion activation map and background activation map P i d With P i b Perform bilinear interpolation upsampling.

[0036] In steps 4 and 5, the training is performed by minimizing the similarity between the lesion region and the normal region and maximizing the similarity contrast within the lesion region and the normal region, and the semantic information between the lesion and normal region representations is used to locate the lesion target region, respectively using the negative contrast loss L neg and the forward contrast loss L pos Two loss functions are used to represent it.

[0037] For multiple diseased images, the representations of the lesion region and the normal region contain different semantic information, and the distance between the lesion region and the normal region is large.

[0038] For n samples of diseased images, n lesion region feature vectors and n normal region feature vectors are separated. For one of the feature vector pairs of lesion and normal region, Negative contrast loss L neg The definition is as follows:

[0039]

[0040]

[0041] in, Represents the feature vector of the lesion area and normal region feature vector The similarity of , sim() represents the cosine similarity.

[0042] For multiple diseased images, the distances between lesion areas and normal areas are small.

[0043] The cosine similarity between the lesion region and the normal region is calculated as follows:

[0044]

[0045]

[0046] in, Represents the feature vector of the lesion area of sample i and the lesion region feature vector of sample j The similarity, Represents the normal region feature vector of sample i and the normal region feature vector of sample j The similarity of , sim() represents the cosine similarity.

[0047] Calculate the forward contrast loss of the lesion area separately and positive contrast loss of normal area as follows:

[0048]

[0049]

[0050] in, Represents the feature vector of the lesion area of sample i and the lesion region feature vector of sample j The similarity, Represents the normal region feature vector of sample i and the normal region feature vector of sample j similarity.

[0051] The overall forward contrast loss L pos The definition is as follows:

[0052]

[0053] Among them, λ1 represents the positive contrast loss of the lesion area The weight coefficient of .

[0054] The final loss function is the sum of the negative contrast loss and positive contrast loss of low pixels and the contrast loss of high pixels, that is:

[0055]

[0056] Among them, L represents the final loss function of the model, and λ2 represents the negative contrast loss and The weight coefficient of represents the negative contrast loss of low pixels, represents the positive contrast loss of low pixels, represents the negative contrast loss of high pixels, Represents the high-pixel positive contrast loss.

[0057] The implementation process of self-supervised segmentation based on contrastive learning mainly relies on diseased images. The images themselves have no segmentation annotations, and there is no need to use normal images as comparison. To achieve feature extraction and comparison of the lesion area and normal area of the diseased image, it is necessary to add a module for contrastive learning of the lesion area and normal area after the feature extraction network layer built by each layer of the network in the model. In recent years, the self-supervised learning method based on contrastive learning has gradually matured. The contrastive learning method constructs a dataset for contrastive learning on an unlabeled dataset (or does not use the label information of labeled data) through data enhancement methods such as image cropping, scaling, and transformation to obtain the implicit representation of the data itself. The corresponding model has made substantial progress in natural image processing and can achieve various tasks such as image classification and clustering. In image applications, it uses self-supervised contrastive learning to enable the neural network to extract the implicit features of the image during the training process through the defined loss function on different groups of data obtained by different methods, thereby achieving downstream tasks such as image classification and clustering.

[0058] Therefore, the present invention uses the idea of self-supervised contrastive learning and uses Vision Transformer (ViT) as the basic framework. On the basis of ViT's feature extraction of the image, self-supervised contrastive learning is used to identify the lesion area and the normal area. During the training process, the class activation maps of the two areas are obtained by maximizing the similarity within the lesion area and the normal area, and minimizing the similarity between the lesion area and the normal area. In order to improve the fineness of the segmentation, the model also performs an upsampling operation on the Feature Map obtained by feature extraction using ViT, and the network parameters are trained using reconstruction error. Self-supervised contrastive learning is also used on the upsampled Feature Map to obtain the upsampled class activation map, and then the fully connected conditional random field (Dense CRF) can be used to obtain the final segmentation result. In order to make the model training more stable and converge faster, the original image information is also added when generating the lesion activation map.

[0059] The overall segmentation process is as follows Figure 1 As shown in Figure 2, the proposed method consists of two phases: training and application. During the training phase, diseased image data is used as training data, without segmentation labels. Self-supervised contrastive learning is used to compare the features of diseased and normal regions in diseased images. Data augmentation uses rotation, translation, and scaling operations within the TorchVision data augmentation tool to eliminate the negative effects of irrelevant features such as lesion size, orientation, and location. The code for this method is based on the PyTorch framework, and the entire model is executed on an Ubuntu 18.04.2LTS operating system equipped with a Titan RTX graphics card.

[0060] Unlike the time-consuming training phase, the application phase directly uses the trained model to perform segmentation prediction on new data outside the training set, generating corresponding segmentation labels. Prediction time can be controlled within seconds, enabling rapid auxiliary diagnosis.

[0061] The model network structure is as follows Figure 2 As shown, the training set is diseased image data, with a diseased image X i For example. X iis the initial input of the entire network, with a size of 256x256x3, where 256 is the width and height of the image, and 3 is the number of channels of the image. During the training process, data augmentation is first performed, and the original image is randomly cropped to a size of 224x224x3, that is, H = 224, W = 224. Then the image is input into the encoder using ViT as the basic framework to generate the corresponding feature map. For example: using the ViT-Small architecture with a patch size of 16*16, the dimension of the obtained feature map is C = 384, H = 224 / 16, W = 224 / 16. A convolutional neural network with 2 output channels and a softmax layer are used on the feature map to obtain the initial lesion activation map and background activation map. To make the model training more stable and the convergence speed faster, first obtain the mean M of the original image in the three channels i , and then multiply M i and 1 - M i respectively on the initial lesion activation map and background activation map to obtain the low-pixel lesion activation map and background activation map P i d and P i b . The two activation maps are multiplied by the feature map after matrix transformation respectively to obtain the corresponding low-pixel lesion feature vector V i d and background feature vector V i b , which are used for self-supervised contrast learning.

[0062] To improve the segmentation fineness, the model also performs an upsampling operation on the feature map obtained by using ViT for feature extraction. The upsampling decoding network uses a four-layer convolutional neural network to restore the feature map to the original size, and the network parameters are trained using the reconstruction error. At the same time, the low-pixel lesion activation map and background activation map P i d and P i b are upsampled by bilinear interpolation to obtain the high-pixel lesion activation map and background activation map Q i d and Q i b . Then, the two activation maps are multiplied by the upsampled feature map after matrix transformation respectively to obtain the high-pixel lesion feature vector U i d and background feature vector U i b , which are used for self-supervised contrast learning.

[0063] For the high-pixel lesion activation map Q obtained by the trained model id , and the original image are output to the Dense CRF module to obtain the final segmentation result.

[0064] Since there are no segmentation labels for diseased images during training, we train by comparing the lesion and normal regions of multiple images, and only use the semantic information between the lesion and normal region representations to locate the lesion target area. During the training process, the similarity between the lesion region and the normal region is minimized, and the similarity within the lesion region and the normal region is maximized. The two optimization objectives are respectively expressed using negative contrast loss L neg and the forward contrast loss L pos To represent (taking low pixels as an example).

[0065] (1) Negative contrast loss:

[0066] For a diseased image, the representations of the lesion area and the normal area contain different semantic information, so there should be a large distance between them in the feature space. This relationship also exists between multiple images, and the distance between the lesion area and the normal area should also be large. Based on the above assumptions, we use contrastive learning to distinguish the lesion area from the normal area. For n samples of diseased images X 1:n , we can separate n feature vectors of the lesion region and n feature vectors of the normal region. For one of the feature vector pairs of the lesion and the normal region, for example: The negative contrast loss can be defined as follows:

[0067]

[0068]

[0069] in, Represents the feature vector of the lesion area and normal region feature vector The similarity of , sim() represents the cosine similarity.

[0070] (2) Forward contrast loss:

[0071] For multiple images, the distance between lesion areas and lesion areas, as well as the distance between normal areas and normal areas should be relatively small. Therefore, the cosine similarity between the lesion area and the normal area can be calculated as follows:

[0072]

[0073]

[0074] in, Represents the feature vector of the lesion area of sample i and the lesion region feature vector of sample j The similarity, Represents the normal region feature vector of sample i and the normal region feature vector of sample j The similarity of , sim() represents the cosine similarity.

[0075] Afterwards, the forward contrast loss of the lesion area can be calculated separately and positive contrast loss of normal area as follows:

[0076]

[0077]

[0078] in, Represents the feature vector of the lesion area of sample i and the lesion region feature vector of sample j The similarity, Represents the normal region feature vector of sample i and the normal region feature vector of sample j similarity.

[0079] The overall forward contrast loss L pos The definition is as follows:

[0080]

[0081] Among them, λ1 represents the positive contrast loss of the lesion area The weight coefficient of .

[0082] In our model, we calculate the low-pixel negative contrast loss and positive contrast loss separately. and And contrast loss at high pixels and The final loss function is as follows:

[0083]

[0084] Among them, L represents the final loss function of the model, and λ2 represents the negative contrast loss and The weight coefficient of represents the negative contrast loss of low pixels, represents the positive contrast loss of low pixels, represents the negative contrast loss of high pixels, Represents the high-pixel positive contrast loss.

[0085] Example 1

[0086] The BraTS 2018 and BraTS 2019 challenges are for 3D glioma lesion area segmentation. First, the 3D glioma dataset in the dataset is sliced into 2D, and a total of 743 diseased 2D images are collected. Then, 563 images are randomly selected from the diseased dataset as training sets, 90 of the remaining 180 images are used as validation sets, and 90 are used as test sets. After uniformly normalizing to a size of 256x256, the training set is subjected to small random translations, rotations, scaling and other data enhancement strategies. The model is trained using the above training set and training is stopped at the 200th epoch. The final segmentation prediction of the model is as follows Figure 4 shown.

[0087] Example 2

[0088] Task 1 in the ISIC 2018 Challenge is skin lesion area segmentation. First, we filter out images with artificially added scale lines and color position marks. We select 262 lesion images from the ISIC 2018 Challenge as the overall set, of which 200 are randomly selected as the training set. Of the remaining 62, 31 are used as the validation set and 31 as the test set. After the diseased data is uniformly normalized to a size of 256x256, the training set is subjected to data augmentation strategies such as small random translation, rotation, and scaling. The model is trained using the above training set and stopped at the 200th epoch. The final segmentation prediction of the model is as follows Figure 5 shown.

[0089] It should be noted that, in the present invention, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A medical image segmentation method based on self-supervised contrastive learning, characterized in that, It includes the following steps: Step 1: Obtain the diseased data without segmentation labels and perform data augmentation on each sample; Step 2: Use ViT for feature extraction and mapping on the views after data augmentation, and then use a convolutional neural network with 2 channels and a softmax layer to obtain the initial lesion activation map and background activation map; Step 3, calculate the mean value M of the original image in three channels i , and respectively multiply M i and 1 - M i with the initial lesion activation map and background activation map obtained in the previous step to obtain the low-pixel lesion activation map and background activation map P i d and P i b ; Step 4: Perform matrix transformation on the feature map obtained by ViT in Step 2, and then multiply it with P i d Multiply with P i b to obtain the low - pixel lesion feature vector V i d and the background feature vector V i b , and use contrastive loss for self - supervised learning; Step 5: Upsample and perform matrix transformation on the feature map obtained by ViT in Step 2, and for the low-pixel lesion activation map and background activation map P obtained in Step 3 i d and P i b to perform upsampling to obtain a high-pixel lesion activation map and background activation map Q i d and Q i b and then perform multiplication to obtain a high-pixel lesion feature vector U i d and background feature vector U i b and use contrastive loss for self-supervised learning; Step 6, for the high-pixel lesion activation map Q obtained from the trained model i d and the original image are output to the Dense CRF module together to obtain the final segmentation result; In Step 1, the data augmentation adopts rotation, translation, and scaling operations in the data augmentation tools provided by torchvision to eliminate the negative impacts caused by irrelevant features of lesion size, direction, and position in the diseased data; Perform bilinear interpolation upsampling on the low-pixel lesion activation map and background activation map Pid and Pib.

2. The medical image segmentation method based on self-supervised contrastive learning according to claim 1, wherein In Step 5, the upsampling decoding network uses a four-layer convolutional neural network to restore the feature map to the original size, and the network parameters are trained using the reconstruction error.

3. The medical image segmentation method based on self-supervised contrastive learning according to claim 1, wherein, In steps 4 and 5, training is performed by minimizing the similarity between the lesion region and the normal region and maximizing the similarity within the lesion region and the normal region. The semantic information between the representations of the lesion and the normal region is used to locate the lesion target region, and the negative contrast loss L neg and the positive contrast loss L pos are represented by two loss functions respectively.

4. The medical image segmentation method based on self-supervised contrastive learning according to claim 3, characterized in that Since the representations of the lesion regions and normal regions in multiple diseased images contain different semantic information, the distance between the lesion regions and normal regions is large.

5. The medical image segmentation method based on self-supervised contrastive learning according to claim 4, wherein For the diseased images of n samples, n lesion region feature vectors and n normal region feature vectors are separated, and for one pair of feature vectors of a lesion and a normal region , the negative contrast loss L neg is defined as follows: ； ； Among them, represents the similarity of the feature vector of the lesion area and the feature vector of the normal area , indicating cosine similarity.

6. The medical image segmentation method based on self-supervised contrastive learning according to claim 4 or 5, characterized in that The distances between the lesion regions and lesion regions and between the normal regions and normal regions in multiple diseased images are small.

7. The medical image segmentation method based on self-supervised contrastive learning according to claim 6, wherein Calculate the cosine similarity between the lesion region and the normal region directly as follows: ； ； Among them, represents the feature vector of the lesion area of sample i and the feature vector of the lesion area of sample j for similarity, represents the feature vector of the normal area of sample i and the feature vector of the normal area of sample j for similarity, represents cosine similarity; Calculate the positive contrast loss of the lesion area separately and the positive contrast loss of the normal area as follows: ； ； Among them, represents the feature vector of the lesion area of sample i and the feature vector of the lesion area of sample j for similarity, represents the feature vector of the normal area of sample i and the feature vector of the normal area of sample j for similarity; Overall positive contrastive loss L pos is defined as follows: ； Among them, represents the weight coefficient of the positive contrast loss of the lesion area .

8. The medical image segmentation method based on self-supervised contrastive learning according to claim 7, characterized in that, The final loss function is the sum of the negative contrast loss and positive contrast loss of low pixels and the contrast loss at high pixels, that is: ； Among them, represents the final loss function of the model, represents the negative contrast loss and is the weight coefficient of represents the negative contrast loss of low pixels, represents the positive contrast loss of low pixels, represents the negative contrast loss of high pixels, represents the positive contrast loss of high pixels.

Images