Pre-training method based on self-supervisory information of unlabeled medical images

By adopting a self-supervised information pre-training method based on unlabeled medical images in medical image processing, self-supervised information is extracted using Trans-Inf-Net and twin neural networks, the shortcomings of relying on labeled data in the prior art are solved, and medical treatment is realized Improved image processing effect.

CN116258928BActive Publication Date: 2024-11-12XIAN UNIV OF TECH
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310270593.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-03-20
Publication Date
2024-11-12
Estimated Expiration
2043-03-20

AI Technical Summary

Technical Problem

Existing medical image processing methods rely on labeled data and lack general pre-trained models, making it difficult to enhance network performance by increasing data. Medical image labeling requires professionals and involves patient privacy, which makes data acquisition difficult.

Method used

Using a pre-training method based on self-supervised information of unlabeled medical images, the self-supervised information of medical images is extracted by building a Trans-Inf-Net segmentation network and twin neural network, and the Gaussian pyramid decomposition and Laplace transform fusion features are used to extract self-supervised information of medical images and generate Pseudo-tagging is trained in networks to simplify the need for tagging data.

Benefits of technology

It effectively overcomes the burden of manual marking in medical image processing, improves image processing effect, and improves the performance and accuracy of the model through self-supervised information extraction and feature fusion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116258928B_ABST
    Figure CN116258928B_ABST
Patent Text Reader

Abstract

The present invention discloses a pre-training method based on self-supervisory information of unlabeled medical images, specifically: building a Trans-Inf-Net segmentation network, putting the medical CT image x0 into the segmentation network to segment the lesion, and obtaining a lesion map x s , as self-supervisory information; build a twin neural network and combine the medical CT image x0 and the lesion map x s As the two inputs of the twin neural network, information is extracted to obtain the weight map W; the weight map W is transformed using Gaussian pyramid decomposition, and the weight map W is combined with the medical CT image x0 and the lesion map x s The fusion is performed to strengthen the features of the medical image and retain the original information of the medical image. This invention studies the self-supervisory information of the pre-trained model for the first time to effectively overcome the burden of manual labeling on medical images, and uses a simple twin network to extract the inherent features of the unlabeled image to improve the processing effect of medical images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing methods, and in particular relates to a pre-training method based on self-supervisory information of unlabeled medical images. Background Art

[0002] Image processing technology has developed rapidly since the introduction of machine learning methods. After the emergence of deep learning methods based on neural networks, results that surpass traditional methods have been achieved in tasks such as image classification and image segmentation. Deep learning has also led to the advancement of medical image processing methods, and the efficiency and accuracy of the methods have been greatly improved.

[0003] At present, the mainstream method of medical image processing is based on semi-supervised learning. In the supervised learning method, the more labeled data, the better the model effect. However, the labeling of medical images requires professional personnel, and part of it involves the privacy of patients. Therefore, under the existing conditions, it is difficult to enhance the performance of the network model by increasing the available training data. At the same time, there is no universal pre-training model or method in medical image processing. Compared with randomly initialized network models, pre-trained network models can achieve better results. Summary of the invention

[0004] The purpose of the present invention is to provide a pre-training method based on self-supervisory information of unlabeled medical images, which can fully utilize the information of the medical images themselves for pre-processing and improve the processing effect of the medical images.

[0005] The technical solution adopted by the present invention is a pre-training method based on self-supervisory information of unlabeled medical images, which is specifically implemented according to the following steps:

[0006] Step 1: Build the Trans-Inf-Net segmentation network, put the medical CT image x0 into the segmentation network to segment the lesion, and obtain the lesion map x s , as self-supervisory information;

[0007] Step 2: Build a twin neural network and combine the medical CT image x0 and the lesion image x s As two inputs of the twin neural network, information is extracted to obtain the weight graph W;

[0008] Step 3: Use Gaussian pyramid decomposition to transform the weight map W obtained in step 2, and compare the weight map W with the medical CT image x0 and the lesion map x s Fusion is performed to enhance the features of medical images and retain the original information of medical images.

[0009] The present invention is also characterized in that:

[0010] In step 1, specifically:

[0011] Step 1.1, Trans-Inf-Net segmentation network is based on the Inf-Net segmentation network, adding a transformer module to improve edge representation. The specific network framework is: the medical CT image is first sent to two convolutional layers to extract low-level features, and then the low-level features f2 are sent to three convolutional layers to extract high-level features. A parallel partial decoder is used to aggregate these features and obtain a global graph S g , combining high-level features and low-level features f2 in S g Under the guidance of , the output of the last RA is input into the reverse attention module, and finally the output of the last RA is input into a Sigmoid activation function for infection prediction of medical CT images;

[0012] Step 1.2: Use 50 labeled medical CT images and 1600 unlabeled medical CT images as training sets; use 48 labeled medical CT images as test sets, and resize all images to 352×352;

[0013] Step 1.3, first train the network with labeled data, resample the training set images using different scaling ratios {0.75, 1, 1.25}, then use the resampled images to train the Trans-Inf-Net segmentation network, the network is trained using the Adam optimizer, and the learning rate is set to 1e-4, finally, save the trained model;

[0014] Step 1.4: Use the trained Trans-Inf-Net segmentation network to generate pseudo labels for unlabeled medical CT images; the number of images is set to 5, i.e. K = 5, to form K pseudo-label data, and the pseudo-label images are expanded into the labeled dataset as training images for network training, and then these K images are removed from the dataset of unlabeled medical CT images. Until the dataset of unlabeled medical CT images is empty;

[0015] Step 1.5, save the trained model, put the medical CT image x0 in the COVID-HSS dataset into the trained model for lesion segmentation, and obtain the lesion map x s .

[0016] In step 2, specifically:

[0017] The medical CT image x0 and the lesion map x sAs two inputs of the twin neural network, the two images are processed by the encoder network f, which consists of the backbone MLP and the projection MLP head; the encoder network f shares weights between the two images; the prediction MLP head h changes the output of one image and matches this output with the output of another image; the two output vectors are represented as and And minimize the negative cosine similarity, as shown in formula (1):

[0018]

[0019] Among them, ||·||2 means is the normalized vector The mean square error of , with scale of 2.

[0020] Symmetric loss, as shown in formula (2):

[0021]

[0022] Modify formula (1) into formula (3):

[0023]

[0024] Modify formula (2) into formula (4):

[0025]

[0026] The encoder network is ResNet-50 with a base learning rate of 0.05, weight decay of 0.0001, SGD momentum of 0.9, and batch size of 4.

[0027] In step 3, specifically:

[0028] Step 3.1, transform the weight map W using Gaussian pyramid decomposition to obtain G{W};

[0029] Step 3.2, use the Laplacian pyramid to transform the medical CT image x0 and the lesion map x s Perform Laplace transform to obtain L{x s} and L{x0}, and then the obtained G{W} and L{x s} and L{x0} are fused separately to obtain L{C} and L{D}; L{C} and L{D} are fused to obtain L{F}, and L{F} is subjected to Laplace inverse transformation to obtain the final fused image x f .

[0030] In step 3.2, for each Laplacian pyramid decomposition layer, L{x o} and L{x s}, as shown in formula (5);

[0031]

[0032] Among them, L{x o} l and L{x s} l Represents the Laplacian pyramid graph of the lth level; and Respectively represent the image x with the center position (x, y) of the lth layer o and image x s The local area energy;

[0033] The similarity measure in fusion mode determination is expressed as formula (6):

[0034]

[0035] Its range is in the interval [-1,1], t is the threshold determined by the fusion mode, if S l (x,y)≥t, the following formula is used for weighted average fusion:

[0036]

[0037] If S l (x,y)<t, then the local energy for comparison is selected in the fusion as follows:

[0038]

[0039] The beneficial effects of the present invention are: for the first time, the self-supervisory information of the pre-trained model is studied to effectively overcome the burden of manual labeling on medical images, and a simple twin network is used to extract the inherent features of unlabeled images to improve the processing effect of medical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a pre-training method based on self-supervisory information of unlabeled medical images of the present invention;

[0041] Figure 2 is a flow chart of extracting information using a twin neural network in the method of the present invention;

[0042] Figure 3 is a diagram of the image fusion process in the method of the present invention;

[0043] Figure 4 This is the fusion result diagram of the COVID-HHS dataset;

[0044] Figure 5 This is the fusion result diagram of the COVID-CT dataset. DETAILED DESCRIPTION

[0045] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] The present invention is based on a pre-training method of self-supervisory information of unlabeled medical images, such as Figure 1 As shown, the specific implementation steps are as follows:

[0047] Step 1: Build the Trans-Inf-Net segmentation network, put the medical CT image into the segmentation network to segment the lesion, and obtain the lesion map as self-supervisory information; specifically:

[0048] Step 1.1, Trans-Inf-Net segmentation network is based on the Inf-Net segmentation network, adding a transformer module to improve edge representation. The specific network framework is: the medical CT image is first sent to two layers of convolutional layers to extract low-level features. Considering that the low-level features also retain some edge information, the low-level features are sent to the edge attention module to learn the representation of edge attention. At the same time, a Transformer module is added to improve the representation of the target area boundary. Then, the low-level feature f2 is sent to the three-layer convolutional layer to extract high-level features. A parallel partial decoder is used to aggregate these features and obtain a global graph S. g , combining high-level features and low-level features f2 in S g The output of the last RA is input into a Sigmoid activation function for infection prediction of medical CT images.

[0049] The working principle of Transformer is as follows:

[0050] Convert medical CT images into a sequence of patches; In order for Transformer to process image data, the image data must first be converted into sequence data. Suppose there is an image with a patch size of p, then patches of N images can be created, where N is the length of the sequence, similar to the number of words in a sentence in NLP; Figure 4 In the figure, you can see that the image is divided into 9 patches.

[0051] Flatten the patch sequence. The size of a patch is 8, so the shape of a patch is (3,8,8), with a dimension of 3. After flattening, the size is 3×8×8=192, that is, a patch becomes a vector of length 192. At this time, add a Linear transformation to map the patch dimension to the embedding dimension, which is similar to the word vector in NLP.

[0052] Add position embedding. Unlike CNNs, the model does not know the position information of the patches in the sequence data. Therefore, these patches must first be appended with a position information, which is the vector with numbers in the figure. Experiments show that different position encoding embeddings have little effect on the final result. Fixed position encoding is used in Transformer, and learnable position embedding vectors are used in ViT, which are added to the corresponding output patches.

[0053] Add a class token. Before inputting to the Transformer Encoder, you need to add a special class token, which is mainly inspired by the BERT model. The purpose of adding this class token is that the ViT model regards the output of this class token in the Transformer Encoder as the model's encoding feature of the input image, which is used in the subsequent input MLP module to calculate the loss with the image label.

[0054] Input Transformer Encoder. Concatenate the patch embeddings and class tokens and input them into the standard Transformer Encoder.

[0055] Step 1.2: Use 50 labeled medical CT images and 1600 unlabeled medical CT images as training sets; use 48 labeled medical CT images as test sets, and resize all images to 352×352;

[0056] Step 1.3: First, train the network with labeled data, resample the training set images with different scaling ratios {0.75, 1, 1.25}, and then train the Trans-Inf-Net segmentation network with the resampled images, thereby improving the generalization performance of the model; the network is trained using the Adam optimizer, and the learning rate is set to 1e-4. Finally, save the trained model;

[0057] Step 1.4: Use the trained Trans-Inf-Net segmentation network to generate pseudo labels for unlabeled medical CT images; the number of images is set to 5, i.e. K = 5, to form K pseudo-label data, and the pseudo-label images are expanded into the labeled dataset as training images for network training, and then these K images are removed from the dataset of unlabeled medical CT images. Until the dataset of unlabeled medical CT images is empty;

[0058] For 1600 unlabeled images, 320 iterations are performed with a batch size of 16.

[0059] Step 1.5, save the trained model, put the medical CT image x0 in the COVID-HSS dataset into the trained model for lesion segmentation, and obtain the lesion map x s ;

[0060] Step 2: Build a twin neural network, such as Figure 2 As shown, the medical CT image x0 and the lesion image x s As the two inputs of the twin neural network, information is extracted to obtain the weight graph W; specifically:

[0061] The medical CT image x0 and the lesion map x s As two inputs of the twin neural network, the two images are processed by the encoder network f, which consists of the backbone MLP and the projection MLP head; the encoder network f shares weights between the two images; the prediction MLP head h changes the output of one image and matches this output with the output of another image; the two output vectors are represented as and And minimize the negative cosine similarity, as shown in formula (1):

[0062]

[0063] Among them, ||·||2 means is the normalized vector The mean square error of , with a scale of 2;

[0064] Symmetric loss, as shown in formula (2):

[0065]

[0066] Modify formula (1) into formula (3):

[0067]

[0068] Modify formula (2) into formula (4):

[0069]

[0070] The encoder network is ResNet-50, the base learning rate is 0.05, the weight decay is 0.0001, the SGD momentum is 0.9, and the batch size is 4 by default.

[0071] The twin network outputs a dense prediction map. In order to obtain a weight map of the same size as the original image, the predicted value of each patch is flattened to all pixels in the patch as the weight of the patch, and the overlapping parts of the pixels are averaged to finally obtain a weight map W of the same size as the original image.

[0072] Step 3: Use Gaussian pyramid decomposition to transform the weight map W obtained in step 2, and compare the weight map W with the medical CT image x0 and the lesion map x s Fusion is performed to enhance the features of medical images and retain the original information of medical images, such as Figure 3 As shown, specifically:

[0073] Step 3.1, transform the weight map W using Gaussian pyramid decomposition to obtain G{W};

[0074] Step 3.2, use the Laplacian pyramid to transform the medical CT image x0 and the lesion map x s Perform Laplace transform to obtain L{x s} and L{x0}, and then the obtained G{W} and L{x s} and L{x0} are fused separately to obtain L{C} and L{D}; L{C} and L{D} are fused to obtain L{F}, and L{F} is subjected to Laplace inverse transformation to obtain the final fused image x f .

[0075] For each Laplacian pyramid decomposition level, L{x o} and L{x s}, as shown in formula (5);

[0076]

[0077] Among them, L{x o} l and L{x s} l Represents the Laplacian pyramid graph of the lth level; and Respectively represent the image x with the center position (x, y) of the lth layer o and image x s The local area energy;

[0078] The similarity measure in fusion mode determination can be expressed as formula (6):

[0079]

[0080] Its range is in the interval [-1,1]. If the similarity measure value is 1, it means that the two images have a very high similarity. t is the threshold determined by the fusion mode. If S l (x,y)≥t, the following formula is used for weighted average fusion:

[0081]

[0082] If S l (x,y)<t, then the local energy for comparison is selected in the fusion as follows:

[0083]

[0084] Two datasets are used, namely the CoVID-CT dataset and the CoVID-HSS dataset. 1. The CoVID-CT dataset contains 349 COVID-19 positive CT images and 397 COVID-19 negative images. 53 of the 349 labeled positive CT images were selected, 41 of which were used as the dataset for the pre-trained model, and the remaining 12 were used for the test set. The size of the input CT images was uniformly adjusted to 224*224. Each image in the dataset also contains some meta-information, including the patient's age, gender, location, medical history, scanning time, severity of COVID-19, and radiology report. The CoVID-CT dataset has more positive CT images and patients than other datasets. 2. The CoVID-HSS dataset structures all clinical data of more than 3,000 patients in the HSS hospital from more than 3,900 sets of unlabeled raw CT images. The amount of unlabeled data is large enough and the distribution is uniform enough to meet the basic assumptions of self-supervised learning and can be well used for the proposed pre-training model. Figure 4 and Figure 5 The final pre-training results on the CoVID-CT dataset and the CoVID-HSS dataset are shown respectively. The first column represents the general degree, the second column represents the severe degree, and the third column represents the extremely severe degree.

[0085] We selected three state-of-the-art semi-supervised transfer learning methods to evaluate the performance of the SegSiamPT model in classification tasks, namely Mean-teacher17, MixMatch19, and SST21. We also selected two traditional semi-supervised learning methods to evaluate the performance of the SegSiamPT model in segmentation tasks, namely U_Net and Attention U_Net, and used indicators to evaluate the segmentation results, as shown in Tables 1 and 2;

[0086] Table 1 Test results of pre-training methods on classification tasks

[0087] method COVID-CT COVID-HSS MT17 75.5% 73.3% MT17-P 83.3%(↑7.8%) 83.0%(↑9.7%) MM19 66.7% 60.3% MM19-P 75.0%(↑8.3%) 86.0%(↑25.7%) SST21 46.7% 66.7% SST21-P 88.9%(↑42.2%) 86.7%(↑20%)

[0088] The method followed by P indicates that the pre-training method proposed in the present invention is used. As can be seen from Table 1, the accuracy on the COVID-HSS dataset reached the highest 86.7%, among which the accuracy of the Mean teacher algorithm increased by 9.7% and the accuracy of the Mixmatch algorithm increased by 25.7%. For the COVID-CT dataset, the highest accuracy was 88.9%. Among them, the accuracy of the Mean teacher algorithm increased by 7.8%, and the accuracy of Mixmatch increased by 8.3%. It can be seen that when the same method is used for different datasets, the accuracy of the dataset processed by the SegSiamPT model is higher.

[0089] Table 2 Test results of pre-training methods on segmentation tasks

[0090] method Dice Prec. Sα MAE U-Net 0.204 0.878 0.524 0.116 U-Net-P 0.083 0.837 0.483 0.084 Attention-UNet 0.098 0.461 0.489 0.064 Attention-UNet-P 0.117 0.812 0.498 0.076

[0091] From the results in Table 2, it can be seen that even in the segmentation task, the pre-trained model of the present invention can improve the performance of popular segmentation methods, especially in terms of Precision and MAE indicators.

Claims

1. A pre-training method based on self-supervisory information of unlabeled medical images, characterized in that: Follow the steps below to implement it: Step 1: Build the Trans-Inf-Net segmentation network, put the medical CT image x0 into the segmentation network to segment the lesion, and obtain the lesion map x s , as self-supervision information; specifically: Step 1.1, Trans-Inf-Net segmentation network is based on the Inf-Net segmentation network, adding a transformer module to improve edge representation. The specific network framework is: the medical CT image is first sent to two convolutional layers to extract low-level features, and then the low-level features f2 are sent to three convolutional layers to extract high-level features. A parallel partial decoder is used to aggregate these features and obtain a global graph S g , combining high-level features and low-level features f2 in S g Under the guidance of , the output of the last RA is input into the reverse attention module, and finally the output of the last RA is input into a Sigmoid activation function for infection prediction of medical CT images; Step 1.2: Use 50 labeled medical CT images and 1600 unlabeled medical CT images as training sets; use 48 labeled medical CT images as test sets, and resize all images to 352×352; Step 1.3, first train the network with labeled data, resample the training set images using different scaling ratios {0.75, 1, 1.25}, then use the resampled images to train the Trans-Inf-Net segmentation network, the network is trained using the Adam optimizer, the learning rate is set to 1e-4, and finally, save the trained model; Step 1.4: Use the trained Trans-Inf-Net segmentation network to generate pseudo labels for unlabeled medical CT images; the number of images is set to 5, i.e., K=5, constituting K pseudo-label data, and the pseudo-label images are expanded into the labeled data set as training images for network training, and then these K images are removed from the data set of unlabeled medical CT images until the data set of unlabeled medical CT images is empty; Step 1.5, save the trained model, put the medical CT image x0 in the COVID-HSS dataset into the trained model for lesion segmentation, and obtain the lesion map x s ; Step 2: Build a twin neural network and combine the medical CT image x0 and the lesion image x s As the two inputs of the twin neural network, information is extracted to obtain the weight graph W; specifically: The medical CT image x0 and the lesion map x s As two inputs of the twin neural network, the two images are processed by the encoder network f, which consists of the backbone MLP and the projection MLP head; the encoder network f shares weights between the two images; the prediction MLP head h changes the output of one image and matches this output with the output of another image; the two output vectors are represented as and And minimize the negative cosine similarity, as shown in formula (1): Among them, ∥·∥2 represents l2-norm, which is the mean square error of the normalized vector l2, with a scale of 2; Symmetric loss, as shown in formula (2): Modify formula (1) to formula (3): Modify formula (2) into formula (4): The twin network outputs a dense prediction map. In order to obtain a weight map of the same size as the original image, the predicted value of each patch is flattened to all pixels in the patch as the weight of the patch, and the overlapping parts of the pixels are averaged to finally obtain a weight map W of the same size as the original image. Step 3: Use Gaussian pyramid decomposition to transform the weight map W obtained in step 2, and compare the transformed G{W} with the medical CT image x0 and the lesion map x s Fusion is performed to enhance the features of medical images and retain the original information of medical images.

2. The pre-training method based on self-supervisory information of unlabeled medical images according to claim 1, characterized in that: The encoder network is ResNet-50, with a base learning rate of 0.05, a weight decay of 0.0001, an SGD momentum of 0.9, and a batch size of 4.

3. The pre-training method based on self-supervisory information of unlabeled medical images according to claim 1, characterized in that: In the step 3, specifically: Step 3.1, transform the weight map W using Gaussian pyramid decomposition to obtain G{W}; Step 3.2, use the Laplacian pyramid to transform the medical CT image x0 and the lesion map x s Perform Laplace transform to obtain L{x s } and L{x0}, and then the obtained G{W} and L{x s } and L{x0} are fused separately to obtain L{C} and L{D}; L{C} and L{D} are fused to obtain L{F}, and L{F} is subjected to Laplace inverse transformation to obtain the final fused image x f .

4. The pre-training method based on self-supervisory information of unlabeled medical images according to claim 3, characterized in that: In step 3.2, for each Laplacian pyramid decomposition layer, L{x o } and L{x s }, as shown in formula (5); Among them, L{x o } 1 and L{x s } 1 Represents the Laplacian pyramid graph of the lth layer; and Respectively represent the image x with the center position (x, y) of the lth layer o and image x s The local area energy; The similarity measure in fusion mode determination is expressed as formula (6): Its range is in the interval [-1,1], t is the threshold determined by the fusion mode, if S l (x,y)≥t, the following formula is used for weighted average fusion: If S l (x, y) < t, then the local energy for comparison is selected in the fusion as follows:

Citation Information

Patent Citations

  • Face recognition method based on deep transformation learning in unconstrained scene

    CN107506717A

  • Watermark identification model establishment and identification method, device, medium and electronic device

    CN109118420A

  • Action sequence segmentation method based on self-supervised few-sample learning for behavior recognition

    CN114118167A

  • Image hash coding method based on direct push type semi-supervised deep learning

    CN114170333A

  • Multiple instance learner for tissue image classification

    US20220237788A1