Vessel lumen segmentation method based on self-supervised learning and sparse annotation

Abstract

The present application relates to a kind of medical image processing techniques based on machine learning, provide a kind of based on self-supervised learning and sparse annotation intravascular image lesion segmentation method.First, collect intravascular image data in clinic, and it is screened and preprocessed, then randomly selected small part of data is carried out with the way of interval several frames pixel-level sparse annotation;Through self-supervised learning, pre-training is carried out on a large number of unlabeled data, then the segmentation model is constructed by transfer learning, and fine-tuning training is carried out on sparse annotation data.Finally, the prediction of intravascular lesion is generated and quantitatively analyzed.The present application can be used for large-scale intravascular medical image data to quickly establish effective lesion analysis model.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of image processing techniques based on machine learning, and specifically to a kind of target segmentation techniques of medical image based on self-supervised learning and sparse annotation. BACKGROUND

[0002] Intracoronary imaging is a powerful imaging technique for assessing coronary artery disease (CAD) and planning percutaneous coronary intervention (PCI), and can visualize and characterize plaque within the coronary artery. However, during PCI, physicians cannot thoroughly and quickly analyze a large number of images. Even for research purposes, manual analysis of IVUS or IVOCT pullback can take several hours and is also prone to inter-observer and intra-observer analysis bias.

[0003] Deep learning methods have achieved remarkable success in various medical image segmentation tasks. However, most state-of-the-art segmentation models are deep and wide networks, whose strong generalization ability highly depends on large-scale and high-quality pixel-level annotation data. Due to the need for experienced clinical experts to guide, pixel-level manual annotation of medical images is an expensive and time-consuming process. In addition, since IVUS and IVOCT images are acquired in a fast helical scanning manner, adjacent slices have a large amount of redundant information. Most existing three-dimensional image segmentation methods such as 3D U-Net are driven by frame-by-frame dense annotation, which is a great challenge for IVUS or IVOCT image annotators. In contrast, 2D processing methods such as 2D U-Net ignore the 3D continuity of intravascular lesions in space and have limited performance in lesion detection. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a method for training an intravascular fully automatic lesion segmentation model using massive unlabeled data and limited sparse annotation data.

[0005] The technical solution adopted by the present application to solve the above technical problem is an intravascular image lesion segmentation method based on self-supervised learning and sparse annotation, comprising the steps of:

[0006] I. Data set construction:

[0007] 1) Collect intravascular image data and perform preprocessing;

[0008] 2) Randomly select a small part of the preprocessed intravascular image data as to-be-labeled data, and the remaining unselected intravascular image data as unlabeled data.

[0009] 3) constructing a labeled dataset based on sparse annotation of interval frames for the selected to-be-labeled data, wherein the sparse annotation of interval frames is to select a frame at a certain interval in the preprocessed intravascular image data, and to perform pixel-level annotation on the selected whole-frame image as a labeled frame, and the frames that are not selected are not annotated as unlabeled frames;

[0010] II. Training steps:

[0011] 1) self-supervised pre-training:

[0012] using unlabeled data to pre-train a Transformer encoder for capturing intravascular image features;

[0013] 2) inputting the training set in the labeled dataset into the automatic segmentation model for fine-tuning training, and the specific training method is:

[0014] first, migrating the Transformer encoder obtained by self-supervised pre-training to the automatic segmentation model;

[0015] then, performing two kinds of reversible random data augmentation on the same image sequence in the training set in the sparse labeled dataset, to obtain a first input image sequence and a second input image sequence; the two kinds of reversible random data augmentation are that the image sequence obtained after using one kind of random data augmentation on the original image sequence and then performing data augmentation inverse transformation is the same as the image sequence obtained by using another kind of random data augmentation on the original image sequence;

[0016] after the first input image sequence and the second input image sequence pass through the automatic segmentation model based on deep learning for capturing intravascular image features and predicting 3D lesion segmentation, first and second prediction results are obtained; then, a first inverse transformation result is obtained by performing data augmentation inverse transformation on the first prediction result;

[0017] in the fine-tuning training process, a segmentation loss that compares the similarity between the predicted part corresponding to the labeled frame in the first prediction result and the true segmentation label, and a contrastive loss that compares the consistency of the first inverse transformation result obtained by performing data augmentation inverse transformation on the first prediction result and the second prediction result in the same feature space are used to constrain the fine-tuning training process of the encoding and decoding module;

[0018] in the verification process of training, the performance indicators of the automatic segmentation model on the validation set in the sparse labeled dataset are calculated, and then the best automatic segmentation model hyperparameters are selected, wherein the performance indicators on the validation set are the segmentation performance evaluation for the labeled frames; although two loss functions, the segmentation loss and the contrastive loss, are used in the fine-tuning training process, only the labeled frames, i.e., the results of true manual annotation, are used for evaluation when selecting hyperparameters and evaluating performance.

[0019] III. Test steps:

[0020] The intravascular image data to be processed is input into the trained automatic segmentation model, and the automatic segmentation outputs the prediction result of the intravascular image lesion segmentation. Although the automatic segmentation model uses sparse annotation data in the training process, the trained automatic segmentation model can generate dense prediction of the lesion, that is, there is a prediction result about the lesion for all the input frames.

[0021] The beneficial effects of the present application are: compared with the traditional intravascular image data annotation and analysis method, and different from the existing sparse annotation method of marking a small part of the region pixels in one frame, the present application uses the sparse annotation method of marking intravascular medical image data at intervals of several frames, which can greatly reduce the workload of the annotators. The pre-training framework based on self-supervised learning makes full use of the massive unannotated data, effectively improves the extraction ability of the deep learning model for the deep features of the intravascular image, and further provides high-quality initialization parameters for the intravascular lesion segmentation. The lesion segmentation network designed based on sparse annotation data effectively utilizes the plane and spatial information, and can realize high-performance segmentation of the lesion under limited annotation data. Thus, it is helpful to quickly establish an effective lesion analysis model for large-scale intravascular medical image data. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the annotation and model training and evaluation schematic diagram proposed by the embodiment.

[0023] Figure 2 is a schematic diagram of an intravascular image sparse annotation method proposed by the embodiment.

[0024] Figure 3 is a schematic diagram of an intravascular image automatic segmentation model training framework based on sparse annotation proposed by the embodiment. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings.

[0026] In order to make full use of massive unannotated data and establish a full-automatic intravascular lesion segmentation model under limited annotation data, as shown in Figure 1 , a kind of intravascular image lesion automatic segmentation method based on self-supervised learning and sparse annotation, comprising the following steps:

[0027] S1, intravascular image data collection; the intravascular image data is intravascular ultrasound data or intravascular optical coherence tomography (OCT) data;

[0028] S2, intravascular image data screening and preprocessing;

[0029] S3, sparse annotation based on interval frames is performed on a small part of the data selected randomly;

[0030] S4, self-supervised pre-training of the feature extractor is performed on a large amount of unlabeled data; the feature extractor is a Transformer encoder;

[0031] S5, fine-tuning training of the automatic segmentation model is performed on the sparse labeled data;

[0032] S6, the trained automatic segmentation model is used to complete the three-dimensional spatial prediction analysis of intravascular lesions.

[0033] Specifically, the collection of intravascular image data in step S1 is to collect intravascular image data from hospital and other medical center data systems according to task requirements and under the guidance of clinicians, determine the modality and data size of the collected data elements, and organize the data according to patients as the basic unit and group them according to the patient's condition.

[0034] As shown in Figure 2 steps S2 and S3 specifically include:

[0035] The collected intravascular image data is converted from polar coordinate domain to Cartesian coordinate system which is more familiar to clinicians after data processing such as fast Fourier transform (FFT); data screening is performed to exclude data with poor imaging quality and too short imaging time; and finally, the data is preprocessed such as contrast adjustment and size transformation.

[0036] The preprocessed intravascular image data is divided into (to be) labeled data and unlabeled data. The sparse labeling process is completed on the (to be) labeled data. A small part of the data is randomly selected as the to-be-labeled data, and the data that is not selected is used as the unlabeled data.

[0037] For example, a total of 100 intravascular imaging pullbacks (each pullback contains several image frames) are collected and organized, 10 of which are randomly selected as (to be) labeled data, and the remaining 90 are used as unlabeled data. The sparse labeling labels the 10 (to be) labeled videos, which are divided into labeled frames and unlabeled frames according to a certain interval. In other words, the unlabeled frames come from the pullbacks that are sparsely labeled, not from the unlabeled pullbacks.

[0038] For the to-be-labeled data, in the polar coordinate domain or the Cartesian coordinate domain, a frame is selected along the pullback direction according to a certain labeling interval, and a pixel-level annotation is performed on the selected whole frame image by an intravascular imaging expert or an experienced labeler to serve as a labeled frame. The unselected image frames are not annotated as unlabeled frames.

[0039] The annotation interval can be set as a fixed value, or can be set as different values according to the length, size, rarity and identification difficulty of different lesions in the blood vessels, and the like. For example, for cholesterin crystals and the like in the blood vessels which are short in length and small in size, at least one frame of image containing the lesion is annotated; for lesions which are not high in clarity or difficult to identify, a frame or a plurality of frames which are relatively clear and easy to identify are selected from the related consecutive frames for annotation.

[0040] The sparsely annotated intravascular image data is divided into a training set, a validation set and a test set according to a certain ratio (for example, 6:2:2) as training data.

[0041] In specific operation, in order to more rigorously complete the annotation work, the annotation consensus can be reached by no less than two primary or intermediate annotators, and the annotation conflict is solved by a senior annotator or a clinical image specialist.

[0042] Step S4 includes:

[0043] Step S4 is implemented by an intravascular image pre-training framework based on self-supervision, self-supervised pre-training is performed on a large amount of unlabeled intravascular image data, so as to obtain a feature encoder which fully captures the features of the intravascular image. The pre-training framework obtains a to-be-input image sequence in which a part of the region is occluded by randomly occluding a region of an unlabeled image sequence. Then, the to-be-input image sequence is input into a Transformer encoder to capture high-dimensional and deep-level features of the intravascular image, and after further passing through a self-supervised agent task head, a loss evaluation for image reconstruction, rotation angle prediction, local and global feature consistency and other self-supervised agent tasks is obtained, and iterative training is performed.

[0044] The random region occlusion refers to that a 3D image sequence is divided into a plurality of patches according to a pre-set patch size; then a part of the patches in the divided patch set is occluded by taking one patch as a basic unit according to a certain ratio. Finally, only the unoccluded patches are positionally encoded and input into the Transformer encoder. The pre-set patch can be a 2D patch, a 3D patch, an A-line or a plurality of continuous A-lines, and the like.

[0045] The Transformer encoder is a Vision Transformer encoder or a Swin Transformer encoder.

[0046] Wherein, the self-supervised agent task head is different according to different agent tasks. For example, the agent task head of image reconstruction is a Transformer or CNN decoder; and the agent task head of rotation angle prediction is a multi-layer perceptron (MLP).

[0047] Step S5 comprises:

[0048] Through the way of transfer learning, the pre-training encoder obtained by self-supervised pre-training in step S4 is transferred to the sparse annotation-based intravascular image automatic segmentation framework.

[0049] As shown in Figure 3 The training framework of the automatic segmentation model for capturing intravascular image features and predicting 3D lesion segmentation includes a Transformer pre-training encoder, a CNN decoder, data augmentation transformation and inverse transformation, and segmentation loss and contrast loss calculation.

[0050] The Transformer pre-training encoder has the same structure as the Transformer encoder in the self-supervised-based intravascular image pre-training framework. The Transformer pre-training encoder is parameterized after the Transformer encoder is pre-trained. The Transformer pre-training encoder and the CNN decoder are combined to build the intravascular image automatic segmentation model.

[0051] Through the sparse annotation-based intravascular image automatic segmentation model training framework, the intravascular image automatic segmentation model is fine-tuned on the sparse annotation image data.

[0052] During the training process, the automatic segmentation model calculates the segmentation performance indicators on the annotated frames on the validation set, and then selects the best hyperparameters; after the training is completed, the final performance evaluation of the automatic segmentation model is obtained on the test set.

[0053] Wherein, the CNN decoder is composed of a series of convolutional blocks; each convolutional block is composed of a unit of multiple convolutional layers-batch normalization-activation function, and the convolutional blocks are connected by upsampling modules.

[0054] The specific process of fine-tuning training is as follows:

[0055] For a sparse-labeled image sequence, two image sequences to be input, View 1 and View 2, are obtained through two kinds of reversible random data augmentation; wherein, the data augmentation transformation and inverse transformation are a series of reversible data augmentation transformation pairs. For example, plane flipping, back direction flipping, rotation resampling, rotation direction re-stitching, etc. The two kinds of reversible random data augmentation are that the image sequence obtained after using one kind of random data augmentation on the original image sequence and then performing data augmentation inverse transformation is the same as the image sequence obtained by using another kind of random data augmentation on the original image sequence.

[0056] The image sequences to be input, View 1 and View 2, are input into the intravascular image automatic segmentation model composed of the Transformer pre-training encoder and the CNN decoder, and 3D prediction results Pred 1 and Pred 2 of the respective lesions are formed. For the prediction results corresponding to the labeled frames in View 1, the segmentation loss loss 1 between the prediction results and the true manual labeling results is calculated. In addition, the inverse transformation result 1 of the 3D prediction Pred 1 of the image sequence View 1 is obtained by data augmentation inverse transformation, and the contrast loss loss 2 between the inverse transformation result 1 and the 3D prediction Pred 2 of the image sequence View 2 is further calculated. Then, the above process is repeated so that the intravascular image automatic segmentation model is iteratively trained on the sparse-labeled dataset with a small learning rate.

[0057] The segmentation loss is used to reflect the similarity between the prediction part corresponding to the labeled frames in the 3D prediction result Pred 1 and the true segmentation label, and is realized by using common segmentation loss functions such as Dice loss and combinations thereof. The contrast loss is used to reflect the consistency of the inverse transformation result 1 and the 3D prediction result Pred 2 in the same feature space, and is realized by using common contrast loss functions such as mean square error (MSE) and combinations thereof.

[0058] Step S6 includes:

[0059] For the intravascular image data collected in clinical practice, the same data preprocessing as in S2 is performed; for the preprocessed data, X consecutive frames are extracted from the starting frame and input into the trained intravascular image automatic segmentation model to form a prediction of the lesions / tissues in the X frames, and then the above process is repeated at intervals of Y until all the data is processed, the 3D prediction of the intravascular lesions is completed, and then quantitative analysis is performed. The values of X and Y are pre-set according to actual needs.

[0060] The above-described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

Claims

1. A method for intravascular image lesion segmentation based on self-supervised learning and sparse annotation, characterized in that, The method comprises the steps of: I. Data set construction step: 1) Collecting intravascular image data and preprocessing; 2) Randomly selecting a small part of the preprocessed intravascular image data as the to-be-labeled data, and the remaining intravascular image data as the unlabeled data; 3) Constructing a sparse annotation data set based on interval frame sparse annotation for the selected to-be-labeled data, wherein the interval frame sparse annotation is to select a frame at a certain interval in the preprocessed intravascular image data, and to perform pixel-level annotation on the selected whole frame image as the annotation frame, and the unselected frame image is not annotated as the unannotated frame; II. Training step: 1) Self-supervised pre-training: Using the unlabeled data to pre-train the Transformer encoder for capturing intravascular image features; 2) Inputting the training set in the sparse annotation data set into the automatic segmentation model for fine-tuning training, wherein the automatic segmentation model comprises a Transformer pre-training encoder and a CNN decoder, and the specific training method is as follows: First, the pre-trained Transformer encoder is migrated to the automatic segmentation model, that is, the Transformer encoder obtained by self-supervised pre-training is used to initialize the parameters of the Transformer pre-training encoder in the automatic segmentation model; Two kinds of reversible random data augmentation are performed on the same image sequence in the training set, respectively obtaining a first input image sequence and a second input image sequence; the two kinds of reversible random data augmentation are the same as the image sequence obtained by using another kind of random data augmentation on the original image sequence after data augmentation inverse transformation; Then, the first input image sequence and the second input image sequence are respectively input into the automatic segmentation model based on deep learning for capturing intravascular image features and predicting 3D lesion segmentation, to obtain a first prediction result and a second prediction result; and the first prediction result is subjected to data augmentation inverse transformation to obtain a first inverse transformation result; In the fine-tuning training process, the segmentation loss between the predicted part corresponding to the annotation frame in the first prediction result and the true segmentation label, and the consistency comparison loss of the first inverse transformation result obtained by data augmentation inverse transformation of the first prediction result and the second prediction result in the same feature space are used to constrain the fine-tuning training process of the encoding and decoding module; In the verification process of the fine-tuning training process, the performance indicators of the automatic segmentation model on the validation set in the sparse annotation data set are calculated, and then the best automatic segmentation model hyperparameters are selected, wherein the performance indicators on the validation set are the segmentation performance evaluation for the annotation frame; III. Testing step: Input the to-be-processed intravascular image data into the trained automatic segmentation model, and output the lesion segmentation prediction result of all intravascular image frames by the automatic segmentation. The iterative process of the self-supervised pre-training comprises: obtaining an image sequence to be input in which a partial region is blocked by random region blocking from the unlabeled data pair, inputting the to-be-input voxel into a Transformer encoder to capture high-dimensional and deep-level features of the intravascular image, and obtaining loss evaluation of the self-supervised proxy task for image reconstruction, rotation angle prediction, and local and global feature consistency after the self-supervised proxy task head.

2. The method of claim 1, wherein, The sparse annotation based on the interval frame is to select a frame for annotation along the pullback direction at a certain annotation interval in the polar coordinate domain or the Cartesian coordinate domain in the preprocessed intravascular image data; the annotation interval can be a pre-set fixed value or different values according to the length, size, rarity and identification difficulty of different intravascular lesions.

3. The method of claim 1, wherein, The random region blocking is random blocking of the image sequence of the intravascular image by using 2D Patch, 3D Patch, A-line and multiple continuous A-lines at a certain ratio.

4. The method of claim 1, wherein, The Transformer encoder is a VisionTransformer encoder and its variants or a Swin Transformer encoder and its variants.

5. The method of claim 1, wherein, The self-supervised proxy task loss function comprises a regression loss and a contrast loss and their variants and combinations.

Images