Weakly supervised interstitial lung disease lesion identification method based on multiple-instance learning

Abstract

The application belongs to the technical field of image recognition, and discloses a weakly supervised interstitial lung disease lesion recognition method based on multiple example learning, which comprises the following steps: step 1: acquiring CT image samples; step 2: selecting part of the CT images, and manually labeling the lung parenchyma in the images; step 3: establishing a lung parenchyma segmentation model through a saliency segmentation algorithm, inputting the manually labeled CT image samples to perform training and testing, and obtaining a trained lung parenchyma segmentation model; step 4: training a lesion recognition model using a multiple example learning algorithm; step 5: acquiring a to-be-recognized CT image sample, inputting the lung parenchyma segmentation model to perform segmentation, and then inputting the segmented data sample into the lesion recognition model to obtain a lesion position. The application can realize the visual labeling of interstitial lung disease through a small amount of labeling, and greatly improves the recognition efficiency.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and particularly relates to a weakly supervised interstitial lung disease lesion recognition method based on multiple example learning. BACKGROUND

[0002] Interstitial lung disease is a clinical-pathological entity composed of different types of diseases with diffuse lung parenchyma, alveolar inflammation and interstitial fibrosis as the basic pathological changes, and with active dyspnea, diffuse infiltrative shadow on X-ray chest film, restrictive ventilation disorder, decreased diffusion function and hypoxemia as the clinical manifestations. Chest CT or high-resolution CT can more clearly show the morphological structural changes of lung tissue and interstitium, and the high-resolution CT image is obviously superior to the ordinary X-ray chest film in the diagnosis of interstitial lung disease, and has important value for the diagnosis of early lung fibrosis and honeycomb lung.

[0003] Deep learning is a new powerful tool for machine vision and pattern recognition. Through a hierarchical network structure, the original data can be effectively converted into high-level and abstract features. After training, the deep learning model can effectively segment the lung structure. Multiple example learning is a kind of weakly supervised classification, in which only the patient level label is known. In the study of interstitial lung disease detection, the CT scan result only has the label of interstitial lung disease or non-interstitial lung disease, without information about the lesion site or the severity of the lesion site.

[0004] Interstitial lung disease is a highly heterogeneous disease, and its lesions are multi-component, multi-dimensional and multi-site. At present, the lung interstitial involvement on CT images is mainly evaluated by radiologists through visual assessment, and is evaluated by subjective semi-quantitative scoring method. However, these evaluations are subjective and time-consuming, which may affect the analysis results of ILD. Therefore, it is necessary to objectively quantitatively evaluate the interstitial lung involvement on CT images by computer. And the existing methods all rely on doctor's annotation, which needs to consume a lot of energy and time of doctors.

[0005] Therefore, it is necessary to provide a new recognition method for accurately recognizing interstitial lung disease lesions, so as to reduce the requirement for sample annotation, improve the recognition efficiency and accuracy. SUMMARY

[0006] The present application overcomes the deficiencies of the prior art, and solves the technical problems, that is, to provide a weakly supervised interstitial lung disease lesion recognition method based on multiple example learning, so as to improve the accuracy and efficiency of lesion recognition, and reduce the difficulty and workload of manual annotation.

[0007] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows: a weakly supervised interstitial lung disease lesion recognition method based on multiple example learning, comprising the following steps:

[0008] Step 1: Obtain a plurality of CT image data, sample and extract each CT image data to obtain a plurality of images to establish corresponding CT image samples, each CT image sample includes a plurality of CT images extracted from the corresponding CT image, and the CT image sample is marked as an interstitial lung disease sample or a healthy sample;

[0009] Step 2: Select part of the CT image, and manually label the lung parenchyma in the image;

[0010] Step 3: Establish a lung parenchyma segmentation model through a saliency segmentation algorithm, input the manually labeled CT image obtained in step 2 into the lung parenchyma segmentation model for training and testing to obtain the lung parenchyma segmentation model;

[0011] Step 4: Train a lesion recognition model using a multi-instance learning algorithm, the process is as follows:

[0012] Step 4.1: Establish a lesion recognition model using a multi-instance learning algorithm;

[0013] Step 4.2: Select CT image samples for training the lesion recognition model, segment the lung parenchyma in the CT image through the lung parenchyma segmentation model; and cut the lung parenchyma in the CT image into a plurality of small blocks using a fixed size sliding window;

[0014] Step 4.3: Take the CT image sample of the interstitial lung disease patient as a positive bag, take the CT image sample of the healthy patient as a negative bag, take the corresponding CT image in the positive bag and the negative bag as an example, and train and test the lesion recognition model;

[0015] Step 5: Obtain a to-be-recognized CT image sample, input the lung parenchyma segmentation model obtained in step S3 for segmentation, and then input the segmented data sample into the lesion recognition model obtained in step S4 to obtain the lesion position.

[0016] When each CT image data is sampled and extracted, the corresponding CT image sample is obtained by equidistant extraction.

[0017] In step 3, the lung parenchyma segmentation model is established by a GCPANet saliency segmentation algorithm.

[0018] In step 4.2, the lung parenchyma obtained in step 3 is cut into a plurality of small blocks using a 48*48 sliding window.

[0019] The step 4 further includes the following steps:

[0020] Step 4.4: Establish a multi-instance CAM model, the multi-instance CAM model includes:

[0021] Global sampling module: used for calculating the global context information and the coarse mask of the lung parenchyma data sample;

[0022] Causal relationship module: used for sending the global context information and the coarse mask to the causal relationship module to obtain the category causal relationship graph Mc and the anatomical causal relationship graph Ms;

[0023] Cascade module: used for cascading the category causal relationship graph Mc and the lesion position feature obtained by the lesion identification model to obtain the cascade feature;

[0024] Classification head: used for performing global average pooling and full connection operation on the cascade feature to obtain the feature vector of the category in the training stage;

[0025] Mapping module: used for performing class activation mapping on the cascade feature to generate the saliency mapping feature CAMcc representing only the category causal relationship;

[0026] Multiplication module: used for multiplying the anatomical causal relationship graph Ms and the saliency mapping feature CAMcc to obtain the saliency map CAMac;

[0027] Up-sampling module: used for up-sampling the saliency map CAMac to obtain the lesion segmentation result;

[0028] Step 4.5: sending the lesion position output by the lesion identification model and the lung parenchyma data output by the lung parenchyma segmentation model to the multi-instance CAM model to train the multi-instance CAM model.

[0029] The step 5 further comprises the following steps:

[0030] Sending the segmented data sample and the lesion position obtained by the lesion identification model to the trained multi-instance CAM model to output the CT image and the lesion position thereof.

[0031] In the step 4.2, the number of CT image samples of the interstitial lung disease patients in the selected CT image samples is equal to the number of CT image samples of the healthy patients.

[0032] In the step 4.3, the ratio of the training data and the test data is 2:1.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] This invention provides a weakly supervised method for identifying interstitial lung disease lesions based on multiple instance learning. First, CT images are extracted hierarchically. Then, a saliency segmentation algorithm is used to segment the patient's lung structure, requiring only a small number of labeled lung samples for segmentation. A multiple instance learning algorithm is introduced to address the issue of different lesion locations in interstitial lung disease. Finally, two causal chains—anatomical and categorical—are used to drive the process. The anatomical causal chain addresses the co-occurrence problem in CT images, while the categorical causal chain resolves the issue of blurred lesion region boundaries, thus visualizing the lesion location. This method eliminates the need for extensive sample labeling, providing a novel approach for the automated and visual identification of interstitial lung disease lesions from CT images. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating a weakly supervised interstitial lung disease lesion identification method based on multi-instance learning, provided in Embodiment 1 of the present invention.

[0036] Figure 2 This is a schematic diagram of the lung parenchyma segmentation model in Embodiment 1 of the present invention;

[0037] Figure 3 A flowchart illustrating a weakly supervised interstitial lung disease lesion identification method based on multi-instance learning provided in Embodiment 2 of the present invention;

[0038] Figure 4 This is a schematic diagram of the global sampling module in Embodiment 2 of the present invention;

[0039] Figure 5 This is a schematic diagram of the causal relationship module in Embodiment 2 of the present invention;

[0040] Figure 6 This is a schematic diagram illustrating the visualization of lesion locations in different slice CT images of a patient obtained through the method of Embodiment 2 of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] like Figure 1 As shown, this embodiment of the invention provides a weakly supervised method for identifying interstitial lung disease lesions based on multi-instance learning, comprising the following steps:

[0043] Step 1: Obtain CT image data of a plurality of patients, sample each CT image data to obtain a plurality of images to establish corresponding CT image samples, wherein the CT image data of each patient corresponds to a CT image sample, and each CT image sample includes a plurality of CT images extracted from the corresponding CT image, and the CT image sample is marked as an interstitial lung disease sample or a healthy sample.

[0044] Since a 3D CT image of a patient can include 200-500 images, and adjacent images are highly similar, in this embodiment, the CT image of each patient is extracted every 20 images starting from number 0 as a sample, and a CT image sample including a plurality of CT images can be obtained, which can be specifically represented as: ID n = {ID n-1 , ID n-2 , ID n-3 , ……, ID n-m}. ID n represents the nth CT image sample, ID n-1 , ID n-2 , ID n-3 , ……, ID n-m indicates the 0th, 1st, 2nd, ……, mth CT image in the nth CT image sample ID n .

[0045] Step 2: Select part of the CT images for manual annotation.

[0046] In this embodiment, only part of the CT images are selected for manual annotation. When annotating, the lung parenchyma in part of the samples can be annotated using the Labelme annotation software. The class label of the lung parenchyma annotation is set as lung. Since the annotated file is in json format, this embodiment converts it into png format, and the generated folder includes the class name file, the original image file in png format, the segmented image in png format, and the png format of the segmented image superimposed on the original image.

[0047] Step 3: Establish a lung parenchyma segmentation model through a saliency segmentation algorithm, input the CT image samples manually annotated in step 2 into the lung parenchyma segmentation model for training and testing, and through training and testing of the lung parenchyma segmentation model, the lung parenchyma segmentation model obtained can segment the lung parenchyma of all CT images in all CT image samples.

[0048] Specifically, in step 3, the lung parenchyma segmentation model is established through a GCPANet saliency segmentation algorithm, such as Figure 2As shown, GCPANet is a symmetric encoder-decoder architecture, in which the encoder component extracts multi-level features based on ResNet-50, and the decoder component progressively integrates multi-level comprehensive features to generate saliency maps in a supervised manner.

[0049] The model comprises:

[0050] A feature interweaved aggregation (FIA) module that fully fuses three levels of features, i.e., high-level features output by the previous layer, low-level features corresponding to the bottom layer, and global context features generated by the GCF module, to generate features with global perception, distinction, and comprehensiveness.

[0051] A self-refinement (SR) module that combines complementary features between different levels of features to obtain comprehensive expression of the features. This method is simple and intuitive, and can directly apply a softmax layer after the FIA module to obtain a saliency map, but still has some defects. For example, there are some holes in the predicted saliency target, which is caused by contradictory responses of different levels. Therefore, after passing through the HA module and the FIA module, an SR module is developed to further refine and enhance the feature map using multiplication and addition operations.

[0052] A head attention (HA) module that is designed to follow the top layer of the encoder component to learn more selective and representative features by utilizing attention mechanisms in space and channels, since the top layer features of the encoder component are usually redundant for prominent target detection.

[0053] A cascade module that comprises three FIA modules and three SR modules, wherein an SR module is cascaded after each FIA module.

[0054] A global context flow (GCF) module that captures global context information embedded in the FIA module at each stage. First, a global average pool is used to obtain global context information, and then different weights are re-assigned to different channels of the global context feature map at each stage.

[0055] In the lung parenchyma segmentation model, first, the HA module is used to enhance the spatial region and feature channel with high response to the salient object, and then the SR module is used to generate high-level features in the first stage through feature optimization and enhancement. Then, the FIA module (Feature Interweaved Aggregation) and the SR module are cascaded in three stages step by step to understand more discriminative functions and generate more accurate saliency maps. In the FIA module, low-level detailed information, high-level semantic information and global context information are fused in an interwoven manner. The SR module after each FIA module will perfect the coarse aggregation function. Note that the global context information is generated by the Global Context Flow (GCF) module, which captures the relationship between different salient regions and limits more complete saliency prediction. To facilitate optimization, the auxiliary loss branch of each sub-stage is combined with the main loss.

[0056] Step 4: training a lesion identification model using a multi-instance learning algorithm to identify the lesions of interstitial lung disease in the lung parenchyma, the process is as follows:

[0057] Step 4.1: establishing a lesion identification model using a multi-instance learning algorithm;

[0058] Step 4.2: selecting CT image samples for training the lesion identification model, segmenting the lung parenchyma in the CT images in the CT image samples through the lung parenchyma segmentation model; and cutting the lung parenchyma in the CT images in the CT image samples into a plurality of small blocks using a fixed-size sliding window;

[0059] Step 4.3: taking the CT image samples of patients with interstitial lung disease as positive bags, the CT image samples of healthy patients as negative bags, and the corresponding CT images in the positive and negative bags as examples, training and testing the lesion identification model;

[0060] Wherein, the lesion identification model completed through training and testing can identify the lesion positions in all CT images.

[0061] In step 4.2, the lung parenchyma obtained in step 3 is cut into a plurality of small blocks using a 48*48 sliding window. In step 4.2, the number of CT image samples of patients with interstitial lung disease in the selected CT image samples is equal to the number of CT image samples of healthy patients, that is, the number of positive bags is equal to the number of negative bags, so that the data of the lesion identification model is more balanced and the identification is more accurate.

[0062] Further, in step 4.2, when the lesion identification model is trained and tested, 2 / 3 of the CT images therein are selected as training data and 1 / 3 of the images are selected as test data, that is, the ratio of training data to test data is 2:1.

[0063] Step 5: Obtain the CT image sample to be identified, input it into the lung parenchyma segmentation model obtained in step S3 to segment it and obtain the corresponding lung parenchyma data, and then input the segmented lung parenchyma data into the lesion identification model obtained in step S4 to obtain the lesion location.

[0064] Example 2

[0065] like Figure 3 As shown, unlike Embodiment 1, in this embodiment, step 4 further includes the following step:

[0066] Step 4.4: Establish a multi-instance CAM model, which includes:

[0067] Global sampling module: used to calculate the global context information and coarse mask of lung parenchymal data samples;

[0068] Causal relationship module: Used to send global context information and coarse mask to the causal relationship module to obtain the category causal relationship graph Mc and the dissection causal relationship graph Ms;

[0069] Cascade module: used to cascade the category causal relationship graph Mc with the lesion location features obtained from the lesion identification model to obtain cascade features;

[0070] Classification head: Used during the training phase to perform global average pooling and fully connected operations on cascaded features to obtain the feature vector of the category;

[0071] Mapping module: used to perform class activation mapping on cascaded features to generate salient mapping features CAMcc that represent only class causal relationships;

[0072] Multiplication module: Used to multiply the anatomical causal relationship map Ms with the saliency mapping feature CAMcc to obtain the saliency map CAMac;

[0073] Upsampling module: used to upsample the saliency map CAMac to obtain lesion segmentation results;

[0074] Step 4.5: Send the lesion location output by the lesion recognition model and the lung parenchyma data output by the lung parenchyma segmentation model to the multi-instance CAM model to train the multi-instance CAM model.

[0075] In this embodiment, the multi-instance CAM model generates a global context M through the global sampling module (GS). GCand coarse masks. Moreover, the category causal graph Mc and the anatomical causal graph Ms are calculated by the causal relationship module. Then the Mc is concatenated with the multi-instance learned features. The concatenated features are fed into the classification head (global average pooling + fully connected layer) in the training phase. In the inference phase, the class activation map CAMcc representing the saliency map with only category causal relationship is generated. The saliency map CAMac representing the saliency map with both category causal relationship and anatomical causal relationship is generated by multiplying the saliency map CAMcc and the anatomical causal graph Ms. Finally, the saliency map CAMac is up-sampled to obtain the segmentation result.

[0076] As shown in Figure 4 , in the embodiment, the global sampling module is used to extract the global context containing category and anatomical information. The global sampling module includes a convolutional neural network, a classification head, a mapping operation module and an up-sampling module.

[0077] First, after the picture is input, the feature map is generated by the convolutional neural network CNN, and then input into the classification head to obtain the category feature vector. Then the mapping operation module generates a feature map for each category, and finally the up-sampling module is used for up-sampling operation to obtain the coarse mask. Then the coarse masks are summed to obtain the global context information M GC .

[0078] As shown in Figure 5 , the causal relationship module inputs the coarse mask obtained by the global sampling module and the global context information M GC after segmenting the lung parenchyma into the reconstruction layer for reconstruction, and then the reconstructed coarse mask and the global context information are respectively mapped to the same space by one convolutional layer, and then a class-aware attention vector Ac is obtained by a softmax function. Finally, the category causal graph Mc is obtained by formula (1); in the anatomical relationship chain, a 1 / 0 indicator is designed, as shown in formula (2), to obtain the possible position of each category, and the anatomical causal graph Ms is obtained; Ms is down-sampled and multiplied by CAMcc to obtain CAMac, and then up-sampled to obtain the final pseudo-segmentation mask.

[0079] M c = Down (A c x M GC ) (1)

[0080]

[0081] The step 5 further includes the following steps: sending the segmented data sample and the lesion position obtained by the lesion identification model to the trained multi-instance CAM model, and outputting the CT image and the lesion position thereof.

[0082] In the embodiment, two causal chains of anatomical causal chain and category causal chain are introduced by the multi-instance CAM model, wherein the anatomical causal chain solves the co-occurrence problem existing in the CT image, avoids different lesions in the same image, the category causal chain solves the problem of fuzzy boundary of lesion region segmentation, and then realizes the visualization of the lesion position.

[0083] In the embodiment of the application, 405 patient (19000) CT image data are used as source data, 200 image samples are selected for manual annotation, then the lung parenchyma segmentation model is trained by the 200 CT images, and the remaining 18800 CT images are segmented for lung parenchyma; subsequently, 99 patients with diseases and 99 normal patients are selected, and a total of 15999 CT image data are used as data samples, 10666 image data are set as training samples, and 5333 image data are set as test samples, the lesion recognition model is trained by the multi-instance learning algorithm, and the training is performed by taking the patient as a package and the picture as an example. Figure 6 As shown in the figure, the lesion position in the CT image of a certain patient at different layers is obtained by the method of the application.

[0084] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, but not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A weakly supervised interstitial lung disease lesion identification method based on multiple-instance learning, characterized in that, The method comprises the following steps: Step 1: Obtain a plurality of CT image data, sample and extract each CT image data to obtain a plurality of images to establish corresponding CT image samples, each CT image sample comprising a plurality of CT images extracted from the corresponding CT image, and the CT image sample being marked as an interstitial lung disease sample or a healthy sample; Step 2: Select part of the CT images, and manually label the lung parenchyma in the images; Step 3: Establish a lung parenchyma segmentation model through a saliency segmentation algorithm, input the manually labeled CT images obtained in step 2 into the lung parenchyma segmentation model for training and testing to obtain the lung parenchyma segmentation model; Step 4: Train a lesion recognition model using a multi-instance learning algorithm, and the process is as follows: Step 4.1: Establish a lesion recognition model using a multi-instance learning algorithm; Step 4.2: Select CT image samples for training the lesion recognition model, segment the lung parenchyma in the CT images in the CT image samples through the lung parenchyma segmentation model, and cut the lung parenchyma in the CT images in the CT image samples into a plurality of small blocks using a fixed-size sliding window; Step 4.3: Take the CT image samples of interstitial lung disease patients as positive bags, take the CT image samples of healthy patients as negative bags, take the corresponding CT images in the positive bags and the negative bags as instances, and train and test the lesion recognition model; Step 4.4: Establish a multi-instance CAM model, and the multi-instance CAM model comprises: A global sampling module for calculating global context information and a coarse mask of the lung parenchyma data sample; A causal relationship module for sending the global context information and the coarse mask to the causal relationship module to obtain a category causal relationship graph Mc and an anatomical causal relationship graph Ms; A cascading module for cascading the category causal relationship graph Mc and lesion position features obtained by the lesion recognition model to obtain cascaded features; A classification head for performing global average pooling and full connection operations on the cascaded features to obtain a feature vector of the category in the training stage; A mapping module for performing class activation mapping on the cascaded features to generate a saliency mapping feature CAMcc representing only the category causal relationship; A multiplication module for multiplying the anatomical causal relationship graph Ms and the saliency mapping feature CAMcc to obtain a saliency map CAMac; An up-sampling module for up-sampling the saliency map CAMac to obtain a lesion segmentation result; Step 4.5: Send the lesion position output by the lesion recognition model and the lung parenchyma data output by the lung parenchyma segmentation model to the multi-instance CAM model to train the multi-instance CAM model; Step 5: Obtain a to-be-recognized CT image sample, input the lung parenchyma segmentation model obtained in step S3 for segmentation, and then input the segmented data sample into the lesion recognition model obtained in step S4 to obtain a lesion position.

2. The weakly supervised interstitial lung disease lesion identification method based on multiple-instance learning according to claim 1, characterized in that, When sampling and extracting each CT image data, equidistant extraction is performed to obtain the corresponding CT image sample.

3. The weakly supervised interstitial lung disease lesion identification method based on multiple-instance learning according to claim 1, characterized in that, In step 3, the lung parenchyma segmentation model is established through a GC-PANet saliency segmentation algorithm.

4. The weakly supervised interstitial lung disease nodule identification method based on multiple-instance learning according to claim 1, characterized in that, The step 4.2, use 48 48 sliding window will be cut into a plurality of small pieces of lung parenchyma obtained in step 3.

5. The weakly supervised interstitial lung disease nodule identification method based on multiple-instance learning according to claim 1, characterized in that, The step 5 further comprises the following steps: The segmented data sample and the lesion position obtained by the lesion identification model are sent to the trained multi-instance CAM model, and a CT image and its lesion position are output.

6. The weakly supervised interstitial lung disease nodule identification method based on multiple-instance learning according to claim 1, characterized in that, In step 4.2, the number of CT image samples of the interstitial lung disease patients in the selected CT image samples is equal to the number of CT image samples of the healthy patients.

7. The weakly supervised interstitial lung disease nodule identification method based on multiple-instance learning according to claim 1, characterized in that, In step 4.3, the ratio of the training data to the test data is 2:1.

Images