An intelligent analysis method and system for chest X-rays based on deep self-supervised contrastive learning

Abstract

The present invention proposes an X-ray chest film intelligent analysis method and system based on deep self-supervised contrastive learning. This method extracts image features from the X-ray chest film in the image feature encoding module and outputs an image representation. The image feature mapping module obtains an image feature view based on the image representation. The lesion feature enhancement module extracts radiomics information based on the X-ray chest film and the image representation. The radiomics feature encoding module extracts radiomics features from the radiomics information and outputs the radiomics representation. The radiomics feature mapping module obtains an radiomics feature view based on the radiomics representation. The image feature view and the radiomics feature view perform self-supervised deep contrastive learning in a feedback loop to update the network parameters. The discrimination module performs disease classification and lesion localization on the image representation output by the image feature encoding module. This method improves the accuracy of disease classification and significantly improves the localization accuracy of the lesion area, and its localization result is more robust and accurate.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an X-ray chest film intelligent analysis method and system based on deep self-supervised comparative learning. Background Art

[0002] Chest X-rays (CXRs) are quick and inexpensive to obtain and are one of the most commonly used screening methods for heart, chest, and lung diseases. In clinical practice, each chest X-ray requires radiologists to extract pathological information from complex images and analyze and determine the disease category. This process is time-consuming and labor-intensive, placing a tremendous strain on medical resources. Due to the complex and variable texture details of diseases, missed diagnoses and misdiagnoses are prone to occur. Furthermore, there is uncertainty in physicians' subjective decisions, and even two professional radiologists may still have different diagnoses. Therefore, utilizing new-generation artificial intelligence technologies, such as deep learning, to implement computer-aided diagnosis (CAD) on chest X-rays is of great significance, as it can quickly and accurately assist in analysis and provide a reference for clinical decision-making. Specifically, realizing intelligent analysis of chest X-rays includes two subtasks: one is to realize category judgment for chest X-rays, that is, to determine whether the chest X-ray is in a normal state or an abnormal state. Furthermore, it also includes analysis of the specific abnormal type of the abnormal chest X-ray; the other is to locate the lesion area of the abnormal chest X-ray; the lesion area is the local image area that can most closely represent the pathological condition. It can usually provide richer analysis information for clinical decision-making (based on automatic or manual), so its accurate positioning is of great significance.

[0003] At present, building an intelligent auxiliary analysis system for chest X-rays faces a series of challenges, including:

[0004] (1) Establishing a highly accurate analysis system for intelligent assisted analysis of chest X-rays usually requires a large number of manually labeled state labels and abnormal region (lesion region) bounding boxes. However, identifying abnormalities in chest X-rays usually requires professional radiologists. Therefore, it is costly to obtain accurate annotations of chest X-ray data; in particular, obtaining labeled bounding boxes for lesion regions is even more time-consuming and costly. To address this problem, contrastive learning in self-supervised learning proposed in recent years has shown excellent ability to process unlabeled data. Popular self-supervised learning models include MoCo, SimCLR, PIRL, and BYOL, which have shown great potential in utilizing unlabeled natural images to generate highly generalizable and identifiable features and have achieved success in natural image analysis tasks such as image classification and object detection. In addition, when only a small amount of labeled data is available, contrastive learning is more robust than semi-supervised learning. Moreover, contrastive learning is also robust to data with imbalanced data categories. Therefore, contrastive learning provides a promising direction for realizing intelligent analysis of a large number of unlabeled chest X-rays. However, at present, there is little research on extending its functionality to the field of medical imaging, and it is more difficult because medical images have much stricter requirements on the modifiability of data augmentation.

[0005] (2) The biggest difficulty in using contrastive learning in intelligent analysis of chest X-rays is that most existing contrastive learning frameworks rely heavily on maximizing the similarity between two “feature views”, namely the anchor and its positive sample, which are usually generated by applying random data augmentation methods to the same image. However, this data augmentation strategy is not easy to apply directly to chest X-ray images. Unlike natural images, chest X-rays have very subtle and similar image features. The most discriminative features only exist in local areas. Therefore, chest X-rays are very sensitive to distortion and typical natural image data augmentation methods (such as random cropping, color jittering, adding noise, etc.) cannot be directly used.

[0006] (3) The simultaneous demand for classification features and positioning features further complicates the problem. Summary of the Invention

[0007] In order to overcome the defects in the above-mentioned prior art, the purpose of the present invention is to provide an X-ray chest film intelligent analysis method and system based on deep self-supervised comparative learning.

[0008] In order to achieve the above-mentioned object of the present invention, the present invention provides an X-ray chest film intelligent analysis method based on deep self-supervised contrastive learning, comprising the following steps:

[0009] Obtain chest X-ray samples;

[0010] Construct an analysis model, which includes an image feature extraction subnetwork, an imaging genomics feature extraction subnetwork and a lesion feature enhancement module; the image feature extraction subnetwork includes an image feature encoding module, an image feature mapping module and a discrimination module; the imaging genomics feature extraction subnetwork includes an imaging genomics feature encoding module RFE r and radiomics feature mapping modules;

[0011] The image feature encoding module, the image feature mapping module, the lesion feature enhancement module, the radiomics feature encoding module and the radiomics feature mapping module form a feedback loop;

[0012] The analysis model is trained using X-ray chest film samples: the X-ray chest film samples are subjected to image feature extraction in the image feature encoding module, and an image representation is output; the image feature mapping module outputs an image feature map based on the image representation to obtain an image feature view; the lesion feature enhancement module extracts radiomics information based on the X-ray chest film samples and the image representation; the radiomics feature encoding module extracts radiomics features from the radiomics information and outputs a radiomics representation; the radiomics feature mapping module outputs a radiomics feature map based on the radiomics representation to obtain a radiomics feature view; the image feature view and the radiomics feature view undergo self-supervised deep contrast learning in the feedback loop to update the network parameters of the image feature extraction subnetwork and the radiomics feature extraction subnetwork; the discrimination module performs disease classification and lesion localization on the image representation output by the image feature encoding module.

[0013] This method uses radiomics features as auxiliary knowledge to enhance the comparative learning effect with image features. The constructed feedback loop enables image features and radiomics features to reinforce each other, thereby improving the accuracy of disease classification. It can also generate more accurate lesion area prediction frames using only a small amount of lesion area annotation data, significantly improving the positioning accuracy of the lesion area, and its positioning results are more robust and accurate.

[0014] In one alternative, during the training of the analysis model, the consistency between the image feature map and the radiomics feature map is maximized and the classification error of the image representation is minimized.

[0015] In an optional solution, the image feature encoding module IFE i It includes N1 first multimodal feature extraction modules connected in series; the first multimodal feature extraction module includes 3 image feature extraction submodules, the image feature extraction submodule includes M1 image feature extraction units, and the image feature extraction unit includes two groups of feature expression update subunits and feature screening update subunits.

[0016] In an optional solution, the radiomics feature encoding module RFE r It includes N2 second multimodal feature extraction modules, the second multimodal feature extraction module includes 3 imaging genomics feature extraction submodules, the imaging genomics feature extraction submodule includes M2 imaging genomics feature extraction units, and the imaging genomics feature extraction unit includes two groups of feature expression update subunits and feature screening update subunits.

[0017] In an optional solution, the feature expression update subunit is composed of 6 residual submodules, 9 convolutional layers, and 1 Relu excitation layer;

[0018] The feature screening and updating subunit consists of 2 residual submodules, 5 convolutional layers, and 3 ReLU excitation layers;

[0019] The residual submodule consists of three convolutional layers, two ReLU excitation layers, and also includes two cross-layer shortcut connections.

[0020] In an optional solution, the lesion feature enhancement module includes a bounding box generation unit and an information acquisition unit;

[0021] The bounding box generation unit determines whether the X-ray chest image is an unlabeled image or a labeled image according to whether the image has corresponding lesion labeling information; for unlabeled images, the image feature encoder IFE i The features output by the last multimodal feature extraction module in the first step are subjected to gradient weighted class activation mapping to generate a heat map, and a predicted bounding box of the lesion area is generated on the heat map; for annotated images, the actual bounding box of the lesion area is directly used;

[0022] The information acquisition unit uses the predicted bounding box and the real bounding box in combination to extract radiomics information. The extracted radiomics information is a combination of the precise radiomics information of the labeled image and the "pseudo" radiomics information of the unlabeled image.

[0023] In an optional solution, the loss function L of the analysis model includes a self-supervised knowledge enhancement contrast loss function and a supervised classification loss function;

[0024] The self-supervised knowledge enhancement contrast loss function is used for learning the task of lesion localization in chest X-rays, and the supervised classification loss function is used for learning the task of lesion classification in chest X-rays;

[0025] Loss function L = λ × L cl +(1-λ)×L fl ,

[0026] Where λ is the self-supervised knowledge enhancement contrast loss function L cl The weight, L fl is the supervised classification loss function.

[0027] In an optional solution, the steps of constructing the self-supervised knowledge enhanced contrast loss function are:

[0028] Randomly extract a mini-batch of samples, the number of samples is N, and define the comparative prediction task on the paired amplified samples derived from the mini-batch: Assume that in a mini-batch, IFView bd RFView is an image representation of a body part b with lesion d. bd is the radiomics representation of body part b with lesion d, sim(u,v) is the cosine similarity;

[0029] The positive samples (IFView bd ,RFView bd ) Defined as:

[0030]

[0031] Among them, L [K＝b,l≠d] ∈{0 , 1} is a loss function, τ is a constant, IFView(v bd ) refers to the image feature view corresponding to the lesion d and the body part b; RFView(v' bd ) refers to the radiomics feature view corresponding to the lesion d and the body part b, wherein the positive samples include abnormal chest X-ray samples with lesion annotations and samples with predicted bounding boxes generated by the lesion feature enhancement module LFEM;

[0032] The final self-supervised contrastive loss is calculated on all lesion-positive images in the mini-batch:

[0033]

[0034] In one alternative, the supervised classification loss function is defined as follows:

[0035]

[0036] Among them, α allows different weights to be given to positive samples and negative samples, γ is used to distinguish easy samples from similar difficult samples, y refers to the actual status of the chest X-ray, 0 represents a normal / negative sample, 1 represents an abnormal / positive sample, and y' refers to the model's prediction result; the negative sample is a normal chest X-ray sample or an X-ray sample of the same body part but with different diseases; before disease classification, the correspondence between body parts and disease types is constructed.

[0037] The present invention also provides an X-ray chest film intelligent analysis system, comprising a control module, a storage module and a sample acquisition module;

[0038] The sample acquisition module is connected to the control module and sends the acquired X-ray chest film samples to the control module. The control module is connected to the storage module and communicates with each other. The storage module is used to store at least one executable instruction. The executable instruction enables the control module to perform operations corresponding to the above-mentioned X-ray chest film intelligent analysis method based on deep self-supervised comparative learning based on the X-ray chest film sample, to locate the lesions on the X-ray chest film and classify the disease.

[0039] The beneficial effects of the present invention are:

[0040] The present invention adopts a positive sample and negative sample selection method to amplify X-ray chest image data, which solves the problem that typical natural image data amplification methods (such as random cropping, color dithering, noise addition, etc.) cannot be directly used for X-ray chest images.

[0041] When conventional image data augmentation methods cannot be used, the present invention uses radiomics features as auxiliary knowledge to enhance the comparative learning effect with image features. The two contrasting "feature views" come from two different knowledge domains that characterize the same patient: the image features of the chest X-ray and the radiomics features. Since these features will be dynamically updated, a "feedback loop" is formed during the training process. In this feedback loop, the features of the two modalities reinforce each other during the learning process, and their contrast produces robust knowledge-enhanced features. Here, the key to realizing the feedback loop is the lesion feature enhancement module. For unlabeled chest X-ray images, a gradient-weighted class activation map is used to generate an input heat map from the image modality backbone, and a predicted bounding box is generated after thresholding; then the radiomics feature is extracted from this predicted bounding box, which becomes an alternative feature view that is contrasted with the image feature view.

[0042] The present invention utilizes a lesion feature enhancement module to generate radiomics features, using radiomics features as prior knowledge. The lesion feature enhancement module dynamically generates predicted lesion bounding boxes during training, solving the problem that chest X-rays usually do not have real lesion annotated bounding boxes.

[0043] The present invention improves abnormality recognition and lesion localization in chest X-rays (where most chest X-ray image data is unlabeled) through knowledge-enhanced contrast learning. The present invention can simultaneously complete disease classification and lesion localization in chest X-rays with only a small amount of labeled lesion data.

[0044] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments with reference to the following drawings, in which:

[0046] Figure 1 It is a schematic diagram of the framework structure of the analysis model;

[0047] Figure 2 It is a schematic diagram of the network structure of the image feature extraction sub-network;

[0048] Figure 3 This is a schematic diagram of the network structure of the radiomics feature extraction subnetwork;

[0049] Figure 4 It is a schematic diagram of the network structure of the image feature extraction submodule;

[0050] Figure 5 It is a schematic diagram of the network structure of the image feature extraction unit;

[0051] Figure 6 This is a schematic diagram of the network structure of the radiomics feature extraction submodule;

[0052] Figure 7 This is a schematic diagram of the network structure of the radiomics feature extraction unit;

[0053] Figure 8 It is an optional structure of the feature expression update subunit;

[0054] Figure 9 It is an optional structure of the feature screening and updating subunit;

[0055] Figure 10 It is an optional structure of the Residual submodule;

[0056] Figure 11 This is a schematic diagram of the framework structure of the lesion feature enhancement module;

[0057] Figure 12 1 is a schematic diagram comparing the disease classification performance (average values for eight diseases) of Example 1 with seven typical methods;

[0058] Figure 13 It is a schematic diagram of the performance comparison of disease (atelectasis) classification;

[0059] Figure 14 It is a performance comparison diagram of disease (cardiac hypertrophy) classification;

[0060] Figure 15 It is a schematic diagram of the performance comparison of disease (exudation) classification;

[0061] Figure 16 It is a schematic diagram of the performance comparison of disease (infiltration) classification;

[0062] Figure 17 It is a performance comparison diagram of disease (tumor) classification;

[0063] Figure 18 It is a schematic diagram of the performance comparison of disease (nodule) classification;

[0064] Figure 19 It is a performance comparison diagram of disease (pneumonia) classification;

[0065] Figure 20 It is a schematic diagram of the performance comparison of disease (pleural effusion) classification;

[0066] Figure 21 1 is a schematic diagram comparing the performance of embodiment 1 and two typical existing methods for lesion localization (average values for eight diseases);

[0067] Figure 22 This is a schematic diagram of the performance comparison of lesion (atelectasis) localization;

[0068] Figure 23 It is a schematic diagram of the performance comparison of lesion (cardiac hypertrophy) localization;

[0069] Figure 24 It is a schematic diagram of the performance comparison of lesion (exudate) localization;

[0070] Figure 25 It is a schematic diagram of the performance comparison of lesion (infiltration) localization;

[0071] Figure 26 It is a schematic diagram of the performance comparison of lesion (mass) localization;

[0072] Figure 27 It is a schematic diagram of the performance comparison of lesion (nodule) localization;

[0073] Figure 28 It is a schematic diagram of the performance comparison of lesion (pneumonia) localization;

[0074] Figure 29 It is a schematic diagram of the performance comparison of lesion (pleural effusion) localization;

[0075] Figure 30 1 is a schematic diagram comparing the ablation experiment performance (average values for eight diseases) of abnormality classification / disease classification in Example 1;

[0076] Figure 31 This is a schematic diagram comparing the performance of abnormality classification / disease classification (atelectasis) ablation experiments;

[0077] Figure 32 This is a schematic diagram comparing the performance of abnormality classification / disease classification (cardiac hypertrophy) ablation experiments;

[0078] Figure 33 This is a schematic diagram comparing the performance of the abnormality classification / disease classification (exudation) ablation experiment;

[0079] Figure 34 This is a schematic diagram comparing the performance of abnormality classification / disease classification (infiltration) ablation experiments;

[0080] Figure 35 This is a schematic diagram comparing the performance of abnormality classification / disease classification (tumor) ablation experiments;

[0081] Figure 36 This is a schematic diagram comparing the performance of abnormality classification / disease classification (nodule) ablation experiments;

[0082] Figure 37 This is a schematic diagram comparing the performance of the abnormality classification / disease classification (pneumonia) ablation experiment;

[0083] Figure 38 This is a schematic diagram comparing the performance of the abnormality classification / disease classification (pleural effusion) ablation experiment;

[0084] Figure 39 It is a diagram of the hierarchical relationship of DR chest radiograph diseases predefined based on domain expert knowledge. DETAILED DESCRIPTION

[0085] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0086] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal communication between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0087] Example 1

[0088] This paper provides an intelligent analysis method for chest X-rays based on deep self-supervised contrastive learning, which can locate and classify lesions on chest X-rays. The specific steps are as follows:

[0089] Obtain a sample chest X-ray.

[0090] In this example, the proposed method was evaluated using the publicly available chest X-ray dataset, ChestX-ray14, as a sample of chest X-ray images. This dataset contains 112,120 chest X-ray images collected from 30,805 patients. Like other large medical datasets, the number of images corresponding to different disease categories in this dataset is highly imbalanced: healthy samples (84,321 images) far outnumber abnormal (disease) samples (24,624 images). Furthermore, the frequency of different diseases within these abnormal (disease) samples varies significantly. The chest X-ray images in the dataset have nine class labels, including "No findings" and eight disease labels: Atelectasis, Cardiomegaly, Effusion, Infiltration, Mass, Nodule, Pneumonia, and Pneumothorax.

[0091] In addition, the dataset includes 984 lesion bounding boxes of 8 types of chest diseases annotated by radiologists for 880 images. This embodiment separates the images that provide lesion bounding boxes from the entire dataset. Therefore, the dataset has two groups of images, namely 880 images with "lesion annotations" and 111,240 images without "lesion annotations". During the experiment, the image dataset with "no lesion annotations" was divided into three subsets: 70% for training, 10% for validation, and 20% for testing. For the image dataset with "lesion annotations", we randomly divided the dataset into two subsets: 20% for training and 80% for testing. Note that there is no overlap of patient data between all subsets.

[0092] Build analytical models.

[0093] like Figure 1 As shown, the analysis model in this embodiment includes an image feature extraction subnetwork IFESNet, a radiomics feature extraction subnetwork RFESNet, and a lesion feature enhancement module LFEM.

[0094] Among them, the image feature extraction subnetwork IFESNet includes the image feature encoding module IFE i , Image Feature Mapping Module IFM i And the discrimination module. Figure 2 As shown, the image feature encoding module IFE i It includes N1 first multimodal feature extraction modules MMFEMs-1 connected in series, where N1 is a configurable parameter with a value range of 2-8, preferably 4 or 5, and a default value of 4. Each first multimodal feature extraction module MMFEMs-1 includes three image feature extraction submodules MFE i ,like Figure 4 As shown, each image feature extraction submodule MFE i Includes M1 image feature extraction units FE i , M1 is a configurable parameter with a value range of 2-8, with preferred values of 3 and 4 and a default value of 3. Figure 5 As shown, each image feature extraction unit FE i It includes two groups of feature expression update subunits FRU and feature screening update subunits FFU. Image feature mapper IFM i Two-layer MLP with ReLU.

[0095] The radiomics feature extraction subnetwork RFESNet includes the radiomics feature encoding module RFE r and radiomics feature mapping module RFM r .like Figure 3 As shown, the radiomics feature encoding module RFE r It includes N2 second multimodal feature extraction modules MMFEMs-2, where N2 is a configurable parameter with a value range of 2-8, a preferred value of 3 or 4, and a default value of 3. The second multimodal feature extraction module MMFEMs-2 includes three radiomics feature extraction submodules MFE r ,like Figure 6 As shown, the radiomics feature extraction submodule MFE r Includes M2 radiomics feature extraction units FE r , where M2 is a configurable parameter with a value range of 2-8, with preferred values of 3 and 4 and a default value of 3. Figure 7 As shown, the radiomics feature extraction unit FE r It includes two groups of feature expression update subunits FRU and feature screening update subunits FFU. Radiomics feature mapper RFM r Three-layer MLP with ReLU.

[0096] The feature expression update subunit FRU in the image feature extraction subnetwork IFESNet and the radiomics feature extraction subnetwork RFESNet consists of 6 Residual submodules, 9 convolutional layers, and 1 Relu excitation layer; the feature screening update subunit FFU consists of 2 Residual submodules, 5 convolutional layers, and 3 Relu excitation layers; among them, the Residual submodule consists of three convolutional layers, 2 Relu excitation layers, and also contains 2 cross-layer shortcut connections. Figure 8 、 Figure 9 、 Figure 10 The optional structures of the feature expression update subunit FRU, the feature screening update subunit FFU and the Residual submodule are given respectively.

[0097] like Figure 11 As shown in FIG, the lesion feature enhancement module LFEM specifically includes a bounding box generation unit and an information acquisition unit.

[0098] The bounding box generation unit determines whether the X-ray chest image has corresponding lesion annotation information and whether it is an unlabeled image or a labeled image; for unlabeled images, the image feature encoder IFE is used. i The features of the last first multimodal feature extraction module MMFEMs-1 are used for gradient weighted class activation mapping, that is, the Grad-CAM algorithm is used to generate a heat map, and an adaptive threshold is applied to generate a predicted bounding box of the lesion area on the heat map; for annotated images, the real bounding box of the lesion area is used directly.

[0099] The information acquisition unit uses a combination of predicted bounding boxes and true bounding boxes, and uses the Pyradiomic tool to extract radiomic information. The extracted radiomic information is a combination of the precise radiomic information of the annotated image, that is, the true radiomic information, and the "pseudo" radiomic information of the unannotated image, that is, the radiomic information of the simulated lesion area.

[0100] The radiomics information used consists of the following three categories of features:

[0101] First-order statistical features: used to measure the distribution of pixel intensities within the bounding box, including energy (a measure of pixel value intensity), entropy (a measure of pixel value uncertainty), and the maximum, average, and median values of pixel grayscale within the region of interest (ROI).

[0102] Shape-based features: including mesh surfaces, pixel surfaces, perimeters, etc.

[0103] Grayscale features: including Gray Level Co-occurance Matrix (GLCM) features, Gray Level Size Zone Matrix (GLSZM) features, Gray Level Run Length Matrix (GLRLM) features, Neighboring Gray Tone Difference Matrix (NGTDM) features and Gray Level Dependence Matrix (GLDM) features, etc.

[0104] In this analysis model, the image feature encoding module IFE i , Image Feature Mapping Module IFM i , lesion feature enhancement module LFEM, imaging genomics feature encoding module RFE r and radiomics feature mapping module RFM r The lesion feature enhancement module (LFEM) uses the feedback loop to learn regional localization features from the acquired radiomics information and obtain radiomics information of positive samples (X-ray chest images corresponding to patients with the disease).

[0105] The entire analysis model can be trained end-to-end, and the image representation IR i It will be used for subsequent disease classification and lesion localization tasks.

[0106] Use chest X-ray samples to train the analysis model: X-ray chest samples are encoded in the image feature encoding module IFE i Extract image features and output image representation IR i , IR i =IFE i (IFView), Image Feature Mapping Module IFM i Image-based IR i Output image feature map IP i , IP i =IFM i (IR i ) to obtain the image feature view IFView. The bounding box generation unit in the lesion feature enhancement module LFEM directly uses the real bounding box of the lesion area or uses the gradient weighted class activation mapping algorithm Grad-CAM to generate a heat map, and applies an adaptive threshold to generate a predicted bounding box of the lesion area on the heat map, highlighting the key (abnormal) area of the X-ray chest film (even if it is not marked). The information extraction unit then uses the Pyradiomic tool to extract the radiomics feature information. The extracted radiomics feature information is encoded in the radiomics feature encoding module RFE.r Perform radiomics feature extraction and output radiomics representation RR r , RR r =RFE r (RFView), Radiomics Feature Mapping Module RFM r Characterizing RR based on radiomics r Output radiomics feature map RIP r , RIP r =RFM r (RR r ) to obtain the radiomics feature view RFView; the image feature view IFView and the radiomics feature view RFView perform self-supervised deep contrast learning in the feedback loop, use radiomics features to enhance image features, iteratively optimize during the training process, and continuously update the network parameters of the image feature extraction subnetwork IFESNet and the radiomics feature extraction subnetwork RFESNet; the discriminant module performs self-supervised deep contrast learning on the image feature encoding module IFE i Output image representation IR i =IFE i (IFView) for disease classification and lesion localization.

[0107] Maximize image feature map IP during analysis model training i and radiomics feature mapping RIP r The consistency between them, minimizing the image representation IR i Specifically, by contrasting the loss function, the image feature map IP is maximized. i and radiomics feature mapping RIP r The consistency between them is used to locate the lesion in the chest X-ray; the classification loss function is minimized from the image representation IR i The classification error is reduced to achieve the classification of lesions.

[0108] In this example, for the task of lesion localization in chest X-rays, a knowledge-enhanced contrast loss is designed to implement self-supervised contrastive learning. In this example, the contrast loss function is called the self-supervised knowledge-enhanced contrast loss function, which utilizes cross-modal information (image features and radiomics features) and is constructed as follows:

[0109] Randomly extract a mini-batch of samples, the number of samples is N, and define the comparative prediction task on the paired amplified samples derived from the mini-batch: Assume that in a mini-batch, IFView bd RFView is an image representation of a body part b with lesion d. bdis the radiomics representation of body part b with lesion d, sim(u,v) is the cosine similarity;

[0110] The positive samples (IFView bd ,RFView bd ) Defined as:

[0111]

[0112] Among them, L [K＝b,l≠d] ∈{0,1} is a loss function, k represents the body part b, k takes different values to traverse all the key body parts in the X-ray chest film; l represents the lesion d, l takes different values to traverse the lesion types of specific body parts in the X-ray chest film, τ is a constant, IFView(v bd ) refers to the image feature view corresponding to the lesion d and the body part b; RFView(v' bd ) refers to the radiomics feature view corresponding to the lesion d and the body part b. Positive samples include abnormal chest X-ray samples with lesion annotations and samples with predicted bounding boxes generated by the lesion feature enhancement module LFEM.

[0113] In this example, positive sampling involves randomly selecting an image labeled with a specific disease and generating an image feature view and a radiomics feature view for it. Because traditional image augmentation strategies are not applicable here, this example uses specific radiomics features in the radiomics feature view, and this process is implemented using the lesion feature enhancement module (LFEM).

[0114] The final self-supervised knowledge-enhanced contrast loss is calculated on all lesion-positive images in the mini-batch:

[0115]

[0116] For the classification of lesions, this embodiment uses the image feature encoder IFE i The output of is sent to a simple linear classifier using a supervised classification loss function, which is defined as:

[0117]

[0118] Among them, α allows different weights to be given to positive samples and negative samples, γ is used to distinguish easy samples from similar difficult samples to drive the model to learn more information from difficult samples, y refers to the actual status of the chest X-ray, 0 represents a normal / negative sample, 1 represents an abnormal / positive sample (one of eight diseases), and y' refers to the model's prediction result (the maximum probability value of the nine predicted categories).

[0119] In order to enhance the ability to identify and distinguish subtle and similar image features and improve the performance of intelligent analysis of chest X-rays, unlike randomly selecting images, in this embodiment, similar difficult samples are used as negative samples. Here, normal chest X-ray samples or chest X-ray samples of the same body part but with different diseases are collectively referred to as similar difficult samples. All samples except similar difficult samples are easy samples. Similar difficult sample negative sampling first obtains prior knowledge from the pre-constructed disease hierarchy, such as Figure 39 As shown in the figure, before disease classification, a correspondence between body parts and disease types is pre-established based on expert knowledge in the field of chest X-ray diagnosis. In this hierarchy, each disease belongs to a body part. For example, negative samples can be used as "similar difficult sample" images for the corresponding body part; if the "base anchor" image is labeled "pneumonia / lung", its "similar difficult sample" images should include "atelectasis / lung", "edema / lung", or "normal / lung", but not "fracture".

[0120] The loss function L of the analysis model in this embodiment includes a self-supervised knowledge enhancement contrast loss function and a supervised classification loss function. The loss function of the analysis model is set to L = λ × L cl +(1-λ)×L fl , where λ is the self-supervised knowledge enhancement contrast loss function L cl The weight of .

[0121] During training, the batch size was set to 16, and the analysis model was trained for 100 epochs. The model was optimized using the Adam method, and the initial learning rate of 0.001 was decayed by 10% every 20 epochs. The default value of 0.5 was used for the weight λ of the loss function. The relevant models in the proposed method were all implemented in the PyTorch framework. In this example, two Nvidia Tesla V100 GPUs were used to train the model.

[0122] Testing process: only retain the image feature encoder IFE i , which acts on the input chest X-ray image. If it is an abnormal chest X-ray, the output result shows the location of the abnormality and also shows the location of the abnormality.

[0123] This embodiment also evaluates the performance of the two subtasks: (1) disease classification for chest X-rays; and (2) lesion area localization for abnormal chest X-rays, and compares them with existing typical methods. For the disease classification task, the area under the curve (AUC) of the receiver operating characteristic curve (ROC) is used to measure the model performance. For the lesion area localization task of abnormal chest X-rays, the intersection over union (IoU) is used, that is, the model's localization performance is measured based on the annotated true bounding box and the model's predicted bounding box. The localization results are only calculated on the test set of the annotated dataset. The lesion area localization is defined as correct only when IOU>T(IOU), where T(·) is the set intersection over union threshold.

[0124] Disease classification experiments and analysis

[0125] Table 1 shows the AUC scores of this embodiment (denoted as proposed) for specific disease classification and the average AUC scores of eight disease classifications, and compares them with seven typical existing X-ray chest film disease classification methods (denoted as meth[1] to meth[7], respectively).

[0126] meth[1]: Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestxray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of commonthorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097–2106, 2017.

[0127] meth[2]:Xiaosong Wang,Yifan Peng,Le Lu,Zhiyong Lu,and RonaldMSummers.Tienet:ext-image embedding network for common thorax diseaseclassification and reporting in chest x-rays.In Proceedings of the IEEEconference on computer vision and pattern recognition,pages 9049–9058,2018.

[0128] meth[3]:Li Yao,Eric Poblenz,Dmitry Dagunts,Ben Covington,DevonBernard,and Kevin Lyman.Learning to diagnose from scratch by exploitingdependencies among labels.arXiv preprint arXiv:1710.10501,2017.

[0129] meth[4]:Pranav Rajpurkar,Jeremy Irvin,Kaylie Zhu,Brandon Yang,HershelMehta,Tony Duan,Daisy Ding,Aarti Bagul,Curtis Langlotz,Katie Shpanskaya,etal.Chexnet:Radiologist level pneumonia detection on chest x-rays with deeplearning.arXiv preprint arXiv:1711.05225,2017.

[0130] meth[5]: Pulkit Kumar, Monika Grewal, and Muktabh Mayank Srivastava. Boosted cascaded convnets for multilabel classification of thoracic diseases in chest radiographs. In International Conference ImageAnalysis and Recognition, pages 546–552. Springer, 2018.

[0131] meth[6]: Jingyu Liu, Gangming Zhao, Yu Fei, Ming Zhang, Yizhou Wang, and Yizhou Yu. Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision. In Proceedings of the IEEE / CVFInternational Conference on Computer Vision, pages 10632–10641, 2019.

[0132] meth[7]: Laleh Seyyed-Kalantari, Guanxiong Liu, Matthew McDermott, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-rayclassifiers. arXiv preprint arXiv:2003.00827, 2020.

[0133] Figure 12 This is a comparison of the average performance (AUC score) of this embodiment and seven typical methods for eight disease classifications (average value for eight diseases). Figures 13-20 It is a comparison of the performance (AUC score) of a specific disease classification experiment, which is atelectasis, cardiomegaly, exudate, infiltration, mass, nodule, pneumonia, and pleural effusion.

[0134] Table 1. AUC and mean AUC for each disease classification compared with the comparison methods. For each column, the bold font represents the best result.

[0135]

[0136] The comparative experiments on disease classification show that this embodiment achieved better AUC scores in most disease classification experiments, and its overall performance in disease classification was significantly improved. Specifically, the classification effect of diseases with large lesion areas on X-ray chest images has been significantly improved, such as "atelectasis", "cardiac hypertrophy" and "pleural effusion". In addition, small lesion features such as "mass" and "nodule" can also be effectively identified by the proposed method. In summary, the results of the disease classification experiments demonstrate the superiority of this embodiment over the seven existing typical methods in the task of X-ray chest disease classification.

[0137] Lesion localization experiment and analysis

[0138] Table 2 shows the experimental results of this embodiment (referred to as proposed) for lesion localization on abnormal chest X-rays at different IoU thresholds (including 0.3, 0.5, and 0.7), and compares it with two existing typical methods (referred to as meth[1] and meth[8], respectively). The experimental results are shown in Table 2.

[0139] meth[8]: Zhe Li, Chong Wang, Mei Han, Yuan Xue, Wei Wei, Li-Jia Li, and LiFei-Fei. Thoracic disease identification and localization with limited supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8290–8299, 2018.

[0140] Figure 21 This is a comparison of the average performance (IoU value) of this embodiment and seven typical methods for lesion area localization of eight diseases (average value for eight diseases). Figures 22-29 The performance (IoU value) of the lesion area localization experiment for a specific disease is compared, which is atelectasis, cardiomegaly, exudate, infiltration, mass, nodule, pneumonia, and pleural effusion. Since lesion localization is a relatively difficult task in X-ray chest images, we did not find as many typical methods for comparison as for disease classification tasks. As far as we know, there are only two methods in meth[1] and meth[8]. According to the experimental results of lesion area localization shown in Table 2, we can see that:

[0141] (1) The lesion localization capability of this embodiment is significantly better than that of the comparison method at different IoU thresholds (0.3, 0.5, 0.7), and the average value of its lesion area localization accuracy is significantly higher than that of the comparison method.

[0142] (2) This example outperforms the comparison method in localizing lesion regions for most diseases at different IoU thresholds (0.3, 0.5, and 0.7), with higher IoU values than the comparison method. Importantly, the proposed method not only performs well in simple lesion localization tasks, but also achieves good performance in difficult lesion localization tasks, such as locating disease categories such as “lumps” and “nodules” that only have small lesion regions.

[0143] (3) Compared with the comparison method, the lesion localization accuracy of this embodiment maintains this superior performance as the IoU threshold increases. For example, when the threshold increases, the decrease in the IoU value corresponding to "cardiac hypertrophy" is smaller than that of the comparison method.

[0144] Based on the experimental results and analysis above, this embodiment outperforms the two existing typical methods used for comparison. It is important to note that for some diseases, such as pneumonia and infiltration, lesions may appear in multiple locations, while each annotated image in the dataset only provides a bounding box. Overall, this embodiment outperforms the comparison methods in lesion region localization performance for the eight diseases mentioned above, as well as its robustness with increasing IoU thresholds.

[0145] Table 2 Comparison of lesion localization accuracy at different IoU thresholds. Bold font represents the best result in each column.

[0146]

[0147]

[0148] The experimental results show that by introducing a cross-modal contrastive learning branch and a network structure carefully designed for X-ray chest images and imaging genomics features, the performance of X-ray chest image analysis can be effectively improved, and it has good robustness for different types of disease classification tasks.

[0149] Ablation experiments and analysis

[0150] This example conducts an ablation experiment on the disease classification task to verify the proposed supervised classification loss function L fl The lesion feature enhancement module LFEM is used to improve the analysis performance of the X-ray chest image dataset with imbalanced disease categories to verify the rationality and effectiveness of the above technology.

[0151] Table 3 shows the ablation experiments of supervised classification loss function and LFEM module for disease classification tasks.

[0152] Bold font indicates the best result in each column.

[0153] Atelectasis Cardiac hypertrophy exudate infiltration Lump nodules pneumonia Pleural effusion average value Base 0.752 0.854 0.838 0.672 0.694 0.647 0.703 0.794 0.7443 Model.FL 0.786 0.841 0.805 0.686 0.767 0.725 0.727 0.827 0.7705 Model.LEFM 0.825 0.907 0.857 0.711 0.829 0.758 0.746 0.863 0.8120 Full model 0.842 0.931 0.882 0.724 0.876 0.792 0.774 0.902 0.8404

[0154] The performance of the basic model (using the standard cross entropy loss function for disease classification and not using the lesion feature enhancement module LFEM, marked as "Base") was compared with the performance of the supervised classification loss function L fl The experimental results are shown in Table 3. Figure 30 is the average performance (AUC score) comparison of the proposed method and the above ablation experimental method for eight disease classifications (average value for eight diseases), Figures 31-38 The performance (AUC score) of a specific disease classification experiment is compared, which is atelectasis, cardiomegaly, exudate, infiltration, mass, nodule, pneumonia, and pleural effusion. The ablation experiment results of the disease classification task show that the supervised classification loss L fl The use of and the introduction of the lesion feature enhancement module LFEM significantly improve the performance of disease classification tasks, and the contribution of the LFEM module is greater because the LFEM module tends to generate more robust radiomics features, which further enhances the image feature encoder's attention to image regions containing specific diseases.

[0155] Example 2

[0156] This embodiment provides an X-ray chest film intelligent analysis system, including a control module, a storage module and a sample acquisition module; the sample acquisition module is connected to the control module and sends the X-ray chest film samples it has acquired to the control module; the control module is connected to the storage module and communicates with each other; the storage module is used to store at least one executable instruction, and the executable instruction enables the control module to perform operations corresponding to the above-mentioned X-ray chest film intelligent analysis method based on deep self-supervised comparative learning based on the X-ray chest film sample, to locate lesions on the X-ray chest film and classify diseases.

[0157] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0158] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for intelligent analysis of chest X-rays based on deep self-supervised contrastive learning, characterized in that: The following steps are involved: Obtain chest X-ray samples; An analysis model is constructed, which includes an image feature extraction subnetwork, an imaging genomics feature extraction subnetwork and a lesion feature enhancement module; the image feature extraction subnetwork includes an image feature encoding module, an image feature mapping module and a discrimination module; the image feature encoding module includes N1 first multimodal feature extraction modules connected in series; the first multimodal feature extraction module includes 3 image feature extraction submodules, the image feature extraction submodule includes M1 image feature extraction units, and the image feature extraction unit includes two groups of feature expression update subunits and feature screening update subunits; the imaging genomics feature extraction subnetwork includes an imaging genomics feature encoding module RFE r and an imaging omics feature mapping module; the lesion feature enhancement module includes a bounding box generation unit and an information acquisition unit; the bounding box generation unit determines whether the X-ray chest image is an unlabeled image or an annotated image based on whether the image has corresponding lesion annotation information; for an unlabeled image, a gradient-weighted class activation mapping is performed on the features output by the last first multimodal feature extraction module in the image feature encoding module to generate a heat map, and a predicted bounding box of the lesion area is generated on the heat map; for an annotated image, the true bounding box of the lesion area is directly used; the information acquisition unit uses a combination of the predicted bounding box and the true bounding box to extract imaging omics information, and the extracted imaging omics information is a combination of the precise imaging omics information of the annotated image and the "pseudo" imaging omics information of the unlabeled image; The image feature encoding module, the image feature mapping module, the lesion feature enhancement module, the radiomics feature encoding module, and the radiomics feature mapping module form a feedback loop, and the lesion feature enhancement module uses the feedback loop to learn regional positioning features from the obtained radiomics information and obtain radiomics information of the positive sample; The analysis model is trained using chest X-ray samples: the chest X-ray samples are subjected to image feature extraction in an image feature encoding module, and an image representation is output; the image feature mapping module outputs an image feature map based on the image representation to obtain an image feature view; the lesion feature enhancement module extracts radiomics information based on the chest X-ray samples and the image representation; the radiomics feature encoding module extracts radiomics features from the radiomics information and outputs a radiomics representation; the radiomics feature mapping module outputs a radiomics feature map based on the radiomics representation to obtain a radiomics feature view; the image feature view and the radiomics feature view undergo self-supervised deep contrast learning in the feedback loop to update the network parameters of the image feature extraction subnetwork and the radiomics feature extraction subnetwork; the discrimination module performs disease classification and lesion localization on the image representation output by the image feature encoding module; The loss function L of the analysis model includes a self-supervised knowledge enhancement contrast loss function and a supervised classification loss function; The self-supervised knowledge enhancement contrast loss function is used for learning the task of lesion localization in chest X-rays, and the supervised classification loss function is used for learning the task of lesion classification in chest X-rays; Loss function L = λ × L cl +(1-λ)×L fl , Where λ is the self-supervised knowledge enhancement contrast loss function L cl The weight, L fl is the supervised classification loss function; The steps for constructing the self-supervised knowledge enhanced contrast loss function are: Randomly extract a mini-batch of samples, the number of samples is N, and define the comparative prediction task on the paired amplified samples derived from the mini-batch: Assume that in a mini-batch, IFView bd RFView is an image representation of a body part b with lesion d. bd is the radiomics representation of body part b with lesion d, sim(u,v) is the cosine similarity; The positive samples (IFView bd ,RFView bd )’s loss function l vbd Defined as: Among them, L [K＝b,l≠d] ∈{0,1} is a loss function, τ is a constant, IFView(v bd ) refers to the image feature view corresponding to the lesion d and the body part b; RFView(v' bd ) refers to the radiomics feature view corresponding to the lesion d and the body part b, wherein the positive samples include abnormal chest X-ray samples with lesion annotations and samples with predicted bounding boxes generated by the lesion feature enhancement module LFEM; The final self-supervised contrastive loss is calculated on all lesion-positive images in the mini-batch: The supervised classification loss function is defined as follows: Among them, α allows different weights to be given to positive samples and negative samples, γ is used to distinguish easy samples from similar difficult samples, y refers to the actual status of the chest X-ray, 0 represents a normal / negative sample, 1 represents an abnormal / positive sample, and y' refers to the model's prediction result; the negative sample is a normal chest X-ray sample or an X-ray sample of the same body part but with different diseases; before disease classification, the correspondence between body parts and disease types is constructed.

2. The method for intelligent chest X-ray analysis based on deep self-supervised contrastive learning according to claim 1, characterized in that: During the training of the analysis model, the consistency between the image feature map and the radiomics feature map is improved to minimize the classification error of the image representation.

3. The method for intelligent chest X-ray analysis based on deep self-supervised contrastive learning according to claim 1, characterized in that: The radiomics feature encoding module RFE r It includes N2 second multimodal feature extraction modules, the second multimodal feature extraction module includes 3 imaging genomics feature extraction submodules, the imaging genomics feature extraction submodule includes M2 imaging genomics feature extraction units, and the imaging genomics feature extraction unit includes two groups of feature expression update subunits and feature screening update subunits.

4. The method for intelligent chest X-ray analysis based on deep self-supervised contrastive learning according to claim 3, characterized in that: The feature expression update subunit consists of 6 residual submodules, 9 convolutional layers, and 1 Relu excitation layer; The feature screening and updating subunit consists of 2 residual submodules, 5 convolutional layers, and 3 ReLU excitation layers; The residual submodule consists of three convolutional layers, two ReLU excitation layers, and also includes two cross-layer shortcut connections.

5. An X-ray chest film intelligent analysis system, characterized in that: It includes a control module, a storage module and a sample acquisition module; The sample acquisition module is connected to the control module and sends the acquired X-ray chest film samples to the control module. The control module is connected to the storage module and communicates with each other. The storage module is used to store at least one executable instruction. The executable instruction enables the control module to perform the operation corresponding to the X-ray chest film intelligent analysis method based on deep self-supervised comparative learning as described in any one of claims 1 to 4 based on the X-ray chest film sample, to locate the lesions on the X-ray chest film and classify the disease.

Images