X-ray film lung disease screening system fusing multi-level features

Abstract

The application discloses a kind of X-ray film lung disease screening systems of fusing multi-level features, including lung parenchyma segmentation module, lung disease preliminary screening module, multi-level feature construction module, lung disease screening module;Lung parenchyma segmentation module is used to obtain lung parenchyma part from complete chest X-ray film segmentation;Lung disease preliminary screening module is used to construct based on feature consistency variation auto-encoding generation adversarial network and train it, and obtains preliminary screening abnormal score using trained generation adversarial network;Multi-level feature construction module is used to extract features of different semantic levels, including the extraction of bottom visual features, middle layer depth features and high layer experience features;Lung disease screening module is used to construct lung disease screening model using multi-level features and the abnormal score obtained by lung disease preliminary screening module, and obtains the final lung disease classification result.The present application can effectively solve the problem that current computer-aided lung disease screening field research is difficult to be applied to practical clinical scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent medical image analysis, and relates to an X-ray film lung disease screening system fusing multi-level features. BACKGROUND

[0002] For example, common lung diseases such as pneumonia, lung cancer, tuberculosis, and chronic obstructive pulmonary disease have become global high-incidence diseases, and therefore, efficient screening of lung diseases plays a crucial role in ensuring people's healthy life.

[0003] Currently, there are various examination methods for diagnosing lung diseases, such as sputum cell examination, bronchoscopy examination, CT examination, and chest X-ray examination. Among them, chest X-ray has become one of the most commonly used imaging examination methods and the main examination method for lung disease screening in the world due to its easy accessibility, low cost, low radiation, and ability to observe most lung lesions. With the development of medical reform and the increasing demand for physical examination, the output of X-ray images has increased exponentially, and the workload of radiologists has also increased. However, doctors still have high misdiagnosis and missed diagnosis rates, limited reading speed, and other shortcomings due to physiological fatigue and personal subjective factors when reading X-ray films by eye. Moreover, there is a significant shortage of professional doctors in the current medical imaging field.

[0004] When human resources are insufficient, more and more researchers attempt to use computer means to assist doctors in disease screening, and thus introduce computer-aided detection or diagnosis (CAD) systems to assist medical services. Computers can quickly, comprehensively, and accurately calculate image information due to their powerful computing and storage capabilities, allowing doctors to rely on more accurate calculations for diagnosis instead of relying on experience, which greatly reduces the workload of doctors while ensuring the accuracy of screening results.

[0005] The current main research idea of computer-aided lung disease screening is to model certain types of diseases, seek improvements in classification or detection models for these types of diseases, and thus complete disease classification. However, in real life, there are numerous types of lung diseases that are difficult to exhaust, and image signs are complex and variable, so unknown types of diseases in the testing process can easily lead to missed diagnosis. Moreover, current research mainly aims to improve the accuracy of screening results, seeks a compromise between sensitivity and specificity, and further provides disease diagnosis results. However, the reading process of radiologists is often complex, and the disease diagnosis results directly provided by computers often still need to be reviewed by doctors, which increases the time for disease diagnosis. The above problems further prevent current research from being applied to actual clinical scenarios. SUMMARY

[0006] In view of the problem that current research in the field of computer-aided lung disease screening is difficult to apply to actual clinical scenarios, the purpose of the present application is to provide an X-ray film lung disease screening system fusing multi-level features, which can better adapt to clinical scenarios and play the role of computer-aided screening.

[0007] The present application believes that the focus of computer-aided diagnosis is assistance, i.e. providing a "second reference opinion" to assist the doctor's diagnosis decision, rather than replacing the doctor to give a diagnosis result. Therefore, the present application does not replace the doctor to diagnose the disease type, but assists the doctor to exclude the images that are 100% normal in the screening process, and hands over the images that may have diseases to the doctor or subsequent tasks for judgment. In this way, in the face of physical examination and other large-scale screening scenarios, the present application can better fit the doctor's reading process in the clinical scenario, and thus can greatly reduce the workload of the doctor and subsequent tasks. Based on the above clinical application purpose, the goal of the screening system will be summarized as ensuring the missed diagnosis rate of the screening result to be 0 while further reducing the misdiagnosis rate of the screening result.

[0008] To achieve the above goal, the present application will construct multi-level features by fusing the complementary advantages of different semantic level features, and only model the above features of normal images, thereby achieving the goal of assisting doctors to exclude 100% normal images. The present application will be implemented by using the following technical solutions:

[0009] An X-ray film lung disease screening system fusing multi-level features, comprising a lung parenchyma segmentation module, a lung disease preliminary screening module, a multi-level feature construction module, and a lung disease screening module; wherein:

[0010] The lung parenchyma segmentation module is used to segment the lung parenchyma part from a complete chest X-ray film;

[0011] The lung parenchyma segmentation module comprises a preprocessing submodule, a lung parenchyma segmentation model training submodule, and a segmentation submodule;

[0012] The preprocessing submodule is used to implement: simultaneously rotating, translating, cropping, and horizontally flipping the Montgomery CXR dataset chest X-ray film and the segmentation mask corresponding thereto, respectively, to obtain preprocessed pictures as a training set;

[0013] The lung parenchyma segmentation model training submodule is used to train an Attention U-net model using the training set to obtain a trained lung parenchyma segmentation model;

[0014] The segmentation submodule is used to input the chest X-ray film image to be segmented into the trained lung parenchyma segmentation model to obtain its corresponding segmentation mask binary image;

[0015] The lung disease preliminary screening module is configured to construct and train a generative adversarial network, and obtain a preliminary screening abnormality score by using the trained generative adversarial network.

[0016] The lung abnormality preliminary screening module comprises a generative adversarial network construction submodule, a generative adversarial network training submodule, and a lung disease preliminary screening submodule.

[0017] The generative adversarial network construction submodule is configured to construct a generative adversarial network.

[0018] The generative adversarial network training submodule is configured to perform the following functions: obtaining a complete lung region by cropping the original chest X-ray film according to a segmentation mask binary image obtained by the lung parenchyma segmentation module, that is, obtaining a cropped X-ray film; dividing the cropped X-ray film into a training set, a validation set, and a test set, wherein the training set only contains normal X-ray films; and training the training set to obtain a trained generative adversarial network.

[0019] The lung disease preliminary screening submodule is configured to input the test set into the trained generative adversarial network to obtain a reconstructed image, and calculate a preliminary screening abnormality score according to a difference between the reconstructed image and an original image.

[0020] The multi-level feature construction module is configured to extract multi-level features, including extraction of bottom-level visual features, middle-level deep features, and high-level experience features.

[0021] The lung disease screening module is configured to construct a lung disease screening model by using the multi-level features obtained by the multi-level feature construction module and the abnormality score obtained by the lung disease preliminary screening module, and obtain a final lung disease classification result.

[0022] Further, the lung parenchyma segmentation module further comprises a connected region detection submodule configured to perform connected region detection on the segmentation mask binary image output by the segmentation submodule, and only retain the largest two connected regions, i.e., left and right lung regions, to obtain a final segmentation mask binary image of the lung region.

[0023] Further, the generative adversarial network comprises two encoders Enc, one decoder Dec, one discriminator D, and a hidden layer encoding classifier C, and the two encoders share weights.

[0024] Further, in the generative adversarial network training submodule, the specific operation of training the training set to obtain the trained generative adversarial network is as follows:

[0025] First, the image in the training set is taken as an input image x, and a hidden layer encoding feature with a dimension of 128 is obtained through the encoder.

[0026] Then, 128-dimensional features are sampled from the standard Gaussian distribution, and the sampled features are sent to the hidden layer encoding classifier together with the hidden layer encoding features to be classified, and the binary cross-entropy loss is used as the hidden layer encoding adversarial loss L adv_code updating the parameters of the hidden layer encoding classifier and the encoder;

[0027] Meanwhile, the hidden layer encoding features are sent to the decoder to obtain the reconstructed image of the input image x The sampled features obtained from the standard Gaussian distribution are sent to the decoder to obtain the generated image Further, the generated image the reconstructed image and the input image x are sent to the discriminator for classification; here, the cross-entropy loss is used as the image adversarial loss L adv_img updating the parameters of the discriminator and the decoder; and using the L1 loss as the image reconstruction loss L rec updating the parameters of the encoder and the decoder;

[0028] In addition, the reconstructed image is sent to a second encoder which shares weights with the first encoder; the features fmp of the second convolutional layer and the encoding features code of the last layer of the two encoders are extracted, wherein fmp contains shallow style features of the image, and code contains deep semantic features of the image; the L2 loss is used as the feature Figure One consistency loss L fmp and the L1 loss as the encoding consistency loss L code updating the parameters of the decoder;

[0029] Finally, the parameters of the encoder the parameters of the decoder and the hidden layer encoding classifier are updated respectively

[0030] Further, the hyperparameters λ1, α1, α2, α3 are set to 4, 10, 4, and 2 respectively.

[0031] Further, in the lung disease preliminary screening sub-module, the image reconstruction loss L rec and the feature Figure One consistency loss L fmp are calculated to obtain the anomaly score of each test sample in the test set:

[0032]

[0033] Further, the bottom layer visual features include geometric shape features, intensity features, and texture features; the middle layer depth features adopt image encoding features extracted by an encoder of a trained generative adversarial network; and the high layer experience features include left and right lung height difference, lung width-height ratio, cardiothoracic ratio, left and right lung costophrenic angle, and upper and lower mediastinal width features extracted from a segmented chest X-ray film output by a lung parenchyma segmentation module.

[0034] Further, the lung disease screening module includes a feature grouping module, a feature selection and extraction module, and a classification output module, wherein:

[0035] The feature grouping module is configured to divide the multi-level features into seven groups, i.e., high layer experience features, middle layer depth features, geometric shape features, left lung intensity features, right lung intensity features, left lung texture features, and right lung texture features, according to left and right lung region division and feature category division.

[0036] The feature selection and extraction module is configured to perform feature selection and extraction on the seven groups of features by using a univariate analysis method and a multivariate analysis method, respectively, to obtain seven feature subsets.

[0037] The classification output module is configured to construct OC-SVM base classifiers for the seven feature subsets, respectively, and further combine abnormal scores obtained by the seven OC-SVM base classifiers and preliminary screening abnormal scores output by the lung disease preliminary screening module as inputs of an OC-SVM meta-classification model to obtain a final lung disease classification result.

[0038] Further, in the feature selection and extraction module, the univariate analysis method adopts t-test and mutual information method, and the multivariate analysis method adopts factor analysis method; and the optimal seven feature subsets are selected by taking the AUC value of the classification result of a validation set as an evaluation criterion.

[0039] Compared with the prior art, the system of the present application has the following beneficial effects:

[0040] 1. The present application does not replace doctors to diagnose disease types, but assists doctors to exclude 100% normal images in the screening process. Compared with the prior art of replacing doctors to classify diseases, the present application can better fit the doctor's reading process in the clinical scene, thereby greatly reducing the workload of doctors and subsequent tasks.

[0041] 2. The present application only models features of normal chest X-ray films, and does not model features of any type of disease, thereby enabling the learned classifier to correctly identify normal chest X-ray films in the reasoning process, and uniformly classifying other types of X-ray films as unknown categories and temporarily as abnormal categories. Compared with the prior art of modeling specific several types of diseases, the present application can effectively reduce the missed diagnosis phenomenon caused by misclassification of unknown diseases.

[0042] 3. This invention proposes a variational autoencoder generative adversarial network based on feature consistency. This network adds an encoder with weights shared with the original encoder to encode the reconstructed image and adds features to the traditional variational autoencoder network. Figure One Consistency loss and encoding consistency loss make the original and reconstructed images closer in terms of low-level style features and deep semantic features, further improving the image reconstruction effect. This network can make the features learned by the model from normal images more discriminative of those from abnormal images, even when only normal images are used for model training, thus enhancing the initial screening effect for lung diseases in this invention.

[0043] 4. This invention integrates the complementary advantages of features at different semantic levels to construct a multi-level feature set. Compared with most existing computer-aided screening methods that use deep networks, it supplements the prior knowledge brought by doctors' experience features and the image detail features brought by low-level visual features, and to a certain extent enhances the interpretability of screening results. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the X-ray lung disease screening system that integrates multi-level features according to the present invention;

[0045] Figure 2 This is a framework diagram of the generative adversarial network in this invention.

[0046] Figure 3 This is an example diagram of lung region partitioning on the lung parenchyma segmented image in the embodiment.

[0047] Figure 4 This is a visualization of the segmentation results of the TCLD_CXR dataset in the embodiment. Detailed Implementation

[0048] Reference Figure 1 The present invention provides an X-ray lung disease screening system that integrates multi-level features, including a lung parenchyma segmentation module, a lung disease initial screening module, a multi-level feature construction module, and a lung disease screening module.

[0049] I. Lung Parenchyma Segmentation Module

[0050] The lung parenchyma segmentation module is used to segment the lung parenchyma from a complete chest X-ray to reduce redundant information in irrelevant lung areas and eliminate the impact of this redundant information on screening results. The lung parenchyma segmentation module includes a preprocessing submodule, a lung parenchyma segmentation model training submodule, and a segmentation submodule, wherein:

[0051] Preprocessing submodule: in order to enhance the generalization performance of the model, the Montgomery CXR data set chest X-ray film and the corresponding segmentation mask are simultaneously rotated, translated, cropped and horizontally flipped, and the preprocessed pictures are taken as the training set to further expand the diversity of training samples.

[0052] Lung parenchyma segmentation model training submodule: the Attention U-net model is trained by using the training set to obtain the trained lung parenchyma segmentation model; the Attention U-net is an improved model of the U-net, which has the same advantages as the U-net in small sample data training, and at the same time, the soft-attention method can effectively suppress the irrelevant regions of the image and highlight the significant features of the specific region, which can further improve the sensitivity and prediction accuracy of the model.

[0053] Segmentation submodule: used for inputting the chest X-ray film image to be segmented into the trained lung parenchyma segmentation model, which can classify each pixel of the input image to obtain the corresponding segmentation mask binary image.

[0054] In the above lung parenchyma segmentation module, the Montgomery CXR data is used for model training to realize automatic segmentation of the lung parenchyma region, without the need for manual outlining by doctors, which can save a lot of work burden.

[0055] Preferably, in order to avoid that the model may predict part of other small tissues as lung regions, a connected region detection submodule is designed in the lung parenchyma segmentation module, which is used to realize connected region detection on the segmentation mask binary image output by the segmentation submodule, and only the largest two connected regions, i.e. left and right lung regions, are retained to further obtain the final segmentation mask binary image of the lung region. According to the segmentation mask, the lung region can be cropped from the original image for subsequent feature extraction.

[0056] II. Lung disease preliminary screening module

[0057] The feature consistency based variational auto-encoding generative adversarial network is constructed and trained, and the trained generative adversarial network is used to obtain the preliminary screening abnormal score. The lung abnormality preliminary screening module includes a generative adversarial network construction submodule, a generative adversarial network training submodule and a lung disease preliminary screening submodule.

[0058] The generative adversarial network construction submodule is used to construct the generative adversarial network, and the network structure is as shown in Figure 2 The generative adversarial network construction submodule is used to construct the generative adversarial network, and the network structure is as shown in

[0059] The generative adversarial network training submodule is configured to: crop a complete lung region from the original chest X-ray according to a segmentation mask binary image obtained by the lung parenchyma segmentation submodule, to obtain a cropped X-ray; divide the cropped X-ray into a training set, a verification set and a test set, wherein the training set only contains normal X-rays; and train the training set to obtain a trained generative adversarial network.

[0060] The specific operation of training the training set to obtain the trained generative adversarial network is as follows:

[0061] First, an image in the training set is taken as an input image x, and an encoded feature with a dimension of 128 is obtained by an encoder.

[0062] Next, a 128-dimensional feature is sampled from a standard Gaussian distribution, and the sampled feature is sent to a hidden layer coding classifier together with the above-mentioned encoded feature for classification, and a binary cross-entropy loss is taken as a hidden layer coding adversarial loss L adv_code The hidden layer coding classifier and the encoder parameters are updated so that the hidden layer coding feature is closer to the standard Gaussian distribution.

[0063] Meanwhile, the hidden layer coding feature is sent to a decoder to obtain a reconstructed image of the input image x The sampled feature in the above-mentioned standard Gaussian distribution is sent to the decoder to obtain a generated image The generated image is further sent to the decoder to obtain a reconstructed image The reconstructed image and the input image x are sent to a discriminator for classification. Here, a cross-entropy loss is used as an image adversarial loss L adv_img The discriminator and the decoder parameters are updated to further promote the decoder to generate higher-quality images. And an L1 loss is used as an image reconstruction loss L rec The encoder and the decoder parameters are updated so that the input image x is as similar as possible to the reconstructed image .

[0064] In addition, the reconstructed image is sent to a second encoder, which shares weights with the first encoder. The features fmp of the second convolutional layer and the encoded features code of the last layer of the two encoders are extracted, wherein fmp contains shallow style features of the image, and code contains deep semantic features of the image. An L2 loss is used as a feature Figure One consistency loss L fmp and an L1 loss is used as an encoding consistency loss L code The decoder parameters are updated so that the input image x is as similar as possible to the reconstructed image The feature maps fmp obtained after encoding by two encoders and the encoded features code are as close as possible to enhance the reconstruction effect of normal X-ray films.

[0065] Finally, the encoder parameters are updated respectively decoder parameters and the hidden layer encoding classifier parameters wherein the hyperparameters λ1, α1, α2, α3 are set to 4, 10, 4, and 2 respectively.

[0066] Pulmonary disease preliminary screening sub-module: Since the aforementioned model training stage only learns the reconstruction of normal X-ray films, the reconstruction effect on normal samples in the test stage is relatively good, and the reconstruction effect on abnormal samples not learned is relatively poor. Therefore, the test set can be input into the trained generative adversarial network to obtain the reconstructed image, and the difference before and after reconstruction is calculated to obtain the preliminary screening abnormal score. Preferably, the image reconstruction loss L rec and the features Figure One consistency loss L fmp The abnormal score of each test sample in the test set is calculated:

[0067] III. Multi-level feature construction module

[0068] The multi-level feature construction module is used to extract different semantic level features, including bottom-level visual features, middle-level deep features and high-level experience features. Among them, the bottom-level features include bottom-level visual features such as lung shape, texture, etc. in X-ray films, which contain less semantic information, but have finer feature granularity and accurate description purpose; the middle-level features include deep features extracted by deep neural networks, which are semantically abstract but lack interpretability; the high-level features include some hand-crafted features guided by doctors' experience and reflecting lung disease abnormalities, which can be better understood by doctors and can reflect accurate disease information. The specific design of the three-level feature extraction operation is as follows:

[0069] Bottom-level visual feature extraction: From the perspective of radiologists, abnormal chest X-ray films often exhibit visual abnormalities such as irregular shape, unusual size or structure, complex texture or opacity. Therefore, in the present application, three types of bottom-level visual features, including geometric shape features, intensity features and texture features, are collected. Among them, the geometric shape features include grid surface, pixel surface, perimeter and perimeter ratio, etc. commonly used in imageomics. In addition, in order to extract more fine-grained sample intensity features, the present application uses the lung partition method commonly used in medicine to divide the segmented chest X-ray film image into inner, middle and outer zones longitudinally, and upper, middle and lower fields transversely, and respectively based on Figure 3The first-order statistics features (FOSF, i.e. intensity features) of each individual are extracted from 10 sub-regions in the left and right lung and the whole left and right lung. Meanwhile, the Gray Level Co-occurrence Matrix (GLCM, i.e. texture features) of each individual is extracted based on the four directions of 0°, 45°, 90° and 135°, respectively. The key motivation of using GLCM texture features is that it can capture the spatial relationship and predefined distance of pixel pairs with predefined gray intensity in different directions, and it is a widely accepted texture feature coding method in the field of medical image analysis.

[0070] Mid-level deep feature extraction: Since the encoding feature distribution of normal samples is forced to fit the standard Gaussian distribution during the training of the aforementioned generative adversarial network, the encoding features of normal samples are more consistent with the standard Gaussian distribution during the inference process, while the encoding feature distribution of abnormal samples is more discrete. At the same time, the semantic information contained in the features obtained after encoding by the encoder is stronger, so the image encoding features extracted by the encoder in the aforementioned trained generative adversarial network are used as the mid-level deep learning features of the image.

[0071] High-level experience feature extraction: Existing research based on doctors' experience shows that abnormalities such as diffuse lung can directly cause abnormalities in the width or height of the lung in chest X-ray imaging; cardiomegaly can cause abnormal cardiothoracic ratio; pleural effusion can cause blunting of the costophrenic angle; and the width of the mediastinum differs between normal images and images of patients with pulmonary tuberculosis. Therefore, the height difference between the left and right lungs, the lung width-height ratio, the cardiothoracic ratio, the left and right lung costophrenic angle, and the upper and lower mediastinal width features of the segmented chest X-ray image obtained by the lung segmentation module are extracted as high-level features of the image.

[0072] As shown in Table 1, the present application comprehensively collects a total of 646 features, including 502 bottom-level visual features, 128 mid-level deep features, and 16 high-level experience features.

[0073] Table 1: Summary of multi-level features

[0074]

[0075] Pulmonary disease screening module: used to construct a pulmonary disease screening model using the multi-level features obtained by the multi-level feature construction module and the abnormality scores obtained by the pulmonary disease preliminary screening module, to obtain the final pulmonary disease classification result. The pulmonary disease screening module includes a feature grouping submodule, a feature selection and extraction submodule, and a classification output submodule. Among them:

[0076] Feature grouping submodule: the multi-level feature is divided into seven groups, i.e. high-level experience feature, middle-level depth feature, geometric shape feature, left lung intensity feature, right lung intensity feature, left lung texture feature and right lung texture feature according to left and right lung area division and feature category division.

[0077] Feature selection and extraction submodule: used for performing feature selection and feature extraction on the seven groups of features respectively by using univariate analysis method and multivariate analysis method, so as to eliminate irrelevant or redundant features as much as possible, and obtain seven groups of feature subsets.

[0078] Specifically, the univariate analysis adopts t-test and mutual information method to screen out features below the threshold. It should be noted that the above method needs to compare and evaluate normal and abnormal samples, and the method only exists one normal class during training, therefore, the above feature selection methods are completed on the validation set. In addition, considering that there may be multiple features that jointly affect the classification result, factor analysis method is used for multivariate analysis to extract common factors of multiple features for classification. Finally, the AUC value of the validation set classification result is used as the evaluation criterion to select the optimal seven groups of feature subsets.

[0079] Classification output submodule: used for constructing OC-SVM base classifier for the seven groups of feature subsets, and further combining the abnormal scores obtained by the seven OC-SVM base classifiers and the preliminary screening abnormal scores output by the lung disease preliminary screening module as the input of the OC-SVM meta-classification model to obtain the final lung disease classification result.

[0080] Embodiment 1:

[0081] The chest X-ray data TCLD_CXR data set used in this embodiment is from a domestic first-class hospital, a total of 5299 chest X-ray images are collected, of which 2383 images are normal images without any disease, and 2806 images are accompanied by various pulmonary tuberculosis diseases. The collected image format is DICOM format, and the image resolution is between 858*1004 pixels and 3480*4240 pixels. All the annotation work of the data is completed by three professional imaging doctors, and the final annotation is determined by voting for the annotation results with objection. In addition, data desensitization work is completed for all DICOM images. And the 16-bit gray value DICOM image is mapped to 8-bit gray value, and the image is saved in lossless compression type PNG format.

[0082] The data set is divided into 1200 normal images as a training set, 500 normal images and 500 abnormal images as a validation set, and the remaining images are used as a test set. In the test process, the test is simulated by the physical examination data set. A total of 50 tests are performed, each test randomly extracts 500 normal samples and 25 abnormal samples from the test set for testing, and the abnormality is less than 5%.

[0083] Then, the following steps are implemented:

[0084] Lung parenchyma segmentation module: The Attention U-net model trained by the above disclosed Montgomery CXR data set is used to automatically segment the lung parenchyma region of the data set, and the predicted segmentation mask is subjected to connected region detection to obtain the final lung region segmentation mask and segmentation result as shown in the following figure: Figure 4

[0085] Lung disease screening module: The feature consistency-based variational auto-encoding generative adversarial network proposed in the present application is trained using normal X-ray films. The trained network is used to reconstruct the test X-ray film, and the image reconstruction loss L rec and feature Figure One consistency loss L fmp together as the abnormal score (AS) of the test sample, that is, In this embodiment, a1 and a2 are set to 3 and 2, respectively.

[0086] Multi-level feature construction module: The corresponding bottom visual features and high-level experience features are extracted from the segmented chest X-ray film obtained from the lung parenchyma segmentation module. At the same time, the feature consistency-based variational auto-encoding generative adversarial network trained by the lung disease screening module is used to extract the encoder hidden layer encoding features as the middle layer deep features of the X-ray film.

[0087] ​Pulmonary disease screening module: The constructed multi-level features are divided into seven groups, including high-level experience features, middle-level depth features, geometric shape features, left lung intensity features, right lung intensity features, left lung texture features, and right lung texture features. T-test and mutual information method are used to evaluate the importance of the features, and features below the threshold are removed. Meanwhile, factor analysis method is used for multivariate analysis to extract common factors of multiple features for classification. The AUC value of the classification results of the validation set is used as the evaluation criterion to select the optimal seven feature subsets. Finally, OC-SVM base classifiers are constructed for the seven feature sets, and the abnormal scores obtained by the seven base classifiers and the initial screening abnormal scores are used to construct an OC-SVM meta-classification model to obtain the final classification results. In this embodiment, the OC-SVM kernel function is a Gaussian kernel function, and the kernel coefficient gamma is set to'scale' mode, i.e. 1 / (n_features*X.var()) as the gamma value. The upper limit of the allowed training error score nu for each base model is set to nu_shape=0.25, nu_left_fosf=0.08, nu_right_fosf=0.1, nu_left_glcm=0.15, nu_right_glcm=0.2, nu_exp=0.3, and nu_deep=0.25. The upper limit of the allowed training error score nu for the integrated meta-model is selected according to the misdiagnosis rate when the validation set screening result has a 0 false negative rate, and finally set to nu_ensemble=0.25.

[0088] The comparison results of the generative adversarial network in this embodiment and several existing deep learning models for single-class classification are shown in Table 2. The evaluation index AUC is used to represent the performance of the classifier, with a value range of [0.5, 1], and a higher value indicates better performance. False positive rate (FPR) represents the misdiagnosis rate, and false negative rate (FNR) represents the missed diagnosis rate.

[0089] From Table 2, it can be seen that although the classification effect of the feature consistency-based variational auto-encoding generative adversarial network proposed in this embodiment has improved significantly compared to other methods, it still cannot achieve the screening goal of a 0 false negative rate in this paper. In this regard, we believe that deep learning models lack the guidance of doctors' experience, and it is difficult to learn distinguishing features that distinguish other classes from diseased images when only normal images of one class are involved in training, especially when the difference between the diseased images and the normal images is small during inference.

[0090] Table 2 Comparison of experimental results of existing single-class classification deep learning models

[0091]

[0092] Further, the embodiment sets several groups of ablation tests as shown in Table 3. It can be seen that the X-ray lung disease screening method proposed by the present application which fuses multi-level features has obvious improvement compared with using only the doctor's experience features and using only the deep learning model, and the method can ensure that the misdiagnosis rate is 0 under the premise of ensuring the misdiagnosis rate is greatly reduced. At the same time, it can be seen that the prediction result variance of the method is also obviously reduced, which proves that the method has obvious improvement in robustness. Further prove the effectiveness of fusing doctor's experience.

[0093] Table 3 Ablation experiment results

[0094]

[0095] Comparative Example 2:

[0096] The public data set Chest X-ray14 used in the embodiment is provided by the National Institutes of Health, and contains 112,120 individually labeled front chest X-ray images of 14 different lung diseases of 30,805 patients. Researchers use NLP methods to label the images. This data set is currently widely used in the field of lung disease screening.

[0097] Here, the embodiment is mainly used to compare several relatively advanced lung disease screening technologies for specific diseases. In the experimental implementation process, the other several technologies only screen for pneumonia disease, and the training set contains 800 normal X-ray images and 800 pneumonia disease images, and the validation set and the test set both contain 200 normal images and 200 abnormal images. Among them, the abnormal images of the validation set and the test set-1 only contain pneumonia disease, while the abnormal images of the test set-2 contain 10% of other diseases. The training set applied to the present method only contains 800 normal X-ray images, and the validation set and the test set are set in the same way as the above validation set and test set-2. Here, the above several technologies are compared in the test process to add unknown diseases.

[0098] The comparison results are shown in Table 4, wherein ResNet50, VGG19, DesNet121, Xception, and MobileNet are several commonly used basic classification models in the field of X-ray lung disease screening. As can be seen from Table 4, the system of the present application can well classify unknown diseases, and the missed diagnosis rate of the screening result is 0. The missed diagnosis rate of other technologies is more or less improved after adding unknown diseases in the test process, and the performance of the classifier also decreases to a certain extent. Among them, the reason why the FPR value does not change is that the normal samples of the two test sets do not change, so the classification result does not change. Based on this result, it can be seen that the present application can well ensure that the unknown disease is not misclassified and not missed.

[0099] Table 4 Comparison results of Chest X-ray14 dataset

[0100]

Claims

1. A system for screening lung diseases from X-ray images by fusing multi-level features, characterized in that, The application relates to a lung disease screening system, which comprises a lung parenchyma segmentation module, a lung disease preliminary screening module, a multi-level feature construction module and a lung disease screening module. The lung parenchyma segmentation module is used for segmenting lung parenchyma from a complete chest X-ray film. The lung parenchyma segmentation module comprises a pretreatment sub-module, a lung parenchyma segmentation model training sub-module and a segmentation sub-module. The pretreatment sub-module is used for simultaneously performing rotation, translation, cutting and horizontal flip operations on Montgomery CXR dataset chest X-ray films and corresponding segmentation masks to obtain pretreated pictures as a training set. The lung parenchyma segmentation model training sub-module is used for training an Attention U-net model by using the training set to obtain a trained lung parenchyma segmentation model. The segmentation sub-module is used for inputting a chest X-ray film image to be segmented into the trained lung parenchyma segmentation model to obtain a corresponding segmentation mask binary graph, and further obtaining a segmented lung region according to the segmentation mask binary graph. The lung disease preliminary screening module is used for constructing and training a feature consistency variational auto-encoding generative adversarial network to obtain a preliminary screening abnormality score by using the trained generative adversarial network. The lung disease preliminary screening module comprises a generative adversarial network construction sub-module, a generative adversarial network training sub-module and a lung disease preliminary screening sub-module. The generative adversarial network construction sub-module is used for constructing a generative adversarial network. The generative adversarial network training sub-module is used for the following functions: obtaining a complete lung region by cutting the original chest X-ray film according to the segmentation mask binary graph obtained by the lung parenchyma segmentation module, dividing the cut X-ray film into a training set, a verification set and a test set, wherein only normal X-ray films are contained in the training set, and training the generative adversarial network by using the training set to obtain a trained generative adversarial network. The lung disease preliminary screening sub-module is used for inputting the test set into the trained generative adversarial network to obtain a reconstructed image, and calculating a preliminary screening abnormality score by calculating the difference between before and after reconstruction. The multi-level feature construction module is used for extracting multi-level features, including extraction of bottom visual features, middle deep features and high-level experience features. The lung disease screening module is used for constructing a lung disease screening model by using the multi-level features obtained by the multi-level feature construction module and the abnormality score obtained by the lung disease preliminary screening module, and obtaining a final lung disease classification result.

2. The integrated multi-level feature fusing chest radiograph lung disease screening system of claim 1, wherein, The lung parenchyma segmentation module further comprises a connected region detection sub-module, which is used for performing connected region detection on the segmentation mask binary graph output by the segmentation sub-module, and only retaining the largest two connected regions, i.e. left and right lung regions, to obtain a final lung region segmentation mask binary graph.

3. The fusion multi-scale feature integrated x-ray chest radiography system of claim 1, wherein, The generative adversarial network comprises two encoders Enc , one decoder Dec , one discriminator D , and a hidden layer encoding classifier C , and the two encoders share weights.

4. The fusion multi-scale feature X-ray chest radiography system for lung disease screening of claim 3, wherein, In the generative adversarial network training sub-module, the specific operation of training the generative adversarial network by using the training set is as follows: First, the image in the training set is taken as an input image , and the dimension size of the hidden layer coding feature obtained by the encoder is 128. Then, 128-dimensional features are sampled from the standard Gaussian distribution, and the sampled features are sent to the hidden layer encoding classifier together with the hidden layer encoding features described above for classification, and the binary cross-entropy loss is used as the hidden layer encoding adversarial loss for updating the parameters of the hidden layer encoding classifier and the encoder Meanwhile, the hidden layer encoding features are sent into the decoder to obtain a reconstructed image of the input image The sampled features obtained in the above standard Gaussian distribution are sent into the decoder to obtain a generated image ; the generated image , the reconstructed image , and the input image are jointly sent into the discriminator for classification; here, the cross-entropy loss is used as the image adversarial loss to update the parameters of the discriminator and the decoder; and the L1 loss is used as the image reconstruction loss to update the parameters of the encoder and the decoder;​ Furthermore, the reconstructed image is fed into a second encoder which shares weights with the first encoder; features from the second convolutional layer of both encoders fmp and the encoded features of the last layer code are used as inputs to a discriminator fmp which contains shallow features of the image , code which contains deep semantic features of the image; uses L2 loss as feature map consistency loss and L1 loss as encoding consistency loss to update the decoder parameters; Finally, the encoder parameters , the decoder parameters , , and the hidden layer encoding classifier parameters are updated, respectively.

5. The fusion multi-scale feature integrated x-ray chest radiography system of claim 4, wherein, Hyperparameters , , , are set to 4, 10, 4, 2, respectively.

6. The fusion multi-scale feature integrated x-ray chest radiography system of claim 4, wherein, In the lung disease preliminary screening sub-module, according to the image reconstruction loss The feature map consistency loss The abnormal score of each test sample in the test set is calculated: Wherein, And And are hyperparameters, and are set to 3 and 2.

7. The fusion multi-scale feature integrated x-ray chest radiography system of claim 4, wherein, The bottom-level visual features include geometric features, intensity features and texture features of the lung region; the middle-level deep features adopt image encoding features extracted by an encoder of a trained generative adversarial network; and the high-level experience features include left and right lung height difference, lung width-height ratio, cardiothoracic ratio, left and right lung costophrenic angle and upper and lower mediastinal width features extracted from the segmented chest X-ray film output by the lung parenchyma segmentation module.

8. The fusion multi-scale feature X-ray chest radiography system for lung disease screening of claim 7, wherein, The lung disease screening module comprises a feature grouping module, a feature selection and extraction module and a classification output module, wherein: The feature grouping module is used for dividing the multi-level feature into seven groups, i.e., high-level experience features, middle-level deep features, geometric features, left lung intensity features, right lung intensity features, left lung texture features and right lung texture features, according to left and right lung region division and feature category division; The feature selection and extraction module is used for performing feature selection and feature extraction on the seven groups of features by using univariate analysis method and multivariate analysis method respectively to obtain seven groups of feature subsets; The classification output module is used for constructing OC-SVM base classifiers for the seven groups of feature subsets, and further combining the abnormal scores obtained by the seven OC-SVM base classifiers and the preliminary screening abnormal scores output by the lung disease preliminary screening module as inputs of an OC-SVM meta-classification model to obtain a final lung disease classification result.

9. The fusion multi-scale feature X-ray chest radiography system for lung disease screening of claim 8, wherein, In the feature selection and extraction module, the univariate analysis method adopts t -Test and mutual information method, multivariate analysis method adopts factor analysis method; the AUC value of the classification result of the verification set is taken as the evaluation criterion to select the optimal seven feature subsets.

Images