Intelligent monitoring and early warning method for unknown pneumonia based on multi-modal AI big data

Abstract

The invention provides an intelligent monitoring and early warning method for unknown pneumonia based on multi-modal AI big data, belongs to the technical field of medical detection, solves the problem of low sensitivity during differential diagnosis of different pneumonia, and comprises the following steps: step 1, constructing a multi-modal and multi-classification model, and training the multi-modal and multi-classification model; 2, obtaining a known sample and an unknown sample, and inputting the known sample and the unknown sample into the trained multi-modal and multi-classification model to obtain the distance between the center of the known sample feature and the center of the unknown sample feature; 3, constructing a self-encoding anomaly detection model, and inputting the known sample and the unknown sample into the self-encoding anomaly detection model to obtain reconstruction difference features; and 4, inputting the distance between the known sample feature center and the unknown sample feature center and the reconstruction difference feature into the discrimination network to obtain an anomaly detection result, and sending out early warning information according to the anomaly detection result.

Description

Technical Field

[0001] The present invention relates to an intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data, and belongs to the field of medical detection technology. Background Art

[0002] How to achieve the early diagnosis, identification and reporting of new PUE is of great significance to improving the current epidemic prevention and control system. When the AI ​​model is used to classify pneumonia and normal imaging data, it can achieve very ideal discrimination performance (sensitivity and specificity are close to 100%), but when used for differential diagnosis of different pneumonia, its sensitivity is lower (about 90%).

[0003] Combined with the patient's clinical characteristics, it can provide more dimensional information and help AI achieve more accurate multi-classification differential diagnosis. So far, AI-based multimodal data anomaly detection has not been used to identify PUE and other classified pneumonia. Therefore, the establishment of this method in PUE identification and early warning is innovative and of great medical value. In the AI-assisted diagnosis model. On the premise of greatly improving the classification accuracy of different types of pneumonia, the AI-based multimodal data anomaly detection method can be used to identify PUE. Summary of the invention

[0004] The present invention aims to solve the problem of low sensitivity in the differential diagnosis of different pneumonias, and further proposes an intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data.

[0005] The technical solution adopted by the present invention to solve the above-mentioned problem is: the specific steps of the present invention include:

[0006] Step 1: Build a multimodal and multi-classification model and train the multimodal and multi-classification model;

[0007] Step 2: Obtain known samples and unknown samples, and input them into the trained multimodal and multi-classification model to obtain the distance between the centers of known sample features and unknown sample features;

[0008] Step 3: Build an autoencoder anomaly detection model, input known samples and unknown samples into the autoencoder anomaly detection model to obtain reconstructed difference features;

[0009] Step 4: Input the distance between the known sample features and the unknown sample feature centers and the reconstructed difference features into the discriminant network to obtain the anomaly detection results, and issue warning information based on the anomaly detection results.

[0010] Optionally, the known samples in step 2 include normal lung CT images, bacterial lung CT images, fungal lung CT images, other viral lung CT images, and COVID-19 lung CT images, and the unknown samples include unknown chest CT images.

[0011] Optionally, the step of obtaining the distance between the center of the known sample feature and the center of the unknown sample feature in step 2 includes:

[0012] Step 2.1: Input normal lung CT images, bacterial lung CT images, fungal lung CT images, other viral lung CT images, and COVID-19 lung CT images into the classifier module to obtain deep feature maps of five categories;

[0013] Step 2.2: Reduce the dimension of the deep feature maps of the five categories and map them to the deep embedding space to obtain five clusters. The center point and range of each category are calculated based on the clusters. Gaussian distribution is used to model each cluster, and the Gaussian distribution modeling function is optimized to minimize the Gaussian distribution modeling function.

[0014] Step 2.3: Map the depth feature map of the unknown sample to the depth embedding space, and calculate the position of the unknown sample in the depth space;

[0015] Step 2.4: Calculate the distance between the unknown sample and the center points of the five clusters, and determine whether the unknown sample is within the range of the five clusters;

[0016] The calculation formula of Gaussian distribution is:

[0017]

[0018] In formula (1), μ is the center of the category and σ is the variance of the category.

[0019] Optionally, the autoencoding anomaly detection model in step 3 includes an encoder, a feature point storage module, and a decoder;

[0020] The encoder consists of several 3D convolutional layers to obtain embedded codes of samples;

[0021] The feature point storage module is used to store the feature points of known samples obtained from encoding and decoding;

[0022] The convolution layer includes several deconvolution layers, which are used to reconstruct the sample feature points.

[0023] Optionally, in step 3, the step of inputting the known sample and the unknown sample into the autoencoder anomaly detection model to obtain the reconstructed difference features includes:

[0024] Step 3.1: Input the known sample into the encoder and decoder for self-encoding and decoding until the loss between the input of the known sample and the reconstruction of the self-encoding anomaly detection model is less than a preset threshold, obtain the feature points of the known sample, and store the feature points of the known sample in the feature point storage module;

[0025] Step 3.2: Input the known samples and the unknown samples into the encoder to obtain the embedded codes of the known samples and the embedded codes of the unknown samples;

[0026] Step 3.3: Find the feature points closest to the embedded code of the known sample and the embedded code of the unknown sample in the feature point storage module and send them to the decoder for reconstruction;

[0027] Step 3.4: Calculate the difference between the reconstructed sample and the input sample, and use the difference between the reconstructed sample and the input sample as the reconstruction difference feature;

[0028] The expression of the reconstruction process is:

[0029]

[0030] In formula (2), Encoder is the encoder, Decoder is the decoder, and MSE is the reconstruction loss function.

[0031] Optionally, the step of obtaining the abnormal detection result in step 4 and issuing a warning message according to the abnormal detection result includes:

[0032] Step 4.1: Input the distance and reconstructed difference features of the known sample features and the unknown sample feature centers into the discriminant network in the same channel dimension, connect to several fully connected layers and activation layers to obtain the abnormal confidence probability, and judge whether the current unknown sample is abnormal data according to the confidence abnormal probability;

[0033] Step 4.2: Establish a normal reference baseline through diagnostic analysis of the period without epidemics. If the unknown data exceeds the baseline by 1.96 standard deviations for three consecutive days, issue an early warning message;

[0034] The calculation formula for abnormal confidence probability is:

[0035]

[0036] In formula (3), F θ is the discriminant function.

[0037] The beneficial effects of the present invention are:

[0038] 1. The present invention introduces multi-center, multi-modal big data to establish an AI pneumonia differential diagnosis model, and based on this, innovatively cross-introduces an outlier detection AI model that has been maturely used in other fields for the identification of PUE. In the later stage, through real-time dynamic analysis of multiple centers, the established AI model is continuously fine-tuned, and ultimately the automatic detection and timely warning of PUE are achieved, assisting the CDC in the early control of infectious PUE.

[0039] 2. Based on the established public database platform iCTCF, the present invention expands the data of known and unknown samples and adds additional dimensional information input, such as multi-classification pneumonia, patient admission records, etc. By further upgrading and improving the database, expanding the types of diseases and the number of cases, and refining the input of multi-phase chest CT images and multiple review clinical related indicators, it is expected to provide basic data support for the later research based on multi-center, multi-modal AI differential diagnosis models and PUE early warning platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of the intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data provided by the present invention;

[0041] Figure 2 This is a workflow diagram of the self-encoding anomaly detection model provided by the present invention. DETAILED DESCRIPTION

[0042] Combination Figure 1-2 This embodiment is described as follows. Figure 1 As shown, the steps of the intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data described in this embodiment include:

[0043] S1: Obtain known samples and unknown samples and input them into the trained multimodal and multi-classification model;

[0044] S101: Known samples include normal lung CT images, bacterial lung CT images, fungal lung CT images, other viral lung CT images, and COVID-19 lung CT images. Unknown samples include unknown chest CT images. Normal lung CT images, bacterial lung CT images, fungal lung CT images, other viral lung CT images, and COVID-19 lung CT images are input into the classifier module to obtain deep feature maps of five categories.

[0045] S102: Take the deep features of each category obtained in the previous classifier module. Then reduce the dimension of each category feature map and map it to the deep embedding space;

[0046] S103: Each type of feature point will form a cluster in the deep embedding space, and the center point and range of each category are calculated. The distribution of each category is modeled using Gaussian distribution, and the Gaussian distribution modeling function is optimized to minimize the Gaussian distribution modeling function;

[0047] S104: For an unknown chest CT image, calculate the position of the unknown sample in the depth space according to S101-S103;

[0048] S105: The distance between the unknown sample and the center point of the other five categories and whether it is within the range of the other five categories are taken as the detection result;

[0049] The calculation formula of Gaussian distribution is:

[0050]

[0051] In formula (1), μ is the center of the category and σ is the variance of the category.

[0052] Several radiologists manually annotated, segmented and corrected some samples in turn, and added them to the admission text. For known and unknown samples in the admission text record, the present invention used the Chinese Word2Vec pre-trained model to convert the Chinese text into a vector form, and then used the transformer layer (Attention Is All You Need) to convert the text vector into a deep embedding space. After that, the present invention can use the attention mechanism (Non-local Neural Networks) to extract image features of other modalities. In the final classifier module, the present invention can use several layers of fully connected layers to obtain the probability values ​​of each category; in the synchronized decoder module, the present invention uses several 3D convolutional layers to obtain the final lesion localization heat map. The present invention uses Score-CAM (Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks) to visualize the model's location of interest in the image input, and the attention map (Attention Map) to extract high response values ​​in the matrix to obtain the clinical data and admission text of interest to the model. After the model converges and the performance meets the actual requirements, the model is deployed on the diagnostic platform to realize integrated automated analysis. That is, for newly collected chest CT samples, the model can directly output the category of the current sample. If it is a pneumonia sample, the model outputs the specific location of the lesion.

[0053] In the deep learning process of multimodal and multi-classification models, in addition to chest CT images, medical diagnosis also includes oral descriptions of the condition and laboratory tests, which serve as important bases for clinicians to judge the disease. However, oral descriptions of the condition are highly unstructured text information, and laboratory tests are structured data information. The deep model of the present invention has the ability to process information from different sources and in multiple modes to process highly unstructured text information and structured data information.

[0054] S2: Build an autoencoding anomaly detection model, input known samples and unknown samples into the autoencoding anomaly detection model to obtain reconstructed difference features;

[0055] S201: Figure 2 As shown, all known samples are sent to the network for self-encoding and decoding, and the model is trained until the loss between the input of the known samples and the network reconstruction is less than a certain threshold, and several typical feature points of the known samples are obtained;

[0056] S202: For the newly input sample, after passing through the encoder, the nearest feature point is found among the recorded typical feature points and sent to the decoder for reconstruction. If it is an abnormal sample, the reconstruction loss will be very large, while for a normal sample, the reconstruction loss will be small. The encoder includes several 3D convolutional layers, and the convolutional layer includes several deconvolutional layers.

[0057] S203: inputting the difference between the reconstructed sample and the input sample into the discriminant network as a reconstructed difference feature;

[0058] The expression of the reconstruction process is:

[0059]

[0060] In formula (2), Encoder is the encoder, Decoder is the decoder, MSE is the reconstruction loss function, and finally it is determined whether the L(θ) function is too large.

[0061] S3: Input the distance between the known sample features and the unknown sample feature centers and the reconstructed difference features into the discriminant network to obtain the anomaly detection results, and issue warning information based on the anomaly detection results;

[0062] S301: Input the unknown sample’s distance from the center of other classes and the reconstructed difference features into the subsequent discriminant network in the channel dimension, access several fully connected layers and activation layers to obtain the final abnormal confidence probability, and then determine whether the current unknown sample is abnormal;

[0063] The calculation formula for abnormal confidence probability is:

[0064]

[0065] In formula (3), F θ is the discriminant function.

[0066] S302: The discrimination network will output the probability value of sample I being PUE. The central cloud server cloud will input a large amount of patient information uploaded and saved from multiple centers every day. The present invention can automatically perform dynamic multimodal data analysis on the patient data of daily chest CT examinations through the AI ​​diagnostic model established above. Through the PUEAI diagnostic analysis within a sufficient observation time during the period without an epidemic (≥1 year), the present invention can establish a normal baseline reference. If the AI ​​model indicates that the count data and / or growth rate of a single day and / or multiple days (such as 3 days and 5 days) exceeds 1.96 standard deviations of the baseline (p<0.05), the system will automatically issue a warning message, and the radiologists and clinical physicians will jointly identify the warning information to determine whether it needs to be reported.

[0067] On the basis of establishing the above-mentioned multimodal, multi-classification model and autoencoding anomaly detection model, the present invention builds a continuously expanding data platform. Every day, a large number of pneumonia patients come to the hospital for treatment, and doctors will make detailed diagnoses of the chest CT samples of these patients. Therefore, the present invention can batch add samples after daily subjective identification by doctors to the data platform, and mark samples with different results from the AI ​​model and the doctor to focus on the errors of these models; and focus on samples with different diagnostic results from the model and the doctor. The present invention fine-tunes the trained AI model and enters the self-optimization algorithm, so that the multimodal, multi-classification model has better generalization ability.

[0068] In summary, the present invention introduces multi-center, multi-modal big data to establish an AI pneumonia differential diagnosis model, and based on this, innovatively cross-introduces an outlier detection AI model that has been maturely used in other fields for the identification of PUE. In the later stage, through real-time dynamic analysis of multiple centers, the established AI model is continuously fine-tuned, and ultimately the automatic detection and timely warning of PUE are achieved, assisting the CDC in the early control of infectious PUE.

[0069] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. An intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data, characterized in that: The steps of the intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data include: Step 1: Build a multimodal and multi-classification model and train the multimodal and multi-classification model; Step 2: Obtain known samples and unknown samples, and input them into the trained multimodal and multi-classification model to obtain the distance between the centers of known sample features and unknown sample features; Step 3: construct an autoencoder anomaly detection model, and input the known samples and unknown samples into the autoencoder anomaly detection model to obtain reconstructed difference features; Step 4: Input the distance between the known sample feature and the unknown sample feature center and the reconstructed difference feature into the discriminant network to obtain anomaly detection results, and issue warning information based on the anomaly detection results.

2. The intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data according to claim 1 is characterized in that: In step 2, the known samples include normal lung CT images, bacterial lung CT images, fungal lung CT images, other viral lung CT images, and COVID-19 lung CT images, and the unknown samples include unknown chest CT images.

3. The intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data according to claim 1 is characterized in that: The steps of obtaining the distance between the centers of known sample features and unknown sample features in step 2 include: Step 2.1: Input the normal lung CT images, bacterial lung CT images, fungal lung CT images, other viral lung CT images, and COVID-19 lung CT images into a classifier module to obtain deep feature maps of five categories; Step 2.2: Reduce the dimension of the deep feature maps of the five categories and map them to the deep embedding space to obtain five clusters. The center point and range of each category are calculated based on the clusters. Gaussian distribution is used to model each cluster, and the Gaussian distribution modeling function is optimized to minimize the Gaussian distribution modeling function. Step 2.3: Map the depth feature map of the unknown sample to the depth embedding space, and calculate the position of the unknown sample in the depth space; Step 2.4: Calculate the distance between the unknown sample and the center points of the five clusters, and determine whether the unknown sample is within the range of the five clusters; The calculation formula of Gaussian distribution is: In formula (1), μ is the center of the category and σ is the variance of the category.

4. The intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data according to claim 1 is characterized in that: The autoencoding anomaly detection model in step 3 includes an encoder, a feature point storage module, and a decoder; The encoder includes several 3D convolutional layers for obtaining embedded coding of samples; The feature point storage module is used to store the feature points of known samples obtained by self-encoding and decoding; The convolution layer includes several deconvolution layers, which are used to reconstruct the sample feature points.

5. The intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data according to claim 1, characterized in that: In step 3, the steps of inputting known samples and unknown samples into the autoencoder anomaly detection model to obtain reconstructed difference features include: Step 3.1: Input the known sample into the encoder and decoder for self-encoding and decoding until the loss between the input of the known sample and the reconstruction of the self-encoding anomaly detection model is less than a preset threshold, obtain the feature points of the known sample, and store the feature points of the known sample in the feature point storage module; Step 3.2: Input the known samples and the unknown samples into the encoder to obtain the embedded codes of the known samples and the embedded codes of the unknown samples; Step 3.3: Find the feature points closest to the embedded code of the known sample and the embedded code of the unknown sample in the feature point storage module and send them to the decoder for reconstruction; Step 3.4: Calculate the difference between the reconstructed sample and the input sample, and use the difference between the reconstructed sample and the input sample as the reconstruction difference feature; The expression of the reconstruction process is: In formula (2), Encoder is the encoder, Decoder is the decoder, and MSE is the reconstruction loss function.

6. The intelligent monitoring and early warning method for pneumonia of unknown cause based on multimodal AI big data according to claim 1, characterized in that: The steps of obtaining the abnormal detection result in step 4 and issuing warning information according to the abnormal detection result include: Step 4.1: Input the distance and reconstructed difference features of the known sample features and the unknown sample feature centers into the discriminant network in the same channel dimension, connect to several fully connected layers and activation layers to obtain the abnormal confidence probability, and judge whether the current unknown sample is abnormal data according to the confidence abnormal probability; Step 4.2: Establish a normal reference baseline through diagnostic analysis of the period without epidemics. If the unknown data exceeds the baseline by 1.96 standard deviations for three consecutive days, issue an early warning message; The calculation formula for abnormal confidence probability is: In formula (3), F θ is the discriminant function.

Images