Chest pathology artificial intelligence aggregation prediction method based on multi-modal data

Through the artificial intelligence aggregation method of multimodal data, table, image and text data are integrated, and feature embedding is extracted using multiple machine learning models, which solves the problem of multimodal data processing in medical diagnosis, and improves the accuracy of chest pathological diagnosis and efficiency.

CN118942679APending Publication Date: 2024-11-12BEIHANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411003019.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In medical diagnosis, it is difficult for the prior art to effectively integrate and process a variety of different data modes, resulting in limited clinical diagnostic efficiency and accuracy, especially in complex chest pathology.

Method used

Using an artificial intelligence aggregation method based on multimodal data, the data is extracted by converting table, image and text data into a unified format, and using machine learning models such as multi-layer perceptron, random forest and XGBoost, combined with TabNet, ResNet101 and BERT models. Feature embedding, Shapley values ​​are calculated to analyze the contributions of each data source and modality, and a chest pathology prediction model is constructed.

Benefits of technology

It improves the accuracy and efficiency of chest pathological diagnosis, provides an auxiliary medical tool, and enhances the multimodal data processing capability of clinical diagnosis.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942679A_ABST
    Figure CN118942679A_ABST
Patent Text Reader

Abstract

The invention provides a chest pathology artificial intelligence aggregation prediction method based on multi-modal data. Comprising the following steps: converting original medical data from different modalities into formats suitable for processing, establishing three machine learning general models for training, respectively extracting multi-modal features of a patient, embedding and inputting the multi-modal features into the models for chest pathology aggregation prediction, calculating Shapley values of different data sources, and analyzing the influence of each modal on the performance of the models. The artificial intelligence aggregation prediction method based on the multi-modal data can become an auxiliary tool for clinical practice and medical diagnosis, and the clinical diagnosis efficiency and accuracy are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0024] The present invention provides a chest pathology prediction method, and specifically relates to a chest pathology prediction method based on artificial intelligence aggregation of multimodal data, and belongs to the technical field of multimodal medical big data intelligent analysis. Background Art

[0025] Artificial intelligence (AI) methods are expected to become foundational tools in the next generation of clinical practice and healthcare operations, especially in improving clinical efficiency and patient outcomes. The effectiveness of such methods depends on integrating comprehensive knowledge from multiple data sources and different input modalities. The multimodal structure of AI methods is particularly attractive because of their ability to mimic the diverse input conditions that clinicians and healthcare administrators currently use to make predictions and deal with complex decision-making environments (King Z, Farrington J et al.).

[0026] In typical clinical practice, patient electronic health records (EHRs) contain information in multiple formats, such as tabular data (including age, demographics, and medical history), image data (such as photographs, X-rays, computed tomography, and magnetic resonance imaging), time series data (such as blood analysis, electrocardiograms, ultrasound, and wearable sensors), and unstructured sequence data (such as notes, forms, voice recordings, and videos).

[0027] Recently, AI models that fuse multiple data modalities have been validated in areas such as cardiology, gastroenterology, hematology, neurology, oncology, and healthcare operations management (e.g., mortality, length of stay, and discharge prediction). In addition, studies have shown that multimodal systems generally outperform single-modality approaches in most of these areas.

[0028] The present invention not only exploits multimodal inputs, but also attempts to address multiple bottleneck challenges of AI approaches in healthcare in a more unified way than previously achieved, including the possibility of processing data from multiple different modalities. By aggregating predictions from multimodal data, the model can be useful for daily practice in managing complex clinical situations. Summary of the invention

[0029] The purpose of this invention is to provide a chest pathology aggregation prediction method, which uses artificial intelligence technology and multimodal data to predict the classification of patient chest pathology categories, and can become an auxiliary tool for clinical practice and medical diagnosis, improving clinical diagnosis efficiency and accuracy. It is based on the current development of multimodal databases in the medical field, and is in line with the current trend of artificial intelligence being applied to medical diagnosis.

[0030] The technical solution of the present invention is as follows:

[0031] S1. Converting raw medical data from different modalities into a format suitable for processing, including the following steps:

[0032] S1.1. The table data is de-identified according to the HIPAA Act, so that the patient uses a special unordered number as the unique identifier;

[0033] S1.2, the image data is uniformly reduced to 224×224 size, and the “resize()” function in the “opencv” library is used to implement image downsampling;

[0034] S1.3, the text data comes from the patient examination text report, the "FINDINGS" part is extracted and uniformly processed into a fixed length n;

[0035] S2. Establish three general machine learning model training, including the following steps:

[0036] S2.1. We explored three machine learning architectures, including Multi-Layer Perceptron, Random Forest, and XGBoost, to heuristically determine the best model choice for subsequent experiments.

[0037] S3, extracting the patient's multimodal feature embedding and inputting it into the model for chest pathology aggregation prediction, including the following steps:

[0038] S3.1. Use TabNet network as table feature encoder to fully mine the information in preprocessed table data. Use the data embedding extraction module in TabNet model to embed common inspection indicator data into E om , drug code data embedded in E dr , laboratory measurement index value data embedded in E la , drug dosage and unit data embedding E in Different treatment methods are embedded in E pr And use drugs and frequency embedding E ph ;

[0039] S3.2, ResNet101 network is used as the image feature encoder to extract details and global information from ultrasound image data. The preprocessed uniform size image data is input into the model, and the output of the fully connected layer is used as the image data to embed E im ;

[0040] S3.3. Use the "bert-base" pre-trained model to build a text feature encoder for text embedding extraction, and get the corresponding text data embedding E according to the input text re ;

[0041] S3.1,Modeling experiments were conducted using every possible combination of input embeddings, extracted from the 8 allowed data sources and 3 modalities;

[0042] S3.2. In this process, each data stream is permuted to produce a fused embedding, and the model is trained using unimodal, bimodal, and trimodal combinations of the input;

[0043] S4. Calculating the Shapley values ​​of different data sources, including the following steps:

[0044] S4.1. Analyze the impact of different data source embeddings on the model prediction task, calculate the Shapley values ​​of different data sources to quantitatively derive the contribution of different data sources to the prediction task. The larger the Shapley value, the greater the contribution of the embedding to the model prediction in the current combination.

[0045] S4.2. In order to understand the contribution of each data source and modality to the final performance, the Shapley value of each of the eight data sources and three modalities was calculated;

[0046] S5. Analyze the impact of each mode on model performance, including the following steps:

[0047] S5.1. Analyze the impact of different modal combination inputs on the model prediction task. For single-modal input, bi-modal input, and tri-modal input, they are respectively input into the tumor predictor according to different data source embedding combinations to obtain the predicted output of chest pathology type, and analyze the prediction accuracy of different combination models.

[0048] Through the above steps, an artificial intelligence aggregation prediction method based on machine learning was constructed to assist in the medical diagnosis of chest pathology and improve the accuracy and efficiency of disease diagnosis.

[0049] The advantage of the present invention is that it proposes an artificial intelligence chest pathology aggregation prediction method paradigm, which improves the accuracy and efficiency of medical diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flow chart of a chest pathology artificial intelligence aggregation prediction method based on multimodal data provided by the present invention;

[0051] Figure 2 This is the ResNet101 network structure diagram;

[0052] Figure 3 This is the XGBoost tree building step provided by the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The present invention is a chest pathology artificial intelligence aggregation prediction method based on multimodal data. Figure 1 As shown, the steps are as follows:

[0055] S1. Converting raw medical data from different modalities into a format suitable for processing, including the following steps:

[0056] S1.1. For tabular data, considering the particularity of medical data, in order to better provide patients with health insurance rights and protect the privacy of their health information, the patient identifier is de-identified according to the "HIPAA Act", and the patient's name, surgery date and other information in the original table are hidden. A special unordered number is used as the unique identifier for the patient. In order to limit the preprocessed data to a certain range and eliminate the adverse effects caused by singular sample data, the original tabular data needs to be normalized. This operation can speed up the gradient descent to find the optimal solution, and thus may improve the accuracy. The Z-Score normalization method is used for normalization, as shown in formula (1):

[0057]

[0058] in is the original data mean, is the standard deviation of the original data. After normalization, the data distribution falls between 0 and 1.

[0059] S1.2. For image data, since the input of the image feature extractor is fixed, the image is uniformly reduced to a size of 224×224. Image reduction is achieved by reducing the number of pixels. Therefore, according to the reduced size, appropriate pixels should be selected from the original image so that the features of the original image can be maintained after the image is reduced. Here, the equal-interval sampling method is mainly used, which is to take pixels at a certain distance from the original image and put them on the output image. Suppose the size of the original image is W×H, and the reduction factors of width and length are k1 and k2 respectively, then the sampling intervals are W / k1 and W / k2 respectively. That is to say, a pixel is taken every W / k1 in the horizontal direction of the original image and every W / k2 in the vertical direction. When the reduction factors k1 and k2 of length and width are equal, the image is reduced in equal proportion; when they are not equal, the image is reduced in unequal proportions, and the ratio of the length and width of the reduced image will change. The "resize()" function in the "opencv" library is used to implement image downsampling, and the output image size is defined as 224×224.

[0060] S1.3. For text data, collect the "FINDINGS" part of the pathology diagnosis report, count the number of words in the text data of each patient, select a fixed length n, and add unified invalid characters to all text data for embedding or have professional doctors delete useless characters and uniformly process them into a fixed length n.

[0061] S2. Establish three general machine learning model training, including the following steps:

[0062] S2.1. Multilayer perceptron is a generalized linear model. The difference is that a hidden layer is added between the original input and output. If the input is represented by x, the output is represented by y, and the hidden layer is represented by h, then the relationship between the input and output of the multilayer perceptron can be expressed as shown in formula (2):

[0063] y=v[1]h[1]+v[2]h[2]+s (2)

[0064] h[1]=f(w[1,1]x[1]+w[2,1]x[2]+w[3,1]x[3]+b[0])

[0065] h[2]=f(w[1,2]x[1]+w[2,2]x[2]+w[3,2]x[3]+b[1])

[0066] Wherein f(·) is a nonlinear activation function, and currently the most widely used ones are corrected nonlinearity, tangent hyperbola or sigmoid function. The present invention adopts the sigmoid function, and its expression is shown in formula (3):

[0067]

[0068] The optimization process is to minimize the loss function, which is expressed as shown in formula (4):

[0069]

[0070] In the above formula, v, w, b and s are the variables to be optimized. The gradient descent method is used to find According to the chain rule, we have formula (5):

[0071]

[0072] The Scikit-learn library has integrated a toolkit for multi-layer perceptrons. The present invention uses its classification function "MLPClassifier", sets the activation function to the sigmoid function, and has two hidden layers, each of which contains 10 hidden units. Due to the shallow number of layers, the maximum number of iterations max_epoch=1000, and the output is set to 14 according to the chest pathology classification label.

[0073] S2.2, Random Forest is an ensemble learning algorithm based on the decision tree algorithm. A decision tree is a tree structure in which each internal node represents a feature or attribute, each branch represents a value of this feature or attribute, and each leaf node represents a classification or regression result. Through the decision tree, the data set can be divided into multiple subsets, each of which contains data with the same features or attributes. Each subset is then analyzed and used to classify the type of chest pathology.

[0074] The present invention uses the "RandomForestClassifier" in the Scikit-learn library to build a random forest classification model, and some parameters that need to be set in the algorithm, such as the number of trees, the maximum depth of the trees, etc., use default parameters.

[0075] S2.3, XGBoost is an additive model based on boosted trees, which is a type of gradient boosted trees. Each base classifier used is a regression tree, and the training method is to use a forward step-by-step algorithm to gradually optimize each base learner. The objective function of XGBoost consists of a loss function and a regularization term. For the t-th tree, the model prediction value of the i-th sample is shown in formula (6):

[0076]

[0077] The present invention uses the xgboost library in the Scikit-learn library, inputs the system kernel nthread=12, the learning rate eta=0.01, the maximum tree depth max_depth=5, and the number of iterations num_boost_round=1000.

[0078] S3, extracting the patient's multimodal feature embedding and inputting it into the model for chest pathology aggregation prediction, including the following steps:

[0079] S3.1. TabNet is a new deep learning architecture for table learning. It implements instance-level feature selection through a sequential attention mechanism similar to the additive model, and also implements self-supervised learning through an encoder-decoder framework, thus combining the interpretability of tree models with the representation ability of deep neural networks (DNNs).

[0080] The present invention uses TabNet network as a table feature encoder to fully mine the information in the preprocessed table data. In terms of parameter setting, the present invention sets batch size = 128 (table information occupies less memory than image information, and a larger batch size can be used to increase the training speed), decision steps N_steps = 3, maximum iterations max_epoch = 100, sparse regularization weight lambda_sparse = 1e-3. Different from image operations, since the table data has a more detailed classification, the present invention only uses the data embedding extraction module in the TabNet model to obtain the common inspection indicator data embedding E respectively. om , drug code data embedded in E dr , laboratory measurement index value data embedded in E la , drug dosage and unit data embedding E in Different treatment methods are embedded in E pr And use drugs and frequency embedding E ph , as shown in formula (7):

[0081]

[0082] Where TabPreprpcessor refers to the embedding extraction module of TabNet, cols of embed is the input parameter of TabPreprocessor, and cols is the number of table columns actually used, which is 29 in this invention. Transform of fit is an embedding extraction method of TabPreprocessor, and the input data is X train , refers to the table data in the present invention. x refers to the extracted data embedding, where x = ba, pe, sy, tu, co, ca.

[0083] S3.2, ResNet101 is a deep residual network (Kaiming He et al.), which is one of the ResNet series. There are 101 layers in the ResNet101 network structure diagram, of which the first layer is a 7×7 convolution layer, followed by 4 stages, each stage containing several residual blocks. Finally, there is a layer of global average pooling and a layer of fully connected layer. Each residual block consists of two 3×3 convolution layers, each of which is followed by a batch normalization and ReLU activation function. There are also batch normalization and ReLU activation functions between residual blocks. The first residual block in each stage uses a 1×1 convolution layer to convert the number of input channels to the number of output channels for addition operations with subsequent residual blocks. The specific network of ResNet101 is as follows Figure 2 shown.

[0084] The main feature of the ResNet101 network structure is that it uses residual units to build a deep network, and solves the gradient vanishing problem through jump connections, making the network deeper and easier to train. The present invention uses the ResNet101 network as an image feature encoder to extract details and global information from ultrasound image data. The preprocessed unified 224×224 size image data is input into the ResNet101 model in batch size=64. When only the image corresponding embedding needs to be extracted, the last output layer of the model is removed, and the output of the fully connected layer is directly embedded as the image data into E. im .

[0085] S3.3. Currently, the most widely used model in the field of natural language processing is BERT, which uses a bidirectional Transformer encoder structure and is well suited for various natural language processing tasks and text information extraction. Unlike image encoders, which extract information from images through operations such as convolution and pooling, text encoders extract text information through the attention module in the Transformer encoder. The formula for the self-attention mechanism is shown in formula (8):

[0086]

[0087] Where Q refers to a series of concatenated queries, while K and V refer to the key and value obtained by the same operation, respectively. Q, K, and V are all obtained by multiplying the input vector with a specific matrix. Refers to the dimension of input queries and keys. The multi-head attention mechanism enables the model to learn spatial information representations at different locations. Its formula is shown in formula (9):

[0088]

[0089] in The present invention adopts the hyperparameter h=12, that is, 12 parallel attention layers are used, each of which adopts d k =d v =d model / h=64.

[0090] The present invention adopts the "bert-base" pre-trained model. By installing the pytorch version of the transformer library and the BERT model related modules, the corresponding pre-trained parameters of the model can be downloaded and loaded into the text feature encoder built by the present invention for text embedding extraction. Due to the high integration of the transformer library, after the steps of importing the word segmenter through the dictionary, importing the configuration file, modifying the configuration, and importing the model through the configuration and path, the corresponding text data embedding E can be directly obtained according to the input text. re .

[0091] S4. Calculating the Shapley values ​​of different data sources, including the following steps:

[0092] S4.1. Analyze the impact of different data source embeddings on the model prediction task. The present invention calculates the Shapley values ​​of different data sources to quantitatively obtain the contribution of different data sources to the prediction task. Let I = {1, 2, ..., n} be a set of n different embeddings, and its calculation formula is shown in formula (10):

[0093] (10)

[0095] Where S i is the set of all subsets of I that contain embedding i, |s| refers to the number of embeddings in set s, and ω(|s|) is the weighting factor s / {i}, which represents the new set of embeddings after removing embedding i from set s.

[0096] There are many combinations of embedding i. The present invention lists all the combinations containing embedding i, and then calculates the marginal contribution of embedding i in each combination in turn, and multiplies the marginal contribution by the probability of the combination, and adds up the result values ​​to get the Shapley value of embedding i. The larger the Shapley value of embedding i, the greater the contribution of the embedding to the model prediction in the current combination.

[0097] S5. Analyze the impact of each mode on model performance, including the following steps:

[0098] S5.1. From S3, we can see that there are three modalities: image, table and text. The image data is extracted by the ResNet101 model and embedded in E im , the data embedding extraction module is extracted from the TabNet model, and the common inspection indicator data embedding E om, drug code data embedded in E dr , laboratory measurement index value data embedded in E la , drug dosage and unit data embedding E in Different treatment methods are embedded in E pr And use drugs and frequency embedding E ph There are 8 data source embeddings in total. Therefore, for single-modal input, depending on the number of input embeddings, there are possible inputs; for bimodal input, since both image data and text data have only one embedding, there are a total of possible inputs; for trimodal inputs, there are According to different data source embedding combinations, they are input into the chest pathology predictor to obtain the predicted output of the pathology type, and the prediction accuracy of different combination models is analyzed.

[0099] Finally, it should be noted that the above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included in the scope of the present invention.

Claims

1. A chest pathology artificial intelligence aggregation prediction method based on multimodal data, characterized in that: include: Step 1: Convert raw medical data from different modalities into a format suitable for processing; Step 2: Establish three general machine learning model training; Step 3: Extract the patient's multimodal features and embed them into the model for chest pathology aggregation prediction; Step 4: Calculate the Shapley value of different data sources; Step 5: Analyze the impact of each mode on model performance.

2. The chest pathology artificial intelligence aggregation prediction method based on multimodal data according to claim 1 is characterized by: In step 1, the raw medical data from different modalities are converted into a format suitable for processing. The specific implementation steps are as follows: (1) The table data is de-identified according to the "HIPAA Act" and the patient identifier is used as a unique identifier for the patient; (2) The image data is uniformly reduced to 224×224 size, and the "resize()" function in the "opencv" library is used to implement image downsampling; (3) The text data comes from the patient examination text report. The "FINDINGS" part is extracted and uniformly processed into a fixed length n.

3. The chest pathology artificial intelligence aggregation prediction method based on multimodal data according to claim 1 is characterized by: In step 2, three general machine learning model trainings are established as follows: (1) Three machine learning architectures were established and explored, including Multi-Layer Perceptron, Random Forest, and XGBoost, to heuristically determine the best model selection for subsequent experiments.

4. The chest pathology artificial intelligence aggregation prediction method based on multimodal data according to claim 1 is characterized by: In step 3, the patient's multimodal features are extracted and embedded and input into the model for chest pathology aggregation prediction. The specific implementation steps are as follows: (1) TabNet network is used as the table feature encoder to fully mine the information in the preprocessed table data. The data embedding extraction module in the TabNet model is used to obtain the common inspection indicator data embedding E om , drug code data embedded in E dr , laboratory measurement index value data embedded in E la , drug dosage and unit data embedding E in 、Different treatment methods are embedded in E pr And use drugs and frequency embedded E ph ; (2) ResNet101 network is used as the image feature encoder to extract the details and global information in the ultrasound image data. The pre-processed uniform size image data is input into the model, and the output of the fully connected layer is used as the image data to embed E im ; (3) Using the "bert-base" pre-training model, build a text feature encoder for text embedding extraction, and get the corresponding text data embedding E according to the input text re。 5. The chest pathology artificial intelligence aggregation prediction method based on multimodal data according to claim 1 is characterized by: In step 4, the Shapley values ​​of different data sources are calculated as follows: (1) Analyze the impact of different data source embeddings on the model prediction task and calculate the Shapley value of different data sources to quantitatively derive the contribution of different data sources to the prediction task. The larger the Shapley value, the greater the contribution of the embedding to the model prediction in the current combination. (2) In order to understand the contribution of each data source and modality to the final performance, the Shapley value of each of the 8 data sources and 3 modalities was calculated separately.

6. The chest pathology artificial intelligence aggregation prediction method based on multimodal data according to claim 1 is characterized by: In step 5, the impact of each mode on model performance is analyzed as follows: (1) Analyze the impact of different modal combination inputs on the model prediction task. For single-modal input, dual-modal input, and tri-modal input, they are respectively input into the tumor predictor according to different data source embedding combinations to obtain the predicted output of chest pathology type, and analyze the prediction accuracy of different combination models.