Medical missing data completion method based on graph attention mechanism and language large model

Abstract

The application provides a medical missing data completion method based on a graph attention mechanism and a language large model, and has the characteristics that the method comprises data set preprocessing, training of a graph attention network, extraction of an attention matrix as the weight of each edge in the graph, screening of strongly correlated nodes, traversal of the whole data set, judgment of the feature to which a missing value belongs when the missing value is found, input of the strongly correlated feature name and feature value of the missing value as text prompt information into a pre-trained BERT language large model, obtaining of a model output, weighted average of the output, obtaining of a final interpolation result, and completion of missing value interpolation. Compared with the prior art, the application can more quickly and accurately realize the function of missing data completion, can fully mine the correlation degree between features through a graph attention network, can better model the hidden layer feature space based on the strong fitting capability of a BERT model, can obtain more accurate prediction results, is simple in method, has good use effect, and has good application prospect and commercial value.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an imputation method for missing diagnostic data of chronic diseases in the elderly based on graph attention mechanism and language large model. Background Technology

[0002] Chronic diseases in the elderly are a major concern due to their prevalence. Studies have shown that timely follow-up visits and data collection, along with appropriate data mining and analysis, can reveal hidden information, helping doctors make effective clinical warnings for chronic diseases in the elderly and thus reducing their mortality rate. However, due to their long disease cycle and the accumulation of follow-up visits, some diagnostic data (such as physiological indicators and medication use) are easily lost. Therefore, how to fill in the missing values ​​in diagnostic data is an important research direction in the medical field.

[0003] To address these challenges, several statistically based methods have been proposed, such as using statistical features like the mean, median, and mode for imputation of missing values. However, these imputation methods are overly simplistic and do not consider the overall characteristics of patients, often resulting in the same missing feature being replaced by the same value across different patients. This severely disrupts the data distribution and hinders doctors from making accurate clinical predictions.

[0004] In addition, some traditional machine learning methods have been used for missing data imputation, such as linear interpolation and multiple iterative interpolation, clustering-based interpolation, and random forest-based interpolation. These methods have narrowed the distribution gap between the interpolated data and the real data, allowing the interpolated data to better fit the real data. However, due to the poor modeling ability of traditional machine learning algorithms in the latent space of features, there are still considerable discrepancies between the interpolated data and the real data.

[0005] With the rise of deep learning, neural network-based data interpolation methods have emerged. Some researchers have used Generative Adversarial Networks (GANs) to generate missing data, noting that the main challenge they face is the class attribute of the data. GAN-based missing data generation schemes require reinforcement learning algorithms to learn a bidirectional mapping between class labels and the true latent feature space. However, due to the extreme instability of GANs during training, the loss function rarely converges to Nash equilibrium, hindering the large-scale generalization of this method. Other researchers have used Graph Neural Networks (GNNs) to model different features as nodes in a graph, predicting missing values ​​in each node through forward propagation. However, because the edges between nodes are unweighted, each participating node contributes equally to the final prediction result. In reality, different nodes should contribute differently based on their relevance to the predicted feature. Therefore, ordinary GNNs do not achieve good interpolation results. In addition, some researchers have used Long Short Time Memory Networks (LSTM) to predict missing values ​​for time-dependent data, but in reality, many features are still unrelated to time information.

[0006] In summary, existing research algorithms all have various shortcomings and lack representativeness and universality. Furthermore, the accuracy of interpolation methods for missing diagnostic data in elderly chronic diseases still needs improvement. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by providing a medical missing data completion method based on graph attention mechanisms and a large-scale language model. This method trains an undirected fully graph attention network using training samples, extracts the attention matrix from the graph attention network as the weights of each edge in the graph, and selects strongly correlated nodes according to their weights. It then traverses the entire dataset. When a missing value is found, its associated feature is first determined, and then the names and values ​​of its strongly correlated features are input as text prompts into a pre-trained BERT large-scale language model. After obtaining the model output, a weighted average is applied to the output to obtain the final interpolation result, thus completing the missing value imputation. The method is simple and can infer missing data from some non-missing data in electronic medical records, thereby completing diagnostic information and achieving fast and accurate missing data completion. The graph attention network can fully explore the correlation between features, and the powerful fitting ability of the BERT model helps to better model the hidden feature space and obtain more accurate prediction results. The method is simple, effective, and has good application prospects and commercial value.

[0008] The objective of this invention is achieved as follows: a method for completing missing medical data based on graph attention mechanism and large language model, characterized by employing the following three technical solutions for completing missing medical data:

[0009] Firstly, in order to establish an effective interpolation strategy, data preprocessing of the entire electronic medical record is required, which mainly includes:

[0010] 1-1: Electronic medical records containing missing data are filtered based on whether there are any missing data in each patient's information. Patient information without missing data is grouped together, and patient information with missing data is grouped together. Subsequently, the data without missing data is used for training and testing of the graph neural network, while the data with missing data is imputed using the BERT language model.

[0011] 1-2: Divide the dataset into training, validation, and test sets for data that does not have missing data.

[0012] 1-3: Perform batch standardization on features in data without missing features, initialize the adjacency matrix for batch standardization, and encode the labels using one-hot encoding.

[0013] 1-4: Arrange the missing data in ascending order of the number of missing values ​​in each patient's information to facilitate faster imputation during inference.

[0014] Secondly, regarding the question of which data to use to infer the currently missing data, this invention reduces it to the problem of mining the correlation between different features in the diagnostic data. To measure the correlation between different features, this invention employs a graph attention network-based method to model the implicit function space between features and mine the hidden relationships between features. This method specifically includes:

[0015] 2-1: The network input consists of a feature matrix and an adjacency matrix. Each row of the feature matrix represents the information of a node in the graph, while the adjacency matrix represents the connectivity between nodes. Both are in PyTorch tensor format.

[0016] 2-2: The feature matrix consists of patient information without missing values, with rows representing features and columns representing patient IDs. The adjacency matrix is ​​a square matrix with all elements equal to 1, and its dimension is the number of features × the number of features, representing a complete graph where all nodes are connected by edges.

[0017] 2-3: This invention uses a graph attention neural network that includes a multi-head attention mechanism. The calculation results of multiple attention heads jointly determine the final prediction value. Each node updates its own features based on the features of its neighboring nodes by weighted summation. Introducing the attention mechanism is equivalent to assigning different weights to each edge. The actual meaning of the edge weight is the degree of correlation between nodes. The larger the weight, the closer the correlation.

[0018] 2-4: After the graph attention neural network training converges, the attention matrix of each attention head is extracted and then fused by weighted averaging to obtain the final attention matrix. The correlation between nodes is then obtained through this matrix.

[0019] 2-5: Sort each row in the attention matrix from largest to smallest, obtain the top k index values ​​after sorting, which is equivalent to knowing the k other features that are most relevant to the current row (current feature), and save the results in the form of a dictionary to a .json file. The keys of the dictionary are the indices of the current feature, and the values ​​of the dictionary are a list of indices of the k other features.

[0020] Thirdly, to further predict the current missing features based on relevant features, this invention utilizes the pre-trained BERT large language model, which has strong generalization ability and can be directly used for zero-shot learning, to create a method for predicting missing data in the diagnosis of chronic diseases in the elderly. Specifically, this includes:

[0021] 3-1: Establish a pipeline that can be used for loading and inputting / outputting data for pre-trained BERT language models, and determine the format of text prompt statements.

[0022] 3-2: Traverse the electronic medical records with missing data. If a missing value is found, first find its corresponding feature, and then obtain the index value of its corresponding feature.

[0023] 3-3: After obtaining the index value, read the relevant feature dictionary from the .json file and find the relevant features of the feature to which the missing value belongs based on the index value. Read the relevant features and their values ​​for the first m features that are not missing from the electronic medical record of the same patient.

[0024] 3-4: Based on the obtained relevant features and their feature values, fill in the text prompts according to the previously agreed-upon format, and finally add the features to which the missing values ​​belong, leaving a [MASK] mark as a clue for network prediction.

[0025] 3-5: After obtaining the network prediction values, sort them according to their confidence levels, take the top n prediction values ​​with higher confidence levels and their respective confidence levels, apply softmax processing to the above confidence levels and use them as weights to perform weighted averaging on the prediction values ​​to obtain the final interpolation result.

[0026] 3-6: Fill the original missing value positions with the final imputation results, iterate through the entire electronic medical record according to the above steps, and save the imputed content as a .csv file.

[0027] Compared with existing technologies, this invention has the advantages of simple method, inference of missing data from partial non-missing data in electronic medical records, and completion of diagnostic information, achieving fast and accurate missing data completion. Through graph attention network, the correlation between features can be fully explored. Based on the powerful fitting ability of BERT model, it helps to better model the hidden layer feature space and obtain more accurate prediction results. The method is simple, has good effect, and has good application prospects and commercial value. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the process of the present invention;

[0029] Figure 2 Here is a flowchart of the data preprocessing process;

[0030] Figure 3 This is a block diagram of an undirected complete graph neural network structure.

[0031] Figure 4 Extract and save the flowchart for the attention matrix;

[0032] Figure 5 Use a flowchart for the attention matrix;

[0033] Figure 6 A flowchart illustrating the input / output pipeline for the BERT model and the process for obtaining interpolation results. Detailed Implementation

[0034] The present invention will be further described in detail through the following specific implementation examples.

[0035] Example 1

[0036] See Figure 1 The imputation of missing diagnostic data for chronic diseases in the elderly according to a preferred embodiment of the present invention specifically includes the following steps:

[0037] Step 1: Collect complete patient data and missing patient data from the raw data. The source of the raw data is the patient's electronic medical record. The processing procedure is as follows:

[0038] Step 1.1: Obtain the original electronic medical records and perform data segmentation and preprocessing on the raw data.

[0039] Step 1.2: Build a graph attention network, and train and test the network based on the training and test sets obtained from preprocessing;

[0040] Step 1.3: Obtain the attention matrix after training, perform a weighted average, and save the result;

[0041] Step 1.4: Read the attention matrix and save it to a file. After restoring the matrix, record the k most relevant features for each feature according to the specified k value.

[0042] Step 1.5: Establish the input and output pipeline of the BERT language model and set a fixed prompt statement format;

[0043] Step 1.6: Traverse the data with missing values. When a missing value is encountered, fill the prompt statement with the most relevant features and their feature values ​​to guide the BERT model to give the correct prediction value. The final imputation result is obtained after weighted averaging.

[0044] Step 2: Divide the complete patient data into two parts: features and labels. Remove features that are irrelevant to label prediction and save it as a PyTorch tensor format, where rows represent each patient and columns represent various physiological indicators and diagnostic information in their electronic medical records.

[0045] See Figure 2 The data preprocessing process specifically includes:

[0046] Step 2.1: Obtain the original electronic medical records and divide the original data to distinguish between complete data and data with missing data;

[0047] Step 2.2: The main body of the electronic medical record is in tabular form. Its rows represent the patient's ID, and the columns represent the patient's characteristics, including physiological signs and medication. The reasons for missing diagnostic data include the failure to collect records in a timely manner during long follow-up visits, resulting in accidental loss, or forgetting to collect some characteristics, etc.

[0048] Step 2.3: Perform data preprocessing on the complete data, divide it into training and test sets, and standardize the feature matrix and adjacency matrix. The feature matrix is ​​the transpose of the matrix composed of the complete electronic medical record data, that is, its rows represent patient features and its columns represent patient IDs; the adjacency matrix is ​​obtained from an undirected complete graph in which all nodes are interconnected; in order to facilitate the calculation of backpropagation loss, the labels need to be one-hot encoded.

[0049] Step 3: Using each feature as a node, construct an undirected complete graph neural network, introduce an attention mechanism, and use the attention matrix as the weight of each edge to measure the feature correlation between adjacent nodes; the input of the attention matrix is ​​that the rows represent features and the columns represent patients, and the output information is the confidence of the feature type, with the feature type as the label.

[0050] See Figure 3 The undirected complete graph neural network structure specifically includes:

[0051] Step 3.1: The input to the undirected complete graph neural network consists of three parts: the feature matrix, the adjacency matrix, and the one-hot label;

[0052] Step 3.2: Each node represents a feature. The feature matrix is ​​initialized with all 1s, which means that each node has an edge with every other node, that is, the graph is a complete graph;

[0053] Step 3.3: Employ a multi-head attention mechanism, setting the number of attention heads to 8 and the number of hidden layer units to 8;

[0054] Step 3.4: Use the CE loss function without softmax as the loss function, as shown in equation (c) below:

[0055]

[0056] Among them, y k Indicates the predicted value. Represents the actual value.

[0057] Step 3.5: The initial training learning rate is 0.005, the learning rate decay coefficient is 0.0005, the optimizer is Adam, and the number of training generations is 100.

[0058] Step 4: Divide the complete patient data into training, validation and test sets, train and test the graph attention neural network model, extract the attention matrix after training and save it as a .npy file.

[0059] See Figures 4-5 The processing procedure for the attention matrix is ​​as follows:

[0060] Step 4.1: After training is complete, extract an attention matrix from each multi-head attention mechanism;

[0061] Step 4.2: Based on the assumption that different attention heads focus on different information, the above 8 attention matrices are weighted and averaged.

[0062] Step 4.3: Use the np.save() function to save the weighted average attention matrix as a .npy file for later processing.

[0063] Step 5: Load each line of information in the .npy file, sort the feature relevance in each line from largest to smallest, retain the k values ​​with the largest feature relevance, find their corresponding features, and save them to the .json file. The specific steps are as follows:

[0064] Step 5.1: Load the attention matrix using the np.load() function;

[0065] Step 5.2: Initialize a dictionary data, traverse the attention matrix row by row, use the index value of each row as the key, use the np.argsort() function to sort the feature correlation coefficients of each row from largest to smallest and return their index values, save the first m index values ​​as a list and use them as values, and combine them with the above keys to form key-value pairs, and store them together in the dictionary;

[0066] Step 5.3: Create and open a .json file, and save the above dictionary into the .json file.

[0067] Step 6: Call the pre-trained BERT model and set up the pipeline and prompt statement format for data input and output.

[0068] Step 7: Begin iterating through the patient data with missing values. When a missing value is found, determine its corresponding feature and obtain the k most relevant features from the .json file. Then, read the corresponding non-missing feature values ​​from the electronic medical record of the same patient. Use the feature + feature value to form a text prompt statement and input it into the BERT model. Take a weighted average of the obtained model prediction results to obtain the final imputation result.

[0069] See Figure 6 The process of predicting missing values ​​using the BERT language large model is as follows:

[0070] Step 6.1: Download the Clinical_BERT model pre-trained on the MIMICⅢ electronic medical record dataset and import the model;

[0071] Step 6.2: Set the prompt statement format, complete the BERT model input / output pipeline, and load the dictionary in the .json file;

[0072] Step 6.3: Traverse the data with missing values. When a missing value is encountered, first find its corresponding feature. After obtaining the corresponding feature, query the dictionary for the top m features with the highest relevance.

[0073] Step 6.4: Fill in the above m feature names and their corresponding feature values ​​into the corresponding positions of the prompt statement in sequence, and fill in the feature name of the feature and the feature value represented by [MASK];

[0074] Step 6.5: Encode the prompt statement with a token and input the encoded token into the BERT model;

[0075] Step 6.6: Obtain the BERT model prediction results, and select the top n predictions with the highest confidence and their confidence scores;

[0076] Step 6.7: Calculate the softmax value corresponding to the confidence level above as the weight of the predicted value, and use the weighted average predicted value as the final imputation result;

[0077] Step 6.8: Insert the output results into the positions of the missing values ​​and continue the traversal process;

[0078] Step 6.9: Once all missing data has been traversed, the imputation process ends. At this point, the missing data has become complete data.

[0079] Step 6.10: Reassemble the complete data and the imputed missing data to complete the missing value imputation work of the electronic medical record.

[0080] Implementing all or part of the processes in the embodiments of this invention is accomplished by scheduling relevant hardware through computer software programming. The computer software program can be stored as an entity in a computer-readable storage medium, which can be any physical device capable of carrying computer software programs or code, including but not limited to: USB flash drives, portable hard drives, disks, optical discs, etc. This invention can effectively solve the problem of completing missing data in the diagnosis of chronic diseases in the elderly. It can use existing data in electronic medical records to predict missing data, thereby supplementing the electronic medical records and providing higher-quality data for future early warning of chronic diseases in the elderly.

[0081] The above is merely a further description of the present invention and is not intended to limit the scope of this patent. Any equivalent implementation of the present invention should be included within the scope of the claims of this patent.

Claims

1. A method for completing missing medical data based on graph attention mechanism and large language model, characterized in that, The completion method specifically includes the following steps: Step 1: Collect complete patient data and missing patient data from the raw data. The source of the raw data is the patient's electronic medical record. Step 2: Divide the complete patient data into two parts: features and labels. Remove features that are irrelevant to label prediction and save it as a PyTorch tensor format, where rows represent each patient and columns represent various physiological indicators and diagnostic information in their electronic medical records. Step 3: Using each feature as a node, construct an undirected complete graph neural network, introduce an attention mechanism, and use the attention matrix as the weight of each edge to measure the feature correlation between adjacent nodes; the input of the attention matrix is ​​that the rows represent features and the columns represent patients, and the output information is the confidence of the feature category, with the feature category as the label. Step 4: Divide the complete patient data into training, validation and test sets, train and test the graph attention neural network, extract the attention matrix after training and save it as a .npy file; Step 5: Load each line of information in the .npy file, sort the feature relevance in each line from largest to smallest, keep the k values ​​with the largest feature relevance, find their corresponding features, and save them to the .json file; Step 6: Call the pre-trained BERT model and set up the pipeline and prompt statement format for data input and output; Step 7: Start traversing the patient data with missing values. When a missing value is found, determine its corresponding feature and obtain the k features most relevant to it from the .json file. Then, read the corresponding non-missing feature values ​​from the electronic medical record of the same patient. Use the feature + feature value to form a text prompt statement and input it into the BERT model. The obtained model prediction results are weighted and averaged to obtain the final imputation result. The attention matrix obtains the k features most relevant to the current predicted value. After removing features with missing values, a text prompt is formed based on the above information and input into the pre-trained BERT for prediction. After obtaining the prediction results, only the top n prediction results with high confidence are retained, and prediction results that are not in numerical form are removed. Finally, a weighted average is performed based on the confidence of the predicted value to obtain the final imputed value as shown in equation (e) below: （e）； Where pred is the final interpolated value; β n The confidence level of the nth predicted value; pred n This represents the nth predicted value.

2. The medical missing data completion method based on graph attention mechanism and language large model according to claim 1, characterized in that, The graph neural network structure constructed in step 3 based on the computational feature correlation of the graph attention network is an undirected complete graph. The nodes represent various features. A multi-head attention mechanism is introduced, and the attention matrix obtained after weighted averaging is used as the weight of the undirected edges. The node values ​​are updated using the following equation (a): （a）； Among them, h ’ i Represents the updated features of the i-th node; α ij The attention coefficient between the i-th node and the j-th node is represented by W; the adjacency matrix is ​​represented by h. j Let represent the feature of the j-th node; σ is the Leaky-ReLU activation function expressed in equation (b) below: （b）； Where x is the input feature; α is the attenuation coefficient, and its value range is α∈(0,1).

3. The medical missing data completion method based on graph attention mechanism and language large model according to claim 1, characterized in that, In step 4, the attention neural network is trained using the following equation (c) without softmax processing: (Equation c) The cross-entropy loss function is used for backpropagation and gradient update. （c）； Among them, y k Indicates the predicted value; ÿ k Represents the actual value; After training, the attention matrix of each attention head is extracted and weighted averaged according to the following formula (d): （d）； Where, α ij w is the weighted average attention value of the i-th node and the j-th node; k The attention matrix weights of the k-th attention head; α kij Let be the attention values ​​of the i-th and j-th nodes in the k-th attention matrix.

Images