Universal disease prediction method and system based on BERT framework

Abstract

The invention relates to the technical field of disease prediction, and particularly discloses a universal disease prediction method and system based on a BERT framework, and the method comprises the following steps: S1, obtaining multiple diagnosis data of a patient, the diagnosis data comprising age during treatment, diagnosis date, diagnosis code, abnormal laboratory detection result code and operation information; s2, generating an embedded representation of the sequence according to the diagnosis data; the position in the embedded representation of the sequence is embedded with age representation during treatment; and S3, inputting the embedded representation of the sequence into a pre-training model based on a BERT framework to obtain prediction results of various disease risks. According to the method, a pre-training model based on a BERT framework is designed and is specially used for processing structured electronic health record data, discontinuous age during treatment is used as a position for embedding, complex features and relations in EHR data can be captured, and the performance of a disease prediction task is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of disease prediction technology, and in particular to a universal disease prediction method and system based on the BERT framework. Background Art

[0002] Disease prediction is a crucial task in healthcare. Its goal is to predict a patient's next diagnosis based on their historical medical records. Disease prediction not only provides greater convenience for patients and doctors but also brings significant value to the healthcare industry. In recent years, many traditional neural network models, such as recurrent neural networks and graph neural networks, have been applied to disease prediction. However, the following challenges exist: 1. Feature engineering is complex and relies on domain knowledge: Traditional machine learning methods (such as logistic regression and random forests) typically require extensive feature engineering. This requires domain experts to manually select and design features based on medical knowledge, which is not only time-consuming and labor-intensive, but also relies on the experts' experience and knowledge.

[0003] 2. Difficulty capturing long-term dependencies: Traditional machine learning methods often struggle to capture dependencies in long-term data. For example, when processing a patient’s multiple medical records, traditional methods may not be able to effectively leverage the impact of earlier medical records on their current health status.

[0004] 3. High data requirements: Traditional machine learning methods typically require large amounts of labeled data to train models to avoid overfitting. In the medical field, obtaining labeled data is often very expensive and time-consuming.

[0005] 4. Limited model interpretability: Although some traditional machine learning methods (such as logistic regression and decision trees) have certain interpretability, they are usually difficult to explain complex feature interactions and model decision processes.

[0006] 5. Difficulty handling missing data: Existing techniques often use prescription medications obtained during medical consultations as part of the input data. However, this approach fails to fully account for the issue of missing information in real-world applications. For example, many patients with chronic conditions do not frequently visit hospitals to fill prescriptions for their medications, instead choosing to purchase common medications at pharmacies. In such cases, medication information may be missing. Directly incorporating this missing information into the training process could negatively impact the model's predictive performance.

[0007] 6. Continuous values ​​as position embeddings are not suitable for disease prediction: In medical scenarios, the time intervals between patients' visits may be very uneven. Using continuous values ​​as position embeddings (such as simple serial numbers) cannot accurately reflect this uneven time interval. The model may mistakenly interpret the continuous values ​​as indicating that the time intervals between two adjacent visits are the same, causing the model to ignore important time-dependent information when making predictions. In disease prediction, a patient's health status may be affected by multiple past visits. When using continuous values ​​as position embeddings, the model may find it difficult to capture this long-term dependency. In actual medical scenarios, patients' visit patterns are often irregular. Some patients may visit the doctor regularly, while others may only visit when their condition is serious. When using continuous values ​​as position embeddings, the model has difficulty adapting to this irregular visit pattern. Summary of the Invention

[0008] This invention aims to address the aforementioned issues. To this end, it provides a universal disease prediction method and system based on the BERT framework. A pre-trained model based on the BERT framework is designed specifically for processing structured electronic health record (EMR) data. The model uses non-continuous age at visit (calculated in months) as position embeddings and incorporates EMR surgical information for training. This method can capture the complex features and relationships in EHR data, significantly improving the performance of disease prediction tasks.

[0009] The present invention provides a universal disease prediction method based on the BERT framework, which adopts the following technical solution: S1: Obtain multiple diagnostic data of the patient, including age at diagnosis, diagnosis date, diagnosis code, abnormal laboratory test result code, and surgical information; age at diagnosis is calculated in months; S2: Generate an embedding representation of the sequence based on the diagnosis data; concatenate the tag embedding, visit embedding, and position embedding to obtain the embedding representation of the sequence; the tag embedding is a low-dimensional representation of the diagnosis code, abnormal laboratory test result code, and surgical information; the visit embedding is represented by the visit ID, which is generated by sorting the diagnosis data in chronological order according to the diagnosis date; and the position embedding is represented by the age at the time of the visit; S3: Input the sequence embedding representation into the pre-trained model based on the BERT framework to obtain prediction results for multiple disease risks.

[0010] Furthermore, the multiple diagnostic data of the patient are the diagnostic data of the patient in the past three years, and the number of diagnostic data is greater than or equal to 4.

[0011] Furthermore, the generation process of token embedding is: The diagnostic codes in the diagnostic data are sorted according to their priorities, and then the abnormal laboratory test result codes and surgical information are spliced ​​in sequence; using a dictionary, the sorted diagnostic codes, abnormal laboratory test result codes and surgical information are converted into tags, and then data filling is performed to obtain tag embedding.

[0012] Furthermore, the pre-training model based on the BERT framework includes 12 Transformer layers, and each Transformer layer includes a 12-head multi-head self-attention mechanism and a feedforward neural network.

[0013] Furthermore, the training of the pre-trained model based on the BERT framework includes: a pre-training stage and a fine-tuning training stage.

[0014] Furthermore, the prediction results of multiple disease risks are the risk probabilities of the diseases within the next 12 months.

[0015] Furthermore, the prediction results of multiple disease risks are the prediction results of 234 diseases.

[0016] The present invention also provides a universal disease prediction system based on the BERT framework, which adopts the following technical solutions: including: a diagnostic data acquisition module, a diagnostic data processing module and a pre-training model based on the BERT framework, The diagnostic data acquisition module is used to acquire multiple diagnostic data of the patient, wherein the diagnostic data includes the age at the time of consultation, diagnosis date, diagnosis code, abnormal laboratory test result code and operation information; The diagnostic data processing module is used to generate an embedded representation of a sequence based on the diagnostic data; the position embedding in the embedded representation of the sequence uses the age representation at the time of the visit; The BERT framework-based pre-training model is used to calculate the prediction results of various disease risks based on the embedded representation of the sequence.

[0017] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. This paper uses a pre-trained model based on the BERT framework, uses non-continuous age at the time of consultation as position embedding, and considers surgical information in the diagnostic data. It has generalization capabilities and has achieved good prediction results for 234 disease risks, with an accuracy rate significantly better than existing methods.

[0018] 2. This invention uses non-continuous age at visit as a positional embedding, which can reflect the actual interval between visits and helps the model capture long-term dependencies, enabling the model to more accurately predict patients' health status and disease progression. Using non-continuous age at visit as a positional embedding is particularly adaptable to irregular visit patterns, allowing the model to understand the timing of each visit through age at visit, thereby more accurately modeling changes in patients' health status.

[0019] 3. The diagnostic data of this invention includes surgical information, which can more comprehensively reflect the patient's health status, enhance the model's generalization ability, and promote multi-task learning. The diagnostic data of this invention does not use medication information, which can prevent medication information containing missing data from adversely affecting the model's prediction results.

[0020] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 It is a flow chart of the method provided by the present invention.

[0023] Figure 2 It is a flowchart of the pre-training stage provided by the present invention.

[0024] Figure 3 It is a flow chart of the fine-tuning training phase provided by the present invention.

[0025] Figure 4 It is a structural block diagram of the system provided by the present invention.

[0026] Reference numerals: 1. Diagnostic data acquisition module; 2. Diagnostic data processing module; 3. Pre-training model based on the BERT framework. DETAILED DESCRIPTION

[0027] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0028] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0029] The following combination Figures 1 to 4 The present invention is further described in detail, describing a universal disease prediction method and system based on the BERT framework of the present invention: In this embodiment, Figure 1 As shown, a general disease prediction method based on the BERT framework is provided, including the following steps: S1: Acquire multiple diagnostic data of the patient, wherein the diagnostic data includes five types of data, specifically: age at the time of consultation, diagnosis date, diagnosis code, abnormal laboratory test result code, and surgery information.

[0030] The age at the time of consultation is calculated in months, the diagnosis code is in ICD-10 format, the abnormal laboratory test result code is in Loinc format, and the surgical information is in ICD-10PCS format.

[0031] In this embodiment, the multiple diagnostic data of the patient are the diagnostic data of the patient in the past three years, and the number of diagnostic data is greater than or equal to 4.

[0032] This embodiment adds surgical information to the commonly used diagnosis codes and abnormal laboratory test result codes. The reasons are as follows: (1) Surgical information can more comprehensively reflect the patient’s health status The Importance of Surgical Information: Surgery is a critical event in a patient's care, reflecting the severity of their disease and the treatment approach. For example, for heart disease patients, information such as coronary artery bypass graft (CABG) surgery and heart valve replacement surgery can provide detailed information about the specific type and severity of the patient's heart disease. This information is crucial for predicting postoperative complications and long-term outcomes.

[0033] Enriched semantic information: Surgical information can enrich the model's understanding of a patient's health status. The original Med-BERT model was trained primarily on diagnostic information, but the addition of surgical information allows the model to learn more semantic information related to treatment. For example, for a patient with a fracture, surgical information (such as internal fixation) can provide additional information about the fracture type, treatment method, and possible recovery process, helping the model more accurately predict the patient's recovery.

[0034] Improving the accuracy of disease prediction: Surgical information can serve as an important marker of disease progression. For example, in cancer treatment, the extent and method of surgical resection (e.g., radical surgery vs. palliative surgery) directly impacts patient survival and recurrence risk. Incorporating surgical information into training allows the model to better capture the trajectory of disease progression, thereby improving the accuracy of disease prediction tasks such as predicting cancer recurrence and postoperative infection.

[0035] (2) Generalization ability of surgical information enhancement model Covering More Clinical Scenarios: Surgical procedures occur across a wide range of diseases and treatment scenarios. By incorporating surgical information into training, the model can learn a wider range of clinical patterns, thus enhancing generalization across different disease prediction tasks. For example, when dealing with diseases requiring surgical treatment (such as cancer and cardiovascular disease), the model can better understand the complexity of these conditions, resulting in improved predictive performance on new, unseen clinical data.

[0036] Adaptability to diverse medical environments: While surgical records may differ across medical institutions, surgical information itself is relatively standardized. By incorporating surgical information into the model for training, the model can better adapt to data from diverse medical environments, improving its robustness in cross-institutional data applications.

[0037] (3) Surgical information promotes multi-task learning Correlating Surgery and Disease: Surgical information is closely linked to disease diagnosis. Incorporating surgical information into the model facilitates multi-task learning, such as simultaneously predicting disease onset and associated surgical needs. This is extremely valuable for clinical decision support systems, helping doctors plan potential treatment options while diagnosing a disease, thereby improving the efficiency and quality of healthcare services.

[0038] Supporting postoperative management tasks: The inclusion of surgical information can also support tasks related to postoperative management, such as predicting postoperative complications and developing rehabilitation plans. This enables the model to not only predict the onset and progression of the disease, but also provide support for the management and care of postoperative patients, thereby playing a greater role in the entire healthcare process.

[0039] This embodiment does not use the drug information in the diagnostic data, mainly because for some long-term diseases, patients generally choose to go to the pharmacy to get medicine, and the hospital's diagnostic data does not have corresponding drug information, which will lead to inaccurate prediction results.

[0040] S2: Generate an embedding representation of the sequence based on the diagnosis data.

[0041] In this embodiment, the embedding representation of the sequence includes label embedding, position embedding, and visit embedding.

[0042] Token Embedding: Low-dimensional representation of diagnosis codes, abnormal laboratory test result codes, and surgical information.

[0043] Position Embedding: Uses the age at the time of visit to accurately reflect the time interval between visit records.

[0044] Visit Embedding: Uses the visit ID to represent the location embedding used to distinguish each visit.

[0045] According to the diagnosis date, the patient's diagnosis data is sorted in chronological order and a visit ID is generated from 1 to N.

[0046] Sort the diagnosis codes in each diagnosis data according to their priority. If the diagnosis data contains abnormal laboratory test result codes, they will be added after the diagnosis code. If the diagnosis data contains surgical information, they will be added after the abnormal laboratory test result codes.

[0047] One or two of the diagnosis code, abnormal laboratory test result code, and surgical information may be missing. For missing data, simply leave them blank. That is, concatenate the diagnosis code, abnormal laboratory test result code, and surgical information in the order listed above.

[0048] Using the dictionary, the sorted diagnosis codes, abnormal laboratory test result codes and surgical information are converted into tags, and then data filling is performed to obtain tag embedding.

[0049] In this embodiment, the data filling process is as follows: the length of the tag is set to 512, if the length is insufficient, it is filled with 0, and if the length exceeds 512, it is cut.

[0050] The tag embedding, visit embedding, and position embedding are concatenated to obtain the embedding representation of the sequence.

[0051] This example uses the non-continuous monthly age at the time of consultation as the location embedding for the following reasons: (1) Age at the time of consultation can more accurately reflect the time interval Reflecting the actual intervals between visits: Patients' visits are often discontinuous, and the intervals between visits can be long. Using discontinuous age at visit as the position embedding can more accurately reflect these intervals. For example, if a patient visited the doctor twice at the age of 20 and 23, using age at visit as the position embedding can clearly indicate that there was a three-year gap between the two visits.

[0052] Avoiding misleading time intervals: Compared to continuous serial numbers, non-continuous age at visit can avoid misleading results due to uneven time intervals. For example, if two visits are separated by several years, continuous serial numbers may mislead the model into thinking that the two visits are consecutive, while using age at visit can clearly indicate the actual time interval.

[0053] (2) Age at presentation helps capture long-term dependencies Long-term health status changes: A patient's health status may be affected by multiple past visits, even if there are long intervals between these visits. Using discrete visit-time ages as positional embeddings can help the model better capture such long-term dependencies. For example, certain health problems a patient experiences in their youth may affect their health status in middle age.

[0054] Improved prediction accuracy: By better capturing long-term dependencies, the model can more accurately predict a patient's health status and disease progression. This is particularly important for predicting chronic diseases (such as diabetes and cardiovascular disease), as the progression of these diseases often takes years or even decades.

[0055] (3) The age at the time of consultation is particularly suitable for irregular consultation patterns The prevalence of irregular visits: In real-world medical settings, patients' visit patterns are often irregular. Some patients may visit the doctor regularly, while others may only visit when their condition is serious.

[0056] Improve model flexibility: Using non-continuous age at visit as position embedding allows the model to better adapt to these irregular visit patterns. For example, for patients who visit the doctor irregularly, the model can use the age at visit to understand the timing of each visit, thereby more accurately modeling the patient's health status changes.

[0057] (4) Age at presentation enhances the explanatory power of the model Explicit representation of temporal information: Using non-continuous age at visit rather than non-continuous visit ID as location embedding allows for a more explicit representation of temporal information. During model interpretation, this explicit representation of temporal information can help clinicians better understand the model's decision-making process. For example, when a model predicts that a patient may develop a certain disease, clinicians can understand how the model considers the patient's health status at different time points by looking at the non-continuous age at visit, thereby increasing their confidence in the model's predictions.

[0058] Linking time points with health status: This positional embedding approach more intuitively links time points with patients' health status. Model visualization and interpretation tools clearly demonstrate changes in patients' health status at different time points and how these changes affect disease prediction. This is crucial for clinicians to understand and use the model in real-world applications.

[0059] S3: Input the sequence embedding representation into a pre-trained model based on the BERT framework (EMR-BERT model) to obtain prediction results for multiple disease risks.

[0060] The EMR-BERT model adopts BERT's bidirectional Transformer structure, which specifically includes 12 Transformer layers. Each Transformer layer includes a 12-head multi-head self-attention mechanism and a feedforward neural network.

[0061] The following takes the first Transformer layer as an example to illustrate its multi-head self-attention mechanism and feedforward neural network.

[0062] Multi-Head Self-Attention Mechanism: Self-attention mechanism: allows the model to dynamically pay attention to other tokens in the sequence as it processes each token, thereby capturing global dependencies.

[0063] Input: Embedding representation of the sequence.

[0064] Query, Key, Value: The embedded representation of the sequence generates query, key, and value respectively through three different linear transformations.

[0065] in, is the embedding representation of the sequence, is the query vector, is the key vector, is a value vector, is the query weight matrix, is the key weight matrix, is the value weight matrix, 、 and is a learnable weight matrix.

[0066] Attention score calculation: The attention score is obtained by calculating the dot product between the query and the key, and is normalized by scaling and softmax function.

[0067] in, is the attention function, is the softmax function, is the dimension of the key vector, used to scale the dot product to prevent the gradient from disappearing or exploding, and T is the transposed matrix.

[0068] Multi-head mechanism: The input is divided into multiple heads, each head performs self-attention calculation independently, and finally the results of these heads are spliced ​​together, which increases the model's expressive power and ability to learn different subspaces.

[0069] Number of Heads: The EMR-BERT model has 12 heads.

[0070] Multi-head self-attention calculation: in, is the multi-head attention function, For splicing operations, is the linear transformation weight matrix after splicing, is the attention score of the i-th head, is the query weight matrix of the i-th head, is the key weight matrix of the i-th head, is the value weight matrix of the i-th head.

[0071] Feed-Forward Neural Network: Each encoder layer contains a feedforward neural network that performs a nonlinear transformation on the label at each position to increase the fitting ability of the model. It contains two linear transformations with a ReLU activation function in the middle.

[0072] Input: Output of the multi-head self-attention mechanism.

[0073] The first layer of linear transformation: in, is the first linear transformation output, To find the maximum value, is the first weight, is the first bias. is the output after multi-head attention mechanism and residual connection.

[0074] The second layer of linear transformation: in, is the second linear transformation output, is the first weight, is the first bias.

[0075] Residual Connection and Layer Normalization Residual connections: Allow inputs to be passed directly to the next layer, thus avoiding the vanishing gradient problem. The output of each sub-layer (multi-head self-attention and feedforward neural network) is added to the input to form a residual connection.

[0076] Residual connections for multi-head self-attention: in, is the output after multi-head attention mechanism and residual connection.

[0077] Residual connections of feedforward neural networks: in, is the output after the feedforward neural network and residual connection, It is a feed-forward neural network.

[0078] Layer normalization: used to stabilize the training process and reduce internal covariate shift. After the residual connection, the output of each sub-layer is normalized.

[0079] in, is the result of layer normalization, is the mean of the input, is the variance of the input, is a small constant used to prevent division by zero. is the scaling parameter, is the offset parameter, During the pre-training phase, the model outputs a context embedding vector.

[0080] The EMR-BERT model used in this example is a pre-trained model based on the BERT (Bidirectional Encoder with Transformers) framework, specifically designed for processing structured electronic health record (EMR) data to improve the accuracy of disease prediction. The contextual embeddings learned during pre-training enable the EMR-BERT model to capture the complex features and relationships in EHR data, significantly improving performance in disease prediction tasks during fine-tuning. The following is an overview of the complete training process for the EMR-BERT model.

[0081] 1. Build a dataset 1.1. Data preprocessing: We selected 89,490 patients in Miami, USA, who had four or more visits to the hospital between 2019 and 2021.

[0082] 1.2. Data Cleaning: Extract diagnostic data from patient samples and ensure that the patient's diagnostic information is complete and chronologically correct. Diagnostic data includes age at presentation, diagnosis date, diagnosis code, abnormal laboratory test result code, and surgical information. Diagnostic codes are in ICD-10 format, abnormal laboratory test result codes are in Loinc format, and surgical information is in ICD-10 PCS format.

[0083] 1.3. Data Sorting: Sort each patient's diagnostic data in chronological order and generate visit IDs from 1 to N. Sort the diagnostic codes in each visit data according to their priority, and then splice in abnormal laboratory test results and surgical information.

[0084] 1.4. Tokenizer: Build a dictionary based on the sorted diagnostic data of all patients according to independent medical codes, so that each code corresponds one-to-one with a token in the dictionary.

[0085] 1.5. Data padding: The length of each sample is defined as 512 tokens. Samples with insufficient length are padded with 0, and samples with length exceeding 512 are cut.

[0086] 2. Pre-training phase Masked Language Model (MLM): Predicts masked diagnosis codes. Specifically, the code is replaced with the mask [MASK] with an 80% probability; replaced with a random code with a 10% probability; and left unchanged with a 10% probability.

[0087] The pre-training process is as follows Figure 2 As shown in the figure, the masked embeddings of the sequence are replaced and then fed into the model. After going through the Transformer layer, feedforward neural network, residual connection, and layer normalization, the prediction result is obtained. The loss is calculated and backpropagation is performed.

[0088] Optimization algorithm: Use Adam optimizer, learning rate 1e-5, and dropout rate 0.1.

[0089] 3. Fine-tuning training phase 3.1. Fine-tuning task: Achieve prediction of 234 obstetric-related diseases within the next 12 months, including perineal and vulvar injuries, prolonged pregnancy, cephalopelvic disproportion, obstruction, umbilical cord complications, diabetic complications, prostate cancer, etc.

[0090] 3.2. Fine-tuning process Data segmentation: From the above dataset, the diagnostic data with more than or equal to 4 visits during 2019 were divided into training set and test set with a ratio of 8:2, and the diagnostic data with more than or equal to 4 visits during 2020 were used as the validation set.

[0091] Model initialization: Load the parameters learned in the pre-training phase into the EMR-BERT model and update these parameters in the fine-tuning phase.

[0092] Adding a classification layer: A classification layer is added before the output layer in the pre-training framework of the EMR-BERT model. In this embodiment, the classification layer uses a feed-forward fully connected layer. Its function is to reduce the dimension of the word embedding vector to the same dimension as the disease type, which is 234 in this embodiment. The final output is the probability of each disease classification for disease prediction.

[0093] Optimization algorithm: Use Adam optimizer with a learning rate of 1e-5.

[0094] The process of fine-tuning training is as follows Figure 3 As shown in Figure 2, the predicted labels are 234 predictable diseases, and the input labels are the one-hot encoded vectors of the 234 diseases.

[0095] 4. Performance Evaluation This example uses two evaluation metrics: Loss (loss value), which is used to evaluate the degree of training convergence; and AUC (area under the curve), which is used to evaluate the predictive performance of the model.

[0096] 5. Experimental Results Training set: Loss (loss value) is 0.1310, AUC (area under the curve) is 0.9180.

[0097] Validation set: Loss (loss value) is 0.1380, AUC (area under the curve) is 0.9110.

[0098] The prediction results of the EMR-BERT model in this embodiment for 234 diseases are shown in Table 1. The threshold is a critical value for distinguishing the categories of prediction results, and the evaluation indicators are F1 score, accuracy, and area under the curve.

[0099] Table 1

[0100] This example also analyzes 38 diseases from the paper Med-BERT: Pretrained Contextualized Embeddings on Large-Scale Structured Electronic Health Records for Disease Prediction. Table 2 shows the comparison of the EMR-BERT model with the paper in terms of accuracy and area under the curve.

[0101] Table 2

[0102] This example also analyzes two key diseases: type 2 diabetes and heart failure. Table 3 shows the comparison of the area under the curve (AUC) and accuracy between the EMR-Bert model and eight existing methods. The existing methods selected include M-Bert, BEHRT, MedBert, CEHR-Bert, LR, XGB, and LSTM.

[0103] Table 3

[0104] This example predicts disease risk based on four medical records of a patient, as shown below: First visit data: Basic information: Age: 69 years and 0 months. Visit type: Outpatient. Abnormal laboratory test data: 1920-8: Aspartate aminotransferase; 2345-7: Glucose; 48642-3: Glomerular filtration rate; 718-7: Hemoglobin; 3094-0: BUN.

[0105] Second visit data: Basic information: Age: 69 years and 3 months. Visit type: Outpatient. Diagnosis: I50.32: Chronic diastolic heart failure; J44.1: Acute exacerbation of chronic obstructive pulmonary disease; Severe stage.

[0106] Third visit data: Basic information: Age: 70 years and 1 month. Visit type: Inpatient. Diagnosis: D64.89: Anemia; E66.01: Obesity; G47.33: Obstructive sleep apnea; E11.22: Chronic kidney disease; I25.10: Atherosclerotic heart disease. Abnormal laboratory test data: 1988-5: C-reactive protein (CRP); 2345-7: Glucose; 48642-3: Glomerular filtration rate; 2502-3: Iron saturation; 2951-2: Sodium.

[0107] Fourth Visit Data: Basic Information: Age: 70 years, 2 months. Visit Type: Inpatient. Diagnosis: E78.5: Hyperlipidemia; I13.0: Hypertensive Heart Disease and Chronic Kidney Disease with Heart Failure; I25.10: Atherosclerotic Heart Disease; I27.20: Pulmonary Hypertension; N18.9: Chronic Kidney Disease. Abnormal Laboratory Test Data: 48642-3: Glomerular Filtration Rate; 6598-7: Troponin; 6690-2: Leukocytes. Procedure: 4A023N6: Cardiac Sampling and Pressure Measurement; B2111ZZ: Coronary Angiography.

[0108] The four visits above correspond to visit IDs 1-4. All four visits include age at the time of visit, calculated in months. The first visit includes the abnormal laboratory test result code; the second visit includes the diagnosis code; the third visit includes the diagnosis code and the abnormal laboratory test result code; and the fourth visit includes the diagnosis code, the abnormal laboratory test result code, and surgical information.

[0109] Based on the data from the four medical visits above, this method is used to predict the risk of multiple diseases. The disease risk prediction results for the next 12 months are as follows (only diseases with a probability greater than 75% are listed in this example) Nephritis; Kidney disease; Nephrosclerosis: Risk level: high risk; Risk probability: 78%.

[0110] Cardiac arrest and ventricular fibrillation: Risk level: high risk; risk probability: 85%.

[0111] Hypertensive complications and secondary hypertension: Risk level: high risk; risk probability: 91%.

[0112] Uncomplicated diabetes: Risk level: high risk; risk probability: 97%.

[0113] Aortic and peripheral arterial embolism or thrombosis: Risk level: high risk; risk probability: 89%.

[0114] This embodiment also provides a universal disease prediction system based on the BERT framework, such as Figure 4 As shown, the technical solution adopted is as follows: including: a diagnostic data acquisition module 1, a diagnostic data processing module 2 and a pre-training model 3 based on the BERT framework.

[0115] The diagnostic data acquisition module is used to acquire multiple diagnostic data of the patient, wherein the diagnostic data includes the age at the time of consultation, diagnosis date, diagnosis code, abnormal laboratory test result code and operation information; The diagnostic data processing module is used to generate an embedded representation of a sequence based on the diagnostic data; the position embedding in the embedded representation of the sequence uses the age representation at the time of the visit; The BERT framework-based pre-training model is used to calculate the prediction results of various disease risks based on the embedded representation of the sequence.

[0116] The workflow of this system is: S1: The diagnostic data acquisition module acquires multiple diagnostic data for the patient. This diagnostic data includes age at visit, diagnosis date, diagnosis code, abnormal laboratory test result code, and surgical information. Age at visit is calculated monthly, diagnosis codes are in ICD-10 format, abnormal laboratory test result codes are in Loinc format, and surgical information is in ICD-10 PCS format. Multiple diagnostic data for the patient covers the past three years, and the number of diagnostic data points must be greater than or equal to four.

[0117] S2: The diagnostic data processing module generates an embedding representation of the sequence based on the diagnostic data. The embedding representation of the sequence is obtained by concatenating the tag embedding, the visit embedding, and the position embedding. The tag embedding is a low-dimensional representation of the diagnosis code, abnormal laboratory test result code, and surgical information. The visit embedding is represented by the visit ID. The visit ID is generated by sorting the diagnostic data chronologically according to the diagnosis date. The position embedding is represented by the age at the time of the visit.

[0118] S3: The sequence embedding is fed into a pre-trained model based on the BERT framework. The sequence embedding is then fed through a 12-layer Transformer layer with a multi-head self-attention mechanism, a feedforward neural network, residual connections, layer normalization, and a classification layer to calculate the predicted risk of multiple diseases. In this example, the predicted risk of multiple diseases is the risk probability of 234 diseases within the next 12 months.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A universal disease prediction method based on the BERT framework, characterized by: The following steps are involved: S1: Obtain multiple diagnostic data of the patient, including age at diagnosis, diagnosis date, diagnosis code, abnormal laboratory test result code, and surgical information; age at diagnosis is calculated in months; S2: Generate an embedding representation of the sequence based on the diagnosis data; concatenate the tag embedding, visit embedding, and position embedding to obtain the embedding representation of the sequence; the tag embedding is a low-dimensional representation of the diagnosis code, abnormal laboratory test result code, and surgical information; the visit embedding is represented by the visit ID, which is generated by sorting the diagnosis data in chronological order according to the diagnosis date; and the position embedding is represented by the age at the time of the visit; S3: Input the sequence embedding representation into the pre-trained model based on the BERT framework to obtain prediction results for multiple disease risks.

2. A universal disease prediction method based on the BERT framework according to claim 1, characterized in that: The patient's multiple diagnostic data refers to the patient's diagnostic data for the past three years, and the number of diagnostic data is greater than or equal to 4.

3. The universal disease prediction method based on the BERT framework according to claim 1, characterized in that: The generation process of the marker embedding is: The diagnostic codes in the diagnostic data are sorted according to their priorities, and then the abnormal laboratory test result codes and surgical information are spliced ​​in sequence; using a dictionary, the sorted diagnostic codes, abnormal laboratory test result codes and surgical information are converted into tags, and then data filling is performed to obtain tag embedding.

4. A universal disease prediction method based on the BERT framework according to claim 1, characterized in that: The pre-training model based on the BERT framework includes 12 Transformer layers, and each Transformer layer includes a 12-head multi-head self-attention mechanism and a feedforward neural network.

5. A universal disease prediction method based on the BERT framework according to claim 4, characterized in that: The training of the pre-trained model based on the BERT framework includes: pre-training stage and fine-tuning training stage.

6. The universal disease prediction method based on the BERT framework according to claim 1, characterized in that: The prediction results of multiple disease risks are the risk probabilities of the diseases within the next 12 months.

7. The universal disease prediction method based on the BERT framework according to claim 1, characterized in that: The prediction results of multiple disease risks are the prediction results of 234 diseases.

8. A universal disease prediction system based on the BERT framework, characterized by: A method for predicting a disease based on a BERT framework according to any one of claims 1 to 7, comprising: a diagnostic data acquisition module, a diagnostic data processing module, and a pre-trained model based on a BERT framework. The diagnostic data acquisition module is used to acquire multiple diagnostic data of the patient, wherein the diagnostic data includes the age at the time of consultation, diagnosis date, diagnosis code, abnormal laboratory test result code and operation information; The diagnostic data processing module is used to generate an embedded representation of a sequence based on the diagnostic data; the position embedding in the embedded representation of the sequence uses the age representation at the time of the visit; The BERT framework-based pre-training model is used to calculate the prediction results of various disease risks based on the embedded representation of the sequence.

Images