Liver disease early warning method based on pre-training language model and machine learning

Abstract

The invention discloses a liver disease early warning method based on a pre-training language model and machine learning, and the method specifically comprises the following steps: 1, collecting patient information data, and screening an original text of a patient admission record as original data; 2, preprocessing the original data of the patient, and performing data formatting processing on the original data of the patient by using a large language model to obtain formatted patient medical history information, examination information and image report information; 3, training the medical history information and the examination information of the patient by using a gradient boosting decision tree algorithm, and learning the relationship between the medical history information and the examination information and the liver disease; 4, using a pre-training language model for the image report information of the patient, and learning the relationship between the image report and the liver disease; and 5, adjusting and optimizing model parameters by using Bayesian optimization to obtain optimal model parameters, and obtaining a medical history model, an inspection and examination model and an image report model. The method provides support for early discovery and early warning of liver diseases.

Description

Technical Field

[0001] The present invention relates to a liver disease early warning method, and in particular to a liver disease early warning method based on a pre-trained language model and machine learning. Background Art

[0002] Liver disease is a major health problem that affects hundreds of millions of people worldwide and has a wide range of social and economic impacts. Common liver diseases include cirrhosis, fatty liver disease, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, liver hemangiomas, and so on. Although these diseases all fall into the category of liver disease, the pathogenesis and clinical manifestations of each liver disease are significantly different. Therefore, in clinical practice, the accurate identification of various types of liver diseases is often complex and challenging. And because the early symptoms of liver disease are often not obvious, many patients are not diagnosed until their condition is serious. Therefore, it is particularly important to establish an effective early warning mechanism for liver disease.

[0003] In recent years, with the rapid development of computer technology, data science and artificial intelligence have shown significant potential in disease early warning and management. Using computer systems to warn of diseases can effectively assist doctors in making accurate disease assessments and diagnostic decisions. Therefore, liver disease early warning based on computer technology has become possible and has broad application prospects.

[0004] In the field of artificial intelligence, a pre-trained language model refers to a model that is pre-trained on large-scale text data. These models are often widely used in various natural language processing (NLP) tasks, such as text classification, translation, dialogue, question and answer, etc. Pre-trained language models are first pre-trained on large-scale unlabeled text (for example, using autoregressive tasks or masked language model tasks), and then fine-tuned according to specific downstream tasks (such as sentiment analysis, question and answer systems, text classification, etc.). In liver disease early warning, these models can process medical literature, patient medical records, and other relevant text data, extract valuable information from them, and assist doctors in disease analysis and decision-making.

[0005] Among them, a large language model refers to a pre-trained language model with a very large parameter scale, usually with billions or even hundreds of billions of parameters. The scale of large language models enables them to perform well in handling more complex language tasks, with stronger context understanding and generation capabilities. In the field of liver disease early warning, large language models can process complex medical literature and patient information, provide more accurate disease information and early warning recommendations, thereby helping doctors better understand the patient's condition and make corresponding decisions.

[0006] Machine learning is a technology that uses data to train models so that they can automatically learn from data and make predictions or decisions. Machine learning is a subfield of artificial intelligence that aims to simulate human learning ability through algorithms and statistical models without explicit programming instructions. Machine learning is often divided into supervised learning, unsupervised learning, and reinforcement learning. Supervised learning refers to the training of models through labeled input and output data to learn the mapping relationship from input to output. Its typical tasks include classification and regression. Unsupervised learning refers to the model performing pattern recognition based on the inherent structure of data without labeled data. Its common tasks include clustering and dimensionality reduction. Reinforcement learning refers to the model learning how to make decisions through reward and punishment mechanisms through interaction with the environment. In liver disease early warning, machine learning methods can be used to analyze patients' medical data, identify potential symptom patterns, and provide prompts and suggestions to doctors.

[0007] Gradient boosted decision tree (GBDT) is a powerful machine learning algorithm and belongs to the ensemble learning method. It uses a gradient boosting strategy to continuously reduce the prediction error of the model by gradually combining multiple weak learners (usually decision trees). Each new tree attempts to correct the errors of all previous trees, making the model perform better and better on the training set. GBDT works by gradually training new trees to fit the residuals of the previous model based on the initial model (such as the predicted mean). By repeating this process, the model is continuously improved. It uses the gradient descent method to optimize the loss function, and each new tree is constructed to reduce the error along the gradient direction of the loss function. In liver disease early warning, the GBDT algorithm can be used to predict the patient's disease progression and identify key features related to the disease, thereby assisting doctors in diagnosis and increasing the possibility of early detection of liver disease.

[0008] In summary, using pre-trained language models and machine learning methods to automatically analyze and warn patients can not only significantly save medical resources and labor costs, but also improve the accuracy and efficiency of liver disease warnings. These technologies provide doctors with powerful auxiliary tools, help optimize the doctor's diagnosis process, and promote the implementation of subsequent treatment for patients. Summary of the invention

[0009] In view of this, the present invention discloses a liver disease early warning method based on a pre-trained language model and machine learning to solve the liver disease early warning problem faced in the prior art background. This method formats the patient's original data by using a large language model, and uses a gradient boosting decision tree algorithm and a trained language model to perform liver disease early warning, thereby realizing automated liver disease early warning and risk screening, and achieving the purpose of improving the accuracy and efficiency of disease early warning.

[0010] The present invention adopts the following technical solutions to solve the above technical problems:

[0011] The present invention provides a liver disease early warning method based on a pre-trained language model and machine learning, and the specific steps are as follows:

[0012] Step 1: Patient information data collection: collect patient electronic medical record information and filter out the original text of the patient's admission record as the original data.

[0013] Step 2: Preprocess the original patient data. Use the large language model to format the original patient data to obtain the formatted patient medical history information, test information, and imaging report information.

[0014] Step 3: Use the gradient boosting decision tree algorithm to train the patient's medical history information and test information to learn the relationship between medical history information, test information and liver disease.

[0015] Step 4: Use the pre-trained language model on the patient's imaging report information to learn the relationship between the imaging report and liver disease.

[0016] Step 5: Use Bayesian optimization to tune the model parameters to the optimal model parameters, and obtain the medical history model, the test and examination model, and the imaging report model. The present invention obtains high-quality patient formatted information through a large language model, and converts the liver disease early warning problem into a classification problem of three types of key information, so as to obtain accurate liver disease classification results.

[0017] As a further optimization solution of the present invention, in step 2, the data preprocessing operation includes the following steps:

[0018] (1) Design a formatted prompt based on medical knowledge of liver disease, and subdivide the patient's original data into three different types of formatted data: medical history information, test information, and imaging report information.

[0019] (2) Prompt design method: convert all information into numeric types, convert yes and no into 1 and 0, convert null values ​​into -1, delete the signs of range data, extract only the patient's preoperative data, and eliminate irrelevant data.

[0020] (3) Medical history information includes: gender, age, physical examination, smoking and drinking history, medical history, allergy history, surgery and blood transfusion history, and family history.

[0021] (4) The test information includes: five items of hepatitis B, liver function test, virus-related test, autoimmune antibody test, blood lipid and blood sugar test, blood routine test, renal function test, electrolyte and metabolism-related test, coagulation function test, tumor markers, and pancreatic islet function test.

[0022] (5) The imaging report information includes: examination findings and examination conclusions,

[0023] (6) Limit the amount of text that is input into the big model each time, input the original data into the big model in batches until all information is processed, and save all formatted information as a json file.

[0024] (7) All formatted information is divided into training set and test set for subsequent model training and testing.

[0025] As a further optimization scheme of the present invention, in step 3, the gradient boosting decision tree algorithm includes the following steps:

[0026] (1) The medical history model and the test model are trained using the training sets of formatted medical history information and formatted test information respectively, using a gradient boosting decision tree algorithm.

[0027] (2) Different weights are given to samples of different categories according to their sample numbers. The weight formula is: where w i is the weight of the i-th sample, N is the total number of samples, n1, n2...n N , are the number of each sample,

[0028] (3) First, use the relevant data to train a binary classification model to determine whether it is a liver space-occupying disease.

[0029] (4) The loss function of the binary classification gradient boosting decision tree algorithm is: in is the loss function for the binary classification, y i is the true label of the i-th sample, which takes a value of 0 or 1. is the probability that the i-th sample predicted by the model is a positive class,

[0030] (5) Whether it is a liver space-occupying disease is then added to the dataset as a new one-dimensional feature. That is, a one-dimensional feature of whether it is a liver space-occupying disease is added. For patients with liver space-occupying diseases (primary liver cancer, liver cysts, liver hemangioma), this feature is 1, and for patients with non-liver space-occupying diseases (cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis), this feature is 0. The dataset is used to train an eight-classification model. The classification categories are: cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, liver hemangioma.

[0031] (6) The loss function of the eight-class gradient boosting decision tree algorithm is: in is the loss function for the eight categories, y i,k is the true label of the i-th sample, where yi,k =1 means the true category of the sample is k, otherwise y i,k =0, is the probability that the model predicts that the i-th sample belongs to class k.

[0032] As a further optimization scheme of the present invention, in step 4, pre-training the language model includes the following steps:

[0033] (1) Use the training set of formatted image data to train the image model by pre-training the language model.

[0034] (2) First, the imaging information of the same patient is spliced ​​together, and the spliced ​​data is used to train a binary classification model to determine whether it is a liver space-occupying disease.

[0035] (3) Whether it is a liver space-occupying disease is then added to the dataset as a new one-dimensional feature. That is, the patient data of liver space-occupying diseases (primary liver cancer, liver cysts, and liver hemangioma) are concatenated with the "liver space-occupying" field, and the patient data of non-liver space-occupying diseases (cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, and autoimmune hepatitis) are concatenated with the "non-liver space-occupying" field. The concatenated data are used to train an eight-classification model. The classification categories are: cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, and liver hemangioma.

[0036] The present invention adopts the above technical solution, and the beneficial effects are: the original data of the patient is classified into three types of data: medical history, test and imaging, and prompts are written according to liver disease knowledge. The three types of data are structured by using a large model, so as to accurately extract key risk indicator information. And the gradient boosting decision tree algorithm and the pre-trained language model method are used for the three types of structured data to obtain the connection between the data and liver disease, and the model parameters are optimized by using the Bayesian optimization method to obtain the probability of the patient suffering from liver disease. This solution uses a pre-trained language model and a machine learning method to realize automatic analysis and early warning of patient information, thereby assisting doctors in making diagnostic decisions and increasing the possibility of early detection of liver disease. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The figure is a flowchart of a liver disease early warning method based on a pre-trained language model and machine learning. DETAILED DESCRIPTION

[0038] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0039] The gradient boosting decision tree algorithm and the training language model proposed in the present invention include a training part and a testing part. The method needs to be trained first, and the model parameters are updated using Bayesian optimization. After the training is completed, they are saved, and the trained model is directly used for prediction in the testing phase.

[0040] Embodiment: A schematic diagram of a liver disease early warning method based on a pre-trained language model and machine learning proposed by the present invention is as follows Figure 1 The figure mainly consists of two parts: data formatting and prediction model training. The following uses the original data of a patient as an example to explain the steps of the present invention in detail.

[0041] Experimental conditions: A computer is selected for network training. The computer is configured with an Intel(R) processor (2.4GHz) and 128GB random access memory (RAM), Ubuntu 20.04.6 operating system, and NVIDIA GTX 4090 (24GB) graphics card; the software environment is the deep learning framework pytorch2.0.1, tensorflow2.17.0, and transformers4.37.0.

[0042] The embodiment of the present invention provides a liver disease early warning method based on a pre-trained language model and machine learning; the method comprises the following steps:

[0043] Step 1: Patient information data collection: collect patient electronic medical record information and filter out the original text of the patient's admission record as the original data.

[0044] Step 2: Preprocess the original patient data and use a large language model to format the original patient data to obtain formatted patient medical history information, test information, and imaging report information.

[0045] (1) Design a formatted prompt based on medical knowledge of liver disease, and subdivide the patient's original data into three different types of formatted data: medical history information, test information, and imaging report information.

[0046] (2) Prompt design method: convert all information into numeric types, convert yes and no into 1 and 0, convert null values ​​into -1, delete the signs of range data, extract only the patient's preoperative data, and eliminate irrelevant data.

[0047] (3) The medical history information includes: gender, age, physical examination, history of smoking and drinking, history of disease, history of allergies, history of surgery and blood transfusion, and family history. The specific information is shown in Table 1. The prompt example is: "Smoking history: Does the patient have a smoking history? If yes, fill in 1; if no, fill in 0".

[0048] (4) The test information includes: five items of hepatitis B, liver function test, virus-related test, autoimmune antibody test, blood lipid and blood sugar test, blood routine test, renal function test, electrolyte and metabolism-related test, coagulation function test, tumor marker, and pancreatic islet function test. The specific information is shown in Table 1. The prompt sample is: "AFP: Alpha-fetoprotein (AFP) test results, only fill in the value, no unit, if not recorded, fill in -1",

[0049] (5) The imaging report information includes: examination findings and examination conclusions. The specific information is shown in Table 1. The prompt sample is: "Examination findings: the examination conclusions of the imaging report, no additional annotations, explanations, or descriptions are required",

[0050] (6) Limit the amount of text that is input into the big model each time, with medical history information limited to 3,500 tokens, test examination information limited to 2,500 tokens, and imaging report information limited to 4,000 tokens. Input the raw data into the big model in batches until all information is processed, and save all formatted information as a json file.

[0051] (7) All formatted information is divided into a training set and a test set with a ratio of 7:3 for subsequent model training and testing.

[0052] Table 1 Formatted processing data

[0053]

[0054]

[0055] Step 3: Use the gradient boosting decision tree algorithm to train the patient's medical history information and test information to learn the relationship between the medical history information, test information and liver disease.

[0056] (1) The medical history model and the test model are trained using the training sets of formatted medical history information and formatted test information respectively, using a gradient boosting decision tree algorithm.

[0057] (2) Different weights are given to samples of different categories according to their sample numbers. The weight formula is: where w i is the weight of the i-th sample, N is the total number of samples, n1, n2...n N , are the number of each sample,

[0058] (3) First, use the relevant data to train a binary classification model to determine whether it is a liver space-occupying disease.

[0059] (4) The loss function of the binary classification gradient boosting decision tree algorithm is: in is the loss function for the binary classification, y i is the true label of the i-th sample, which takes a value of 0 or 1. is the probability that the i-th sample predicted by the model is a positive class. The gbtree model is used to improve the binary:logistic learning task objective function.

[0060] (5) Whether it is a liver space-occupying disease is then added to the dataset as a new one-dimensional feature. That is, a one-dimensional feature of whether it is a liver space-occupying disease is added. For patients with liver space-occupying diseases (primary liver cancer, liver cysts, liver hemangioma), this feature is 1, and for patients with non-liver space-occupying diseases (cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis), this feature is 0. The dataset is used to train an eight-classification model. The classification categories are: cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, liver hemangioma.

[0061] (6) The loss function of the eight-class gradient boosting decision tree algorithm is: in is the loss function for the eight categories, y i,k is the true label of the i-th sample, where y i,k =1 means the true category of the sample is k, otherwise y i,k =0, It is the probability that the model predicts that the i-th sample belongs to the k-th class. The gbtree model is improved and the multi:softprob learning task objective function is used.

[0062] Step 4: Use the pre-trained language model on the patient’s imaging report information to learn the relationship between the imaging report and liver disease.

[0063] (1) Use the training set of formatted image data to train the image model by pre-training the language model.

[0064] (2) Set the model direction to bidi, the activation function to GELU, the pooling layer type to first_token_transform, the model input dimension to 512, and the output dimension to 768.

[0065] (3) First, the imaging information of the same patient is spliced ​​together, and the spliced ​​data is used to train a binary classification model to determine whether it is a liver space-occupying disease.

[0066] (4) Whether it is a liver space-occupying disease is then added to the data set as a new one-dimensional feature, that is, the patient data of liver space-occupying diseases (primary liver cancer, liver cysts, liver hemangioma) are spliced ​​into the "liver space-occupying" field, and the patient data of non-liver space-occupying diseases (cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis) are spliced ​​into the "non-liver space-occupying" field. The spliced ​​data are used to train an eight-classification model. The classification categories are: cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, and liver hemangioma.

[0067] Step 5: Use Bayesian optimization to tune the model parameters to the optimal model parameters to obtain the medical history model, test and examination model, and imaging report model.

[0068] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A liver disease early warning method based on a pre-trained language model and machine learning, characterized in that: The following steps are involved: Step 1: Patient information data collection: collect patient electronic medical record information and filter out the original text of the patient's admission record as the original data. Step 2: Preprocess the original patient data. Use the large language model to format the original patient data to obtain the formatted patient medical history information, test information, and imaging report information. Step 3: Use the gradient boosting decision tree algorithm to train the patient's medical history information and test information to learn the relationship between medical history information, test information and liver disease. Step 4: Use the pre-trained language model on the patient’s imaging report information to learn the relationship between the imaging report and liver disease. Step 5: Use Bayesian optimization to tune the model parameters to the optimal model parameters to obtain the medical history model, test and examination model, and imaging report model.

2. A liver disease early warning method based on a pre-trained language model and machine learning according to claim 1, characterized in that: The preprocessing operation in step 2 is as follows: (1) Design a formatted prompt based on medical knowledge of liver disease, and subdivide the patient's original data into three different types of formatted data: medical history information, test information, and imaging report information. (2) Prompt design method: convert all information into numeric types, convert yes and no into 1 and 0, convert null values ​​into -1, delete the signs of range data, extract only the patient's preoperative data, and eliminate irrelevant data. (3) Medical history information includes: gender, age, physical examination, smoking and drinking history, medical history, allergy history, surgery and blood transfusion history, and family history. (4) The test information includes: five items of hepatitis B, liver function test, virus-related test, autoimmune antibody test, blood lipid and blood sugar test, blood routine test, renal function test, electrolyte and metabolism-related test, coagulation function test, tumor markers, and pancreatic islet function test. (5) The imaging report information includes: examination findings and examination conclusions, (6) Limit the amount of text that is input into the big model each time, input the original data into the big model in batches until all information is processed, and save all formatted information as a json file. (7) All formatted information is divided into training set and test set for subsequent model training and testing.

3. The liver disease early warning method based on pre-trained language model and machine learning according to claim 1, characterized in that: The specific operation of the gradient boosting decision tree algorithm in step 3 is: (1) The medical history model and the test model are trained using the training sets of formatted medical history information and formatted test information respectively, using a gradient boosting decision tree algorithm. (2) Different weights are given to samples of different categories according to their sample numbers. The weight formula is: where w i is the weight of the i-th sample, N is the total number of samples, n1, n2...n N , are the number of each sample, (3) First, use the relevant data to train a binary classification model to determine whether it is a liver space-occupying disease. (4) The loss function of the binary classification gradient boosting decision tree algorithm is: in is the loss function for binary classification, yi is the true label of the i-th sample, which takes a value of 0 or 1. is the probability that the i-th sample predicted by the model is a positive class, (5) Whether it is a liver space-occupying disease is then added to the dataset as a new one-dimensional feature. That is, a one-dimensional feature of whether it is a liver space-occupying disease is added. For patients with liver space-occupying diseases (primary liver cancer, liver cysts, liver hemangioma), this feature is 1, and for patients with non-liver space-occupying diseases (cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis), this feature is 0. The dataset is used to train an eight-classification model. The classification categories are: cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, liver hemangioma. (6) The loss function of the eight-class gradient boosting decision tree algorithm is: in is the loss function for the eight categories, y i,k is the true label of the i-th sample, where y i,k =1 means the true category of the sample is k, otherwise y i,k =0, is the probability that the model predicts that the i-th sample belongs to class k.

4. The liver disease early warning method based on pre-trained language model and machine learning according to claim 1, characterized in that: The pre-trained language model described in step 4 is as follows: (1) Use the training set of formatted image data to train the image model by pre-training the language model. (2) First, the imaging information of the same patient is spliced ​​together, and the spliced ​​data is used to train a binary classification model to determine whether it is a liver space-occupying disease. (3) Whether it is a liver space-occupying disease is then added to the dataset as a new one-dimensional feature. That is, the patient data of liver space-occupying diseases (primary liver cancer, liver cysts, and liver hemangioma) are spliced ​​into the "liver space-occupying" field, and the patient data of non-liver space-occupying diseases (cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, and autoimmune hepatitis) are spliced ​​into the "non-liver space-occupying" field. The spliced ​​data are used to train an eight-classification model. The classification categories are: cirrhosis, fatty liver, alcoholic liver disease, drug-induced liver injury, autoimmune hepatitis, primary liver cancer, liver cysts, and liver hemangioma.

Images