A method for intelligent prescription generation based on large language model

Abstract

The present invention discloses a method for intelligently generating prescriptions based on a large language model, and belongs to the field of medical natural language processing. The present invention realizes intelligent generation of prescriptions by utilizing advanced natural language processing technology and deep learning models, combined with a medical knowledge base, to conduct large-scale data training. The method mainly adopts a transfer learning method to train a RoBERTa model with good accuracy and strong applicability using a large language model. Since the amount of data generated by medical prescriptions is small, the trained model is often prone to overfitting. The present method uses transfer learning to train a large prediction model, which improves the performance of the trained model in the prescription generation task to a certain extent. The present invention can quickly and accurately generate personalized prescription results based on patient symptoms, physical characteristics and disease diagnosis, providing a powerful auxiliary tool for medical clinical practice, and has broad application prospects.

Description

Technical Field

[0001] The invention relates to a prescription intelligent generation method based on a large language model, and belongs to the field of medical natural language processing. Background Art

[0002] In recent years, with the continuous development of modern medicine, artificial intelligence technology has shown great potential in the medical field. It is used for the analysis of medical images, such as CT scans, MRIs, and X-rays; monitoring the health status of patients to provide personalized health advice; the application of medical robots in surgery, rehabilitation, and nursing. Nowadays, the application of natural language processing technology in the medical field is also receiving increasing attention. The processing and analysis of medical texts are of great significance for assisting physicians in making diagnosis and treatment decisions. The main purpose of natural language processing technology is to automatically extract key information from various medical diagnosis texts, helping physicians to give prescription results based on diagnosis faster and more accurately, thereby improving diagnostic efficiency. However, compared with general texts, medical texts have more professional terms, complex syntactic structures, and rich semantic information, making medical text processing tasks more challenging.

[0003] With the widespread application of deep learning in natural language processing, medical text processing has developed rapidly. The emergence of models such as recurrent neural networks (RNN), long short-term memory networks (LSTM), and Transformer has enabled deep learning models to have the characteristics of high speed, high accuracy, and full automation. These methods usually require a large amount of annotated data and computing resources to train complex models, such as BERT, GPT, etc. However, these models require more training data during reasoning. However, due to the lack of training data, the model's generalization ability is insufficient, and due to the complexity of medical texts and the diversity of information, it is limited in information extraction and context understanding.

[0004] In order to solve the problem of insufficient model generalization ability, a feasible solution is to use natural language processing and transfer learning methods based on deep learning, and fine-tune the RoBERTa model based on the test results of the word embedding pre-training model to obtain a RoBERTa model that can be suitable for medical prescription generation.

[0005] The present invention is based on transfer learning and large language model technology, and proposes a prescription intelligent generation method based on a large language model. Summary of the invention

[0006] The purpose of the present invention is to provide a method for intelligently generating prescriptions based on a large language model, which mainly uses a transfer learning method to train a RoBERTa model with good accuracy and strong applicability using a large language model. Since the amount of data generated by medical prescriptions is small, the trained model is often prone to overfitting. This patent uses transfer learning to train a large prediction model, which improves the performance of the trained model in the prescription generation task to a certain extent.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A method for intelligently generating prescriptions based on a large language model comprises the following steps:

[0009] Step 1: Input medical data: Collect and input existing medical diagnosis and prescription data to provide specific field information for subsequent model training;

[0010] Step 2: Train word embedding model: Use medical data to train word embedding model, embed the semantic relationship in the medical field into vector space, and provide a basis for subsequent models;

[0011] Step 3: Obtain general Chinese corpus: Input general Chinese corpus such as Chinese Wikipedia, various encyclopedias, news, Q&A, etc. to expand the context and knowledge of the model;

[0012] Step 4: Pre-training RoBERTa model: Using general Chinese corpus to train R o The BERTa model is pre-trained to enable it to have a wider range of language understanding capabilities;

[0013] Step 5: Transfer learning fine-tuning: Use the test results of the word embedding model to perform transfer learning and fine-tune R o The BERTa model is adapted to the special needs of the medical field and improves the model's ability to handle medical problems;

[0014] Step 6: Medical data preprocessing: Preprocess the medical diagnosis and prescription data and patient personal information data to be input to ensure that they meet the requirements of model input;

[0015] Step 7: Model training: Train R on the prepared medical data o The BERTa model is trained to further improve the model's performance on medical tasks;

[0016] Step 8: Evaluate model performance: Evaluate the trained model, including the loss value, accuracy, recall, and F1 score indicators of the training set and test set;

[0017] Step 9: Fine-tuning and optimization: Analyze the evaluation results, identify the weaknesses of the model in specific tasks, adjust the model structure or parameters based on the analysis results, and perform further fine-tuning to further optimize the model's parameters and performance;

[0018] Step 10: Model deployment: Deploy the optimized model so that it can process medical diagnosis and prescription data in practical applications and provide accurate predictions and analysis.

[0019] Preferably, in step 1, the medical diagnosis and prescription data includes the patient's medical history, doctor's orders, and prescription information.

[0020] Preferably, the step 2 specifically includes: using the Gensim library in the natural language processing tool Python to segment the medical data, remove stop words for preprocessing, and then use the Word2Vec word embedding method to train the model to provide a basis for subsequent models. Among them, in the Word2Vec model, there are mainly two training models, Skip-Gram and CBOW. Skip-Gram predicts the context given the current value; and CBOW predicts the current value given the context. Their objective functions are as follows:

[0021]

[0022]

[0023] Preferably, step 4 specifically includes: pre-training the RoBERTa model using common Chinese corpus, RoBERTa uses the Transformer's multi-head self-attention mechanism and feedforward neural network, and the calculation formulas of these layers are similar to those of the standard Transformer; given an input sequence X = {x1, x2, ..., x n}, and transform it by the following formula:

[0024] Multi-head attention mechanism:

[0025] Where Q, K and V are the linear transformation results of query, key and value respectively; k is the dimension of the attention head;

[0026] Residual connection and layer normalization: LayerNorm(X+Attention(Q, K, V))

[0027] Feedforward neural network: FFN(X) = ReLU(XW1+b1)W2+b2

[0028] Among them, W1, W2, b1, b2 are the weights and biases of the network;

[0029] At the same time, R o BERTa uses the masked language model technique to learn a rich representation of language. The model randomly masks some tokens in the input sequence and then predicts these masked tokens.

[0030] P(x i |x1,…,x i-1 , x i+1 , …, x n ).

[0031] Preferably, step 5 specifically includes: using medical data for supervised learning, fine-tuning the RoBERTa model on tasks in the medical field, using the test results of the word embedding model trained in step 2 as auxiliary information to help the RoBERTa model better adapt to the medical field; using an adversarial transfer learning method, for the feature extractor, sigmoid is used as the activation function, and its output is:

[0032] G f (x; W, b) = sigm(Wx+b)

[0033] For the label prediction period, softmax is used as the activation function, and its output is:

[0034] G y (G f (x); V, c) = softmax(VG f (x)+c)

[0035] Given a data point (x i ,y i ), with negative log-likelihood as the loss function, the loss of the label predictor is:

[0036]

[0037] The training optimization objectives are:

[0038]

[0039] in, represents the label prediction loss of the i-th sample, R(W, b) is an optional regularizer, λ is a manually set regularization parameter, and λ·R(W, b) is used to prevent overfitting of the neural network;

[0040] For the domain discriminator, sigmoid is used as the activation function, and its output is:

[0041] G d (G f (x);u,z)=sigm(u T Gf (x)+z)

[0042] Domain Discriminator G d The loss is:

[0043]

[0044] The objective function of the discriminator is:

[0045]

[0046] The total loss consists of two parts: the network training loss and the domain judgment loss:

[0047]

[0048] Among them, the parameters of the label prediction period are updated by minimizing the objective function, and the parameters of the domain judgement are updated by maximizing the objective function:

[0049]

[0050]

[0051] Preferably, step 6 specifically includes: cleaning newly input medical data and patient personal information, processing missing values ​​and abnormal situations, and segmenting and tokenizing text data to make it conform to the format of model input.

[0052] Preferably, step 7 specifically includes: using a deep learning framework to train the RoBERTa model on the prepared medical data, adjusting hyperparameters such as learning rate and batch size, and ensuring that the model converges on the medical task.

[0053] Preferably, the accuracy, recall, and F1 score indicators of the training set and the test set in step 8 are as follows:

[0054]

[0055]

[0056]

[0057] Among them, TP (True Positive) is the number of positive samples correctly identified. FP (False Positive) is the number of falsely reported negative samples. TN (True Negative) is the number of negative samples correctly identified. FN (False Negative) is the number of falsely reported positive samples.

[0058] Beneficial effects: The present invention establishes a data set in the medical field by collecting medical diagnosis and prescription data. Subsequently, this data set is used to train a word embedding model to embed the semantic relationship in the medical field in the vector space. It combines both the diagnosis results and the patient's personal information to achieve personalized customization of prescriptions and ensure the accuracy and effectiveness of prescriptions. General Chinese corpus, such as Chinese Wikipedia, encyclopedia, news, etc., are introduced to expand the context and knowledge base of the model. Such multi-source data fusion helps to build a more comprehensive model with broad context understanding. After data preparation, pre-training is performed through the deep learning framework RoBERTa model. Then, through transfer learning, combined with the test results of the word embedding model, RoBERTa is fine-tuned to adapt to the special needs of the medical field. After completing the model training, its performance is comprehensively evaluated, including indicators such as loss value, accuracy, recall rate, F1 score, etc. of the training set and test set. Based on the evaluation results, the model is fine-tuned to further improve its performance in medical field tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is the training flow chart of the RoBERTa large language model in the present invention. DETAILED DESCRIPTION

[0060] The present invention will be further described below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, the prescription intelligent generation method based on the large language model of the present invention comprises the following steps:

[0062] A method for medical image object segmentation based on large model knowledge distillation, characterized by comprising the following steps:

[0063] Step 1: Input medical data: Obtain existing medical diagnosis and prescription data from medical institutions or medical information systems, including patient medical records, doctor's orders, prescriptions, and other information, to provide specific field information for subsequent model training;

[0064] Step 2: Train word embedding model: Use the Gensim library in Python, a natural language processing tool, to perform preprocessing on medical data, such as word segmentation and stop word removal, and then use the Word2Vec word embedding method to train the model to provide a basis for subsequent models. Among them, there are mainly two training models in the Word2Vec model: Skip-Gram and CBOW. Skip-Gram predicts the context given the current value. CBOW predicts the current value given the context. Their objective functions are as follows:

[0065]

[0066]

[0067] Step 3: Obtain general Chinese corpus: Obtain large-scale Chinese corpus from multiple data sources such as Chinese Wikipedia, various encyclopedias, news, Q&A communities, etc., expand the context and knowledge of the model, clean up the general corpus, and remove noise and irrelevant information;

[0068] Step 4: Pre-train the RoBERTa model: Use the common Chinese corpus to pre-train the RoBERTa model. RoBERTa uses the Transformer's multi-head self-attention mechanism and feed-forward neural network. The calculation formulas of these layers are similar to those of the standard Transformer. Given an input sequence X = {x1, x2, ..., x n}, and transform it by the following formula:

[0069] Multi-head attention mechanism: Among them, Q, K and V are the linear transformation results of query, key and value respectively. k is the dimension of the attention head.

[0070] Residual connection and layer normalization: LayerNorm(X+Attention(Q, K, V))

[0071] Feedforward neural network: FFN(X) = ReLU(XW1+b1)W2+b2

[0072] Among them, W1, W2, b1, b2 are the weights and biases of the network.

[0073] At the same time, R o BERTa uses the masked language model technique to learn a rich representation of language. The model randomly masks some tokens in the input sequence and then predicts these masked tokens.

[0074] P(x i |x1,…,x i-1 , x i+1 , …, x n );

[0075] Step 5: Transfer learning fine-tuning: Use medical data for supervised learning, fine-tune the RoBERTa model on tasks in the medical field, and use the test results of the word embedding model trained in step 2 as auxiliary information to help the RoBERTa model better adapt to the medical field. Use the adversarial transfer learning method, for the feature extractor, sigmoid is used as the activation function, and its output is:

[0076] G f (x; W, b) = sigm(Wx+b)

[0077] For the label prediction period, sofimax is used as the activation function, and its output is:

[0078] G y (G f (x); V, c) = softmax(VG f (x)+c)

[0079] Given a data point (x i ,y i ), with negative log-likelihood as the loss function, the loss of the label predictor is:

[0080]

[0081] The training optimization objectives are:

[0082]

[0083] in, represents the label prediction loss of the i-th sample, R(W, b) is an optional regularizer, λ is an artificially set regularization parameter, and λ·R(W, b) is used to prevent the neural network from overfitting.

[0084] For the domain discriminator, sigmoid is used as the activation function, and its output is:

[0085] G d (G f (x);u,z)=sigm(u T G f (x)+z)

[0086] Domain Discriminator G d The loss is:

[0087]

[0088] The objective function of the discriminator is:

[0089]

[0090] The total loss consists of two parts: the network training loss and the domain judgment loss.

[0091]

[0092] Among them, the parameters of the label prediction period are updated by minimizing the objective function, and the parameters of the domain judgement are updated by maximizing the objective function:

[0093]

[0094]

[0095] Step 6: Medical data preprocessing: clean up the newly input medical data and patient personal information, handle missing values ​​and abnormalities, and process the text data by word segmentation and tokenization to make it conform to the model input format;

[0096] Step 7: Model training: Use a deep learning framework to train the RoBERTa model on the prepared medical data, adjust hyperparameters such as learning rate and batch size, and ensure that the model converges on medical tasks to further improve the performance of the model on medical tasks;

[0097] Step 8: Evaluate model performance: Evaluate the trained model, including the loss value, accuracy, recall rate, F1 score and other indicators of the training set and test set. The formula is as follows:

[0098]

[0099]

[0100]

[0101] Among them, TP (True Positive) is the number of positive samples correctly identified. FP (False Positive) is the number of falsely reported negative samples. TN (True Negative) is the number of negative samples correctly identified. FN (False Negative) is the number of falsely reported positive samples.

[0102] Step 9: Fine-tuning and optimization: Analyze the evaluation results, identify the weaknesses of the model in specific tasks, adjust the model structure or parameters according to the analysis results, and perform further fine-tuning;

[0103] Step 10: Model deployment: Deploy the optimized model so that it can process medical diagnosis and prescription data in practical applications and provide accurate predictions and analysis.

[0104] The present invention has the following characteristics:

[0105] Field customization: In the medical field, the project inputs medical diagnosis and prescription data to train and fine-tune the model for specific medical needs, making the model more suitable for natural language processing tasks in the medical field;

[0106] Multi-source data fusion: Combining medical-specific data with general Chinese corpus, a more comprehensive model with broad context understanding is constructed by utilizing these multi-source data;

[0107] Transfer learning and fine-tuning: We used the transfer learning method to achieve knowledge transfer and model optimization by pre-training the RoBERTa model in general fields and then fine-tuning it for specific tasks in the medical field.

[0108] Model evaluation and optimization: After training, the project conducted a comprehensive evaluation, including loss value, accuracy, recall rate, F1 score and other indicators. The model was fine-tuned and optimized based on the evaluation results to improve the performance of the model in medical tasks.

[0109] Application deployment: The ultimate goal is to successfully deploy the trained model so that it can provide accurate predictions and analysis in actual medical applications and provide valuable support for the medical field.

[0110] In addition to the above embodiments, the present invention may also have other implementation modes. Any technical solutions formed by equivalent replacement or equivalent transformation shall fall within the protection scope required by the present invention.

Claims

1. A method for intelligently generating prescriptions based on a large language model, characterized in that: The following steps are involved: Step 1: Input medical data: Collect and input existing medical diagnosis and prescription data to provide information for subsequent model training; Step 2: Train word embedding model: Use medical data to train word embedding model, embed the semantic relationship in the medical field into vector space, and provide a basis for subsequent models; Step 3: Obtain general Chinese corpus: Input general Chinese corpus such as Chinese Wikipedia, various encyclopedias, news, and Q&A to expand the context and knowledge of the model; Step 4: Pre-train the RoBERTa model: Use common Chinese corpus to pre-train the RoBERTa model to enable it to have a wider range of language understanding capabilities; Step 5: Transfer learning fine-tuning: Use the test results of the word embedding model to perform transfer learning and fine-tune the RoBERTa model to adapt to the special needs of the medical field and improve the model's ability to handle medical problems; Step 6: Medical data preprocessing: Preprocess the medical diagnosis and prescription data and patient personal information data to be input to ensure that they meet the requirements of model input; Step 7: Model training: Train the RoBERTa model on the prepared medical data to further improve the model’s performance on medical tasks. Step 8: Evaluate model performance: Evaluate the trained model, including the loss value, accuracy, recall, and F1 score indicators of the training set and test set; Step 9: Fine-tuning and optimization: Analyze the evaluation results, identify the weaknesses of the model, adjust the model structure or parameters according to the analysis results, and make further fine-tuning to further optimize the parameters and performance of the model; Step 10: Model deployment: Deploy the optimized model to process medical diagnosis and prescription data in actual applications to provide accurate predictions and analysis; The medical diagnosis and prescription data includes the patient's medical history, doctor's orders, and prescription information; In step 5, the total loss of the RoBERTa model consists of two parts: the network training loss and the domain judgment loss: ； Among them, the parameters of the label prediction period are updated by minimizing the objective function, and the parameters of the domain judgement are updated by maximizing the objective function: ； 。 2. The method for intelligently generating prescriptions based on a large language model according to claim 1, characterized in that: The step 2 specifically includes: using the Gensim library in Python, a natural language processing tool, to perform word segmentation and stop word removal preprocessing on the medical data, and then using the Word2Vec word embedding method to perform model training to provide a basis for subsequent models. In the Word2Vec model, there are mainly two training models, Skip-Gram and CBOW. Skip-Gram predicts the context given the current value; and CBOW predicts the current value given the context. Their objective functions are as follows: 。 3. The method for intelligently generating prescriptions based on a large language model according to claim 2, characterized in that: The step 4 specifically includes: using common Chinese corpus to pre-train the RoBERTa model. RoBERTa uses the Transformer's multi-head self-attention mechanism and feedforward neural network. The calculation formulas of these layers are similar to those of the standard Transformer. Given an input sequence , transformed by the following formula: Multi-head attention mechanism: ; Among them, Q, K and V are the linear transformation results of query, key and value respectively; is the dimension of the attention head; Residual connections and layer normalization: ; Feedforward Neural Network: ; in, are the weights and biases of the network; At the same time, RoBERTa uses the masked language model technology to enable it to learn rich representations of language. The model randomly masks some tokens in the input sequence and then predicts these masked tokens. 。 4. The method for intelligently generating prescriptions based on a large language model according to claim 3, characterized in that: The step 5 specifically includes: using medical data for supervised learning, fine-tuning the RoBERTa model on tasks in the medical field, using the test results of the word embedding model trained in step 2 as auxiliary information to help the RoBERTa model better adapt to the medical field; using the adversarial transfer learning method, for the feature extractor, sigmoid is used as the activation function, and its output is: ； For the label prediction period, softmax is used as the activation function, and its output is: ； Given a data point , with negative log-likelihood as the loss function, the loss of the label predictor is: ； The training optimization objectives are: ； in, Indicates The label prediction loss of samples is is an optional regularizer, is a regularization parameter set artificially. The purpose is to prevent the neural network from overfitting; For the domain discriminator, sigmoid is used as the activation function, and its output is: ； Domain Discriminator The loss is: ； The objective function of the discriminator is: 。 5. The method for intelligently generating prescriptions based on a large language model according to claim 4, characterized in that: The step 6 specifically includes: cleaning the newly input medical data and patient personal information, processing missing values ​​and abnormal situations, and segmenting and tokenizing the text data to make it conform to the format of model input.

6. The method for intelligently generating prescriptions based on a large language model according to claim 5, characterized in that: The step 7 specifically includes: using a deep learning framework to train the RoBERTa model on the prepared medical data, adjusting hyperparameters such as learning rate and batch size, and ensuring that the model converges on the medical task.

7. The method for intelligently generating prescriptions based on a large language model according to claim 6, characterized in that: The accuracy, recall, and F1 score indicators of the training set and test set in step 8 are as follows: ； Among them, TP is the number of positive samples correctly identified; FP is the number of false positive negative samples; TN is the number of negative samples correctly identified; FN is the number of false positive samples.

Images