Method and device for generating training sample of large language model

Abstract

One or more embodiments of the invention provide a training sample generation method and device for a large language model, and the method comprises the steps: obtaining medical data containing disease features and diagnosed diseases, and obtaining various diseases and corresponding disease features; generating a training sample set and a label thereof based on the medical data, various diseases and corresponding disease features; the training sample comprises data used for indicating whether disease features corresponding to a target disease exist or not, and the label of the training sample is used for indicating whether the target disease is diagnosed or not; performing cross validation on the training sample set based on a machine learning model to obtain a prediction result of the machine learning model for the training sample set; based on the label and the prediction result, screening out a high-quality training sample matched with the prediction result and the label from the training sample set; the high-quality training sample is used for performing supervised fine tuning on the large language model.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a method and device for generating training samples of a large language model. Background Art

[0002] Nowadays, intelligent dialogue systems are widely used in many fields such as customer service, legal aid, online education, medical consultation, etc. Taking the rapidly developing field of medical consultation as an example, intelligent dialogue systems have become a key tool for improving the quality and efficiency of medical services. Intelligent dialogue systems provide consulting services to patients by simulating human communication methods, including disease diagnosis, treatment recommendations, and drug instructions.

[0003] An intelligent dialogue system is an interactive system developed using artificial intelligence technology. It aims to understand and answer questions raised by users in natural language and generate concise and clear answers. Intelligent dialogue systems are usually based on large language models (LLMs), which understand and answer questions raised by users and generate corresponding answers. Intelligent dialogue systems in the field of medical consultation mainly rely on the knowledge acquired by the large language model during the training process by learning static medical-related corpora. Therefore, it is crucial for intelligent dialogue systems to ensure the correctness of these medical-related corpora. Summary of the invention

[0004] One or more embodiments of the present application provide the following technical solutions:

[0005] The present application provides a method for generating training samples of a large language model, the method comprising:

[0006] Obtain medical data including disease characteristics and diagnosed diseases, and obtain various diseases and their corresponding disease characteristics;

[0007] Based on the medical data, various diseases and their corresponding disease characteristics, a training sample set and its label are generated; wherein the training sample includes data indicating whether it has a disease characteristic corresponding to a target disease, and the label of the training sample is used to indicate whether it is diagnosed as the target disease;

[0008] Performing cross-validation on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set;

[0009] Based on the label and the prediction result, high-quality training samples whose prediction result matches the label are screened out from the training sample set; wherein the high-quality training samples are used for supervised fine-tuning of the large language model.

[0010] The present application also provides a training sample generating device for a large language model, the device comprising:

[0011] An acquisition module, which acquires medical data including disease characteristics and diagnosed diseases, and acquires various diseases and their corresponding disease characteristics;

[0012] A generation module, based on the medical data, various diseases and their corresponding disease characteristics, generates a training sample set and its label; wherein the training sample includes data for indicating whether it has a disease characteristic corresponding to a target disease, and the label of the training sample is used to indicate whether it is diagnosed as the target disease;

[0013] A training module, performing cross-validation on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set;

[0014] A screening module is provided for screening high-quality training samples whose prediction results match the labels from the training sample set based on the labels and the prediction results; wherein the high-quality training samples are used for supervised fine-tuning of the large language model.

[0015] The present application also provides an electronic device, comprising:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] The processor implements the steps of any of the above methods by running the executable instructions.

[0019] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0020] In the above technical solution, a training sample set and its labels can be generated based on medical data containing disease characteristics and diagnosed diseases, as well as various diseases and their corresponding disease characteristics. A training sample in the training sample set may include data for indicating whether it has disease characteristics corresponding to the target disease, and a label for indicating whether it is diagnosed as the target disease. Subsequently, cross-validation can be performed on the training sample set based on the machine learning model to obtain the prediction results of the machine learning model for the training sample set, and high-quality training samples whose prediction results and labels match can be screened out from the training sample set; wherein the screened high-quality training samples can be used for supervised fine-tuning of the large language model.

[0021] By adopting the above method, the correctness of the high-quality training samples selected from the generated training sample set can be guaranteed, so as to ensure the correctness of the knowledge acquired by the large language model by learning these high-quality training samples; since these high-quality training samples are training samples related to the task of diagnosing diseases based on disease characteristics, after using these high-quality training samples to perform supervised fine-tuning on the large language model, the response accuracy of the large language model when diagnosing diseases based on disease characteristics can be improved. Furthermore, for the intelligent dialogue system containing the large language model, its response accuracy when diagnosing diseases based on disease characteristics can also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The following is a description of the accompanying drawings required for use in describing the exemplary embodiments, wherein:

[0023] Figure 1 It is a schematic diagram of an intelligent dialogue system shown in an exemplary embodiment of the present application.

[0024] Figure 2 It is a flowchart of a method for generating training samples of a large language model shown in an exemplary embodiment of the present application.

[0025] Figure 3 It is a flowchart of a training sample evaluation method for a large language model shown in an exemplary embodiment of the present application.

[0026] Figure 4 It is a structural schematic diagram of a device shown in an exemplary embodiment of the present application.

[0027] Figure 5 It is a block diagram of a training sample generating device for a large language model shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0028] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description relates to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. On the contrary, they are only examples consistent with some aspects of one or more embodiments of the present application.

[0029] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the steps included in the method may be more or less than those described in this application. In addition, a single step described in this application may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this application may also be combined into a single step for description in other embodiments.

[0030] In the field of medical consultation, intelligent dialogue systems have become a key tool for improving the quality and efficiency of medical services. Intelligent dialogue systems provide consulting services to patients by simulating human communication methods, including disease diagnosis, treatment recommendations, and drug instructions.

[0031] An intelligent dialogue system is an interactive system developed using artificial intelligence technology. It is designed to understand and answer questions raised by users in natural language and generate concise and clear answers. Intelligent dialogue systems are usually based on a large language model, which understands and answers questions raised by users and generates corresponding answers.

[0032] A large language model is a deep learning model trained with a large amount of text data, which can be used to generate natural language text or understand the meaning of natural language text. A large language model can handle a variety of natural language tasks, such as text classification, named entity recognition, question answering, dialogue, etc., and is an important path to artificial intelligence.

[0033] In the field of natural language processing, large-scale text data sets are usually referred to as corpuses. Corpuses can contain various types of text data, such as literary works, academic papers, legal documents, news reports, daily conversations, emails, online forum posts, etc. By learning the text data in the corpus, large language models can acquire and understand the laws and patterns of natural language, thereby achieving effective processing and generation of human language.

[0034] Large language models usually use the Transformer architecture, that is, large language models are usually deep learning models based on the Transformer architecture. Deep learning models based on the Transformer architecture are a type of neural network model that uses the Transformer architecture. This model performs well in fields such as natural language processing.

[0035] Transformer is a neural network model used for sequence-to-sequence modeling. Transformer does not need to rely on recursive structures, can parallelize training and reasoning, and speed up model processing. In deep learning models based on the Transformer architecture, a multi-layer Transformer encoder is usually used to extract features from the input sequence, and a Transformer decoder is used to convert the extracted features into an output sequence. At the same time, such models usually also use a self-attention mechanism to capture long-distance dependencies in the input sequence, and use residual connections and regularization methods to accelerate training and improve model performance.

[0036] A pre-trained model is a large language model that is pre-trained on large-scale unlabeled text data. A pre-trained model is a general model that is not designed and optimized for a specific task. In order to adapt the pre-trained model to specific application scenarios and task requirements, fine-tuning is required to improve the performance of the model on specific tasks. The large language model that is finally put into use is usually a model that is further fine-tuned on the basis of the pre-trained model and is a supervised learning model based on labeled text data. Pre-training and fine-tuning are complementary processes. Pre-training enables the model to have a wide range of language understanding capabilities, while fine-tuning makes the model more professional and accurate in specific tasks.

[0037] That is, the training process of a large language model can be divided into two stages: pre-training and fine-tuning. In the pre-training stage, unsupervised learning (e.g., self-supervised learning) can be used to perform pre-training on large-scale, unlabeled text data sets (e.g., online encyclopedias, online articles, books, etc.). Specifically, the missing part or the next word can be predicted based on the context, and statistical laws and language structures such as semantics and syntax can be learned. The prediction loss can be minimized through back-propagation and optimization algorithms (e.g., gradient descent), and model parameters can be iteratively updated to gradually improve the model's ability to understand language. In the fine-tuning stage, you can select corresponding supervised learning tasks (such as text classification, named entity recognition, question-answering system, dialogue system, etc.) according to specific application scenarios and task requirements, and prepare task-specific text datasets, so that you can use the pre-trained model as the starting point for fine-tuning, and use supervised learning to fine-tune on the task-specific text dataset. Specifically, you can perform the task based on the text dataset, and minimize the loss used to measure the performance of the model when processing specific tasks through back propagation and optimization algorithms (such as gradient descent), iteratively update model parameters, and gradually improve the performance of the model on specific tasks. In practical applications, fine-tuning can flexibly choose supervised learning, unsupervised learning, or semi-supervised learning methods according to specific application scenarios and the type of available data.

[0038] It should be noted that the pre-trained large language model is usually called the base model of the large language model, and the fine-tuned large language model is called the service model of the large language. The language understanding ability learned by the large language model in the pre-training and fine-tuning stages enables the large language model to understand, analyze and integrate text information, perform logical inference, knowledge reasoning or problem solving when faced with complex problems or tasks. This ability is usually called the reasoning ability of the large language model.

[0039] Large language models usually perform specific tasks under the guidance of prompt text (which can be called prompt). Prompt text is an initial text or text fragment provided to the large language model, which is intended to stimulate the model to produce corresponding output. Through the prompt text, you can clearly tell the large language model what task you expect it to perform, such as: answering a question, simulating a conversation, writing an article, translating text, etc. At the same time, the prompt text can provide the large language model with the necessary background information and context, so that the large language model can understand the logic, style, theme or position that should be followed when generating content. In addition, the prompt text can also stimulate the large language model to show its inherent knowledge reserves or specific language capabilities, such as: explaining complex concepts, citing regulations, imitating the writing style of a specific writer, etc.

[0040] Intelligent dialogue systems in the field of medical consultation mainly rely on the knowledge acquired by the large language model during the training process by learning static medical-related corpora. Therefore, the correctness of these medical-related corpora is crucial to the intelligent dialogue system. The more correct the medical-related corpora used in the training process, the more correct the knowledge acquired by the intelligent dialogue system by learning these medical-related corpora, thereby improving the response accuracy of the intelligent dialogue system.

[0041] One or more embodiments of the present application provide a technical solution for generating training samples for a large language model. In the technical solution, a training sample set and its labels can be generated based on medical data containing disease characteristics and diagnosed diseases, as well as various diseases and their corresponding disease characteristics. A training sample in the training sample set may include data indicating whether it has disease characteristics corresponding to a target disease, and a label indicating whether it is diagnosed as the target disease. Subsequently, cross-validation can be performed on the training sample set based on a machine learning model to obtain a prediction result of the machine learning model for the training sample set, and high-quality training samples whose prediction results and labels match can be screened out from the training sample set; wherein the screened-out high-quality training samples can be used for supervised fine-tuning of the large language model.

[0042] By adopting the above method, the correctness of the high-quality training samples selected from the generated training sample set can be guaranteed, so as to ensure the correctness of the knowledge acquired by the large language model by learning these high-quality training samples; since these high-quality training samples are training samples related to the task of diagnosing diseases based on disease characteristics, after using these high-quality training samples to perform supervised fine-tuning on the large language model, the response accuracy of the large language model when diagnosing diseases based on disease characteristics can be improved. Furthermore, for the intelligent dialogue system containing the large language model, its response accuracy when diagnosing diseases based on disease characteristics can also be improved.

[0043] Please refer to Figure 1 , Figure 1 It is a schematic diagram of an intelligent dialogue system shown in an exemplary embodiment of the present application.

[0044] like Figure 1 As shown, the above intelligent dialogue system may include a server and at least one client accessing the server through any type of wired or wireless network.

[0045] Among them, the above-mentioned server can correspond to a server including an independent physical host, or it can be a server cluster composed of multiple independent physical hosts; or it can correspond to a virtual server, cloud server, etc. carried by a host cluster.

[0046] The above-mentioned client may correspond to terminal devices such as smart phones, tablet computers, laptops, desktop computers, PCs (Personal Computers), PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), smart car devices or game consoles.

[0047] The user can use the intelligent question-and-answer service provided by the intelligent dialogue system through the client; the client and the server can implement the user-oriented intelligent question-and-answer service through data interaction between each other.

[0048] For example, the client can output a corresponding user interface to the user, so that the user can perform operations such as inputting query text (which can be called Query) and uploading documents or pictures for auxiliary query in the user interface to ask questions to the intelligent dialogue system and use the intelligent question-and-answer service provided by the intelligent dialogue system. The client can send the query text input by the user to the server, and the server generates a corresponding answer text (which can be called Answer) for the query text, and outputs the answer text to the user, that is, returns the answer text to the client, and the client displays the answer text to the user through the user interface, so that the user can view the corresponding answer generated by the intelligent dialogue system, thereby realizing a user-oriented intelligent question-and-answer service.

[0049] Specifically, the server can be equipped with a large language model, and the intelligent dialogue system can be based on the large language model, and the large language model can understand and answer the query text input by the user and generate an answer text corresponding to the query text. It should be noted that the large language model at this time can refer to the service model of the large language model.

[0050] In practical applications, the constructed large language model can be pre-trained on a large-scale, unlabeled text dataset using an unsupervised learning approach to obtain a basic model of the large language model. Furthermore, the dialogue task can be used as a supervised learning task for fine-tuning, and a text dataset specific to the dialogue task can be prepared, so that the basic model of the large language model can be used as the starting point for fine-tuning, and fine-tuning can be performed on a text dataset specific to the dialogue task using a supervised learning approach to obtain a serving model of the large language model.

[0051] In addition, the above-mentioned server can also be equipped with other functional components or functional subsystems, such as feature extraction components, knowledge base retrieval components, prompt text generation components, etc. These components or subsystems can work together with the large language model installed on the server to jointly generate answer text corresponding to the query text input by the user.

[0052] In order to improve the adaptability and response accuracy of the intelligent dialogue system, the server can also be equipped with a knowledge base. The knowledge base is an external knowledge base relative to the large language model installed on the server, that is, the data in the external knowledge base is not the knowledge acquired by the large language model through learning during the training process, but serves as auxiliary data in the reasoning process of the large language model to assist the large language model in generating the answer text corresponding to the query text input by the user.

[0053] Please refer to Figure 2 , Figure 2 It is a flowchart of a method for generating training samples of a large language model shown in an exemplary embodiment of the present application.

[0054] In this embodiment, the above-mentioned large language model training sample generation method can be applied to Figure 1 The server shown in the figure completes the generation of training samples on the server, and completes the fine-tuning of the large language model in the intelligent dialogue system based on the generated training samples; or, the above-mentioned large language model training sample generation method can be applied to other electronic devices with certain computing capabilities, and the other electronic devices complete the generation of training samples, and then in the example Figure 1 The server shown completes fine-tuning of the large language model in the intelligent dialogue system based on the generated training samples.

[0055] like Figure 2 As shown, the training sample generation method of the above large language model may include the following steps:

[0056] Step 202: Obtain medical data including disease characteristics and diagnosed diseases, and obtain various diseases and their corresponding disease characteristics.

[0057] In this embodiment, the generated training samples can be obtained based on medical data, so that the model can acquire medical-related knowledge by learning these training samples, and thus can provide medical-related services, such as disease diagnosis, treatment recommendations, drug instructions, etc.

[0058] That is, first of all, medical data can be obtained. Medical data can refer to data sets collected in medical practice and used for various purposes (such as research, diagnosis, treatment, management and policy making, etc.), which can come from patients' clinical data, medical records, laboratory test results, imaging data, biological sample analysis and other data related to health care. Usually, these medical data record the disease characteristics of patients and the diseases diagnosed by patients; for example, these medical data can be open source and desensitized medical records.

[0059] Furthermore, various diseases (usually referred to by disease names) and their corresponding disease characteristics can be obtained, so that relevant content can be extracted from the above medical data to generate training samples based on the various diseases and their corresponding disease characteristics. Specifically, at least one disease can be obtained, and various diseases in the at least one disease can be used as target diseases in turn to obtain disease characteristics corresponding to the target disease, thereby obtaining various diseases and their corresponding disease characteristics.

[0060] Among them, the disease can be referred to by the name of the disease (for example: prostatic hyperplasia, kidney stones, ureteral stones, urinary tract infection, etc.); disease characteristics can not only include symptoms closely related to the disease (specifically, they can be symptom descriptions, such as: frequent urination, urgency, difficulty urinating, urinary pain, low back pain, etc.), but also include gender (for example: male, female), age (specifically, the age group to which the age belongs), and other information that will affect the diagnosis of the disease to a certain extent. For example, men have a prostate, while women do not have a prostate. Therefore, when a male patient has the two symptoms of frequent urination and urgency, he may be diagnosed with prostatic hyperplasia, but when a female patient has the two symptoms of frequent urination and urgency, he cannot be diagnosed with prostatic hyperplasia. Therefore, gender will affect the diagnosis of the disease of prostatic hyperplasia. In addition, according to actual needs, information such as medical history and examination reports can also be used as disease characteristics, and this application does not impose special restrictions on this.

[0061] In practical applications, various diseases and their corresponding disease characteristics can be obtained as shown in Table 1 below:

[0062] Table 1

[0063]

[0064]

[0065] In some embodiments, various diseases and their corresponding disease characteristics can be manually set based on known medical knowledge, actual medical experience, etc.

[0066] In order to improve the text matching degree between the set disease characteristics and the above-mentioned medical data, and to facilitate the extraction of relevant content from the medical data, in some embodiments, when obtaining various diseases and their corresponding disease characteristics, feature extraction can be performed on the above-mentioned medical data first to extract disease characteristics and diseases from the medical data, and then the correlation measure between the extracted disease characteristics and the extracted diseases is calculated, and finally, based on the calculated correlation measure, the disease characteristics corresponding to the various extracted diseases are screened out from the extracted disease characteristics.

[0067] In some embodiments, the above correlation metric may be Pearson Correlation Coefficient, Spearman Rank Correlation Coefficient, Kendall Rank Correlation Coefficient, Mutual Information, cosine similarity, distance metric (such as Euclidean distance, Manhattan distance, Chebyshev distance, etc.), covariance, etc. The selected correlation metric may depend on the type of data, distribution characteristics, and specific application scenarios.

[0068] It should be noted that if a disease feature and a disease appear simultaneously in the same part of the above medical data (for example, the same medical record, the same medical order, etc.), it means that the disease feature may be related to the disease, that is, the patient with the disease feature may be diagnosed with the disease. Therefore, after extracting the disease feature and the disease from the medical data, the correlation measure between the disease feature and the disease can be calculated. However, if a disease feature and a disease do not appear simultaneously in the same part of the medical data, it means that the disease feature is not related to the disease. Therefore, in order to reduce the calculation amount of the correlation measure, the correlation measure between the disease feature and the disease may not be calculated.

[0069] In some embodiments, when extracting features from the medical data to extract disease features and diseases from the medical data, the medical data may be input into a large language model, and the large language model may extract features from the medical data to extract disease features and diseases from the medical data. It should be noted that the large language model at this time may refer to a service model of the large language model.

[0070] In practical applications, the above-mentioned large language model can be pre-trained on a large-scale, unlabeled text dataset using an unsupervised learning method to obtain a basic model of the large language model; further, the feature extraction task can be used as a supervised learning task during fine-tuning, and a medical text dataset specific to the feature extraction task can be prepared, so that the basic model of the large language model can be used as the starting point for fine-tuning, and a supervised learning method can be used to perform fine-tuning on a medical text dataset specific to the feature extraction task to obtain a service model of the large language model.

[0071] Take the medical record shown in Table 2 below as an example:

[0072] Table 2

[0073]

[0074] The diseases extracted from the medical records shown in Table 2 may include "pneumonia" and "chronic obstructive pulmonary disease", and the extracted disease features may include "male", "69 years old", "cough", "sputum", "shortness of breath", etc. In practical applications, the age may be adjusted to an age group as needed, for example, "69 years old" may be adjusted to "age group (60,70]".

[0075] It should be noted that if a disease feature appears less frequently in the above-mentioned medical data, it can be considered that this disease feature has no statistical significance. For example, this disease feature may be a special case that only a few patients have. Therefore, in some embodiments, when calculating the correlation measure between the extracted disease feature and the extracted disease, it is possible to specifically screen out the disease features whose number of appearances in the medical data reaches a preset threshold (which may be referred to as the first threshold, which may be a system default or manually set value, and the present application does not impose any special restrictions on this) from the extracted disease features, and calculate the correlation measure between the screened disease features and the extracted disease.

[0076] Since only the correlation measure between the screened disease features and the extracted diseases is calculated, the calculation amount of the correlation measure can be reduced, and the setting efficiency of the disease features corresponding to the diseases can be improved.

[0077] In some embodiments, when screening out disease features corresponding to various extracted diseases from extracted disease features based on the calculated correlation measure, specifically, after extracting the disease features and the diseases, the various extracted diseases can be taken as target diseases in turn, and a preset number (the value of the number can be a system default or manually set value, and the present application does not impose special restrictions on this) of disease features whose correlation measure with the target disease is the largest and reaches a preset threshold (which can be called a second threshold, which can be a system default or manually set value, and the present application does not impose special restrictions on this) are screened out from the extracted disease features as the disease features corresponding to the target disease.

[0078] For example, assuming that the value of the correlation measurement between the extracted disease feature A1 and a certain target disease extracted is 0.8, the value of the correlation measurement between the extracted disease feature A2 and the target disease is 0.9, the value of the correlation measurement between the extracted disease feature A3 and the target disease is 0.7, and the value of the correlation measurement between the extracted disease feature A4 and the target disease is 0.3, and assuming that the above-mentioned second threshold is 0.5, and the value of the above-mentioned preset number is 2, then since 0.9>0.8>0.7>0.5>0.3, disease feature A1 and disease feature A2 can be screened out from the four extracted disease features, and disease feature A1 and disease feature A2 can be used as disease features corresponding to the target disease.

[0079] By setting the disease features corresponding to the diseases based on the correlation measure, it is possible to ensure that there is a strong correlation between the corresponding diseases and the disease features, and to uniformly control the number of disease features corresponding to each disease.

[0080] Step 204: Generate a training sample set and its label based on the medical data, various diseases and their corresponding disease characteristics; wherein the training sample includes data indicating whether it has disease characteristics corresponding to the target disease, and the label of the training sample is used to indicate whether it is diagnosed as the target disease.

[0081] In this embodiment, when various diseases and their corresponding disease characteristics are determined, a training sample set and its labels can be generated based on the above medical data, various diseases and their corresponding disease characteristics. For any training sample in the generated training sample set, the training sample may include data indicating whether it has a disease characteristic corresponding to a certain disease (for ease of description, the disease is referred to as a target disease); and the label of the training sample can be used to indicate whether the training sample is diagnosed as the target disease.

[0082] For example, assuming that the disease characteristics corresponding to disease A include disease characteristics A1 and disease characteristics A2, and assuming that medical record 1 records that the patient has disease characteristics A1 and disease characteristics A2, and is diagnosed with disease A, medical record 2 records that the patient has disease characteristics A1 and disease characteristics B2, does not have disease characteristics A2, and is diagnosed with disease A, and medical record 3 records that the patient has disease characteristics B1 and disease characteristics B2, and is diagnosed with disease B, then the three medical data of medical records 1, medical records 2 and medical records 3 can be represented as the following Table 3:

[0083] Table 3

[0084] Medical data Disease features A1 Disease features A2 Disease features B1 Disease features B2 Disease A Disease B Medical Record 1 1 1 0 0 1 0 Medical Record 2 1 -1 0 1 1 0 Medical Record 3 0 0 1 1 0 1

[0085] As shown in Table 3, the value 1 indicates that the patient has the corresponding disease characteristics or is diagnosed with the corresponding disease, the value 0 indicates that the patient does not have the corresponding characteristics or is not diagnosed with the corresponding disease, and the value -1 indicates that the patient does not have the corresponding characteristics. In this case, the training samples generated based on medical record 1 may include the vector (1, 1, 0, 0), whose label is 10, the training samples generated based on medical record 2 may include the vector (1, -1, 0, 1), whose label is 10, and the training samples generated based on medical record 3 may include the vector (0, 0, 1, 1), whose label is 01.

[0086] Step 206: Perform cross-validation on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set.

[0087] In this embodiment, when the above training sample set is generated, cross-validation can be performed on the training sample set based on the machine learning model, so as to obtain the prediction result of the machine learning model for the training sample set. The prediction result of the machine learning model for the training sample set can be the prediction result of the machine learning model for each training sample in all the training samples included in the training sample set, or the prediction result of the machine learning model for each training sample in a part of the training samples included in the training sample set.

[0088] Cross-validation is a commonly used evaluation method in machine learning and statistical modeling. It is usually used to evaluate the generalization ability of a model, that is, the performance of the model on new data that has not been seen. The main purpose of cross-validation is to reduce the risk of overfitting and give an estimate of the performance of the model on unknown data.

[0089] The basic process of cross-validation includes: dividing the original dataset into two parts: training set and validation set; using a subset of the training set to train the model; using a subset of the validation set to evaluate the performance of the model; repeating the above steps for different combinations of training set / validation set subsets to ensure that the performance of the model is fully evaluated; summarizing the results of all validation stages and calculating the average performance indicators (such as accuracy, recall, etc.).

[0090] In the present application, the model obtained in the cross-validation process can be used to perform data screening on the validation set based on the prediction results of the validation set (i.e., the output of the model after the validation set is input into the model for calculation), so that the validation set after data screening can be used as a higher quality training set, so that a model with better performance can be trained using the training set.

[0091] Specifically, in some embodiments, when cross-validation is performed on the training sample set based on the machine learning model to obtain the prediction result of the machine learning model for the training sample set, the training sample set can be first divided into at least two training sample groups, and then at least one of the divided training sample groups is used as a verification sample set, so as to perform supervised training on the machine learning model based on the remaining training sample groups except the verification sample set, and the trained machine learning model is used to predict the verification sample set to obtain the prediction result of the machine learning model for the verification sample set. When all training sample groups are used as verification sample sets and the prediction result of the machine learning model for each training sample group is obtained, it can be considered that the prediction result of the machine learning model for each training sample in the training sample set is obtained.

[0092] For example, assuming that the above training sample set is divided into training sample group 1, training sample group 2 and training sample group 3, the above machine learning model can be supervised trained based on training sample group 2 and training sample group 3, and the trained machine learning model can predict each training sample in training sample group 1 to obtain the prediction result of the machine learning model for each training sample in training sample group 1; the above machine learning model can also be supervised trained based on training sample group 1 and training sample group 3, and the trained machine learning model can predict each training sample in training sample group 2 to obtain the prediction result of the machine learning model for each training sample in training sample group 2; the above machine learning model can also be supervised trained based on training sample group 1 and training sample group 2, and the trained machine learning model can predict each training sample in training sample group 3 to obtain the prediction result of the machine learning model for each training sample in training sample group 3. In this way, the prediction result of the machine learning model for each training sample in the above training sample set can be obtained.

[0093] Step 208: Based on the label and the prediction result, high-quality training samples whose prediction result matches the label are selected from the training sample set; wherein the high-quality training samples are used for supervised fine-tuning of the large language model.

[0094] In this embodiment, when the prediction result corresponding to the above training sample set is obtained, the training samples whose prediction result and label match can be screened out from the training sample set based on the label and prediction result corresponding to the training sample set. In the medical scenario, the correct data has a high degree of consistency, that is, when patients have the same disease characteristics, they are likely to be diagnosed with the same disease. Therefore, for the training samples whose prediction results of the model do not match the true labels, the consistency of the results cannot be guaranteed, which may be noise data caused by incomplete records or recording errors. In this case, the screened training samples can be used as high-quality training samples. Specifically, since the label is set for a single training sample, the prediction result is the prediction result of the above machine learning model for a single training sample. For the training sample set, it can be determined whether the label and prediction result corresponding to the training sample match (for example, whether the values ​​of the label and the prediction result are equal) for a single training sample in which there is a corresponding label and prediction result. If it matches, the training sample can be used as a high-quality training sample.

[0095] It should be noted that, when the above high-quality training samples are obtained, the large language model can be fine-tuned in a supervised manner based on these high-quality training samples, for example, the large language model in the intelligent dialogue system can be fine-tuned in a supervised manner. In this way, since high-quality training samples mean that they are more correct, the knowledge obtained by the large language model through learning these high-quality training samples is also more correct.

[0096] In some embodiments, when high-quality training samples whose prediction results and labels match are screened out from the training sample set based on the labels and the prediction results, specifically, for a single training sample in the training sample set that has its corresponding label and prediction result, according to the label of the training sample, it can be determined whether the prediction result of the machine learning model for the training sample is correct. If correct, the training sample can be determined as a high-quality training sample.

[0097] Alternatively, when high-quality training samples whose prediction results and labels match are screened out from the training sample set based on the above-mentioned labels and the above-mentioned prediction results, the loss value corresponding to the training sample can be calculated based on the label and prediction result corresponding to the training sample for a single training sample in the training sample set that has its corresponding label and prediction result, and a preset number of training samples with the smallest loss value (the value of the number can be a system default or manually set value, and the present application does not impose any special restrictions on this) are determined as high-quality training samples. Among them, the loss value of the training sample is an indicator to measure the difference between the model prediction output and the true label; the smaller the loss value, the closer the model's prediction is to the true value. In practical applications, the loss value can be calculated according to a preset loss function; the choice of loss function depends on the specific task type, for example: in regression tasks, the mean squared error (MSE) or the mean absolute error (MAE) can be used as the loss function; in classification tasks, the cross-entropy loss (Cross-Entropy Loss) can be used.

[0098] In the technical solution described above, a training sample set and its labels can be generated based on medical data containing disease characteristics and diagnosed diseases, as well as various diseases and their corresponding disease characteristics. A training sample in the training sample set may include data indicating whether it has disease characteristics corresponding to the target disease, and a label indicating whether it is diagnosed as the target disease. Subsequently, cross-validation can be performed on the training sample set based on the machine learning model to obtain the prediction results of the machine learning model for the training sample set, and high-quality training samples whose prediction results and labels match can be screened out from the training sample set; wherein the screened high-quality training samples can be used for supervised fine-tuning of the large language model.

[0099] By adopting the above method, the correctness of the high-quality training samples selected from the generated training sample set can be guaranteed, so as to ensure the correctness of the knowledge acquired by the large language model by learning these high-quality training samples; since these high-quality training samples are training samples related to the task of diagnosing diseases based on disease characteristics, after using these high-quality training samples to perform supervised fine-tuning on the large language model, the response accuracy of the large language model when diagnosing diseases based on disease characteristics can be improved. Furthermore, for the intelligent dialogue system containing the large language model, its response accuracy when diagnosing diseases based on disease characteristics can also be improved.

[0100] Please Figure 2 Based on the reference Figure 3 , Figure 3It is a flowchart of a training sample evaluation method for a large language model shown in an exemplary embodiment of the present application.

[0101] like Figure 3 As shown, the training sample evaluation method of the above large language model may include the following steps:

[0102] Step 302: Perform cross-validation on the high-quality training samples based on the machine learning model to obtain prediction results of the machine learning model for the high-quality training samples.

[0103] In this embodiment, by Figure 2 In the case where the above-mentioned high-quality training samples are obtained in the illustrated embodiment, the high-quality training samples can be evaluated to obtain an evaluation of the high-quality training samples, and the evaluation can help the user determine whether to use the high-quality training samples for fine-tuning the large language model.

[0104] Specifically, the machine learning model can be cross-validated on the high-quality training samples to obtain the prediction results of the machine learning model for the high-quality training samples. Figure 2 The process of performing cross-validation on the above training sample set in the illustrated embodiment will not be described in detail in this application.

[0105] Step 304: Based on the labels of the high-quality training samples and the prediction results of the machine learning model for the high-quality training samples, calculate the prediction accuracy corresponding to the high-quality training samples.

[0106] In this embodiment, when the prediction result corresponding to the above-mentioned high-quality training sample is obtained, the prediction accuracy corresponding to the high-quality training sample can be calculated based on the label and prediction result corresponding to the high-quality training sample. Specifically, for a single high-quality training sample with its corresponding label and prediction result among the high-quality training samples, it can be determined whether the label and prediction result corresponding to the high-quality training sample match (for example: based on the label of the high-quality training sample, it is determined whether the prediction result of the above-mentioned machine learning model for the high-quality training sample is correct). If so, it can be considered that the prediction result for the high-quality training sample is accurate, so that the proportion of high-quality training samples with accurate prediction results to all high-quality training samples can be calculated, and this proportion can be used as the prediction accuracy corresponding to the high-quality training sample.

[0107] It should be noted that if the prediction accuracy corresponding to the above high-quality training samples is high, it can be considered that the high-quality training samples can be used to fine-tune the large language model. However, if the prediction accuracy corresponding to the above high-quality training samples is low, it means that the correctness of the high-quality training samples obtained this time is not strong, and the correctness of the knowledge obtained by the large language model by learning the high-quality training samples cannot be guaranteed. Therefore, the high-quality training samples can be used not to fine-tune the large language model, but new high-quality training samples can be generated.

[0108] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0109] Please refer to Figure 4 , Figure 4 It is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 402, an internal bus 404, a network interface 406, a memory 408 and a non-volatile memory 410, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 402 reading the corresponding computer program from the non-volatile memory 410 into the memory 408 and then running it. Of course, in addition to the software implementation, one or more embodiments of the present application do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0110] Please refer to Figure 5 , Figure 5 It is a block diagram of a training sample generating device for a large language model shown in an exemplary embodiment of the present application.

[0111] The training sample generation device of the large language model can be applied to Figure 4 The device shown is used to implement the technical solution of the present application. The device includes:

[0112] An acquisition module 502 acquires medical data including disease characteristics and diagnosed diseases, and acquires various diseases and their corresponding disease characteristics;

[0113] A generating module 504 generates a training sample set and its label based on the medical data, various diseases and their corresponding disease characteristics; wherein the training sample includes data indicating whether it has a disease characteristic corresponding to a target disease, and the label of the training sample indicates whether it is diagnosed as the target disease;

[0114] A training module 506 performs cross-validation on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set;

[0115] The screening module 508 screens out high-quality training samples whose prediction results match the labels from the training sample set based on the labels and the prediction results; wherein the high-quality training samples are used for supervised fine-tuning of the large language model.

[0116] In some embodiments, the obtaining of various diseases and their corresponding disease characteristics includes:

[0117] Performing feature extraction on the medical data to extract disease features and diseases from the medical data;

[0118] Calculate correlation measures between extracted disease features and extracted diseases;

[0119] Based on the calculated correlation measure, disease features corresponding to various extracted diseases are screened out from the extracted disease features.

[0120] In some embodiments, the calculating of the correlation measure between the extracted disease features and the extracted diseases comprises:

[0121] Disease features whose occurrence times in the medical data reach a preset first threshold are screened out from the extracted disease features, and correlation metrics between the screened disease features and the extracted diseases are calculated.

[0122] In some embodiments, the step of selecting disease features corresponding to various extracted diseases from the extracted disease features based on the calculated correlation metric includes:

[0123] A preset number of disease features whose correlation measure with the extracted target disease is the largest and reaches a preset second threshold are screened out from the extracted disease features as the disease features corresponding to the target disease.

[0124] In some embodiments, the extracting features from the medical data to extract disease features and diseases from the medical data includes:

[0125] The medical data is input into a large language model, and the large language model performs feature extraction on the medical data to extract disease features and diseases from the medical data.

[0126] In some embodiments, the step of selecting high-quality training samples whose prediction results match the labels from the training sample set based on the labels and the prediction results includes:

[0127] Determine whether the prediction result of the machine learning model for each training sample is correct according to the label of each training sample, and determine the training sample with the correct prediction result as a high-quality training sample; or

[0128] Based on the labels and the prediction results, loss values ​​corresponding to the respective training samples are calculated, and a preset number of training samples with the smallest loss values ​​are determined as high-quality training samples.

[0129] In some embodiments, the cross-validation based on the machine learning model on the training sample set to obtain the prediction result of the machine learning model for the training sample set includes:

[0130] Dividing the training sample set into at least two training sample groups;

[0131] At least one training sample grouping is used as a validation sample set to perform supervised training on a machine learning model based on the remaining training sample groups except the validation sample set, and the trained machine learning model performs prediction on the validation sample set to obtain a prediction result of the machine learning model for the validation sample set.

[0132] In some embodiments, the apparatus further comprises:

[0133] A prediction module, performing cross-validation on the high-quality training samples based on the machine learning model to obtain a prediction result of the machine learning model for the high-quality training samples;

[0134] An evaluation module calculates a prediction accuracy rate corresponding to the high-quality training sample based on the label of the high-quality training sample and the prediction result of the machine learning model for the high-quality training sample.

[0135] For the device embodiment, it basically corresponds to the method embodiment, so the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of the present application.

[0136] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0137] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0139] Computer-readable media include permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0140] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0141] The above specific embodiments of the present application are described. Other embodiments are within the scope of the present application. In some cases, the actions or steps recorded in the present application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0142] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings. The term "and / or" refers to and includes any or all possible combinations of one or more associated listed items.

[0143] The description of the terms "one embodiment", "some embodiments", "example", "specific example" or "an implementation method" etc. used in one or more embodiments of the present application means that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic description of these terms is not necessarily for the same embodiment. Moreover, the specific features or characteristics described can be combined in a suitable manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without contradicting each other.

[0144] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0145] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0146] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A method for generating training samples for a large language model, the method comprising: Obtain medical data including disease characteristics and diagnosed diseases, and obtain various diseases and their corresponding disease characteristics; Based on the medical data, various diseases and their corresponding disease characteristics, a training sample set and its label are generated; wherein the training sample includes data indicating whether it has a disease characteristic corresponding to a target disease, and the label of the training sample is used to indicate whether it is diagnosed as the target disease; Performing cross-validation on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set; Based on the label and the prediction result, high-quality training samples whose prediction result matches the label are screened out from the training sample set; wherein the high-quality training samples are used for supervised fine-tuning of the large language model.

2. According to the method of claim 1, the obtaining of various diseases and their corresponding disease characteristics comprises: Performing feature extraction on the medical data to extract disease features and diseases from the medical data; Calculate correlation measures between extracted disease features and extracted diseases; Based on the calculated correlation measure, disease features corresponding to various extracted diseases are screened out from the extracted disease features.

3. The method according to claim 2, wherein the step of calculating a correlation measure between the extracted disease features and the extracted diseases comprises: Disease features whose occurrence times in the medical data reaches a preset first threshold are screened out from the extracted disease features, and correlation metrics between the screened disease features and the extracted diseases are calculated.

4. The method according to claim 2, wherein the step of selecting disease features corresponding to various extracted diseases from the extracted disease features based on the calculated correlation metric comprises: A preset number of disease features whose correlation measure with the extracted target disease is the largest and reaches a preset second threshold are screened out from the extracted disease features as the disease features corresponding to the target disease.

5. The method according to claim 2, wherein the step of extracting features from the medical data to extract disease features and diseases from the medical data comprises: The medical data is input into a large language model, and the large language model performs feature extraction on the medical data to extract disease features and diseases from the medical data.

6. The method according to claim 1, wherein based on the label and the prediction result, selecting high-quality training samples whose prediction result matches the label from the training sample set comprises: Determine whether the prediction result of the machine learning model for each training sample is correct according to the label of each training sample, and determine the training sample with the correct prediction result as a high-quality training sample; or Based on the labels and the prediction results, loss values ​​corresponding to the respective training samples are calculated, and a preset number of training samples with the smallest loss values ​​are determined as high-quality training samples.

7. The method according to claim 1, wherein cross-validation is performed on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set, comprising: Dividing the training sample set into at least two training sample groups; At least one training sample grouping is used as a validation sample set to perform supervised training on a machine learning model based on the remaining training sample groups except the validation sample set, and the trained machine learning model performs prediction on the validation sample set to obtain a prediction result of the machine learning model for the validation sample set.

8. The method according to claim 1, further comprising: Performing cross-validation on the high-quality training samples based on the machine learning model to obtain prediction results of the machine learning model for the high-quality training samples; Based on the labels of the high-quality training samples and the prediction results of the machine learning model for the high-quality training samples, the prediction accuracy corresponding to the high-quality training samples is calculated.

9. A training sample generating device for a large language model, the device comprising: An acquisition module, which acquires medical data including disease characteristics and diagnosed diseases, and acquires various diseases and their corresponding disease characteristics; A generation module, based on the medical data, various diseases and their corresponding disease characteristics, generates a training sample set and its label; wherein the training sample includes data for indicating whether it has a disease characteristic corresponding to a target disease, and the label of the training sample is used to indicate whether it is diagnosed as the target disease; A training module, performing cross-validation on the training sample set based on the machine learning model to obtain a prediction result of the machine learning model for the training sample set; A screening module is provided for screening high-quality training samples whose prediction results match the labels from the training sample set based on the labels and the prediction results; wherein the high-quality training samples are used for supervised fine-tuning of the large language model.

10. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 8 by running the executable instructions.

11. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions are executed by a processor to implement the method according to any one of claims 1 to 8.

Images