Medical dialogue text generation method and device based on large language model

Abstract

One or more embodiments of the invention provide a medical dialogue text generation method and device based on a large language model, and the method comprises the steps: obtaining a first type of medical dialogue text, and carrying out the feature extraction of the first type of medical dialogue text, so as to extract a first type of medical feature set from the first type of medical dialogue text; wherein the medical dialogue text comprises a question text and an answer text in a medical consultation process; acquiring a medical record text, and performing feature extraction on the medical record text to extract a second type of medical feature set from the medical record text; generating a training sample based on the first type of medical feature set and the first type of medical dialogue text, and performing fine tuning on the first large language model based on the training sample to obtain a dialogue generation large model; and inputting the second type of medical feature set into a dialogue generation large model, and generating a second type of medical dialogue text by the dialogue generation large model based on the second type of medical feature set.

Description

Technical Field

[0001] One or more embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a method and device for generating medical dialogue text based on a large language model. Background Art

[0002] Today, intelligent dialogue systems (IDSs) are widely used in a wide range of fields, including customer service, legal aid, online education, and medical consultation. Taking the rapidly developing field of medical consultation as an example, IDSs have become a key tool for improving the quality and efficiency of medical services. By simulating human communication, IDSs provide patients with consulting services, including disease diagnosis, treatment recommendations, and medication instructions.

[0003] An intelligent dialogue system is an interactive system developed using artificial intelligence technology. It is designed to understand and answer questions posed by users in natural language and generate concise and clear answers. Intelligent dialogue systems are typically based on a large language model (LLM), which understands and answers user questions and generates corresponding answers. Intelligent dialogue systems in the medical consultation field primarily rely on the knowledge acquired by the LLM during training by learning static medical-related corpus. Therefore, how to obtain a large amount of medical-related corpus to facilitate LLM learning has become a pressing issue. Summary of the Invention

[0004] One or more embodiments of the present application provide the following technical solutions:

[0005] This application provides a method for generating medical dialogue text based on a large language model, the method comprising:

[0006] Obtaining a first type of medical conversation text, and performing feature extraction on the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text; wherein the medical conversation text includes question text and answer text in a medical consultation process;

[0007] Acquiring a medical record text, and performing feature extraction on the medical record text to extract a second type of medical feature set from the medical record text;

[0008] Generating training samples based on the first type of medical feature set and the first type of medical conversation text, and fine-tuning the first large language model based on the training samples to obtain a large conversation generation model;

[0009] The second-category medical feature set is input into the dialogue generation model, and the dialogue generation model generates a second-category medical dialogue text based on the second-category medical feature set.

[0010] The present application also provides a device for generating medical dialogue text based on a large language model, the device comprising:

[0011] a first extraction module, which obtains a first type of medical conversation text and performs feature extraction on the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text; wherein the medical conversation text includes question text and answer text in the medical consultation process;

[0012] A second extraction module obtains a medical record text and performs feature extraction on the medical record text to extract a second type of medical feature set from the medical record text;

[0013] a fine-tuning module that generates training samples based on the first medical feature set and the first medical conversation text, and fine-tunes the first large language model based on the training samples to obtain a large conversation generation model;

[0014] A generation module inputs the second-category medical feature set into the dialogue generation model, and the dialogue generation model generates a second-category medical dialogue text based on the second-category medical feature set.

[0015] The present application also provides an electronic device, comprising:

[0016] processor;

[0017] a memory for storing processor-executable instructions;

[0018] The processor implements the steps of any of the above methods by running the executable instructions.

[0019] The present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0020] In the above technical solution, first, the first type of medical dialogue text and medical record text including the question text and answer text in the medical consultation process can be obtained, and feature extraction can be performed on the first type of medical dialogue text and the medical record text respectively, so as to extract the first type of medical feature set from the first type of medical dialogue text and the second type of medical feature set from the medical record text. Then, training samples can be generated based on the first type of medical feature set and the first type of medical dialogue text, and the first large language model can be fine-tuned based on the generated training samples to obtain a dialogue generation large model. Finally, the second type of medical feature set can be input into the dialogue generation large model, and the dialogue generation large model generates a second type of medical dialogue text based on the second type of medical feature set.

[0021] Using this approach, existing medical conversation text can be used to generate training samples based on the medical feature sets extracted from the medical conversation text and the medical conversation text itself. The large language model can then be fine-tuned based on the generated training samples to obtain a large conversation generation model. This large conversation generation model can then be used to restore medical records to possible medical conversation text based on the medical feature sets extracted from the medical records, thereby generating corresponding medical conversation text based on the medical records. If the existing medical conversation text is insufficient in quantity or content, the restored medical conversation text can be used to supplement it, ensuring a large amount of medical-related corpus for large language model learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The following is a description of the accompanying drawings required for describing the exemplary embodiments, in which:

[0023] Figure 1 It is a schematic diagram of an intelligent dialogue system shown in an exemplary embodiment of the present application.

[0024] Figure 2 This is a flowchart of a method for generating medical dialogue text based on a large language model, shown as an exemplary embodiment of the present application.

[0025] Figure 3 This is a flowchart of a medical consultation method based on a large language model shown in an exemplary embodiment of the present application.

[0026] Figure 4 This is a schematic diagram of a medical dialogue text generation and medical consultation process based on a large language model, shown as an exemplary embodiment of the present application.

[0027] Figure 5 It is a structural diagram of a device shown in an exemplary embodiment of the present application.

[0028] Figure 6 It is a block diagram of a medical dialogue text generation device based on a large language model, shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0029] The exemplary embodiments will be described in detail herein, with examples thereof being illustrated in the accompanying drawings. When the following description refers to the drawings, identical numbers in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of the present application. Instead, they are merely examples consistent with some aspects of one or more embodiments of the present application.

[0030] It should be noted that, in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this application. In some other embodiments, the method may include more or fewer steps than those described in this application. In addition, a single step described in this application may be broken down into multiple steps for description in other embodiments; and multiple steps described in this application may be combined into a single step for description in other embodiments.

[0031] In the field of medical consultation, intelligent dialogue systems have become a key tool for improving the quality and efficiency of medical services. By simulating human communication methods, intelligent dialogue systems provide patients with consulting services, including disease diagnosis, treatment recommendations, and drug instructions.

[0032] An intelligent dialogue system is an interactive system developed using artificial intelligence (AI) technology. It is designed to understand and answer questions posed by users in natural language and generate clear and concise answers. These systems are typically based on a large language model, which understands and answers user questions and generates corresponding answers.

[0033] Large language models are deep learning models trained using large amounts of text data. They can be used to generate natural language text or understand its meaning. Large language models can handle a variety of natural language tasks, such as text classification, named entity recognition, question-answering, and conversation, and are an important path to artificial intelligence.

[0034] In the field of natural language processing, large-scale text datasets are often referred to as corpuses. Corpuses can contain a variety of text data, such as literary works, academic papers, legal documents, news reports, everyday conversations, emails, and online forum posts. By learning from the text data in a corpus, large language models can acquire and understand the patterns and regularities of natural language, enabling effective processing and generation of human language.

[0035] Large language models typically use the Transformer architecture, meaning they are deep learning models based on the Transformer architecture. Transformer-based deep learning models are a type of neural network model that excels in fields like natural language processing.

[0036] Transformer is a neural network model used for sequence-to-sequence modeling. Transformer does not rely on recursive structures and can parallelize training and inference, speeding up model processing. In deep learning models based on the Transformer architecture, a multi-layer Transformer encoder is typically used to extract features from the input sequence, and a Transformer decoder is used to convert the extracted features into an output sequence. At the same time, such models typically also use a self-attention mechanism to capture long-distance dependencies in the input sequence, as well as residual connections and regularization methods to accelerate training and improve model performance.

[0037] A pretrained model is a large language model pretrained on large amounts of unlabeled text data. Pretrained models are general-purpose models; they are not designed or optimized for specific tasks. To adapt pretrained models to specific application scenarios and task requirements, they require fine-tuning to improve their performance on specific tasks. The large language model that is ultimately put into use is typically a pretrained model that has been further fine-tuned, performing supervised learning on labeled text data. Pretraining and fine-tuning are complementary processes: pretraining enables the model to acquire broad language understanding capabilities, while fine-tuning makes the model more specialized and accurate for specific tasks.

[0038] In other words, the training process of a large language model can be divided into two stages: pre-training and fine-tuning. During the pre-training stage, unsupervised learning (e.g., self-supervised learning) can be used on large-scale, unlabeled text datasets (e.g., online encyclopedias, online articles, books, etc.). Specifically, the model can predict missing parts or the next word based on the context, learn statistical laws such as semantics and syntax, and language structure. Backpropagation and optimization algorithms (e.g., gradient descent) are used to minimize prediction losses, iteratively update model parameters, and gradually improve the model's understanding of language. During the fine-tuning phase, you can select corresponding supervised learning tasks (for example, text classification, named entity recognition, question-answering systems, dialogue systems, etc.) based on the specific application scenarios and task requirements, and prepare task-specific text datasets. This allows you to use the pre-trained model as the starting point for fine-tuning, and fine-tune it on the task-specific text dataset using a supervised learning approach. Specifically, you can perform the task based on the text dataset, and minimize the loss used to measure the performance of the model in processing specific tasks through backpropagation and optimization algorithms (for example, gradient descent). The model parameters are iteratively updated to gradually improve the model's performance on specific tasks. In practical applications, fine-tuning can flexibly choose supervised learning, unsupervised learning, or semi-supervised learning methods based on the specific application scenarios and the type of available data.

[0039] It's important to note that a pre-trained large language model is typically referred to as the large language model's base model, and a fine-tuned large language model is referred to as the large language model's serving model. The language comprehension capabilities acquired by the large language model during pre-training and fine-tuning enable it to understand, analyze, and synthesize text information to perform logical inference, knowledge reasoning, and problem-solving when faced with complex problems or tasks. This capability is often referred to as the large language model's reasoning ability.

[0040] Large language models typically perform specific tasks under the guidance of prompt text (which can be called prompts). Prompt text is an initial text or text fragment provided to the large language model, intended to stimulate the model to produce corresponding output. Through prompt text, you can clearly tell the large language model what task you expect it to perform, such as: answering a question, simulating a conversation, writing an article, translating text, etc. At the same time, prompt text can provide the large language model with the necessary background information and context, enabling the large language model to understand the logic, style, theme or position that should be followed when generating content. In addition, prompt text can also inspire the large language model to demonstrate its inherent knowledge reserves or specific language capabilities, such as: explaining complex concepts, citing laws and regulations, imitating the writing style of a specific author, etc.

[0041] Intelligent dialogue systems in the medical consultation field mainly rely on the knowledge acquired by the large language model during the training process by learning static medical-related corpus. Therefore, how to obtain a large amount of medical-related corpus to facilitate the learning of the large language model has become an urgent problem to be solved.

[0042] One or more embodiments of the present application provide a technical solution for generating medical conversation text based on a large language model. In this technical solution, first, a first type of medical conversation text and medical record text including question text and answer text in a medical consultation process can be obtained, and feature extraction can be performed on the first type of medical conversation text and the medical record text respectively, so as to extract a first type of medical feature set from the first type of medical conversation text and a second type of medical feature set from the medical record text. Then, training samples can be generated based on the first type of medical feature set and the first type of medical conversation text, and the first large language model can be fine-tuned based on the generated training samples to obtain a conversation generation large model. Finally, the second type of medical feature set can be input into the conversation generation large model, and the conversation generation large model generates a second type of medical conversation text based on the second type of medical feature set.

[0043] Using this approach, existing medical conversation text can be used to generate training samples based on the medical feature sets extracted from the medical conversation text and the medical conversation text itself. The large language model can then be fine-tuned based on the generated training samples to obtain a large conversation generation model. This large conversation generation model can then be used to restore medical records to possible medical conversation text based on the medical feature sets extracted from the medical records, thereby generating corresponding medical conversation text based on the medical records. If the existing medical conversation text is insufficient in quantity or content, the restored medical conversation text can be used to supplement it, ensuring a large amount of medical-related corpus for large language model learning.

[0044] Please refer to Figure 1 , Figure 1 It is a schematic diagram of an intelligent dialogue system shown in an exemplary embodiment of the present application.

[0045] like Figure 1 As shown, the intelligent dialogue system may include a server and at least one client accessing the server via any type of wired or wireless network.

[0046] Among them, the above-mentioned server can correspond to a server including an independent physical host, or a server cluster composed of multiple independent physical hosts; or, it can correspond to a virtual server, cloud server, etc. carried by a host cluster.

[0047] The above-mentioned client can correspond to terminal devices such as smart phones, tablet computers, laptops, desktop computers, PCs (Personal Computers), PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smart watches, etc.), smart car devices or game consoles.

[0048] Users can use the intelligent question-and-answer service provided by the intelligent dialogue system through the client; the client and the server can implement user-oriented intelligent question-and-answer services through data interaction between each other.

[0049] For example, the client can output a corresponding user interface to the user, so that the user can perform operations such as inputting question text (which can be called a query or question) and uploading documents or pictures used to assist in asking questions in the user interface, so as to ask questions to the intelligent dialogue system and use the intelligent question-answering service provided by the intelligent dialogue system. The client can send the question text input by the user to the server, and the server will generate a corresponding answer text (which can be called an answer) for the question text and output the answer text to the user, that is, return the answer text to the client, and the client will display the answer text to the user through the user interface so that the user can view the corresponding answer generated by the intelligent dialogue system, thereby realizing a user-oriented intelligent question-answering service.

[0050] Specifically, the server can be equipped with a large language model, and the intelligent dialogue system can use this large language model as a basis to understand and answer the question text entered by the user and generate an answer text corresponding to the question text. It should be noted that the large language model in this case can refer to the service model of the large language model.

[0051] In practical applications, the constructed large language model can be pre-trained on a large-scale, unlabeled text dataset using an unsupervised learning approach to obtain a basic model of the large language model. Furthermore, the dialogue task can be used as a supervised learning task for fine-tuning, and a text dataset specific to the dialogue task can be prepared. This allows the basic model of the large language model to be used as the starting point for fine-tuning, and supervised learning can be used to perform fine-tuning on a text dataset specific to the dialogue task to obtain a serving model of the large language model.

[0052] In addition, the aforementioned server can also be equipped with other functional components or subsystems, such as feature extraction components, knowledge base retrieval components, and prompt text generation components. These components or subsystems can work in conjunction with the large language model installed on the server to jointly generate the answer text corresponding to the question text entered by the user.

[0053] To improve the adaptability and response accuracy of the intelligent dialogue system, the server can also host a knowledge base. This knowledge base is external to the large language model hosted on the server. That is, the data in this external knowledge base is not the knowledge acquired by the large language model during training. Instead, it serves as auxiliary data during the large language model's reasoning process, assisting the large language model in generating answers corresponding to user-entered questions.

[0054] Please refer to Figure 2 , Figure 2 This is a flowchart of a method for generating medical dialogue text based on a large language model, shown as an exemplary embodiment of the present application.

[0055] In this embodiment, the above-mentioned method for generating medical dialogue text based on a large language model can be applied to Figure 1 The server shown in FIG5 is used to generate training samples on the server and fine-tune the large language model in the intelligent dialogue system based on the generated training samples; alternatively, the above-mentioned method for generating medical dialogue text based on the large language model can be applied to other electronic devices with certain computing capabilities, and the training samples are generated by other electronic devices, and then the training samples are generated in the server. Figure 1 The server shown completes fine-tuning of the large language model in the intelligent dialogue system based on the generated training samples.

[0056] like Figure 2 As shown, the above-mentioned method for generating medical dialogue text based on a large language model may include the following steps:

[0057] Step 202: Obtain a first type of medical conversation text, and perform feature extraction on the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text; wherein the medical conversation text includes question text and answer text in the medical consultation process.

[0058] Step 204: Obtain a medical record text, and perform feature extraction on the medical record text to extract a second type of medical feature set from the medical record text.

[0059] In this embodiment, on the one hand, medical conversation texts (which may be referred to as first-category medical conversation texts) may be obtained. Medical conversation texts refer to records of conversations occurring within the medical and health field. These conversations typically occur between doctors and patients, but may also involve other medical professionals such as nurses and pharmacists. Specifically, medical conversation texts may include question texts and answer texts from the medical consultation process. These questions and answers reflect the content of the communication during the medical consultation process, such as: patient descriptions of symptoms, doctors' diagnostic recommendations, discussions of treatment plans, guidance on medication use, and rehabilitation recommendations.

[0060] The following is an example of a medical conversation text:

[0061] "Patient: I had a little bit of subtle swelling in my lower limbs during late pregnancy, but it went away after I got up. Now it's 15-16 days postpartum. I'm using a wrist-based blood pressure meter, but I'm not sure if it's accurate. My blood pressure is 116 / 72. Some parts of my lower limbs hurt when I press them. What could be the problem?"

[0062] Doctor: Are you still taking heparin? Considering your condition, you have a hypercoagulable state, also known as a prothrombotic state. Edema caused by hypertension usually subsides gradually after delivery and rarely worsens. You should be cautious about the following: 1. Lower limb thrombosis and 2. renal dysfunction. You can return to the postpartum clinic for a follow-up checkup, including blood pressure, urinalysis, renal function tests, and a color Doppler ultrasound of both lower limb vessels. It's also best to check for heart problems.

[0063] Patient: Heparin is not used now.

[0064] Doctor: Do as I said above and check for lower limb venous thrombosis.

[0065] Patient: It will take me another eight days to finish my confinement. Can I go for a check-up after I finish my confinement?

[0066] Doctor: No, if it is a lower limb venous thrombosis, it must not be delayed. The detachment of the blood clot may cause pulmonary embolism.

[0067] Patient: Can I make an appointment for the genetic or high-risk outpatient clinic?

[0068] Doctor: Yes.

[0069] Patient: If the examination shows lower limb venous thrombosis, how should it be treated?

[0070] Doctor: Let's talk about medication once we've confirmed it. This needs to be treated in the cardiology department."

[0071] On the other hand, medical record texts can be obtained. Among them, medical record texts refer to various written documents and electronic records generated in the process of healthcare services. These records usually record in detail the patient's health status, diagnosis results, treatment plan, operation details and other information related to the patient's health. Specifically, the types of medical record texts may include but are not limited to the following: outpatient records, inpatient medical records, test reports, prescription records; outpatient records are used to record the patient's condition during outpatient visits, including chief complaints, physical examination results, diagnosis and treatment recommendations, etc.; inpatient medical records are used to record all medical activities of the patient during hospitalization in the hospital, including admission records, medical records, surgical records, discharge summaries, etc.; test reports include laboratory test results, imaging examination results, etc.; prescription records are used to record the drugs prescribed by doctors and their dosage, usage and other information.

[0072] Table 1 below is an example of a medical record text (a medical record):

[0073] Table 1

[0074]

[0075]

[0076] In actual applications, in order to protect user privacy, the above-mentioned medical conversation text and the above-mentioned medical record text can both be open source and anonymized.

[0077] For the aforementioned first-category medical conversation texts, feature extraction can be performed on the first-category medical conversation texts to extract a medical feature set (referred to as a first-category medical feature set) from the first-category medical conversation texts. Specifically, feature extraction can be performed on each first-category medical conversation text, thereby obtaining a first-category medical feature set corresponding to each first-category medical conversation text.

[0078] Similarly, for the above-mentioned medical record text, feature extraction can be performed on the medical record text to extract a medical feature set (which can be called a second-category medical feature set) from the medical record text. Specifically, feature extraction can be performed on each medical record text, thereby obtaining a second-category medical feature set corresponding to each medical record text.

[0079] In some embodiments, the types of medical features in the first and second medical feature sets may be the same, so that corresponding medical dialogue texts can be subsequently generated based on each of the second medical feature sets. The medical feature sets may include, but are not limited to, one or more of the following types of medical features: gender, age, registered department, symptoms, examination, testing, past medical history, surgical history, medication history, and diagnosis.

[0080] To facilitate subsequent data processing and analysis, the aforementioned medical feature set can be organized into a predefined data model to form structured data. The most common way to organize structured data is as a table, where each column represents an attribute or field, and each row represents a record. Alternatively, structured data can be organized using key-value pairs (KVPs) or relational databases.

[0081] For example, the above medical feature set can be organized into a table as shown in Table 2 below:

[0082] Table 2

[0083]

[0084]

[0085] In some embodiments, when performing feature extraction on the first type of medical conversation text to extract the first type of medical feature set from the first type of medical conversation text, the first type of medical conversation text can be input into a feature extraction macromodel, which then performs feature extraction on the first type of medical conversation text to extract the first type of medical feature set from the first type of medical conversation text. It should be noted that the feature extraction macromodel in this case can be a service model of the macro language model used for feature extraction (which can be referred to as the third macro language model).

[0086] In practical applications, the third language model can be pre-trained on a large-scale, unlabeled text dataset using an unsupervised learning approach to obtain a base model of the third language model. Furthermore, the feature extraction task can be used as a supervised learning task during fine-tuning, and a medical text dataset specific to the feature extraction task can be prepared. Thus, the base model of the third language model can be used as a starting point for fine-tuning, and fine-tuned on a medical text dataset specific to the feature extraction task using a supervised learning approach to obtain a service model of the third language model as a large feature extraction model. The medical text dataset can refer to a text dataset collected in medical practice and used for various purposes (such as research, diagnosis, treatment, management, and policy making). These text data can come from patients' clinical data, medical records, laboratory test results, imaging data, biological sample analysis, and other healthcare-related data.

[0087] Similarly, in some embodiments, when performing feature extraction on the above-mentioned medical record text to extract the above-mentioned second category medical feature set from the medical record text, the medical record text can be input into a feature extraction model, and the feature extraction model can perform feature extraction on the medical record text to extract the second category medical feature set from the medical record text.

[0088] It should be noted that the feature extraction macromodel used for extracting features from the first type of medical conversation texts and the feature extraction macromodel used for extracting features from the medical record texts can be the same model. Thus, although the text content formats of medical conversation texts and medical record texts are different (the text content format of medical conversation texts is dialogue, while the text content format of medical record texts is usually tables), because the same feature extraction macromodel is used to extract features from both medical conversation texts and medical record texts, it is possible to ensure that the medical feature sets extracted from the medical conversation texts and the medical feature sets extracted from the medical record texts are consistent.

[0089] In some embodiments, in order to reduce the amount of training required to obtain the above-mentioned feature extraction large model and reduce the difficulty and complexity of obtaining the feature extraction large model, knowledge distillation (KD) can be used to extract knowledge related to feature extraction from the service model of a large language model that can provide more comprehensive services (including feature extraction services), and compress the knowledge into the basic model of another large language model, so that the basic model becomes the service model of the large language model that can provide feature extraction services. The service model of the large language model obtained at this time can be used as the feature extraction large model.

[0090] Knowledge distillation is a machine learning technique used to reduce the complexity and improve the efficiency of deep learning models. The basic idea behind knowledge distillation is to extract knowledge from a complex, large model (called the teacher model) and "compress" it into a smaller, simpler model (called the student model). The essence of knowledge distillation lies in knowledge extraction, knowledge transfer, and knowledge compression. Knowledge extraction refers to extracting effective decision-making information from the teacher model, knowledge transfer refers to transferring this information to the student model in a learnable form, and knowledge compression ensures that the student model can replicate the teacher model's behavior on a smaller scale.

[0091] Knowledge distillation primarily involves the following steps: First, a high-performing but potentially complex teacher model is trained. The teacher model is used to make predictions on the training data, generating soft labels. Soft labels are typically temperature-scaled probability distributions that provide more information than hard labels. A smaller student model is then trained to match the teacher model's soft labels as closely as possible. Both hard and soft labels may be used as part of the loss function. Finally, the student model's performance is evaluated, and any necessary adjustments are made to optimize its performance.

[0092] That is, you can first define a loss function to measure the difference between the student model's predictions and the teacher model's predictions. Soft targets are usually used to achieve this, that is, using the teacher model's output as the student model's target and calculating the cross-entropy loss or other similar loss functions. Subsequently, you can use the teacher model's predictions as auxiliary targets, combined with the student model's original targets for training, and update the student model's model parameters by iteratively minimizing the loss function.

[0093] Specifically, knowledge distillation can be performed on a preset service model (i.e., a service model of a large language model that can provide relatively comprehensive services, such as ChatGPT) that serves as a teacher model to extract knowledge related to feature extraction from the service model. This knowledge is then compressed into a large language model (which can be called a third large language model) that serves as a student model, thereby obtaining the aforementioned feature extraction model. When performing knowledge distillation on the service model, the training data used can be a medical text dataset, and its labels can be a set of medical features extracted from the medical text data.

[0094] Step 206: Generate training samples based on the first type of medical feature set and the first type of medical conversation text, and fine-tune the first large language model based on the training samples to obtain a large conversation generation model.

[0095] In this embodiment, when the above-mentioned first-category medical feature set is extracted from the above-mentioned first-category medical conversation text, a training sample can be generated based on the first-category medical feature set and the first-category medical conversation text, and the large language model (which can be called the first large language model) can be fine-tuned based on the training sample; at this time, the first large language model after fine-tuning can be called the conversation generation large model.

[0096] As previously mentioned, the purpose of fine-tuning the large language model is to improve its performance on specific tasks. In step 206, the specific task is a dialogue generation task, which involves generating medical dialogue text based on a medical feature set. Therefore, training samples specific to the dialogue generation task can be constructed based on the first type of medical feature set and the first type of medical dialogue text. The first large language model can then be fine-tuned based on these training samples to obtain the large dialogue generation model.

[0097] In some embodiments, if a supervised learning method is used to fine-tune the above-mentioned first large language model, then when generating training samples based on the above-mentioned first-category medical feature set and the above-mentioned first-category medical dialogue text, the first-category medical feature set can be used as the content of the training sample, and the first-category medical dialogue text can be used as the label of the training sample. It should be noted that, for a training sample, the content of this training sample can be the first-category medical feature set extracted from a first-category medical dialogue text, and the label of this training sample can be the first-category medical dialogue text itself. Accordingly, when fine-tuning the first large language model based on the training sample, the first large language model can be fine-tuned in a supervised manner based on the training sample and its label, thereby obtaining the above-mentioned large dialogue generation model.

[0098] Step 208: Input the second-category medical feature set into the dialogue generation macromodel, and the dialogue generation macromodel generates a second-category medical dialogue text based on the second-category medical feature set.

[0099] In this embodiment, once the aforementioned conversation generation macromodel is obtained, since the conversation generation macromodel can perform the task of generating medical conversation text based on a medical feature set, the aforementioned second-category medical feature set can be input into the conversation generation macromodel, and the conversation generation macromodel can generate medical conversation text (hereinafter referred to as second-category medical conversation text) based on the second-category medical feature set. It should be noted that for a second-category medical feature set, the conversation generation macromodel can generate a second-category medical conversation text based on this medical feature set. This achieves the generation of corresponding medical conversation text based on medical record text.

[0100] In the technical solution described above, first, the first type of medical conversation text and medical record text including the question text and answer text in the medical consultation process can be obtained, and feature extraction can be performed on the first type of medical conversation text and the medical record text respectively, so as to extract the first type of medical feature set from the first type of medical conversation text and the second type of medical feature set from the medical record text. Then, training samples can be generated based on the first type of medical feature set and the first type of medical conversation text, and the first large language model can be fine-tuned based on the generated training samples to obtain a large conversation generation model. Finally, the second type of medical feature set can be input into the large conversation generation model, and the large conversation generation model generates a second type of medical conversation text based on the second type of medical feature set.

[0101] Using this approach, existing medical conversation text can be used to generate training samples based on the medical feature sets extracted from the medical conversation text and the medical conversation text itself. The large language model can then be fine-tuned based on the generated training samples to obtain a large conversation generation model. This large conversation generation model can then be used to restore medical records to possible medical conversation text based on the medical feature sets extracted from the medical records, thereby generating corresponding medical conversation text based on the medical records. If the existing medical conversation text is insufficient in quantity or content, the restored medical conversation text can be used to supplement it, ensuring a large amount of medical-related corpus for large language model learning.

[0102] Please refer to Figure 3 , Figure 3 This is a flowchart of a medical consultation method based on a large language model shown in an exemplary embodiment of the present application.

[0103] In this embodiment, the above-mentioned medical consultation method based on the large language model can be applied to Figure 1 The server shown in the figure is based on the Figure 2 The second type of medical dialogue text described in the illustrated embodiment is for fine-tuning of a large language model in an intelligent dialogue system.

[0104] like Figure 3 As shown, the above-mentioned medical consultation method based on the large language model may include the following steps:

[0105] Step 302: Generate training samples based on the second type of medical conversation text, and fine-tune the second large language model based on the training samples to obtain a large medical consultation model.

[0106] In this embodiment, the above-mentioned second type of medical dialogue text can be obtained, and training samples can be generated based on the second type of medical dialogue text, and the large language model (which can be called the second large language model, specifically the large language model in the above-mentioned intelligent dialogue system) can be fine-tuned based on the training samples; at this time, the second large language model after fine-tuning can be called the medical consultation large model.

[0107] In actual applications, the second language model can also be fine-tuned based on the generated second-category medical dialogue text and the existing first-category medical dialogue text to obtain the medical consultation model.

[0108] As previously mentioned, the purpose of fine-tuning the large language model is to improve its performance on specific tasks. In step 302, the specific task is a dialogue system task, specifically enabling a medical consultation dialogue between an AI and a human. Therefore, based on the aforementioned second type of medical dialogue text, training samples specific to the dialogue system task can be constructed. Based on these training samples, the second large language model can be fine-tuned to obtain the aforementioned large medical consultation model.

[0109] In some embodiments, if supervised learning is used to fine-tune the second language model, when generating training samples based on the second type of medical conversation text, the question text in the second type of medical conversation text can be used as the content of the training sample, and the answer text in the second type of medical conversation text can be used as the label of the training sample. Accordingly, when fine-tuning the second language model based on the training samples, the second language model can be fine-tuned in a supervised manner based on the training samples and their labels, thereby obtaining the aforementioned large medical consultation model.

[0110] It should be noted that the generated second-category medical conversation text essentially describes the reasoning process of inferring the corresponding answer from a medical question. Therefore, the training samples generated based on the second-category medical conversation text can be considered chain-of-thought samples.

[0111] Thought chaining is a thinking style or problem-solving strategy, particularly used in artificial intelligence and cognitive science, that emphasizes demonstrating the step-by-step logical reasoning process for problem solving. Thought chaining can be incorporated into the training and use of large-scale language models, aiming to enhance the model's interpretability and ability to solve complex problems by explicitly expressing intermediate reasoning steps.

[0112] Thought chaining emphasizes breaking down the problem-solving process into a series of sequential, logically related steps. This involves identifying the key elements of the problem, listing possible solution paths, and evaluating the feasibility of each path until a final answer is reached. By including the thinking and reasoning behind each step in the model's input or output, the decision-making process becomes transparent, which is critical for understanding the model's decision logic and improving interpretability. Thought chaining aims to make machine thinking processes more human-like. By mimicking how humans think step by step to solve complex problems, it helps models make reasonable inferences in the absence of direct training data.

[0113] It should be noted that the specific implementation of the thought chain can involve constructing a training dataset containing intermediate reasoning steps, or directly embedding this step-by-step reasoning format in the Prompt of the large language model, guiding the large language model to generate the thinking process or reasoning steps behind it while generating the answer.

[0114] Thought chain samples can include key content such as problem statements, intermediate reasoning steps, and final answers, which are used to guide and train large language models for step-by-step logical reasoning and problem solving.

[0115] The problem statement is a specific problem or task description that needs to be solved. It can be a mathematical problem, logical reasoning, factual inquiry, or any inquiry that requires a series of thinking steps to arrive at an answer. In the sample thinking chain constructed in this embodiment, the problem statement can include expression text corresponding to a medical examination.

[0116] The intermediate reasoning step is the core part of the thinking chain sample, which includes a series of logical reasoning, calculations or analyses that start from the problem and gradually derive the answer. Each step should be coherent, and the previous step should reasonably lead to the next step until the final answer is reached. For example, when solving mathematical problems, this may involve formula application, variable substitution, calculation simplification, etc.; in logical reasoning tasks, it may include premise analysis, hypothesis verification, etc. In the thinking chain sample constructed in this embodiment, the intermediate reasoning step may include a descriptive text of the reasoning step of inferring the conclusion text corresponding to the medical examination from the performance text corresponding to the medical examination.

[0117] The final answer is the clear answer to the question after all intermediate reasoning steps. This answer should be the natural outcome of the reasoning chain and closely connected to the previous reasoning steps. In the sample thought chain constructed in this embodiment, the final answer can include the conclusion text corresponding to the medical examination.

[0118] In practical applications, thought chain samples can guide the model through complete examples of questions, intermediate reasoning steps, and answers, allowing it to learn to imitate such reasoning patterns. For example, when further constructing the prompt text of the large language model based on the constructed thought chain sample and the above-mentioned target expression text, the thought chain sample can be placed before the target expression text, and by adding contextual information, the large language model first learns the thought chain sample, learns the reasoning pattern shown by the thought chain sample, and then imitates this reasoning pattern, performs reasoning based on the target expression text, and outputs the reasoning steps in this reasoning process and the inferred conclusion text.

[0119] Step 304: Obtain the question text for medical consultation input by the user, and input the question text into the medical consultation big model, and the medical consultation big model generates an answer text based on the question text, and outputs the answer text to the user.

[0120] In this embodiment, once the aforementioned medical consultation macromodel is obtained, it can be used to execute medical-related dialogue system tasks. Specifically, a user-entered medical consultation question text can be obtained and input into the medical consultation macromodel. The macromodel then generates an answer text based on the question text, which can then be output to the user for review.

[0121] That is, in this application, restoring the medical record text into a medical conversation text can be regarded as a reasoning process of generating a diagnosis of the disease, treatment recommendations, and drug instructions from the gender, age, registration department, symptoms, examinations, tests, past medical history, surgical history, and medication history in the medical record text; and fine-tuning the large language model based on the restored medical conversation text, and using the medical consultation large model obtained after the fine-tuning to perform medical-related conversation tasks, can improve the response accuracy of the medical consultation large model to a certain extent.

[0122] It is important to note that the same large language model can be fine-tuned for different tasks. This approach is often called multi-task learning or multi-stage fine-tuning.

[0123] In multi-task learning, a pre-trained model is used on multiple related tasks, and the model is fine-tuned for each task. The benefit of this is that the model can learn richer representations from data from different tasks, which helps improve generalization capabilities.

[0124] Multi-stage fine-tuning involves first fine-tuning a pre-trained model on one or several related tasks, and then further fine-tuning the fine-tuned model on one or several new tasks. This approach helps the model better adapt to specific task domains.

[0125] In the present application, the first language model, the second language model and the third language model can be different large language models or the same large language model. They can be set according to actual needs, and the present application does not impose any feature restrictions on this.

[0126] like Figure 4 As shown, Figure 4 This is a schematic diagram of a medical dialogue text generation and medical consultation process based on a large language model, shown as an exemplary embodiment of the present application.

[0127] like Figure 4 As shown in the above process, first, knowledge distillation can be performed on ChatGPT as a teacher model to extract knowledge related to feature extraction from ChatGPT, and then the knowledge is compressed into a large language model as a student model to obtain a large feature extraction model.

[0128] Then, the above-mentioned feature extraction model can be used to perform feature extraction on the above-mentioned first-category medical conversation text and the above-mentioned medical record text respectively, so as to extract the above-mentioned first-category medical feature set from the first-category medical conversation text and extract the above-mentioned second-category medical feature set from the medical record text.

[0129] Then, training samples can be generated based on the above-mentioned first-category medical feature set and the above-mentioned first-category medical dialogue text, and the large language model can be fine-tuned based on the generated training samples to obtain a large dialogue generation model.

[0130] Then, the above-mentioned dialogue can be used to generate a large model, and the above-mentioned second-category medical dialogue text can be generated based on the above-mentioned second-category medical feature set.

[0131] Then, training samples can be generated based on the second type of medical conversation texts mentioned above, and the large language model can be fine-tuned based on the generated training samples to obtain a large medical consultation model.

[0132] Finally, the above-mentioned medical consultation model can be used to generate an answer text based on the question text input by the user, and the generated answer text can be output to the user to realize a dialogue with the user.

[0133] Corresponding to the aforementioned method embodiments, the present application also provides device embodiments.

[0134] Please refer to Figure 5 , Figure 5 : This is a structural diagram of a device shown in an exemplary embodiment of the present application. At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and of course may also include other required hardware. One or more embodiments of the present application can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to software implementation, one or more embodiments of the present application do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic module, and can also be hardware or logic devices.

[0135] Please refer to Figure 6 , Figure 6 It is a block diagram of a medical dialogue text generation device based on a large language model, shown as an exemplary embodiment of the present application.

[0136] The above-mentioned medical dialogue text generation device based on large language model can be applied to Figure 5 The device shown in the figure is used to implement the technical solution of this application. The device includes:

[0137] A first extraction module 602 acquires a first type of medical conversation text and performs feature extraction on the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text; wherein the medical conversation text includes question text and answer text in the medical consultation process;

[0138] The second extraction module 604 acquires a medical record text and performs feature extraction on the medical record text to extract a second type of medical feature set from the medical record text;

[0139] A fine-tuning module 606 generates training samples based on the first medical feature set and the first medical conversation text, and fine-tunes the first large language model based on the training samples to obtain a large conversation generation model;

[0140] The generation module 608 inputs the second-category medical feature set into the dialogue generation model, and the dialogue generation model generates a second-category medical dialogue text based on the second-category medical feature set.

[0141] In some embodiments, the apparatus further comprises:

[0142] a second fine-tuning module, which generates training samples based on the second type of medical conversation text and fine-tunes the second large language model based on the training samples to obtain a large medical consultation model;

[0143] The second generation module obtains the question text for medical consultation input by the user, and inputs the question text into the medical consultation big model, and the medical consultation big model generates an answer text based on the question text, and outputs the answer text to the user.

[0144] In some embodiments, generating training samples based on the first type of medical feature set and the first type of medical conversation text, and fine-tuning the first large language model based on the training samples to obtain a large conversation generation model includes:

[0145] The first type of medical feature set is used as a training sample, and the medical conversation text is used as a label of the training sample, so as to perform supervised fine-tuning on the first large language model based on the training sample and its label to obtain a large conversation generation model.

[0146] In some embodiments, the performing feature extraction on the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text includes:

[0147] The first category of medical conversation text is input into a feature extraction model, and the feature extraction model performs feature extraction on the first category of medical conversation text to extract a first category of medical feature set from the first category of medical conversation text.

[0148] In some embodiments, the performing feature extraction on the medical record text to extract a second type of medical feature set from the medical record text includes:

[0149] The medical record text is input into a feature extraction model, and the feature extraction model performs feature extraction on the medical record text to extract a second type of medical feature set from the medical record text.

[0150] In some embodiments, the apparatus further comprises:

[0151] The distillation module performs knowledge distillation on the preset service large model as the teacher model to extract knowledge related to feature extraction from the service large model, and compresses the knowledge into the third large language model as the student model to obtain the feature extraction large model.

[0152] In some embodiments, the medical feature set includes one or more types of medical features shown below: gender, age, registered department, symptoms, examination, test, past medical history, surgical history, medication history, and diagnosis.

[0153] For the device embodiments, they basically correspond to the method embodiments, so for relevant details, please refer to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the technical solution of this application.

[0154] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or physical devices, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0155] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0156] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0157] Computer-readable media include permanent and non-permanent, removable and non-removable media that can implement information storage by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0158] It should be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0159] The above description is of specific embodiments of the present application. Other embodiments are within the scope of this application. In some cases, the actions or steps described in this application can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0160] The terms used in one or more embodiments of the present application are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of the present application. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. The term "and / or" refers to and includes any or all possible combinations of one or more of the associated listed items.

[0161] The terms "one embodiment," "some embodiments," "example," "specific example," or "one implementation" used in one or more embodiments of the present application mean that the specific features or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present application. The schematic descriptions of these terms do not necessarily refer to the same embodiment. Moreover, the specific features or characteristics described can be combined in an appropriate manner in one or more embodiments of the present application. In addition, different embodiments and specific features or characteristics in different embodiments can be combined without conflict.

[0162] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0163] The above description is merely a preferred embodiment of one or more embodiments of the present application and is not intended to limit one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included in the scope of protection of one or more embodiments of the present application.

[0164] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Claims

1. A method for generating medical dialogue text based on a large language model, the method comprising: Obtaining a first type of medical conversation text, and performing feature extraction on the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text; wherein the medical conversation text includes question text and answer text in the medical consultation process; Acquire a medical record text, and perform feature extraction on the medical record text to extract a second type of medical feature set from the medical record text; Generate training samples based on the first type of medical feature set and the first type of medical dialogue text, and fine-tune the first large language model based on the training samples to obtain a large dialogue generation model; The second-category medical feature set is input into the dialogue generation model, and the dialogue generation model generates a second-category medical dialogue text based on the second-category medical feature set.

2. The method according to claim 1, further comprising: Generating training samples based on the second type of medical dialogue text, and fine-tuning the second language model based on the training samples to obtain a medical consultation model; A question text for medical consultation input by a user is obtained, and the question text is input into the medical consultation big model, the medical consultation big model generates an answer text based on the question text, and the answer text is output to the user.

3. The method according to claim 1, wherein generating training samples based on the first type of medical feature set and the first type of medical dialogue text, and fine-tuning the first large language model based on the training samples to obtain a large dialogue generation model, comprises: The first category of medical feature set is used as a training sample, and the medical conversation text is used as a label of the training sample, so as to perform supervised fine-tuning on the first large language model based on the training sample and its label to obtain a large conversation generation model.

4. According to the method of claim 1, the step of extracting features from the first type of medical conversation text to extract a first type of medical feature set from the first type of medical conversation text comprises: The first category of medical conversation text is input into a feature extraction large model, and the feature extraction large model performs feature extraction on the first category of medical conversation text to extract a first category of medical feature set from the first category of medical conversation text.

5. According to the method of claim 1, the step of performing feature extraction on the medical record text to extract a second type of medical feature set from the medical record text comprises: The medical record text is input into a feature extraction large model, and the feature extraction large model performs feature extraction on the medical record text to extract a second type of medical feature set from the medical record text.

6. The method according to claim 4 or 5, further comprising: Knowledge distillation is performed on the preset service big model as the teacher model to extract knowledge related to feature extraction from the service big model, and the knowledge is compressed into the third large language model as the student model to obtain the feature extraction big model.

7. According to the method according to claim 1, the medical feature set includes one or more types of medical features shown below: gender, age, registration department, symptoms, examination, test, past medical history, surgical history, medication history, and diagnosis.

8. A device for generating medical dialogue text based on a large language model, the device comprising: A first extraction module obtains a first type of medical dialogue text, and performs feature extraction on the first type of medical dialogue text to extract a first type of medical feature set from the first type of medical dialogue text; wherein the medical dialogue text includes question text and answer text in the medical consultation process; A second extraction module obtains a medical record text and performs feature extraction on the medical record text to extract a second type of medical feature set from the medical record text; A fine-tuning module, which generates training samples based on the first type of medical feature set and the first type of medical dialogue text, and fine-tunes the first large language model based on the training samples to obtain a large dialogue generation model; A generation module inputs the second-category medical feature set into the dialogue generation model, and the dialogue generation model generates a second-category medical dialogue text based on the second-category medical feature set.

9. An electronic device, comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

10. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions are executed by a processor to implement the method according to any one of claims 1 to 7.

Images