Follow-up data collection method and system based on large language model and knowledge distillation

Abstract

This invention discloses a follow-up data collection method and system based on a large language model and knowledge distillation. The method includes the following steps: generating prompts based on items in the follow-up form and inputting them into a large language model; sending the generated questions to the user and obtaining the user's answers; filtering relevant dialogue history based on the similarity between short-term and long-term dialogue history; using the large language model as the teacher model and generating a labeled dialogue dataset; performing knowledge distillation on the intent extraction language model (used as the student model) in conjunction with downstream tasks; extracting user intent using the knowledge-distilled intent extraction language model based on short-term dialogue history, relevant dialogue history, and item information; and collecting follow-up data item by item until the entire follow-up form is completed. This invention enables follow-up data collection with high dialogue efficiency, high data quality, and low computational overhead, and is applicable to various medical data collection application scenarios, such as outpatient follow-up data collection.

Description

Technical Field

[0001] This invention belongs to the field of medical data acquisition technology, specifically relating to a follow-up data acquisition method and system based on large language models and knowledge distillation. Background Technology

[0002] Patient follow-up data objectively reflects patients' subjective feelings about treatment and is widely used in various clinical applications such as drug evaluation and adverse event monitoring. Early studies collected follow-up information through paper forms; however, this method faced problems with data consistency and the immediacy of interaction. With the popularization of electronic interactive media such as personal computers and smartphones, electronic form follow-up data collection systems have effectively overcome previous difficulties by interacting with patients on mobile platforms and storing data in a structured manner. However, the overuse of electronic forms has led to patient fatigue with this type of interaction, and the lack of sufficient feedback from one-way input of information to maintain patient compliance with the system. In addition, some follow-up content involving privacy may cause patient resistance or confusion, further reducing patient compliance without clarification.

[0003] In recent years, the rise of low-code chatbot building platforms such as Dialogueflow has made it possible to construct rule-based question-and-answer dialogue systems for follow-up data collection. By manually defining rules and knowledge bases, these systems can help patients obtain additional information, encouragement, and feedback during the question-and-answer process. Some studies suggest that dialogue systems can simplify operation and facilitate the establishment of deep relationships with users, thereby improving user compliance and data quality. However, rule-based dialogue systems require the manual construction of sufficiently large rule systems and knowledge bases for specific goals to complete the intelligent question-and-answer process, and the initial manual costs further hinder the widespread adoption of this method. For example, Giuseppe Fenza's HealthConversational Agents, built in 2023, was used to collect patient-reported outcome (PRO) follow-up data. This system requires researchers to predefine a set of virtual decision trees to advance the dialogue content, and it also requires training multiple natural language processing components such as intent recognition, entity extraction, and response generation to connect the user's natural language input with the virtual decision trees. This means that the dialogue system requires a large investment of manpower in the early stages. In addition, due to the limited number of decision branches, the system may fail to understand the input and needs to use backup intents to handle unexpected events, making it difficult to simulate real dialogue scenarios.

[0004] The emergence of Large Language Models (LLMs) has brought new generative solutions to follow-up data collection dialogue systems. The dramatic increase in network size and corpus knowledge enables LLMs to correctly understand complex natural language text and provide responses approaching those of professional physicians on medical datasets. Driven by the collection of follow-up form information, LLMs can proactively communicate with patients, allowing for seamless data collection during conversations. More importantly, this process eliminates the need to build complex rules and knowledge bases for specific clinical applications, significantly reducing the upfront development costs of chatbots. However, LLMs cannot proactively initiate communication, lack sustained attention to complex forms during long, multi-turn dialogues, and cannot directly interface dialogue content with structured follow-up databases. Therefore, further research is needed to develop follow-up data collection dialogue systems based on LLMs.

[0005] In the field of follow-up data collection, Jing Wei et al. attempted to use cue engineering to collect users' self-reported information based on GPT-3 in 2023. This study focused on analyzing the impact of different cues on language model generation questions and summarizing the errors that GPT-3 may make when generating self-reported questions. It explored the design of cue engineering. However, their work was limited to 4 to 5 self-reported questions, lacked attempts on large language models, and did not consider the interaction between the dialogue system and the database. Therefore, it lacked guidance for the construction of dialogue systems with longer follow-up forms (>10 items).

[0006] In summary, there is an urgent need for a system construction method to drive large language models to accurately and efficiently collect complex patient follow-up data through natural language interaction. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a follow-up data collection method and system based on a large language model and knowledge distillation. The method obtains the dialogue history between the large language model and user questions and answers based on the follow-up form, and extracts user intent based on the dialogue history through the intent extraction language model obtained by knowledge distillation, thereby completing the data collection of the follow-up form and improving the efficiency, interactivity and data quality of patient outpatient follow-up data collection.

[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0009] In a first aspect, the present invention provides a follow-up data collection method based on a large language model and knowledge distillation, comprising the following steps:

[0010] The follow-up form is broken down into a set of items, where each item contains a follow-up question and a list of corresponding options;

[0011] Based on the follow-up questions and their corresponding option lists, questions are generated and prompts are input into a large language model. The generated natural language questions are then sent to users and their answers are obtained.

[0012] The natural language question and user answer corresponding to each entry are used as short-term dialogue history. The short-term dialogue history of all entries is spliced ​​together to form long-term dialogue history. The short-term dialogue history is filtered according to the similarity between the short-term dialogue history and the long-term dialogue history to obtain relevant dialogue history.

[0013] The large language model is used as the teacher model to generate an annotated dialogue dataset. The intention extraction language model, which is used as the student model, is then subjected to knowledge distillation in conjunction with downstream tasks.

[0014] Based on short-term dialogue history, related dialogue history, and entry information, the intent extraction language model after knowledge distillation is used to extract user intent, and a response generation prompt input large language model is constructed to generate natural language responses and send them to users, thereby realizing one-time follow-up data collection from the follow-up form.

[0015] Preferably, the question generation prompts include personality prompts and item prompts. The personality prompts are used to construct a questioner with specific personality traits and generate questions according to preset rules. The preset rules include objectives, communication methods, and precautions. The item prompts are used to determine the identifier and attributes of each question. The identifier includes the question content and the list of options, and the attributes include whether it is a mandatory question.

[0016] Preferably, the step of filtering short-term dialogue history based on the similarity between short-term and long-term dialogue history to obtain relevant dialogue history includes:

[0017] The SimCSE language model is used to encode short-term and long-term dialogue history into short-term dialogue history embedding vectors and long-term dialogue history embedding vectors. The similarity between the short-term and long-term dialogue history embedding vectors is calculated to obtain a similarity matrix. Based on the similarity matrix, a threshold is defined to filter the short-term dialogue history and obtain the relevant dialogue history.

[0018] Preferably, the step of using a large language model as a teacher model to generate an annotated dialogue dataset, and combining this with downstream tasks to perform knowledge distillation on the intent extraction language model used as a student model, includes:

[0019] Using the large language model as the teacher model, virtual doctor and virtual patient dialogues based on the large language model are constructed for all entries according to the follow-up questions and their corresponding option lists contained in each entry, resulting in an unlabeled dialogue dataset.

[0020] First, annotate a small number of dialogue datasets based on the unlabeled dialogue dataset. Then, use these small number of dialogue datasets as examples and combine them with item information to construct intent extraction prompts through context learning. This enables the large language model to extract the intent in the dialogue based on the examples and complete the annotation of the unlabeled dialogue dataset based on the intent, thus forming an annotated dialogue dataset.

[0021] An intent extraction language model dataset containing structured input is constructed based on an annotated dialogue dataset. The structured input includes dialogue history, task description, and special tags, including the beginning of the input, sentence segmentation points, and the location of the prediction target.

[0022] Based on the intent extraction language model dataset, the intent for predicting target location is predicted, and the intent extraction language model, which serves as the student model, is subjected to knowledge distillation in conjunction with downstream tasks to obtain a knowledge-distilled intent extraction language model, which is used for automatic intent extraction.

[0023] Preferably, the downstream tasks include multi-label classification tasks, semantic matching tasks, and contrastive learning tasks.

[0024] Preferably, the intent includes a list of options, skipping, or ambiguity.

[0025] Preferably, the response generation prompts include personality prompts, urging prompts, item prompts, and dialogue history prompts. Personality prompts are used to construct a questioner with specific personality traits and generate questions. Urging prompts are used to urge users to answer form items as soon as possible. Item prompts are used to determine the identifier and attributes of each question. The identifier includes the question content and the list of options. The attributes include whether it is a mandatory question. Dialogue history prompts are used to introduce short-term dialogue history and related dialogue history.

[0026] Secondly, in order to achieve the above-mentioned objectives, the present invention also provides a follow-up data collection system based on a large language model and knowledge distillation, including: a follow-up form preprocessing module, a large language model question-and-answer module, a related dialogue history filtering module, an intent extraction language model knowledge distillation module, and a user intent extraction and response module;

[0027] The follow-up form preprocessing module is used to decompose the follow-up form into a set of several items, where each item contains a follow-up question and its corresponding option list;

[0028] The large language model question answering module is used to construct questions and generate prompts based on follow-up questions and their corresponding option lists, input them into the large language model, send the generated natural language questions to users, and obtain user answers.

[0029] The relevant dialogue history filtering module is used to take the natural language question and user answer corresponding to each entry as short dialogue history, splice the short dialogue history of all entries as long dialogue history, and filter the short dialogue history according to the similarity between the short dialogue history and the long dialogue history to obtain relevant dialogue history.

[0030] The intent extraction language model knowledge distillation module is used to use a large language model as a teacher model and generate an annotated dialogue dataset, and to perform knowledge distillation on the intent extraction language model as a student model in conjunction with downstream tasks.

[0031] The user intent extraction and response module is used to extract user intent based on short-term dialogue history, related dialogue history and item information, using an intent extraction language model obtained after knowledge distillation, and to build a response generation prompt input large language model to generate a natural language response and send it to the user, thereby realizing one-time follow-up data collection of the follow-up form.

[0032] Thirdly, to achieve the above-mentioned objectives, embodiments of the present invention also provide a follow-up data acquisition device based on large language models and knowledge distillation, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the above-mentioned follow-up data acquisition method based on large language models and knowledge distillation when the computer program is executed.

[0033] Fourthly, to achieve the above-mentioned objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-mentioned follow-up data collection method based on a large language model and knowledge distillation.

[0034] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0035] (1) This invention decomposes the follow-up form into a set of several items and constructs a question generation prompt to drive the large language model to generate questions and obtain user answers. This enables the large language model to actively initiate communication, help users collect data imperceptibly in the dialogue, and supports the construction of response generation prompts to drive the large language model to generate responses and send them to users, thereby enhancing interactivity.

[0036] (2) This invention designs a context-based prompting engineering, which adds prompt words, examples and dialogue history information to the language model input. It can generate highly accurate question and answer content based on any follow-up form, and can generate dialogue content that conforms to the clinical application environment. It can maintain continuous attention to complex forms during long form multi-turn dialogues, thereby improving dialogue quality and efficiency.

[0037] (3) This invention uses a large language model as the teacher model and an intent extraction language model as the student model. It trains the intent extraction language model offline based on the constructed dataset and obtains the knowledge-distilled intent extraction language model, thereby achieving automatic extraction of intent. It can extract intent with high precision and efficiency and form a high-quality structured follow-up database.

[0038] (4) The intent extraction language model obtained by the knowledge distillation of the present invention can achieve efficient reasoning under low cost conditions, which greatly reduces the computational and time costs. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the follow-up data collection method based on a large language model and knowledge distillation provided in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the intent extraction language model knowledge distillation process provided in an embodiment of the present invention;

[0042] Figure 3 This is a schematic diagram of the follow-up data acquisition system based on large language model and knowledge distillation provided in an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0044] The inventive concept of this invention is as follows: Addressing the problems of existing large language models in collecting data from complex follow-up forms—namely, their lack of proactive question-and-answer capabilities, their lack of sustained attention to complex forms during multi-turn dialogues in long forms, and their inability to connect dialogue content with a structured follow-up database—this invention provides a follow-up data collection method and system based on large language models and knowledge distillation. This method leverages the natural language understanding capabilities of large language models and the ability to transfer information extraction capabilities from large language models to smaller models using knowledge distillation algorithms. Through context-based prompting engineering, the large language model can generate highly accurate question-and-answer content suitable for clinical applications based on any follow-up form. The intent extraction language model trained through knowledge distillation can extract intent with high precision and efficiency, forming a high-quality structured follow-up database. This achieves high dialogue efficiency, high data quality, and low computational overhead in follow-up data collection, making it suitable for various medical data collection application scenarios, such as outpatient follow-up data collection.

[0045] In this embodiment, cue engineering refers to a method of guiding a language model to complete a specific task by designing the language model's input structure. Context learning refers to a cue engineering design method that drives a language model to complete a specific downstream task by incorporating information such as cue words, examples, and dialogue history into the language model's input.

[0046] In this embodiment, knowledge distillation refers to a neural network training method that transfers the output distribution of a larger-scale network on one or more downstream tasks to a smaller-scale network. Offline knowledge distillation refers to passing knowledge to a student model in the form of logits or intermediate features to guide the student model's training. Offline knowledge distillation for non-open-source large language models refers to using the large language model as a teacher model to generate or label datasets, and then using the datasets to fine-tune the student model to achieve similar results on specific downstream tasks.

[0047] In this embodiment, cosine similarity is a method for measuring label similarity in multi-label classification problems. In two-dimensional space, cosine similarity can be regarded as the cosine value of the angle between two vectors in vector space. Extending this concept to high-dimensional space, by calculating the similarity between pairs of labels, the final label cosine similarity matrix can be obtained. By converting the text into vector embedding through a language model, and then calculating the similarity between vectors, semantically similar text content can be effectively identified.

[0048] Part One Figure 1This is a flowchart illustrating the follow-up data collection method based on a large language model and knowledge distillation provided in this embodiment of the invention. Because existing large language models can only passively generate answers based on user input and cannot actively ask questions, and because the dialogue history grows continuously during the collection of long forms, the model suffers from a lack of central attention, making the dialogue unstable and leading to problems such as repeated or missed questions. Furthermore, the unlimited growth of the dialogue history may exceed the maximum length of the language model input, inevitably requiring pruning and resulting in missing context. To address these problems, such as... Figure 1 As shown, this embodiment of the invention provides a follow-up data collection method based on a large language model and knowledge distillation. The follow-up forms selected are Fact-B (The Functional Assessment of Cancer Therapy-Breast), BIBCQ (psychometric features of the Body Image after Breast Cancer Questionnaire), and TMI (Testing Morbidities Index), all widely used in the field of breast cancer, and are used as the follow-up targets for data collection. The specific steps include:

[0049] S101, decompose the follow-up form F in the embodiment into item I = {q I O I An entry is defined as containing only one follow-up question q. I List of options corresponding to follow-up questions The set of n, where n I This represents the total number of options in the entry. Therefore, a follow-up form F can be viewed as a set consisting of several entries. Where n represents the total number of items in the follow-up form. Based on this division, the dialogue system collects information from only one item at a time, moving on to the next item after collection is complete. Therefore, for each item's information collection process, the dialogue system and the user will form a multi-turn dialogue history H. s This is defined as short-term dialogue history. The dialogue history of all entries is pieced together to form the entire follow-up form's dialogue history. This is defined as long-term dialogue history. In this embodiment, the Fact-B, BIBCQ, and TMI forms are implemented by splitting them into 36, 44, and 12 entries, respectively.

[0050] S102, based on item I obtained in S101, incorporate follow-up questions q using the prompting process. I List of options corresponding to follow-up questions O IBuild a problem-generating tooltip Pr q Pr q Requires a large language model M L Generate a natural language question Q=M based on the entry information. L (Pr q The question is sent to the user, and the user's answer A is obtained. In this example, the question prompt Pr is generated. q The construction method is as follows: "Personality prompt: You are a breast surgeon and you need to collect ePRO (electronic Patient Reported-Outcome) data to improve the quality of healthcare and patients' quality of life. Communicate with users using Simplified Chinese. Note: Do not arbitrarily change the content of the question when asking it. Entry prompt: Question content: q" I ; Options list: O I "This question is (not) a mandatory question." The phrase "This question is (not) a mandatory question" indicates that the requirement depends on whether the item itself is mandatory. The large language model M used in this example... L ChatGLM4, released by Beijing Zhipu Huazhang Technology Co., Ltd., obtains large language model services through a legitimate API. User response A is returned as free text, without involving emojis, audio, images, or other multimodal information.

[0051] S103, Update the short conversation history H based on the user's response A obtained in S102. s ={H s ,(Q,A)}。 Short-term dialogue history H s It is a list of question-and-answer content stored in JSON format. The JSON format for question-and-answer pairs is "{"role":"system / assistant / user","content":"prompt / assistant content / user content"}". The "role" for the prompt message "prompt" is "system", the "role" for the model-generated content "assistantcontent" is "assistant", and the "role" for the user's answer message "user content" is "user". Therefore, when the dialogue is updated, the short-term dialogue history H... s The model-generated content and user responses will be added to the list in chronological order.

[0052] S104, based on the short dialogue history H obtained in S103 s Using the language model SimCSE to analyze short-term dialogue history H s and a long history of dialogue H lEncoded as an embedding vector and Where n t This represents the total number of questions and answers answered up to the current dialogue. Calculate the short-term dialogue history embedding vector. and long-term dialogue history embedding vector Each dialogue history The cosine similarity yields the similarity matrix. In the example, list H s and H l Convert the data to a string in the format "role:content" and input it into SimCSE to convert it into a 1024-dimensional embedding vector. The formula for calculating cosine similarity is... In this embodiment, correspond and Corresponding long-term dialogue history embedding vector Each dialogue history

[0053] S105, Based on the similarity matrix Sim obtained in step S104, define a threshold λ to filter similar dialogue history and obtain relevant dialogue history. In this embodiment, through multiple trials, the threshold λ was defined as 0.8 based on experience.

[0054] S106, Based on the short-term dialogue history H obtained from S103 and S105 s and related dialogue history H r Combining item information I, the intent extraction language model M is used. S Extracting user intent T=M S (H s H l Intent T includes a list of options O. I And "skip" and "ambiguous". Intent extraction language model M S It is obtained through knowledge distillation training. The intent extraction language model M selected in this embodiment is... S This is a multilingual version of the BERT model (bert-base-multilingual-cased). The knowledge distillation process will be explained in detail in Part 2.

[0055] S107, based on the short dialogue history H obtained from S101, S103 and S105 s Related dialogue history H r Item information I constructs response generation prompts Pr r The requirement is a large language model M L Generate natural language responses R=M L (Pr rThis is to respond to the user's interactions with the dialogue system during the conversation and obtain the user's response A. The response generates a prompt Pr. r The construction method is as follows: "Personality prompt: You are a breast surgeon and you need to collect ePRO (electronic Patient Reported-Outcome) data to improve the quality of healthcare and patients' quality of life. Urging prompt (optional): Please urge users to answer the form items as soon as possible. Item prompt: Question content: q" I ; Options list: O I This question is (not) mandatory. Dialogue history hint: H s H r The prompt will be added at the beginning of the third round of dialogue, so that the system can maintain its attention to the target of the item collection.

[0056] S108: Based on the intent T extracted in S106, if intent T is "unclear" or intent T is "skip" and the current entry is not allowed to be skipped, then return to S103. Otherwise, proceed to S109 for judgment.

[0057] S109. If the follow-up form F has not been completed, proceed to S110; otherwise, proceed to S111.

[0058] S110, add short-term dialogue history to long-term dialogue history H l ={H l H s Clear short conversation history H s =φ, where φ represents the empty set, and the system returns S102.

[0059] S111, Follow-up form F completed data collection, the dialogue system ends, and the dialogue is closed.

[0060] The second part, extracting structured follow-up information from the dialogue history, is crucial for the advancement of the dialogue system. However, using a large language model combined with prompt engineering to extract information incurs significant computational and time costs, is sensitive to prompt content, and generates unstable content, making it difficult to apply in real-world scenarios. Furthermore, traditional intent recognition models require specific datasets to construct supervision signals for training; however, for arbitrary follow-up forms, it is currently impossible to find real-world dialogue datasets for annotation and training. To address this issue, in step S106 of this embodiment, an offline knowledge distillation method is used to train the intent extraction language model M. S ,like Figure 2 As shown, the specific steps for conducting knowledge distillation based on the multilingual version of the BERT model are as follows:

[0061] S201, according to the entry I = {q} defined in S101 I OI}, Add "Skip" and "Unclear" to the options list to get an expanded options list. In this embodiment, the option lists of Fact-B, BIBCQ, and TMI forms contain 7, 12, and 9 options, respectively.

[0062] S202, according to q in S201 I and O I Build Doctor Tips Pr D The requirement is a large language model M L D, playing the role of a doctor, asks questions to patients t Using prompt engineering to build patient prompts Pr P The requirement is a large language model M L Playing the role of a patient, answering the doctor's questions. t Connecting virtual doctors and virtual patients to conduct multi-round dialogues and creating a dialogue history. D0 = P0 = φ, where t represents the t-th round of dialogue. The question D posed by the virtual doctor... t =M L (P D ,P I M t_r ), The virtual patient's answer P t =M L (P P M t ), Dia t_r Indicating the history of the dialogue Dia t The values ​​are flipped to fit the dialogue context of the virtual doctor. Virtual doctor-patient dialogues are constructed for all entries, resulting in an unlabeled dialogue dataset. In this embodiment, the large language model M is selected. L ChatGLM4, released by Beijing Zhipu Huazhang Technology Co., Ltd., obtains large language model services through a legitimate API. (Doctor's suggestion: Pr) D The construction method is as follows: "Personality prompt: You are a breast surgeon and you need to collect ePRO (electronic Patient Reported-Outcome) data to improve the quality of healthcare and patients' quality of life. Communicate with users using Simplified Chinese. Note: Do not arbitrarily change the content of the question when asking it. Entry prompt: Question content: q" I ; Options list: O I This question is (not) mandatory. Dialogue history hint: Dia t_r The patient indicated PrP The construction method is as follows: "Personality Tip 1: Imagine you are a patient and I am a doctor. I need to collect follow-up data from you. Please do not directly select options or include option content in your answer. You need to answer the question indirectly and express your needs at the appropriate time. Personality Tip 2: Imagine you are a patient and I am a doctor. I need to collect follow-up data from you. You do not want to answer the doctor's questions right now. Please make up a reason to persuade the doctor to give up asking questions. Dialogue History Tip: Dia..." t The patient indicated Pr P Two personality cues were selected to generate content, with only one of them appearing in the cues, to increase the sample balance of patient responses and ensure a balanced proportion of "skip" intentions in the experimental dataset. t and Dia t_r Construction Format and Short Dialogue History H s Consistent.

[0063] S203, based on the unlabeled dialogue dataset C obtained from S202 un First, a small number of dialogue datasets were manually annotated. Then, using a small dataset of dialogues as an example, combined with entry information {q} I O I}, constructing intent extraction cues Pr through context learning T The requirement is a large language model M L Extract the intent T from the dialogue based on the example. The annotated dataset is then manually reviewed to create an annotated dialogue dataset. Intent Extraction Hints Pr T The construction method is as follows: "Personality Prompt: Please provide the user's selection results based on the dialogue history and option list. Note: Only the option content is required; no additional dialogue information is needed. Example: Input: Dia" 1 Output: T 1 Input: Dia 2 Output: T 2 ;……enter: Output: Dialogue: Input: Dia; Output: T; where the number of examples is equal to the number of options in the list O. I The lengths of the examples are kept consistent, and each option has exactly one example to maintain a balance of examples.

[0064] S204, based on the annotated dialogue dataset C obtained from S203 ann Construct an intent extraction language model M SGiven a dataset C, the model is trained to predict the intent at the [MASK] location based on dataset C. For any set of labeled data (Dia, T), the input X of dataset C is defined as "[CLS] Doctor: D1 Patient: P1 ... Doctor: D t Patient: P t [SEP] Based on the above dialogue history, what is the corresponding n for question q? I +2 options O I Which option is selected? [MASK], where [CLS], [SEP], and [MASK] represent the beginning of the input, the statement breakpoint, and the prediction target location, respectively. Prediction target Where only when o i =T time otherwise The dataset is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1 for subsequent steps of model training and validation.

[0065] S205, The multi-label classification task is designed in the example by constructing a system of size n. hidden ×n I linear layer (F) Linear (·)) The hidden layer vector h at the [MASK] position [MASK] Mapping to n I On each prediction label, it is represented as Calculate the loss function using the cross-entropy function:

[0066]

[0067] S206, the semantic matching task is designed to match the options in the option list. The option embedding vector is selected by sequentially passing the BERT model and choosing the hidden layer encoding at [CLS] as the option embedding vector, denoted as... The hidden layer vector h at the [MASK] position of the input X is calculated using the cosine similarity function. [MASK] With option embedding vector The similarity between them is used to calculate the loss function using the cross-entropy function:

[0068]

[0069] S207, the contrastive learning task is designed to be performed on each set of dialogue data in dataset C (Dia i ,T i ), randomly selected from the list of options Choose one from the options except for the correct intention T. i Other options besides these are used as negative samples. Composition of contrastive learning dataset T i and Using the Bert model, the vector at [CLS] is selected as the embedding vector h. T and Use cosine similarity as the loss function to optimize the model:

[0070]

[0071] S208 evaluates the knowledge-distilled intent extraction language model using cases from the test set, based on the training results of models S205, S206, and S207. Considering the consistency of weights among different intents, Macro-Precision (P), Macro-Recall (R), and Macro-F1 (F1 score) are used to calculate the model evaluation metrics and compare the performance of the intent extraction language model on different downstream tasks.

[0072] Experimental example:

[0073] Experiments were conducted using the aforementioned knowledge distillation and follow-up data collection dialogue system on the widely used follow-up forms in the field of breast cancer: Fact-B (The Functional Assessment of Cancer Therapy-Breast), BIBCQ (psychometric features of the Body Image after Breast Cancer Questionnaire), and TMI (Testing Morbidities Index). The system was evaluated from the following three dimensions.

[0074] (1) Evaluation of content generation stability. Virtual datasets were generated on three forms, and dialogue content containing errors under the current prompt engineering design conditions was filtered out through manual screening. Manual screening included inviting two annotators to judge whether the dialogue content met the standards. If the two annotators disagreed, the disagreement was discussed to obtain the final result. The inclusion criteria were: 1. The dialogue revolved around the data collection objective and did not fall into meaningless chatter; 2. The dialogue content was consistent in context; 3. The dialogue content was logically consistent, the language was fluent, and there were no illegal characters; 4. The dialogue content did not include specific times, places, names, or other special private information.

[0075] After screening, the evaluation results of the three forms are shown in Table 1, where n raw n represents the number of datasets generated. CThe result indicates that the number of datasets that meet the filtering criteria is denoted as 'number'. The results show that the system can achieve a correct content generation rate of over 95.7% on three different follow-up forms, which means that the follow-up dialogue system can generate dialogue content that meets expectations in most cases.

[0076] Table 1. Content Generation Stability Evaluation Results

[0077] Form Name <![CDATA[n I ]]> <![CDATA[n raw ]]> <![CDATA[n C ]]> Error rate Fact-B 36 2160 2064 4.4％ BIBCQ 44 1144 1101 3.8％ TMI 12 840 800 4.8％

[0078] (2) Subjective Evaluation of Content Generation. Subjective evaluation requires constructing a question-and-answer dataset using a follow-up data collection system for common breast cancer issues. The Alibaba Cloud Tianchi Toyhom Chinese Medical Dialogue Dataset was chosen as the data source. This dataset contains 792,099 questions from patients, of which 2,128 are related to breast cancer. In the experimental case, these 2,128 questions were sequentially input into the BERT model and converted into embedding vectors. Then, the k-means algorithm was used to cluster the embedding vectors. The experimental case set the number of clusters to 50, and randomly selected one question from each cluster to form the question set input to the follow-up data collection system. The follow-up data collection system independently generated two answers for each question, thus constructing 50 question-and-answer pairs. Since there is currently a lack of authoritative indicators for the subjective evaluation of content generated by large language models, the experimental case constructed an evaluation system through discussions among breast surgeons. A Likert scale from 1 (completely disagree) to 5 (completely agree) was used to quantify the results, evaluating the generated content from five dimensions: 1. Accuracy: The system's answer is completely accurate. Do you agree? 2. Completeness: The system completely answers the question. Do you agree? 3. Comprehension Ability: The system fully understands the meaning of the question. Do you agree? 4. Consistency: The content generated by the system in both instances is completely consistent. Do you agree? 5. Safety: The content generated by the system will not cause any potential physiological or psychological harm to the user. Do you agree? Using Tencent Forms, a questionnaire was created based on the question-and-answer pairs, structured as question, answer 1, answer 2, and evaluation system. The questionnaires were distributed to three breast surgeons for evaluation. The evaluation process was independent and did not interfere with each other. Evaluations with a variance greater than 2 in the subjective evaluations generated by the three doctors were excluded from the statistical scope. The results showed that, for the five indicators of accuracy, completeness, comprehension ability, consistency, and safety, the proportions of positive evaluations (scores > 3) were 76.2% (112 / 147), 75.7% (109 / 144), 82.6% (119 / 144), 80.3% (118 / 147), and 73.6% (103 / 140), respectively. The results indicate that the evaluators had greater confidence in the system's ability to understand and stably generate responses.

[0079] (3) Evaluation of downstream tasks of knowledge distillation. The experimental cases were evaluated on three sets of forms, and the results are shown in Table 2.

[0080] Table 2 Evaluation Results of Downstream Tasks in Knowledge Distillation

[0081]

[0082] The results show that the fine-tuned BERT model achieved the best performance in the contrastive learning task on three completely different sets of forms. The contrastive learning extraction performance on the TMI form reached an F1 score of over 0.95, which is currently the best solution among all downstream tasks. The fine-tuned BERT model was used for inference on an Intel 6th generation Core i5 CPU, with an average inference speed of 1.2 seconds, indicating that the small model after knowledge distillation can achieve efficient inference under low cost.

[0083] In summary, the follow-up data collection method based on a large language model and knowledge distillation provided by this invention firstly, through the design of a context-based prompting engineering, the large language model can proactively generate highly accurate question-and-answer content based on any follow-up form. In this embodiment, the invention was tested and manually reviewed on three follow-up forms in the field of breast cancer, and the results showed that the average accuracy rate could reach over 95.7%. Secondly, the invention can generate dialogue content that conforms to the clinical application environment. In this embodiment, three breast surgeons were invited to evaluate the performance of the dialogue system on breast cancer question-and-answer topics in five dimensions, and the positive evaluation rate reached over 75%. Finally, the intent extraction language model trained after knowledge distillation can extract intent with high accuracy and efficiency and form a high-quality structured follow-up database, achieving a Macro-F1 score of 0.95 on the dataset, while its average inference time is 1.2 seconds. These results indicate that the method of this invention can be applied in a wider range of clinical scenarios and promote the remote monitoring and outpatient management of patient follow-up information by medical institutions.

[0084] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention also provides a follow-up data collection system 300 based on a large language model and knowledge distillation, including: a follow-up form preprocessing module 310, a large language model question-and-answer module 320, a related dialogue history filtering module 330, an intent extraction language model knowledge distillation module 340, and a user intent extraction and response module 350.

[0085] The follow-up form preprocessing module 310 is used to decompose the follow-up form into a set of several items, where each item contains a follow-up question and its corresponding option list;

[0086] The large language model question answering module 320 is used to construct questions and generate prompts based on follow-up questions and their corresponding option lists, input them into the large language model, send the generated natural language questions to users, and obtain user answers.

[0087] The relevant dialogue history filtering module 330 is used to take the natural language question and user answer corresponding to each item as short dialogue history, splice the short dialogue history of all items as long dialogue history, and filter the short dialogue history according to the similarity between the short dialogue history and the long dialogue history to obtain relevant dialogue history.

[0088] The intent extraction language model knowledge distillation module 340 is used to take the large language model as the teacher model and generate an annotated dialogue dataset, and combine it with the downstream task to perform knowledge distillation on the intent extraction language model as the student model.

[0089] The user intent extraction and response module 350 is used to extract user intent based on short-term dialogue history, related dialogue history and item information, using the intent extraction language model obtained after knowledge distillation, and to build a response generation prompt input large language model to generate a natural language response and send it to the user, thereby realizing the one-time follow-up data collection of the follow-up form.

[0090] Based on the same inventive concept, this invention also provides a follow-up data acquisition device based on large language models and knowledge distillation, including a memory and a processor. The memory is used to store a computer program, and the processor is used to implement the above-mentioned follow-up data acquisition method based on large language models and knowledge distillation when the computer program is executed.

[0091] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a computer, implements the above-described follow-up data collection method based on a large language model and knowledge distillation.

[0092] It should be noted that the follow-up data acquisition system, follow-up data acquisition device, and computer-readable storage medium based on large language models and knowledge distillation provided in the embodiments of the present invention all belong to the same inventive concept as the follow-up data acquisition method based on large language models and knowledge distillation. For details of their specific implementation process, please refer to the embodiments of the follow-up data acquisition method based on large language models and knowledge distillation, which will not be repeated here.

[0093] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A follow-up data collection method based on large language models and knowledge distillation, characterized in that, Includes the following steps: The follow-up form is broken down into a set of items, where each item contains a follow-up question and a list of corresponding options; Based on the follow-up questions and their corresponding option lists, questions are generated and prompts are input into a large language model. The generated natural language questions are then sent to users and their answers are obtained. The natural language question and user answer corresponding to each entry are used as short-term dialogue history. The short-term dialogue history of all entries is spliced ​​together to form long-term dialogue history. The short-term dialogue history is filtered according to the similarity between the short-term dialogue history and the long-term dialogue history to obtain relevant dialogue history. A large language model is used as the teacher model to generate an annotated dialogue dataset. Knowledge distillation is then performed on the intent extraction language model, which serves as the student model, in conjunction with downstream tasks. The downstream tasks include multi-label classification, semantic matching, and contrastive learning. The intents include option lists, skip, or ambiguous. Based on short-term dialogue history, related dialogue history, and entry information, the intent extraction language model after knowledge distillation is used to extract user intent, and a response generation prompt input large language model is constructed to generate natural language responses and send them to users, thereby realizing one-time follow-up data collection from the follow-up form.

2. The follow-up data collection method based on large language model and knowledge distillation according to claim 1, characterized in that, The question generation prompts include personality prompts and item prompts. Personality prompts are used to construct a questioner with specific personality traits and generate questions according to preset rules, including objectives, communication methods, and precautions. Item prompts are used to determine the identifier and attributes of each question. The identifier includes the question content and the list of options, and the attributes include whether it is a mandatory question.

3. The follow-up data collection method based on large language models and knowledge distillation according to claim 1, characterized in that, The process of filtering short-term dialogue history based on the similarity between short-term and long-term dialogue history to obtain relevant dialogue history includes: The SimCSE language model is used to encode short-term and long-term dialogue history into short-term dialogue history embedding vectors and long-term dialogue history embedding vectors. The similarity between the short-term and long-term dialogue history embedding vectors is calculated to obtain a similarity matrix. Based on the similarity matrix, a threshold is defined to filter the short-term dialogue history and obtain the relevant dialogue history.

4. The follow-up data collection method based on large language model and knowledge distillation according to claim 1, characterized in that, The process of using a large language model as a teacher model to generate an annotated dialogue dataset, and combining this with downstream tasks to perform knowledge distillation on the intent extraction language model used as a student model, includes: Using the large language model as the teacher model, virtual doctor and virtual patient dialogues based on the large language model are constructed for all entries according to the follow-up questions and their corresponding option lists contained in each entry, resulting in an unlabeled dialogue dataset. First, annotate a small number of dialogue datasets based on the unlabeled dialogue dataset. Then, use these small number of dialogue datasets as examples and combine them with item information to construct intent extraction prompts through context learning. This enables the large language model to extract the intent in the dialogue based on the examples and complete the annotation of the unlabeled dialogue dataset based on the intent, thus forming an annotated dialogue dataset. An intent extraction language model dataset containing structured input is constructed based on an annotated dialogue dataset. The structured input includes dialogue history, task description, and special tags, including the beginning of the input, sentence segmentation points, and the location of the prediction target. Based on the intent extraction language model dataset, the intent for predicting target location is predicted, and the intent extraction language model, which serves as the student model, is subjected to knowledge distillation in conjunction with downstream tasks to obtain a knowledge-distilled intent extraction language model, which is used for automatic intent extraction.

5. The follow-up data collection method based on large language models and knowledge distillation according to claim 1, characterized in that, The response generation prompts include personality prompts, urging prompts, item prompts, and dialogue history prompts. Personality prompts are used to construct a questioner with specific personality traits and generate questions. Urging prompts are used to urge users to answer form items as soon as possible. Item prompts are used to determine the identifier and attributes of each question. The identifier includes the question content and the list of options. The attributes include whether it is a mandatory question. Dialogue history prompts are used to introduce short-term dialogue history and related dialogue history.

6. A follow-up data collection system based on large language models and knowledge distillation, characterized in that, include: The module includes a follow-up form preprocessing module, a large language model question-and-answer module, a relevant dialogue history filtering module, an intent extraction language model knowledge distillation module, and a user intent extraction and response module. The follow-up form preprocessing module is used to decompose the follow-up form into a set of several items, where each item contains a follow-up question and its corresponding option list; The large language model question answering module is used to construct questions and generate prompts based on follow-up questions and their corresponding option lists, input them into the large language model, send the generated natural language questions to users, and obtain user answers. The relevant dialogue history filtering module is used to take the natural language question and user answer corresponding to each entry as short dialogue history, splice the short dialogue history of all entries as long dialogue history, and filter the short dialogue history according to the similarity between the short dialogue history and the long dialogue history to obtain relevant dialogue history. The intent extraction language model knowledge distillation module is used to use a large language model as a teacher model and generate an annotated dialogue dataset, and to perform knowledge distillation on the intent extraction language model, which is used as a student model, in conjunction with downstream tasks; the downstream tasks include multi-label classification tasks, semantic matching tasks, and contrastive learning tasks; the intent includes an option list, skip, or ambiguous. The user intent extraction and response module is used to extract user intent based on short-term dialogue history, related dialogue history and item information, using an intent extraction language model obtained after knowledge distillation, and to build a response generation prompt input large language model to generate a natural language response and send it to the user, thereby realizing one-time follow-up data collection of the follow-up form.

7. A follow-up data acquisition device based on large language models and knowledge distillation, comprising a memory and a processor, wherein the memory is used to store computer programs, characterized in that, The processor is configured to implement the follow-up data acquisition method based on large language model and knowledge distillation as described in any one of claims 1-5 when executing the computer program.

8. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by a computer, it implements the follow-up data collection method based on large language models and knowledge distillation as described in any one of claims 1-5.

Images