Classification-guided generative question answering methods, devices, and readable media

Abstract

This invention discloses a classification-guided generative question-answering method, apparatus, and readable medium, comprising: constructing an original dialogue dataset and training a trained topic classification model, a trained intent classification model, and a trained action recognition model; extracting a portion of processed dialogue data from the original dialogue dataset and constructing a classification-guided instruction dialogue dataset using the trained topic classification model, trained intent classification model, and trained action recognition model; mixing the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuned dataset; fine-tuning the pre-trained generative question-answering model using the fine-tuned dataset to obtain a fine-tuned generative question-answering model; and inputting the historical dialogue to be responded to into the fine-tuned generative question-answering model to obtain the corresponding output statement. This invention enhances the guidance of dialogue and makes the response controllable.

Description

Technical Field

[0001] This invention relates to the field of generative question answering, and more specifically to a classification-guided generative question answering method, apparatus, and readable medium. Background Technology

[0002] Thanks to breakthroughs in natural language processing technology and large language models in generating dialogues, generative chatbots have been widely adopted, providing multifaceted value to businesses and playing a crucial role in customer service and support. By responding to customer inquiries in real time, providing product information, and resolving common issues, chatbots improve customer service efficiency, offer 24 / 7 support, reduce the workload of human customer service representatives, and ultimately enhance customer satisfaction.

[0003] Existing customer service robots generally suffer from low accuracy in identifying the type of question they are answering. Due to the diversity and complexity of natural language, user questions may involve different fields, topics, or expressions, making it difficult for machines to accurately capture user intent. Furthermore, limitations in semantic understanding and contextual inference can lead to positioning errors when faced with complex and ambiguous questions. In addition, industry-specific terminology, emerging topics, or personalized user expressions further complicate the accuracy of positioning.

[0004] The low quality and illogical reasoning in medical customer service chatbots significantly impact their understanding of user symptoms and their ability to provide accurate health advice. Poor guidance prevents the chatbot from accurately acquiring key symptom information, from delving deeper to obtain a complete medical history, or from adjusting further questions based on user responses. This can lead to misunderstandings of the condition, inaccurate diagnoses, or inappropriate advice, affecting the credibility and practicality of the chatbot in medical consultations.

[0005] Customer service robots may exhibit uncontrollable responses due to various interferences. Firstly, the ambiguity of language makes it difficult for machines to accurately capture users' true intentions, especially in situations involving complex contexts or implicit information. Furthermore, users' questioning styles can vary due to personalization, regional differences, and other factors, making it difficult for customer service robots to maintain consistency in their responses. Secondly, the training data for generative language models may contain diverse linguistic expressions, including potentially negative or inappropriate content, leading to uncontrollable wording in customer service robot responses that could cause misunderstandings or inappropriate responses. Summary of the Invention

[0006] The purpose of this application is to propose a generative question-answering method, device, and readable medium based on classification guidance to address the aforementioned technical issues of inaccurate problem type identification, poor guidance in consultation, and uncontrollable answers in customer service robots.

[0007] In a first aspect, the present invention provides a classification-guided generative question-answering method, comprising the following steps:

[0008] Several dialogue corpora were collected and data cleaned and processed separately to obtain several processed dialogue corpora and construct the original dialogue dataset. The processed dialogue corpora include historical dialogues and response statements.

[0009] Construct a topic classification model, an intent classification model, and an action recognition model, and train them separately to obtain the trained topic classification model, the trained intent classification model, and the trained action recognition model.

[0010] A portion of processed dialogue data is extracted from the original dialogue dataset, and the embedding vectors corresponding to the historical dialogues in each processed dialogue data are obtained. The embedding vectors corresponding to the historical dialogues in each processed dialogue data are input into a trained topic classification model and a trained intent classification model to obtain topic classification results and intent classification results, respectively. The topic classification results, intent classification results, and the embedding vectors corresponding to the historical dialogues in each processed dialogue data are input into a trained action recognition model to predict at least one action label corresponding to the response statement. Based on the historical dialogues, topic classification results, intent classification results, and at least one action label, instructions are constructed using a thought chain approach. Different descriptive statements are configured according to different action labels or different combinations of action labels and combined into the corresponding instructions. The instructions are stored in an instruction set. A classification-guided instruction dialogue dataset is constructed based on the historical dialogues in each processed dialogue data and their corresponding instructions, response statements, and descriptive statements. The original dialogue dataset and the classification-guided instruction dialogue dataset are mixed to obtain a fine-tuning dataset.

[0011] A pre-trained generative question answering model based on a hybrid expert model is constructed. The pre-trained generative question answering model is fine-tuned using a fine-tuning dataset to obtain a fine-tuned generative question answering model. The historical dialogue to be replied to is input into the fine-tuned generative question answering model to obtain the corresponding output statement.

[0012] Preferably, a topic classification model, an intent classification model, and an action recognition model are constructed and trained separately to obtain trained topic classification models, trained intent classification models, and trained action recognition models, specifically including:

[0013] Construct a topic classification model based on LSTM and a pre-trained first Roberta model, and collect a sentence-topic dataset. Use the pre-trained first Roberta model as the teacher model and the topic classification model as the student model. Input sentences from the sentence-topic dataset into the teacher model to obtain the output of the teacher model. Use the sentence-topic dataset and the output of the teacher model to train the student model through knowledge distillation to obtain the trained topic classification model.

[0014] We construct an intent classification model based on an LSTM model and a pre-trained second Roberta model, and collect a sentence-intent dataset. We use the pre-trained second Roberta model as the teacher model and the intent classification model as the student model. We input sentences from the sentence-intent dataset into the teacher model and obtain the output of the teacher model. We use the sentence-intent dataset and the output of the teacher model to train the student model through knowledge distillation, and obtain the trained intent classification model.

[0015] Collect an action classification dataset, construct an action recognition model based on the Qianwen Big Data Model or BERT model, and train the action recognition model using the action classification dataset to obtain a trained action recognition model. Alternatively, construct an action recognition model based on the LSTM model and a pre-trained third Roberta model, and collect a sentence-intent dataset. Use the pre-trained third Roberta model as the teacher model and the action recognition model as the student model. Input the sentences in the action classification dataset and their corresponding topic classification results and intent classification results into the teacher model to obtain the output results of the teacher model. Use the action classification dataset and the output results of the teacher model to perform knowledge distillation training on the student model to obtain a trained action recognition model.

[0016] Preferably, each feedforward network layer in the Transformer model of the pre-trained generative question answering model is replaced with a fusion module. The fusion module includes several expert networks and gating networks corresponding to action labels. During the fine-tuning of the pre-trained generative question answering model, historical dialogues or historical dialogues and instructions from the fine-tuning dataset are input into the pre-trained generative question answering model. The topic classification results and intent classification results of the historical dialogues or historical dialogues and instructions are passed through the gating network to obtain a probability vector. The probability vector represents the probability that the expert network corresponding to each action label is selected. According to the probability vector, the historical dialogues are assigned to the expert networks of the corresponding action labels according to different blocks. The output results of all expert networks are aggregated and weighted to obtain the corresponding output statement.

[0017] Preferably, if at least one action label predicted from the historical dialogue and the topic classification results and intent classification results in the instructions is found to have a corresponding description statement in the instruction set, then the reply statement is concatenated with the description statement as the output statement; otherwise, the reply statement is directly used as the output statement.

[0018] As a preferred approach, during the fine-tuning process of the generative question-answering model, the parameters of the expert network are frozen and the parameters of the gating network are adjusted. The fine-tuning process of the generative question-answering model adopts supervised fine-tuning, and the fine-tuning methods used include LoRA fine-tuning, Q-LoRA fine-tuning, or P-tuning fine-tuning.

[0019] As a preferred option, action tags include answering questions, consulting, linking, connecting, and confirming. Action tag combinations are constructed by building at least two action tags in the order required by business needs.

[0020] Secondly, the present invention provides a classification-guided generative question-answering device, comprising:

[0021] The data acquisition module is configured to collect several dialogue data segments and perform data cleaning and data processing respectively to obtain several processed dialogue data segments and construct the original dialogue dataset. The processed dialogue data includes historical dialogues and response statements.

[0022] The model building module is configured to build a topic classification model, an intent classification model, and an action recognition model and train them respectively to obtain trained topic classification models, trained intent classification models, and trained action recognition models.

[0023] The fine-tuning dataset construction module is configured to extract a portion of processed dialogue corpora from the original dialogue dataset and obtain the embedding vectors corresponding to the historical dialogues in each processed dialogue corpus. The embedding vectors corresponding to the historical dialogues in each processed dialogue corpus are then input into a trained topic classification model and a trained intent classification model to obtain topic classification results and intent classification results, respectively. The topic classification results, intent classification results, and the embedding vectors corresponding to the historical dialogues in each processed dialogue corpus are then input into a trained action recognition model to predict at least one action label corresponding to the response statement. Instructions are constructed using a thought chain approach based on the historical dialogues, topic classification results, intent classification results, and at least one action label. Different descriptive statements are configured according to different action labels or combinations of different action labels and incorporated into the corresponding instructions. The instructions are stored in an instruction set. A classification-guided instruction dialogue dataset is constructed based on the historical dialogues in each processed dialogue corpus and their corresponding instructions, response statements, and descriptive statements. The original dialogue dataset and the classification-guided instruction dialogue dataset are then mixed to obtain the fine-tuning dataset.

[0024] The generation module is configured to build a pre-trained generative question answering model based on a hybrid expert model, fine-tune the pre-trained generative question answering model using a fine-tuning dataset to obtain a fine-tuned generative question answering model; and obtain the historical dialogue to be replied to as input into the fine-tuned generative question answering model to obtain the corresponding output statement.

[0025] Thirdly, the present invention provides an electronic device including one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any implementation of the first aspect.

[0026] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0027] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method as described in any of the implementations in the first aspect.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The generative question-answering method based on classification guidance proposed in this invention establishes a more granular classification standard. It identifies the topic classification results, intent classification results and action tags in the historical dialogue through a trained topic classification model, a trained intent classification model and a trained action recognition model, respectively, so as to better capture the user's implicit intent and topic, and construct instructions with good guidance effect. It can solve the problem of inaccurate positioning of question answering type of existing medical customer service robots.

[0030] (2) The generative question answering method based on classification guidance proposed in this invention uses a trained topic classification model, a trained intent classification model and a trained action recognition model to construct a classification guidance instruction dialogue dataset. The original dialogue dataset and the classification guidance instruction dialogue dataset are mixed to obtain a fine-tuning dataset. The pre-trained generative question answering model is supervised and fine-tuned using this fine-tuning dataset, so that it can output corresponding response statements according to different action labels. Furthermore, more complete and logical description statements are established for different needs to connect the response statements corresponding to different actions, so as to better interact with users. This can solve the problem of poor guidance of customer service robots, make the answer more controllable, and thus improve the success rate of achieving task requirements.

[0031] (3) The pre-trained generative question answering model in the classification-guided generative question answering method proposed in this invention adopts the principle of hybrid experts and is fine-tuned through the constructed fine-tuning dataset. Therefore, it has the ability to deeply mine semantic understanding and context awareness, and can generate output statements that are more logical and meet business needs. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a flowchart illustrating a classification-guided generative question-answering method according to an embodiment of this application.

[0034] Figure 2 This is a schematic diagram of a generative question answering model based on a classification-guided generative question answering method according to an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of a classification-guided generative question-answering device according to an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0038] Figure 1 The illustration shows an embodiment of a classification-guided generative question-answering method provided by this application, comprising the following steps:

[0039] S1: Collect several dialogue corpora and perform data cleaning and processing on them respectively to obtain several processed dialogue corpora and construct the original dialogue dataset. The processed dialogue corpora include historical dialogues and response statements.

[0040] Specifically, the process begins by acquiring dialogue data generated from real-person conversations. This data is then cleaned and preprocessed, including noise reduction and handling of missing values ​​to ensure data diversity. Imbalanced metrics are also sampled. The dialogue data includes dialogue role information and dialogue content. In this embodiment, the dialogue role information includes user roles and customer service roles. The processed dialogue data, after cleaning and preprocessing, includes historical dialogues and response statements. Historical dialogues and response statements can be defined based on the dialogue role information. Historical dialogues can have contextual logic and contain several alternating user statements output by user roles and response statements output by customer service roles. The last historical dialogue is a user statement, corresponding to a response statement.

[0041] The following is an example of the dialogue data:

[0042] User: I've been having headaches lately, and I'd like to ask what might be causing this?

[0043] Customer Service: Headaches can have many causes, including lifestyle factors, stress, and vision problems. To better assist you, I need to understand some details. Have you recently experienced sleep deprivation or high work stress?

[0044] Customer: Yes, I've been quite busy with work lately, and I haven't been sleeping very well.

[0045] Customer service: Long working hours and insufficient sleep can indeed affect your health. Besides headaches, do you have any other symptoms?

[0046] Customer: Sometimes I feel a little dizzy, and my appetite isn't very good either.

[0047] Customer service: Dizziness and loss of appetite may be related to headaches, but we need to know more. What's your WeChat ID? I'll add you and help you with your follow-up medical appointment.

[0048] Customer: My WeChat ID is ×××.

[0049] S2, construct a topic classification model, an intent classification model, and an action recognition model, and train them respectively to obtain the trained topic classification model, the trained intent classification model, and the trained action recognition model.

[0050] In a specific embodiment, step S2 specifically includes:

[0051] Construct a topic classification model based on LSTM and a pre-trained first Roberta model, and collect a sentence-topic dataset. Use the pre-trained first Roberta model as the teacher model and the topic classification model as the student model. Input sentences from the sentence-topic dataset into the teacher model to obtain the output of the teacher model. Use the sentence-topic dataset and the output of the teacher model to train the student model through knowledge distillation to obtain the trained topic classification model.

[0052] We construct an intent classification model based on an LSTM model and a pre-trained second Roberta model, and collect a sentence-intent dataset. We use the pre-trained second Roberta model as the teacher model and the intent classification model as the student model. We input sentences from the sentence-intent dataset into the teacher model and obtain the output of the teacher model. We use the sentence-intent dataset and the output of the teacher model to train the student model through knowledge distillation, and obtain the trained intent classification model.

[0053] Collect an action classification dataset, construct an action recognition model based on the Qianwen Big Data Model or BERT model, and train the action recognition model using the action classification dataset to obtain a trained action recognition model. Alternatively, construct an action recognition model based on the LSTM model and a pre-trained third Roberta model, and collect a sentence-intent dataset. Use the pre-trained third Roberta model as the teacher model and the action recognition model as the student model. Input the sentences in the action classification dataset and their corresponding topic classification results and intent classification results into the teacher model to obtain the output results of the teacher model. Use the action classification dataset and the output results of the teacher model to perform knowledge distillation training on the student model to obtain a trained action recognition model.

[0054] Specifically, the topic classification model, intent classification model, and action recognition model employ the Roberta-LSTM knowledge distillation structure and are trained on the sentence-topic dataset, sentence-intent dataset, and action classification dataset, respectively, resulting in trained topic classification models, trained intent classification models, and trained action recognition models. The pre-trained Roberta model, trained on a large-scale unlabeled corpus, has acquired the ability to encode input sentences into embeddings. Downstream tasks are then trained on the sentence-topic dataset, sentence-intent dataset, and action classification dataset. Next, the LSTM model is trained using the outputs of the sentence-topic dataset, sentence-intent dataset, action classification dataset, and the pre-trained Roberta model. This allows the LSTM model, acting as a student model, to acquire the classification capabilities of the Roberta model, acting as a teacher model, while maintaining a smaller number of model parameters and faster processing speed. The trained topic classification model, trained intent classification model, and trained action recognition model are then obtained by identifying the embedding vectors corresponding to historical dialogues, respectively, to obtain topic classification results, intent classification results, and action labels, thus achieving accurate fine-grained segmentation of historical dialogues. The training steps for the topic classification model, intent classification model, and action recognition model include:

[0055] 1) Extraction of single-sentence corpus;

[0056] 2) Data labeling;

[0057] 3) Model training is sufficient to achieve an accuracy rate of 80% or higher.

[0058] When a historical dialogue is received, the embedding vector corresponding to the historical dialogue is first input into a trained topic classification model and a trained intent classification model, respectively, to obtain topic classification results and intent classification results. Then, the topic classification results, intent classification results, and the embedding vector corresponding to the historical dialogue are input into a trained action recognition model to obtain action labels. Examples of input historical dialogues and recognized action labels are shown in Table 1.

[0059] Table 1

[0060] Historical Dialogue Action tags Have I been swimming too much? Q&A, tagged 1 Is it because I eat too many different kinds of food? Q&A, tagged 1 I've been having frequent stomach aches lately. Consultation, tagged as 2 What causes neck pain? Consultation, tagged as 2 I have developed many dark spots. Consultation, tagged as 2 My hand has been swollen for two weeks. Consultation, tagged as 2 Can your hospital handle this? Set, tag number 3 My hand wasn't cured at the hospital last time. Set, tag number 3

[0061] The trained topic classification model, the trained intent classification model, and the trained action recognition model are all pre-trained supervised classification models. The labels of the models are constrained to a range according to business needs. For example, the number of categories in the topic classification model, the intent classification model, and the action recognition model are k1, k2, and k3, respectively. Among them, k1, k2, and k3 are all integers greater than 1.

[0062] Furthermore, the action recognition model can also be directly trained on the 1.5B model, the 0.5B model, or a pre-trained BERT model to obtain a trained action recognition model.

[0063] S3. Extract a portion of the processed dialogue data from the original dialogue dataset and obtain the embedding vector corresponding to the historical dialogue in each processed dialogue data. Input the embedding vector corresponding to the historical dialogue in each processed dialogue data into a trained topic classification model and a trained intent classification model to obtain topic classification results and intent classification results, respectively. Input the topic classification results, intent classification results, and the embedding vector corresponding to the historical dialogue in each processed dialogue data into a trained action recognition model to predict at least one action label corresponding to the response statement. Construct instructions using a thought chain approach based on the historical dialogue, topic classification results, intent classification results, and at least one action label. Configure different descriptive statements according to different action labels or different combinations of action labels and combine them into the corresponding instructions. Store the instructions in an instruction set. Construct a classification-guided instruction dialogue dataset based on the historical dialogue in each processed dialogue data and its corresponding instructions, response statements, and descriptive statements. Mix the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset.

[0064] In specific embodiments, action tags include answering questions, consulting, linking, connecting, and confirming. Action tag combinations are obtained by constructing at least two action tags in the order required by business needs.

[0065] Specifically, embodiments of this application utilize trained topic classification models, trained intent classification models, and trained action recognition models to construct a categorized guidance instruction dialogue dataset. In constructing this dataset, firstly, M% of processed dialogue data can be extracted from the original dialogue dataset and fine-tuned. The trained topic classification model and trained intent classification model are then used to obtain topic classification and intent classification results from the M% processed dialogue data. Furthermore, the topic classification and intent classification results, along with the embedding vectors corresponding to historical dialogues, are input into the trained action recognition model to identify at least one corresponding action label. A descriptive statement is then configured using at least one action label. Further, at least two action labels can be sequentially combined to form an action label combination, which constitutes the guidance interval. In embodiments of this application, three action labels—question answering, diagnosis, and connection—constitute the guidance interval, and a corresponding descriptive statement is configured for each guidance interval. To prevent users from logging off after the customer service robot completes the question answering process, reasonable and effective guidance communication is needed during the diagnosis process to facilitate more natural connection with users. To meet different needs, embodiments of this application configure the description statements to achieve the purpose of "retaining users and obtaining contact information," as shown in Table 2.

[0066] Table 2

[0067]

[0068]

[0069] When establishing the classification-guided dialogue dataset, instructions are first constructed using a chain of thought (COT) based on historical dialogues, topic classification results, intent classification results, and at least one identified action label. Specifically, historical dialogues, topic classification results, intent classification results, and action labels are filled into a suitable instruction template to construct the dialogue generation instruction. This instruction, combined with historical dialogues, generates a response statement that satisfies the semantics corresponding to the action label. This response statement corresponds to at least one action label, and the number of action labels is greater than or equal to 1 and less than k3. In practice, this embodiment selects historical dialogues with no more than 8 action labels for the next step. In the above example, the logical chain must consist of three elements: "answer," "diagnosis," and "link," and its length is limited to avoid logical confusion in the dialogue output. When there are guiding intervals such as "answer-diagnosis" in the logical chain, corresponding descriptive statements are added between each guiding part. By providing instructions and adding pre-defined descriptive statements, the ability to learn dialogue styles is fine-tuned. This allows the fine-tuned generative question-answering model to automatically determine whether it matches the guidance interval when inputting historical dialogues. If it does, the generated response statement is concatenated with the descriptive statement corresponding to the matching guidance interval as the output statement of the generative question-answering model. Otherwise, the generated response statement is directly used as the output statement of the generative question-answering model.

[0070] Below is a complete instruction template format:

[0071] You are currently a gynecologist. Based on the following dialogue between customer service and a visitor on [date], please consider the following: From the dialogue history, you can first understand that the visitor is concerned with [topic categorization results], and the visitor currently wants to know [intent categorization results]. Therefore, we can infer that the customer service representative can provide a response with [action tag]: [response statement].

[0072] Action tag combinations are constructed using specific strategies based on specific business needs. At least one action tag in the combination is identified based on historical dialogues, topic classification results, and intent classification results. Other action tags are selected and constructed sequentially according to specific business requirements, taking into account the number of dialogue rounds. For example, if the question-answering action tag is identified from historical dialogues, topic classification results, and intent classification results, and customer service needs to use a consultation to continue the dialogue, a consultation action tag will be added after the question-answering action tag if no matching action tag is identified, resulting in a question-answering-consultation action tag combination. Then, a corresponding descriptive statement is found based on this action tag combination and incorporated into the instruction for guidance. The guidance generation follows the order of the action tags. However, if the dialogue rounds are greater than 5 and a matching action tag is predicted, only one matching action tag is retained in the instruction for guidance. In the above examples, to better achieve the purpose of matching, the embodiments of this application must interact with the customer to answer questions. Therefore, descriptive statements are configured for the generated content to better connect to the matching process. The descriptions of these different guidance intervals are stored in the instructions of the classification guidance instruction dialogue dataset. Through the fine-tuning process, the fine-tuned generative question answering model learns the ability to link.

[0073] After obtaining the classification guidance instruction dialogue dataset, it is mixed with the original dialogue dataset to form a fine-tuning dataset.

[0074] S4. Construct a pre-trained generative question answering model based on a hybrid expert model. Fine-tune the pre-trained generative question answering model using a fine-tuning dataset to obtain a fine-tuned generative question answering model. Input the historical dialogue to be replied to into the fine-tuned generative question answering model to obtain the corresponding output statement.

[0075] In a specific embodiment, each feedforward network layer of the pre-trained generative question answering model in the Transformer model is replaced with a fusion module. The fusion module includes several expert networks and gating networks corresponding to action labels. During the fine-tuning process of the pre-trained generative question answering model, historical dialogues or historical dialogues and instructions from the fine-tuning dataset are input into the pre-trained generative question answering model. The topic classification results and intent classification results in the historical dialogues or historical dialogues and instructions are passed through the gating network to obtain a probability vector. The probability vector represents the probability that the expert network corresponding to each action label is selected. Based on the probability vector, the historical dialogues are assigned to the expert networks corresponding to the action labels according to different blocks. The output results of all expert networks are aggregated and weighted to obtain the corresponding output statement.

[0076] In a specific embodiment, if at least one action tag predicted by the topic classification results and intent classification results in the historical dialogue and the instructions is found to have a corresponding description statement in the instruction set, then the reply statement is concatenated with the description statement as the output statement; otherwise, the reply statement is directly used as the output statement.

[0077] In a specific embodiment, during the fine-tuning process of the generative question-answering model, the parameters of the expert network are frozen and the parameters of the gating network are adjusted. The fine-tuning process of the generative question-answering model adopts supervised fine-tuning, and the fine-tuning methods used include LoRA fine-tuning, Q-LoRA fine-tuning, or P-tuning fine-tuning.

[0078] Specifically, the generative language model used in the embodiments of this application employs a hybrid expert model and undergoes supervised fine-tuning. Fine-tuning operations utilize methods such as LoRA, Q-Lora, or P-tuning. Specifically, the pre-trained generative language model is first subjected to supervised fine-tuning using a fine-tuning dataset. During the fine-tuning process of the pre-trained generative question-answering model, the parameters of the gating network in the fusion module need to be trained, while the parameters of the expert network are frozen to avoid large-scale parameter changes during subsequent fine-tuning of the generative question-answering model, which would lead to extremely high training overhead. By fine-tuning the pre-trained generative question-answering model using this fine-tuning dataset, it is made to not only satisfy normal dialogue generation but also possess classification guidance capabilities, enabling smoother dialogue and clearer logic.

[0079] For details, please refer to Figure 2 In the example of this application, consultation content is generated after the question is answered based on the context, forming a question-and-answer behavior pattern, and linked statements are generated at appropriate times. The example of this application sets up expert networks corresponding to question answering, consultation, and linked statements, each with different parameters and design purposes. A gating network determines at least one expert network among these networks to input historical dialogues, outputting response statements corresponding to the question answering, consultation, and / or linked statements, i.e., question answering statements, consultation statements, and / or linked statements. Furthermore, if a descriptive statement corresponding to an action tag combination is matched in the instruction set, the response statement is combined with the descriptive statement to obtain the output statement generated by the fine-tuned generative question-and-answer model. Specifically, descriptive statements are used to connect the switching between question answering statements, consultation statements, and linked statements, thus more naturally transitioning to the linked statement stage in the dialogue with the customer.

[0080] Specifically, the pre-trained generative language model in this application employs the Mixture of Experts (MOE) principle. This part consists of an expert network and a gating network. When a historical dialogue is input, the topic classification result and intent classification result are obtained through a logical chain. These results are then used as biases and input into the gating network. The gating network consists of learnable parameters (including a linear layer and a softmax layer). The gating network determines which expert network to use for the historical dialogue input, such as the answering, diagnosis, or connection expert networks, to provide a response statement. By fine-tuning the dataset, the gating network is trained to assign the historical dialogue input to the appropriate expert network for response. Therefore, different types of expert networks can be selected to provide response statements based on the topic classification result and intent classification result corresponding to the input historical dialogue. When the guidance interval (action label combination) is met, a descriptive statement is provided, thus better connecting to the target stage.

[0081] Furthermore, the topic classification results and intent classification results corresponding to the historical dialogues are compared with the guidance intervals (action label combinations) in the instruction set to determine whether the topic classification results and intent classification results match the guidance intervals. If they do, it indicates that there is a matching descriptive statement corresponding to the guidance interval, which is then incorporated into the corresponding instruction. The corresponding descriptive statement is output based on the historical dialogue and the corresponding instruction through a logical chain. The response statement and descriptive statement output by the generative language model can be used as the output statement of the fine-tuned generative question-answering model. Otherwise, if there is no descriptive statement, only the response statement output by the generative language model is used as the output statement of the fine-tuned generative question-answering model. In addition, if a user inputs at least two user statements containing different types in a round, the at least two user statements can be segmented and classified first. Then, each user statement is input into the fine-tuned generative question-answering model to obtain the response statement corresponding to each user statement. Finally, at least two response statements, or at least two response statements and a descriptive statement, are used as the output statement and returned to the user.

[0082] Further reference Figure 3 As an implementation of the methods shown in the above figures, this application provides an embodiment of a classification-guided generative question-answering device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0083] This application provides a classification-guided generative question-answering device, including:

[0084] Data acquisition module 1 is configured to collect several dialogue data segments and perform data cleaning and data processing respectively to obtain several processed dialogue data segments and construct the original dialogue dataset. The processed dialogue data includes historical dialogues and response statements.

[0085] Model building module 2 is configured to build a topic classification model, an intent classification model, and an action recognition model and train them respectively to obtain trained topic classification models, trained intent classification models, and trained action recognition models.

[0086] The fine-tuning dataset construction module 3 is configured to extract a portion of the processed dialogue data from the original dialogue dataset and obtain the embedding vector corresponding to the historical dialogue in each processed dialogue data. The embedding vector corresponding to the historical dialogue in each processed dialogue data is input into a trained topic classification model and a trained intent classification model to obtain topic classification results and intent classification results, respectively. The topic classification results, intent classification results, and the embedding vector corresponding to the historical dialogue in each processed dialogue data are input into a trained action recognition model to predict at least one action label corresponding to the response statement. Based on the historical dialogue, topic classification results, intent classification results, and at least one action label, instructions are constructed in a thought chain manner. Different descriptive statements are configured according to different action labels or different combinations of action labels and combined into the corresponding instructions. The instructions are stored in the instruction set. A classification-guided instruction dialogue dataset is constructed based on the historical dialogue in each processed dialogue data and its corresponding instructions, response statements, and descriptive statements. The original dialogue dataset and the classification-guided instruction dialogue dataset are mixed to obtain the fine-tuning dataset.

[0087] The generation module 4 is configured to build a pre-trained generative question answering model based on a hybrid expert model, fine-tune the pre-trained generative question answering model using a fine-tuning dataset to obtain a fine-tuned generative question answering model; and obtain the historical dialogue to be replied to as input into the fine-tuned generative question answering model to obtain the corresponding output statement.

[0088] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. For example... Figure 4 As shown, the electronic device in this embodiment includes a processor 401 and a memory 402; wherein the memory 402 is used to store computer execution instructions; and the processor 401 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0089] Alternatively, the memory 402 can be either standalone or integrated with the processor 401.

[0090] When the memory 402 is set up independently, the electronic device also includes a bus 403 for connecting the memory 402 and the processor 401.

[0091] This invention also provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the method described above.

[0092] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0093] In the embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0094] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0095] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0096] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0097] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0098] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0099] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0100] The aforementioned storage medium can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium accessible to general-purpose or special-purpose computers.

[0101] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.

[0102] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generative question-answering method based on classification guidance, characterized in that, Includes the following steps: Several dialogue corpora are collected and data cleaning and processing are performed on them respectively to obtain several processed dialogue corpora and construct the original dialogue dataset. The processed dialogue corpora include historical dialogues and response statements. Construct a topic classification model, an intent classification model, and an action recognition model, and train them separately to obtain the trained topic classification model, the trained intent classification model, and the trained action recognition model. A portion of processed dialogue data is extracted from the original dialogue dataset, and the embedding vector corresponding to the historical dialogue in each processed dialogue data is obtained. The embedding vector corresponding to the historical dialogue in each processed dialogue data is input into the trained topic classification model and the trained intent classification model to obtain topic classification results and intent classification results, respectively. The topic classification results, intent classification results, and the embedding vector corresponding to the historical dialogue in each processed dialogue data are input into the trained action recognition model to predict at least one action label corresponding to the response statement. Based on the topic classification results, intent classification results, and at least one action label, instructions are constructed using a thought chain approach. Different descriptive statements are configured according to different action labels or different combinations of action labels and combined into the corresponding instructions. The instructions are stored in the instruction set. A classification guidance instruction dialogue dataset is constructed based on the historical dialogue in each processed dialogue data and its corresponding instructions, response statements, and descriptive statements. The original dialogue dataset and the classification guidance instruction dialogue dataset are mixed to obtain a fine-tuning dataset. A pre-trained generative question answering model based on a hybrid expert model is constructed. The pre-trained generative question answering model is then fine-tuned using the fine-tuning dataset to obtain a fine-tuned generative question answering model. The historical dialogue to be replied to is input into the fine-tuned generative question answering model to obtain the corresponding output statement.

2. The classification-guided generative question-answering method according to claim 1, characterized in that, Construct a topic classification model, an intent classification model, and an action recognition model, and train them separately to obtain trained topic classification models, trained intent classification models, and trained action recognition models, specifically including: Construct a topic classification model based on an LSTM model and a pre-trained first Roberta model, and collect a sentence-topic dataset. Use the pre-trained first Roberta model as the teacher model and the topic classification model as the student model. Input sentences from the sentence-topic dataset into the teacher model to obtain the output of the teacher model. Use the sentence-topic dataset and the output of the teacher model to train the student model through knowledge distillation to obtain a trained topic classification model. An intent classification model based on an LSTM model and a pre-trained second Roberta model are constructed, and a sentence-intent dataset is collected. The pre-trained second Roberta model is used as the teacher model, and the intent classification model is used as the student model. The sentences in the sentence-intent dataset are input into the teacher model to obtain the output of the teacher model. The student model is trained by knowledge distillation using the sentence-intent dataset and the output of the teacher model to obtain the trained intent classification model. Collect an action classification dataset, construct an action recognition model based on the Qianwen Big Data Model or BERT model, and train the action recognition model using the action classification dataset to obtain a trained action recognition model. Alternatively, construct an action recognition model based on the LSTM model and a pre-trained third Roberta model, and collect a sentence-intent dataset. Use the pre-trained third Roberta model as the teacher model and the action recognition model as the student model. Input the sentences and their corresponding topic classification results and intent classification results from the action classification dataset into the teacher model to obtain the output results of the teacher model. Use the action classification dataset and the output results of the teacher model to perform knowledge distillation training on the student model to obtain a trained action recognition model.

3. The generative question-answering method based on classification guidance according to claim 1, characterized in that, In the pre-trained generative question answering model, each feedforward network layer in the Transformer model is replaced with a fusion module. The fusion module includes several expert networks and gating networks corresponding to action labels. During the fine-tuning process of the pre-trained generative question answering model, historical dialogues or historical dialogues and instructions from the fine-tuning dataset are input into the pre-trained generative question answering model. The topic classification results and intent classification results of the historical dialogues or historical dialogues and instructions are passed through the gating network to obtain a probability vector. The probability vector represents the probability that the expert network corresponding to each action label is selected. According to the probability vector, the historical dialogues are assigned to the expert networks corresponding to the action labels according to different blocks. The output results of all expert networks are aggregated and weighted to obtain the corresponding output statement.

4. The classification-guided generative question-answering method according to claim 3, characterized in that, If at least one action tag predicted from the historical dialogue and the topic classification and intent classification results in the instructions is found to have a corresponding description statement in the instruction set, then the reply statement and the description statement are concatenated as the output statement; otherwise, the reply statement is directly used as the output statement.

5. The classification-guided generative question-answering method according to claim 3, characterized in that, During the fine-tuning process of the generative question answering model, the parameters of the expert network are frozen and the parameters of the gating network are adjusted. The fine-tuning process of the generative question answering model adopts supervised fine-tuning, and the fine-tuning methods used include LoRA fine-tuning, Q-LoRA fine-tuning, or P-tuning fine-tuning.

6. The generative question-answering method based on classification guidance according to claim 1, characterized in that, The action tags include answering questions, consulting, linking, connecting, and confirming. The action tag combination is obtained by constructing at least two action tags in the order required by business needs.

7. A generative question-answering device based on classification guidance, characterized in that, include: The data acquisition module is configured to collect several segments of dialogue data and perform data cleaning and data processing respectively to obtain several segments of processed dialogue data and construct the original dialogue dataset. The processed dialogue data includes historical dialogues and response statements. The model building module is configured to build a topic classification model, an intent classification model, and an action recognition model and train them respectively to obtain trained topic classification models, trained intent classification models, and trained action recognition models. The fine-tuning dataset construction module is configured to extract a portion of processed dialogue data from the original dialogue dataset and obtain the embedding vector corresponding to the historical dialogue in each processed dialogue data. The embedding vector corresponding to the historical dialogue in each processed dialogue data is input into the trained topic classification model and the trained intent classification model to obtain topic classification results and intent classification results, respectively. The topic classification results, intent classification results, and the embedding vector corresponding to the historical dialogue in each processed dialogue data are input into the trained action recognition model to predict at least one action label corresponding to the response statement. Based on the topic classification results, intent classification results, and at least one action label, instructions are constructed using a thought chain approach. Different descriptive statements are configured according to different action labels or combinations of different action labels and incorporated into the corresponding instructions. The instructions are stored in the instruction set. A classification-guided instruction dialogue dataset is constructed based on the historical dialogue in each processed dialogue data and its corresponding instructions, response statements, and descriptive statements. The original dialogue dataset and the classification-guided instruction dialogue dataset are mixed to obtain the fine-tuning dataset. The generation module is configured to construct a pre-trained generative question answering model based on a hybrid expert model, fine-tune the pre-trained generative question answering model using the fine-tuning dataset to obtain a fine-tuned generative question answering model; and obtain the historical dialogue to be replied to as input into the fine-tuned generative question answering model to obtain the corresponding output statement.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images