Generative question answering method and device based on classification guidance and readable medium

Abstract

The invention discloses a generative question and answer method and device based on classification guidance and a readable medium, and the method comprises the steps: constructing an original dialogue data set, and training to obtain a trained theme classification model, a trained intention classification model and a trained action recognition model; extracting partially processed dialogue corpora from the original dialogue data set, constructing a classification guide instruction dialogue data set by utilizing the trained subject classification model, the trained intention classification model and the trained action recognition model, and mixing the original dialogue data set and the classification guide instruction dialogue data set to obtain a fine tuning data set; performing fine tuning on the pre-trained generative question and answer model by adopting the fine tuning data set to obtain a fine-tuned generative question and answer model; and obtaining a historical dialogue to be replied, and inputting the historical dialogue to the fine-tuned generative question and answer model to obtain a corresponding output statement. According to the invention, the guidance of the dialogue can be enhanced, and the answer is controllable.

Description

Technical Field

[0001] The present invention relates to the field of generative question answering, and in particular to a generative question answering method, device and readable medium based on classification guidance. Background Art

[0002] Based on the breakthroughs in natural language processing technology and large language models in generating conversations, generative conversational robots have been widely used, providing multi-faceted value to enterprises in business, and they play a key role in customer service and support. By responding to customer inquiries in real time, providing product information and solving common problems, conversational robots improve the efficiency of customer service, provide users with 24-hour support, reduce the workload of manual customer service, and thus improve customer satisfaction.

[0003] Existing customer service robots generally have low accuracy in positioning the question-answering type. Due to the diversity and complexity of natural language, user questions may involve different fields, topics or expressions, making it difficult for machines to accurately capture user intentions. At the same time, due to the limitations of semantic understanding and contextual inference, customer service robots may have positioning deviations when facing complex and ambiguous questions. In addition, industry-specific professional terms, emerging topics or user-personalized expressions all increase the difficulty of positioning accuracy.

[0004] The low quality and confusing logic of the medical customer service robot's consultation guidance also greatly affects the medical customer service robot's understanding of the user's symptoms and correct health advice. Poor guidance results in the robot's inability to accurately obtain key symptom information, failure to ask in-depth questions to obtain a complete medical history, or failure to adjust further questions based on the user's answers, which may lead to misunderstandings of the condition, inaccurate diagnosis or inappropriate advice, affecting the customer service robot's credibility and practicality in medical consultations.

[0005] Customer service robots may give uncontrollable answers to questions due to some interference. On the one hand, the ambiguity of language makes it difficult for machines to accurately capture the user's true intentions, especially in situations involving complex contexts or implicit information. In addition, the way users ask questions may vary due to personalization, regionality, and other reasons, making it difficult for customer service robots to maintain consistency when generating answers. On the other hand, the training data of the generative language model may contain various language expressions, including potential negative or inappropriate content, which may cause uncontrollable words in the customer service robot's answers, which may cause misunderstandings or inappropriate responses. Summary of the invention

[0006] The purpose of this application is to propose a generative question-answering method, device and readable medium based on classification guidance in response to the above-mentioned technical problems such as inaccurate positioning of question types, poor guidance of questioning, and uncontrollable answers of customer service robots.

[0007] In a first aspect, the present invention provides a generative question answering method based on classification guidance, comprising the following steps:

[0008] Collect several dialogue corpora and perform data cleaning and data processing respectively to obtain several processed dialogue corpora and construct the original dialogue dataset. The processed dialogue corpora include historical dialogues and reply sentences.

[0009] Constructing a topic classification model, an intent classification model, and an action recognition model and training them respectively to obtain a trained topic classification model, a trained intent classification model, and a trained action recognition model;

[0010] Extracting part of the processed dialogue corpus from the original dialogue dataset and obtaining the embedding vector corresponding to the historical dialogue in each of the processed dialogue corpus, inputting the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained topic classification model and the trained intent classification model, respectively obtaining the topic classification result and the intent classification result, inputting the topic classification result, the intent classification result and the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained action recognition model, predicting at least one action label corresponding to the reply sentence, constructing instructions in a thought chain manner based on the historical dialogue, the topic classification result, the intent classification result and the at least one action label, configuring different description sentences according to different action labels or different action label combinations and combining them with the corresponding instructions, storing the instructions in an instruction set, constructing a classification-guided instruction dialogue dataset according to the historical dialogue in each processed dialogue corpus and its corresponding instructions, reply sentences and description sentences, and mixing the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset;

[0011] A pre-trained generative question-answering model based on the hybrid expert model is constructed, and the pre-trained generative question-answering model is fine-tuned using a fine-tuning dataset to obtain a fine-tuned generative question-answering model; the historical dialogue to be replied is obtained and input into the fine-tuned generative question-answering model to obtain the corresponding output sentence.

[0012] Preferably, a topic classification model, an intention classification model and an action recognition model are constructed and trained respectively to obtain a trained topic classification model, a trained intention classification model and a trained action recognition model, which specifically include:

[0013] Construct a topic classification model based on the LSTM model and a pre-trained first Roberta model, collect a sentence-topic dataset, use the pre-trained first Roberta model as the teacher model, and the topic classification model as the student model. Input the sentences in the sentence-topic dataset into the teacher model to obtain the output of the teacher model. Use the sentence-topic dataset and the output of the teacher model to perform knowledge distillation training on the student model to obtain a trained topic classification model.

[0014] Build an intent classification model based on the LSTM model and a pre-trained second Roberta model, collect a sentence-intent dataset, use the pre-trained second Roberta model as the teacher model, and the intent classification model as the student model. Input the sentences in the sentence-intent dataset into the teacher model to obtain the output of the teacher model. Use the sentence-intent dataset and the output of the teacher model to perform knowledge distillation training on the student model to obtain a trained intent classification model.

[0015] Collect action classification data sets, build an action recognition model based on the Qianwen model or the BERT model, use the action classification data set to train the action recognition model to obtain a trained action recognition model, or build an action recognition model based on the LSTM model and a pre-trained third Roberta model, and collect sentence-intent data sets, use the pre-trained third Roberta model as the teacher model, and the action recognition model as the student model, input the sentences in the action classification data set and their corresponding topic classification results and intent classification results into the teacher model, obtain the output results of the teacher model, use the action classification data set and the output results of the teacher model to perform knowledge distillation training on the student model, and obtain a trained action recognition model.

[0016] Preferably, each feedforward network layer in the Transformer model of the pre-trained generative question-answering model is replaced with a fusion module, and the fusion module includes several expert networks and gating networks corresponding to action labels; in the fine-tuning process of the pre-trained generative question-answering model, the historical conversations or historical conversations and instructions in the fine-tuning dataset are input into the pre-trained generative question-answering model, and the topic classification results and intention classification results in the historical conversations or historical conversations and instructions are passed through the gating network to obtain a probability vector, which represents the probability of the expert network corresponding to each action label being selected, and the historical conversations are allocated to the expert network corresponding to the action label according to different blocks according to the probability vector, and the output results of all expert networks are aggregated and weighted fused to obtain the corresponding output sentence.

[0017] Preferably, if at least one action label predicted by historical conversations and topic classification results and intention classification results in instructions retrieves a corresponding description statement in the instruction set, the reply statement and the description statement are concatenated as the output statement, otherwise the reply statement is directly used as the output statement.

[0018] Preferably, during the fine-tuning process of the generative question-answering model, the parameters of the expert network are frozen, and the parameters of the gating network are adjusted. The fine-tuning process of the generative question-answering model adopts supervised fine-tuning, and the fine-tuning methods adopted include LoRA fine-tuning method, Q-lora fine-tuning method or P-tuning fine-tuning method.

[0019] Preferably, the action tags include question answering, consultation, association, connection and confirmation, and the action tag combination is obtained by constructing at least two action tags in the order required by business needs.

[0020] In a second aspect, the present invention provides a generative question-answering device based on classification guidance, comprising:

[0021] A data acquisition module is configured to collect several segments of dialogue data and perform data cleaning and data processing respectively, obtain several segments of processed dialogue data and construct an original dialogue data set, wherein the processed dialogue data includes historical dialogues and reply statements;

[0022] A model building module is configured to build a topic classification model, an intent classification model, and an action recognition model and train them respectively to obtain a trained topic classification model, a trained intent classification model, and a trained action recognition model;

[0023] A fine-tuning dataset construction module is configured to extract a portion of processed dialogue corpora from the original dialogue dataset and obtain an embedding vector corresponding to a historical dialogue in each processed dialogue corpus, input the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into a trained topic classification model and a trained intent classification model, respectively obtain a topic classification result and an intent classification result, input the topic classification result, the intent classification result and the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into a trained action recognition model, predict at least one action label corresponding to a reply statement, construct instructions in a thought chain manner based on the historical dialogue, the topic classification result, the intent classification result and the at least one action label, configure different description statements according to different action labels or different action label combinations and combine them with corresponding instructions, store the instructions in an instruction set, construct a classification-guided instruction dialogue dataset according to the historical dialogue in each processed dialogue corpus and its corresponding instructions, reply statements and description statements, and mix the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset;

[0024] The generation module is configured to build a pre-trained generative question-answering model based on the hybrid expert model, fine-tune the pre-trained generative question-answering model using a fine-tuning dataset to obtain a fine-tuned generative question-answering model; obtain the historical dialogue to be replied and input it into the fine-tuned generative question-answering model to obtain the corresponding output sentence.

[0025] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0027] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The generative question-answering method based on classification guidance proposed in the present invention establishes a more fine-grained classification standard, and identifies the topic classification results, intent classification results and action labels in historical conversations through a trained topic classification model, a trained intent classification model and a trained action recognition model, so as to better capture the user's implicit intentions and topics, and construct instructions with good guidance effects, which can solve the problem of inaccurate positioning of the existing question-answering types of customer service robots in the medical field.

[0030] (2) The classification-guided generative question-answering method proposed in the present invention utilizes a trained topic classification model, a trained intent classification model, and a trained action recognition model to construct a classification-guided instruction dialogue dataset, and mixes the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset. The fine-tuning dataset is used to perform supervised fine-tuning on the pre-trained generative question-answering model, so that it can output corresponding reply sentences according to different action labels, and establish more complete and logical description sentences according to different needs to connect the reply sentences corresponding to different actions, so as to better interact with users, solve the problem of poor guidance of customer service robots, make the answers more controllable, and thus improve the success rate of achieving task requirements.

[0031] (3) The pre-trained generative question answering model in the classification-guided generative question answering method proposed in the present invention adopts the hybrid expert principle and is fine-tuned through the constructed fine-tuning dataset. Therefore, it has the ability to deeply explore semantic understanding and context perception, and can generate output sentences that are more logical and meet business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of a flow chart of a generative question-answering method based on classification guidance according to an embodiment of the present application;

[0034] Figure 2 A schematic diagram of a generative question answering model of a generative question answering method based on classification guidance according to an embodiment of the present application;

[0035] Figure 3 A schematic diagram of a generative question-answering device based on classification guidance according to an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] Figure 1 A generative question answering method based on classification guidance provided by an embodiment of the present application is shown, comprising the following steps:

[0039] S1, collect several segments of dialogue corpus and perform data cleaning and data processing respectively, obtain several segments of processed dialogue corpus and construct the original dialogue dataset. The processed dialogue corpus includes historical dialogues and reply sentences.

[0040] Specifically, first obtain the dialogue corpus generated by the real-person dialogue, and perform data cleaning and preprocessing on the dialogue forecast. The preprocessing methods include denoising, processing missing values, etc. to ensure the diversity of the data, and sample some unbalanced indicator data. The dialogue forecast includes dialogue role information and dialogue content. In an embodiment of the present application, the dialogue role information includes user roles and customer service roles. The processed dialogue corpus obtained after data cleaning and preprocessing includes historical dialogues and reply statements. Historical dialogues and reply statements can be defined based on dialogue role information. Historical dialogues can have contextual logic. The historical dialogue contains several user statements output by user roles and reply statements output by customer service roles that are formed alternately. The last sentence of the historical dialogue is a user statement, and the corresponding sentence is a reply statement.

[0041] An example of a dialogue corpus is as follows:

[0042] User: I always have headaches recently. I would like to know what might be going on?

[0043] Customer Service: Headaches can have many causes, including lifestyle, stress, vision problems, etc. In order to better help you, I need to know some details. Have you been sleep deprived or stressed at work recently?

[0044] Customer: Yes, I have been very busy at work recently and haven’t been sleeping well.

[0045] Customer Service: Long working hours and insufficient sleep do affect your health. In addition to headaches, do you have any other symptoms of discomfort?

[0046] Customer: Sometimes I feel a little dizzy and my appetite is not very good.

[0047] Customer Service: Dizziness and loss of appetite may be related to headaches, but we need to know more. What's your WeChat number? I'll add you and provide assistance for your follow-up visit.

[0048] Customer: My WeChat is ×××.

[0049] S2, construct a topic classification model, an intent classification model and an action recognition model and train them respectively to obtain a trained topic classification model, a trained intent classification model and a trained action recognition model.

[0050] In a specific embodiment, step S2 specifically includes:

[0051] Construct a topic classification model based on the LSTM model and a pre-trained first Roberta model, collect a sentence-topic dataset, use the pre-trained first Roberta model as the teacher model, and the topic classification model as the student model. Input the sentences in the sentence-topic dataset into the teacher model to obtain the output of the teacher model. Use the sentence-topic dataset and the output of the teacher model to perform knowledge distillation training on the student model to obtain a trained topic classification model.

[0052] Build an intent classification model based on the LSTM model and a pre-trained second Roberta model, collect a sentence-intent dataset, use the pre-trained second Roberta model as the teacher model, and the intent classification model as the student model. Input the sentences in the sentence-intent dataset into the teacher model to obtain the output of the teacher model. Use the sentence-intent dataset and the output of the teacher model to perform knowledge distillation training on the student model to obtain a trained intent classification model.

[0053] Collect action classification data sets, build an action recognition model based on the Qianwen model or the BERT model, use the action classification data set to train the action recognition model to obtain a trained action recognition model, or build an action recognition model based on the LSTM model and a pre-trained third Roberta model, and collect sentence-intent data sets, use the pre-trained third Roberta model as the teacher model, and the action recognition model as the student model, input the sentences in the action classification data set and their corresponding topic classification results and intent classification results into the teacher model, obtain the output results of the teacher model, use the action classification data set and the output results of the teacher model to perform knowledge distillation training on the student model, and obtain a trained action recognition model.

[0054] Specifically, the topic classification model, intent classification model and action recognition model adopt the Roberta-LSTM knowledge distillation structure and are trained on the sentence-topic dataset, sentence-intent dataset and action classification dataset respectively to obtain the trained topic classification model, the trained intent classification model and the trained action recognition model. The pre-trained Roberta model has been trained on a large-scale unlabeled corpus to obtain the ability to encode input sentences into embeddings. Then, downstream task training is performed on the sentence-topic dataset, sentence-intent dataset and action classification dataset. Then, the LSTM model is trained with the output results of the sentence-topic dataset, sentence-intent dataset, action classification dataset and the pre-trained Roberta model for knowledge distillation, so that the LSTM model as the student model obtains the classification ability of the Roberta model as the teacher model, and has a smaller model parameter amount and faster processing speed. The trained topic classification model, the trained intent classification model and the trained action recognition model are trained respectively, and the embedding vectors corresponding to the historical dialogues are respectively identified to obtain the topic classification results, intent classification results and action labels, so as to accurately divide the historical dialogues into fine-grained parts. The training steps for the topic classification model, intent classification model, and action recognition model include:

[0055] 1) Single sentence corpus extraction;

[0056] 2) Data annotation;

[0057] 3) Model training to achieve an accuracy rate of more than 80%.

[0058] When a historical conversation is received, the embedding vectors corresponding to the historical conversation are first input into the trained topic classification model and the trained intent classification model respectively, so as to obtain the topic classification results and the intent classification results. Then, the topic classification results, the intent classification results and the embedding vectors corresponding to the historical conversation are input into the trained action recognition model to obtain the action label. Table 1 shows examples of the input historical conversation and the recognized action label.

[0059] Table 1

[0060] Historical Dialogue Action Tags Did I swim too much? Q&A, label 1 Is it because I eat too much? Q&A, label 1 I often have stomach pain recently Consultation, label 2 Why does my neck hurt? Consultation, label 2 I have a lot of black spots. Consultation, label 2 My hand has been swollen for two weeks. Consultation, label 2 Can your hospital handle it? Set, label is 3 The last time I went to the hospital, my hand was not healed. Set, label is 3

[0061] The trained topic classification model, the trained intent classification model, and the trained action recognition model are all classification models obtained through supervised training in advance. The labels of the models are constrained within a range according to business needs. For example, the number of categories of the topic classification model, the intent classification model, and the action recognition model are k1, k2, and k3 respectively. Among them, k1, k2, and k3 are all integers greater than 1.

[0062] Furthermore, the action recognition model can also be directly trained on the Qianwen 1.5B large model, the Qianwen 0.5B large model or the pre-trained BERT model to obtain a trained action recognition model.

[0063] S3, extracting part of the processed dialogue corpus from the original dialogue dataset and obtaining the embedding vector corresponding to the historical dialogue in each processed dialogue corpus, inputting the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained topic classification model and the trained intent classification model, respectively obtaining the topic classification results and the intent classification results, inputting the topic classification results, the intent classification results and the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained action recognition model, predicting at least one action label corresponding to the reply sentence, constructing instructions in a thinking chain manner based on the historical dialogue, the topic classification results, the intent classification results and the at least one action label, configuring different description sentences according to different action labels or different action label combinations and combining them with the corresponding instructions, storing the instructions in an instruction set, constructing a classification-guided instruction dialogue dataset according to the historical dialogues in each processed dialogue corpus and their corresponding instructions, reply sentences and description sentences, and mixing the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset.

[0064] In a specific embodiment, the action tags include question answering, consultation, association, connection and confirmation, and the action tag combination is obtained by constructing at least two action tags in the order required by business needs.

[0065] Specifically, the embodiment of the present application uses a trained topic classification model, a trained intent classification model, and a trained action recognition model to construct a classification guidance instruction dialogue data set. In the process of constructing the classification guidance instruction dialogue data set, first, M% of the processed dialogue corpus can be extracted from the original dialogue data set and the instructions can be fine-tuned. The M% of the processed dialogue corpus can be used to obtain the topic classification results and intent classification results using the trained topic classification model and the trained intent classification model. Then, the topic classification results and the intent classification results and the embedding vector corresponding to the historical dialogue are further input into the trained action recognition model to identify at least one corresponding action label, and the description statement is configured by at least one action label. Further, at least two action labels can be sequentially constructed into an action label combination, and the action label combination is the guidance interval. In the embodiment of the present application, the guidance interval is composed of three action labels, namely, answering questions, asking questions, and linking up, and a corresponding description statement is configured for each guidance interval. In order to avoid the situation where the customer service robot will be offline after answering questions, it is necessary to reasonably and effectively guide the communication during the consultation process, so as to link up the user more naturally. According to different requirements, the embodiment of the present application configures the description statement to achieve the purpose of "retaining users and obtaining contact information", as shown in Table 2.

[0066] Table 2

[0067]

[0068]

[0069] When establishing a classified guidance instruction dialogue data set, firstly, a chain of thought (COT) is used to construct an instruction based on historical dialogues, topic classification results, intention classification results, and at least one action tag identified. Specifically, the historical dialogues, topic classification results, intention classification results, and action tags are filled into appropriate instruction templates to construct the instructions generated by the dialogue, so that the instructions can be combined with historical dialogues to generate a reply statement that satisfies the semantics corresponding to the action tag. The reply statement corresponds to at least one action tag, and the number of action tags is greater than or equal to 1 and less than k3. In fact, the embodiment of the present application selects historical dialogues with no more than 8 action tags to proceed to the next step. In the above example, the logic chain must be composed of three elements: "answering questions", "inquiry", and "set", and the length is limited to avoid logical confusion in the dialogue output. When there are guidance intervals such as "answering questions-inquiry" in the logic chain, corresponding description statements are added between each guidance part. Instructions are given and pre-set description sentences are added. By fine-tuning the ability to learn conversational styles, the fine-tuned generative question-answering model can spontaneously determine whether it conforms to the guide interval when historical conversations are input later. If so, the generated reply sentence is concatenated with the description sentence corresponding to the guide interval that conforms to the guide interval as the output sentence of the generative question-answering model. Otherwise, the generated reply sentence is directly used as the output sentence of the generative question-answering model.

[0070] The following is a complete instruction template format:

[0071] You are now a {gynecology} professional practitioner. Please think about the following conversation between the customer service and the visitor on {xxxx.xx month xx day}. From the conversation history, you can first know that the content that the visitor is concerned about is {topic classification results}, and the visitor currently wants to know is {intent classification results}, so we can infer that the customer service can give a reply of {action label}: {reply sentence}.

[0072] The action tag combination is constructed according to specific business needs through a certain strategy. At least one action tag in the action tag combination is identified based on historical conversations, topic classification results, and intent classification results. The other action tags are selected according to specific business needs and constructed in order. The specific business needs need to consider the conversation round. For example: if the question-answering action tag identified by historical conversations, topic classification results, and intent classification results, because the customer service needs to prompt the conversation to proceed through consultation, if the set action tag is not identified, a consultation action tag will be added to the back of the question-answering action tag by default to combine, and the question-answering-consultation action tag combination is obtained. Then, the corresponding description statement is found according to the action tag combination, and the description statement is combined with the instruction for guidance, and the guidance generation is carried out in the order of the action tags. However, if the conversation round is greater than 5, and it is predicted to be a set action tag, then only one set action tag is retained in the instruction for guidance. In the above example, in order to better achieve the purpose of the set, the embodiment of the present application must interact with the customer to answer questions, so the description statement is configured for the generated content, so as to better connect to the set link. The description sentences of these different guide intervals are stored in the instructions of the classified guide instruction dialogue dataset. Through the fine-tuning process, the fine-tuned generative question-answering model can master the ability of set association.

[0073] After obtaining the classification guidance instruction dialogue dataset, it is mixed with the original dialogue dataset to form a fine-tuning dataset.

[0074] S4, construct a pre-trained generative question-answering model based on the hybrid expert model, use the fine-tuning dataset to fine-tune the pre-trained generative question-answering model to obtain a fine-tuned generative question-answering model; obtain the historical dialogue to be replied and input it into the fine-tuned generative question-answering model to obtain the corresponding output sentence.

[0075] In a specific embodiment, each feedforward network layer in the Transformer model of the pre-trained generative question-answering model is replaced with a fusion module, and the fusion module includes several expert networks and gating networks corresponding to action labels; in the fine-tuning process of the pre-trained generative question-answering model, the historical conversations or historical conversations and instructions in the fine-tuning dataset are input into the pre-trained generative question-answering model, and the topic classification results and intention classification results in the historical conversations or historical conversations and instructions are passed through the gating network to obtain a probability vector, which represents the probability of the expert network corresponding to each action label being selected, and the historical conversations are allocated to the expert network corresponding to the action label according to different blocks according to the probability vector, and the output results of all expert networks are aggregated and weighted fused to obtain the corresponding output sentence.

[0076] In a specific embodiment, if at least one action label predicted by the topic classification results and intention classification results in the historical conversation and the instructions retrieves a corresponding description statement in the instruction set, the reply statement and the description statement are concatenated as the output statement, otherwise the reply statement is directly used as the output statement.

[0077] In a specific embodiment, during the fine-tuning process of the generative question-answering model, the parameters of the expert network are frozen, and the parameters of the gating network are adjusted. The fine-tuning process of the generative question-answering model adopts supervised fine-tuning, and the fine-tuning methods adopted include LoRA fine-tuning method, Q-lora fine-tuning method or P-tuning fine-tuning method.

[0078] Specifically, the generative language model used in the embodiments of the present application adopts a hybrid expert model and performs supervised fine-tuning, and the fine-tuning operation uses fine-tuning methods such as LoRA, Q-lora or P-tuning. Specifically, the pre-trained generative language model is first fine-tuned in a supervised manner using a fine-tuning dataset. During the fine-tuning process of the pre-trained generative question-answering model, it is necessary to train the parameters of the gating network in the fusion module, and freeze the parameters of the expert network to avoid large-scale parameter changes in the subsequent fine-tuning process of the generative question-answering model, resulting in extremely high training overhead. The pre-trained generative question-answering model is fine-tuned through the fine-tuning dataset so that it has classification guidance capabilities while meeting normal dialogue generation, making the dialogue smoother and the logic clearer.

[0079] Specifically, refer to Figure 2 In the example of the present application, the content of the inquiry is generated after the question-answering according to the context, forming a behavioral pattern of both questions and answers, and generating set sentences at the appropriate time. The example of the present application sets up expert networks corresponding to question-answering, inquiry and set respectively, which have different parameters and design purposes, and determines through the gating network to input the historical dialogue into at least one of the expert networks corresponding to question-answering, inquiry and set, so as to output the reply sentence corresponding to the question-answering, inquiry and / or set, that is, the question-answering sentence, the inquiry sentence and / or the set sentence, and further, if the description sentence corresponding to the action label combination is matched in the instruction set, the reply sentence is combined with the description sentence to obtain the output sentence generated by the fine-tuned generative question-answering model. Specifically, the description sentence is used to connect the switching between the question-answering sentence, the inquiry sentence and the set sentence, so as to more naturally connect to the set link in the conversation with the customer.

[0080] Specifically, the pre-trained generative language model of the embodiment of the present application adopts the Mixture of Experts (MOE) principle. This part is composed of an expert network and a gating network. When a historical dialogue is input, the topic classification result and the intent classification result are obtained through a logic chain, and the topic classification result and the intent classification result are input into the gating network together as a bias. The gating network is composed of learnable parameters (including a linear layer and a Softmax layer). The gating network judges and inputs the historical dialogue into at least one expert network corresponding to the question-answering, medical consultation and set association to give a reply statement. The historical dialogue is input into the appropriate expert network for answering by training the gating network through fine-tuning the data set. Therefore, different types of expert network types can be selected to give a reply statement according to the topic classification results and intent classification results corresponding to the input historical dialogue, and a description statement is given when the guide interval (action label combination) is met, so as to better connect to the target link.

[0081] Furthermore, the topic classification results and intention classification results corresponding to the historical dialogue are compared with the guide interval (action label combination) in the instruction set to determine whether the topic classification results and intention classification results can match the guide interval. If so, it means that there is a description sentence corresponding to the matching guide interval, and it is combined with the corresponding instruction. Through the logic chain, the description sentence corresponding to the historical dialogue and the corresponding instruction output can be used as the output sentence of the fine-tuned generative question-answering model. Otherwise, there is no description sentence, and only the reply sentence output by the generative language model is used as the output sentence of the fine-tuned generative question-answering model. In addition, if the user inputs at least two user sentences containing different types in one round, the at least two user sentences can be segmented and classified first, and then each user sentence is input into the fine-tuned generative question-answering model to obtain the reply sentence corresponding to each user sentence, and then at least two reply sentences or at least two reply sentences and description sentences are used as output sentences to reply to the user.

[0082] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a generative question-answering device based on classification guidance, and the device embodiment is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0083] The embodiment of the present application provides a generative question-answering device based on classification guidance, including:

[0084] Data acquisition module 1 is configured to collect several segments of dialogue data and perform data cleaning and data processing respectively, obtain several segments of processed dialogue data and construct an original dialogue data set, the processed dialogue data includes historical dialogues and reply sentences;

[0085] Model building module 2 is configured to build a topic classification model, an intention classification model and an action recognition model and train them respectively to obtain a trained topic classification model, a trained intention classification model and a trained action recognition model;

[0086] The fine-tuning dataset construction module 3 is configured to extract part of the processed dialogue corpus from the original dialogue dataset and obtain the embedding vector corresponding to the historical dialogue in each processed dialogue corpus, input the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained topic classification model and the trained intention classification model, respectively obtain the topic classification result and the intention classification result, input the topic classification result, the intention classification result and the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained action recognition model, predict at least one action label corresponding to the reply sentence, construct instructions in a thinking chain manner based on the historical dialogue, the topic classification result, the intention classification result and the at least one action label, configure different description sentences according to different action labels or different action label combinations and combine them with the corresponding instructions, store the instructions in an instruction set, construct a classification-guided instruction dialogue dataset according to the historical dialogue in each processed dialogue corpus and its corresponding instructions, reply sentences and description sentences, and mix the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain the fine-tuning dataset;

[0087] The generation module 4 is configured to build a pre-trained generative question-answering model based on the hybrid expert model, fine-tune the pre-trained generative question-answering model using a fine-tuning dataset to obtain a fine-tuned generative question-answering model; obtain the historical dialogue to be replied and input it into the fine-tuned generative question-answering model to obtain the corresponding output sentence.

[0088] Figure 4 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 4 As shown, the electronic device of this embodiment includes: a processor 401 and a memory 402; wherein the memory 402 is used to store computer-executable instructions; the processor 401 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0089] Optionally, the memory 402 may be independent or integrated with the processor 401 .

[0090] When the memory 402 is independently provided, the electronic device further includes a bus 403 for connecting the memory 402 and the processor 401 .

[0091] An embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the above method is implemented.

[0092] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by a processor, the above method is implemented.

[0093] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0094] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0095] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0096] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0097] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0098] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0099] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0100] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0101] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0102] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A generative question answering method based on classification guidance, characterized in that: The following steps are involved: Collect several segments of dialogue corpus and perform data cleaning and data processing respectively to obtain several segments of processed dialogue corpus and construct an original dialogue data set, wherein the processed dialogue corpus includes historical dialogues and reply sentences; A topic classification model, an intent classification model, and an action recognition model are constructed and trained respectively to obtain a trained topic classification model, a trained intent classification model, and a trained action recognition model; Extracting a portion of processed dialogue corpora from the original dialogue dataset and obtaining an embedding vector corresponding to a historical dialogue in each processed dialogue corpus, inputting the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained topic classification model and the trained intent classification model, respectively obtaining a topic classification result and an intent classification result, inputting the topic classification result, the intent classification result and the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained action recognition model, predicting at least one action label corresponding to the reply statement, constructing instructions in a thought chain manner based on the topic classification result, the intent classification result and the at least one action label, configuring different description statements according to different action labels or different action label combinations and combining them with corresponding instructions, storing the instructions in the instruction set, constructing a classification-guided instruction dialogue dataset according to the historical dialogues in each processed dialogue corpus and their corresponding instructions, reply statements and description statements, and mixing the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset; A pre-trained generative question-answering model based on a hybrid expert model is constructed, and the pre-trained generative question-answering model is fine-tuned using the fine-tuning dataset to obtain a fine-tuned generative question-answering model; a historical dialogue to be replied is obtained and input into the fine-tuned generative question-answering model to obtain a corresponding output sentence.

2. The classification-guided generative question answering method according to claim 1, characterized in that: Construct a topic classification model, an intent classification model, and an action recognition model and train them separately to obtain a trained topic classification model, a trained intent classification model, and a trained action recognition model, which specifically include: Construct a topic classification model based on the LSTM model and a pre-trained first Roberta model, collect a sentence-topic data set, use the pre-trained first Roberta model as a teacher model, and the topic classification model as a student model, input the sentences in the sentence-topic data set into the teacher model, obtain the output result of the teacher model, and use the sentence-topic data set and the output result of the teacher model to perform knowledge distillation training on the student model to obtain a trained topic classification model; Construct an intent classification model based on the LSTM model and a pre-trained second Roberta model, collect a sentence-intent dataset, use the pre-trained second Roberta model as a teacher model, and the intent classification model as a student model. Input sentences in the sentence-intent dataset into the teacher model to obtain the output result of the teacher model. Use the sentence-intent dataset and the output result of the teacher model to perform knowledge distillation training on the student model to obtain a trained intent classification model. Collect an action classification data set, build an action recognition model based on the Qianwen model or the BERT model, use the action classification data set to train the action recognition model to obtain a trained action recognition model, or build an action recognition model based on the LSTM model and a pre-trained third Roberta model, and collect a sentence-intent data set, use the pre-trained third Roberta model as a teacher model, and the action recognition model as a student model, input the sentences in the action classification data set and their corresponding topic classification results and intent classification results into the teacher model to obtain the output result of the teacher model, use the action classification data set and the output result of the teacher model to perform knowledge distillation training on the student model to obtain a trained action recognition model.

3. The classification-guided generative question answering method according to claim 1, characterized in that: The pre-trained generative question-answering model replaces each feed-forward network layer in the Transformer model with a fusion module, and the fusion module includes a plurality of expert networks and a gating network corresponding to action labels. During the fine-tuning process of the pre-trained generative question-answering model, the historical dialogues or the historical dialogues and instructions in the fine-tuning dataset are input into the pre-trained generative question-answering model, and the topic classification results and the intention classification results in the historical dialogues or the historical dialogues and the instructions are passed through the gating network to obtain a probability vector, and the probability vector represents the probability of the expert network corresponding to each action label being selected. According to the probability vector, the historical dialogues are allocated to the expert networks corresponding to the action labels according to different blocks, and the output results of all the expert networks are aggregated and weighted fused to obtain the corresponding output sentences.

4. The classification-guided generative question answering method according to claim 3, characterized in that: If at least one action label predicted by the historical conversation and the topic classification results and the intention classification results in the instructions retrieves a corresponding description statement in the instruction set, the reply statement and the description statement are concatenated as the output statement; otherwise, the reply statement is directly used as the output statement.

5. The classification-guided generative question answering method according to claim 3, characterized in that: During the fine-tuning process of the generative question-answering model, the parameters of the expert network are frozen, and the parameters of the gating network are adjusted. The fine-tuning process of the generative question-answering model adopts supervised fine-tuning, and the fine-tuning methods adopted include LoRA fine-tuning method, Q-lora fine-tuning method or P-tuning fine-tuning method.

6. The classification-guided generative question answering method according to claim 1, characterized in that: The action tags include question answering, consultation, association, connection and confirmation, and the action tag combination is obtained by constructing at least two action tags in the order required by business needs.

7. A generative question-answering device based on classification guidance, characterized in that: include: A data acquisition module is configured to collect several segments of dialogue data and perform data cleaning and data processing respectively to obtain several segments of processed dialogue data and construct an original dialogue data set, wherein the processed dialogue data includes historical dialogues and reply statements; A model building module is configured to build a topic classification model, an intent classification model, and an action recognition model and train them respectively to obtain a trained topic classification model, a trained intent classification model, and a trained action recognition model; A fine-tuning dataset construction module is configured to extract a portion of processed dialogue corpora from the original dialogue dataset and obtain an embedding vector corresponding to a historical dialogue in each processed dialogue corpus, input the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained topic classification model and the trained intention classification model, respectively obtain a topic classification result and an intention classification result, input the topic classification result, the intention classification result and the embedding vector corresponding to the historical dialogue in each processed dialogue corpus into the trained action recognition model, predict and obtain at least one action label corresponding to the reply statement, construct instructions in a thought chain manner based on the topic classification result, the intention classification result and the at least one action label, configure different description statements according to different action labels or different action label combinations and combine them with corresponding instructions, store the instructions in the instruction set, construct a classification-guided instruction dialogue dataset according to the historical dialogues in each processed dialogue corpus and their corresponding instructions, reply statements and description statements, and mix the original dialogue dataset and the classification-guided instruction dialogue dataset to obtain a fine-tuning dataset; The generation module is configured to build a pre-trained generative question-answering model based on the hybrid expert model, fine-tune the pre-trained generative question-answering model using the fine-tuning dataset to obtain a fine-tuned generative question-answering model; obtain the historical dialogue to be replied and input it into the fine-tuned generative question-answering model to obtain the corresponding output sentence.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images