Question and answer model training method and device, storage medium and electronic equipment

Abstract

The invention discloses a question and answer model training method and device, a storage medium and electronic equipment. In the question and answer model training method provided by the specification, a consultation case is obtained, and the consultation case comprises a consultation record between a consultation service provider and a consultation user; semantic disassembling is conducted on the consultation case, question elements, object elements and consultation elements are obtained, the question elements are used for representing question information related to the consultation case, the object elements are used for representing information of consultation users, and the consultation elements are used for representing information of consultation service providers; a triple containing states, actions and rewards is constructed, the states are determined according to the question elements and the object elements, the actions are determined according to the consultation elements, and the rewards are determined according to consultation results of the consultation cases and the object elements; and performing reinforcement learning on a pre-trained first question and answer model according to the state, the action and the reward to obtain a target question and answer model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular, to a method, device, storage medium, and electronic device for training a question-and-answer model. Background Art

[0002] Nowadays, the consulting Q&A technology based on Artificial Intelligence (AI) is being continuously popularized and applied to various fields. With the development of Natural Language Processing (NLP) technology, many consulting systems have started to adopt an NLP-based approach, generating more natural conversations through training on large-scale corpora, being able to recognize the language content input by users and generate appropriate responses according to the context. However, due to the wide range of content involved in consulting, traditional NLP-based methods are difficult to meet diverse consulting needs.

[0003] With the development of large models, methods based on fine-tuning of large models have emerged. However, in the process of training large models, the way of processing data is relatively rough. Only various corpora are directly input into the large model for simple fine-tuning, and the large model usually can only learn knowledge but cannot learn how to use knowledge to give high-quality responses. Therefore, existing question-and-answer models often can only give general suggestions in the actual consulting process and are difficult to give accurate and specific full-process implementation methods.

[0004] Therefore, how to train a question-and-answer model that can give more refined and practical suggestions is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a method, device, storage medium, and electronic device for training a question-and-answer model to at least partially solve the above problems existing in the prior art.

[0006] This specification adopts the following technical solutions:

[0007] This specification provides a method for training a question-and-answer model, including:

[0008] Obtaining a consulting case, where the consulting case includes the consulting record between the consulting service provider and the consulting user;

[0009] Semantically disassembling the consulting case to obtain problem elements, object elements, and consulting elements, where the problem elements are used to represent the problem information involved in the consulting case, the object elements are used to represent the information of the consulting user, and the consulting elements are used to represent the information of the consulting service provider;

[0010] Construct a triple containing state, action, and reward, where the state is determined according to the problem element and the object element, the action is determined according to the consultation element, and the reward is determined according to the consultation result of the consultation case and the object element;

[0011] Perform reinforcement learning on a pre-trained first question-answering model according to the state, the action, and the reward to obtain a target question-answering model.

[0012] Optionally, the problem element at least includes problem content, problem-related parties, problem time, and problem environment, and the object element at least includes the basic information of the consulting user and the first text, first action, and first emotion of the consulting user in the consultation record; the consultation element at least includes the second text and second action of the consultation service provider in the consultation record.

[0013] Optionally, determining the state according to the problem element and the object element, and determining the action according to the consultation element specifically includes:

[0014] For each round of conversation in the consultation case, determine the problem element and object element of this round of conversation as the state of this round of conversation, and determine the consultation element of this round of conversation as the action of this round of conversation.

[0015] Optionally, determining the reward according to the consultation result of the consultation case and the object element specifically includes:

[0016] For each round of conversation in the consultation case, determine the pending reward of this round of conversation according to the first action and first emotion of the consulting user in this round of conversation, and the first action and first emotion of the consulting user in the next round of conversation of this round of conversation;

[0017] Adjust the pending rewards of each round of conversation according to the consultation result of the consultation case to obtain the rewards of each round of conversation.

[0018] Optionally, performing reinforcement learning on a pre-trained first question-answering model according to the state, the action, and the reward to obtain a target question-answering model specifically includes:

[0019] For each round of conversation in the consultation case, determine the state and action of this round of conversation as text corpus, and input it into a pre-trained first question-answering model to adjust the parameters of the first question-answering model to obtain a second question-answering model;

[0020] For each round of conversation in the consultation case, input the state and action of this round of conversation into the second question-answering model to obtain the predicted reward and predicted object element output by the second question-answering model;

[0021] Determine a loss function based on the predicted reward and the predicted object element, and adjust the parameters of the second question-and-answer model using the loss function to obtain an evaluation model;

[0022] Perform joint training using the second question-and-answer model and the evaluation model to obtain a target question-and-answer model.

[0023] Optionally, determining a loss function based on the predicted reward and the predicted object element specifically includes:

[0024] Determine a first difference between the predicted reward and the reward of this round of conversation, and a second difference between the predicted object element and the object element of the next round of conversation of this round of conversation;

[0025] Determine a loss function based on the first difference and the second difference.

[0026] Optionally, performing joint training using the second question-and-answer model and the evaluation model specifically includes:

[0027] Obtain sample question elements and sample object elements;

[0028] Input the sample question elements and the sample object elements into the second question-and-answer model to obtain predicted consultation elements generated by the second question-and-answer model;

[0029] Input the sample question elements, sample object elements, and the predicted consultation elements into the evaluation model to obtain an evaluation reward output by the evaluation model;

[0030] Adjust the parameters of the second question-and-answer model according to the evaluation reward to obtain a target question-and-answer model.

[0031] A question-and-answer model training device provided in this specification, the device includes:

[0032] An acquisition module, configured to acquire consultation cases, where the consultation cases include consultation records between a consultation service provider and a consultation user;

[0033] A disassembling module, configured to semantically disassemble the consultation case to obtain question elements, object elements, and consultation elements, where the question elements are used to represent the problem information involved in the consultation case, the object elements are used to represent the information of the consultation user, and the consultation elements are used to represent the information of the consultation service provider;

[0034] A construction module, configured to construct a triple including a state, an action, and a reward, where the state is determined according to the question elements and the object elements, the action is determined according to the consultation elements, and the reward is determined according to the consultation result of the consultation case and the object elements;

[0035] A training module, configured to perform reinforcement learning on a pre-trained first question-and-answer model according to the state, the action, and the reward, to obtain a target question-and-answer model.

[0036] This specification provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the above-mentioned question-and-answer model training method.

[0037] This specification provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the above-mentioned question-and-answer model training method when executing the program.

[0038] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0039] In the question-and-answer model training method provided in this specification, a consultation case is obtained, where the consultation case includes a consultation record between a consultation service provider and a consultation user; the consultation case is semantically disassembled to obtain a problem element, an object element, and a consultation element, where the problem element is used to characterize the problem information involved in the consultation case, the object element is used to characterize the information of the consultation user, and the consultation element is used to characterize the information of the consultation service provider; a triple including a state, an action, and a reward is constructed, where the state is determined according to the problem element and the object element, the action is determined according to the consultation element, and the reward is determined according to the consultation result of the consultation case and the object element; reinforcement learning is performed on a pre-trained first question-and-answer model according to the state, the action, and the reward, to obtain a target question-and-answer model.

[0040] By disassembling the semantics of existing consultation cases into problem elements, object elements, and consultation elements, clustering and standardizing them, this method can effectively extract and standardize a large amount of consultation data, laying a good foundation for the further development and optimization of subsequent related models. At the same time, this method constructs a generative consultation advice assistance method based on reinforcement learning, which can provide personalized advice for each consultation process. By differentially combining the reward function with the state of the consultation object, it helps to achieve dynamic adjustment for each consultation scenario, thereby improving the fineness and practicality of the consultation effect. In addition, when training the reward function, this method increases the prediction of the feedback of the consultation object, so that the simultaneously generated evaluation model can be effectively applied in consultation teaching and practice. Description of the Drawings

[0041] The accompanying drawings described herein are used to provide a further understanding of the present specification, and constitute a part of the present specification. The schematic embodiments and descriptions thereof in the present specification are used to explain the present specification, and do not constitute an improper limitation to the present specification. In the drawings:

[0042] Figure 1 It is a schematic flowchart of a method for training a question-and-answer model in the present specification;

[0043] Figure 2 It is a schematic diagram of the relationship between each element obtained by semantic decomposition and a triple provided in the present specification;

[0044] Figure 3 It is a schematic diagram of the relationship between each model involved in a joint training process provided in the present specification;

[0045] Figure 4 It is a schematic diagram of a device for training a question-and-answer model provided in the present specification;

[0046] Figure 5 corresponding to the one provided in the present specification Figure 1 schematic diagram of an electronic device. Specific embodiments

[0047] To make the purpose, technical solutions and advantages of the present specification clearer, the technical solutions of the present specification will be clearly and completely described below in conjunction with specific embodiments of the present specification and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present specification, rather than all the embodiments. Based on the embodiments in the present specification, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of this application.

[0048] The following will describe in detail the technical solutions provided by each embodiment of the present specification in conjunction with the drawings.

[0049] Figure 1 It is a schematic flowchart of a method for training a question-and-answer model in the present specification, specifically including the following steps:

[0050] S100: Obtain a consultation case, where the consultation case includes the consultation record between the consultation service provider and the consultation user.

[0051] All steps in the method for training a question-and-answer model provided in the present specification can be implemented by any electronic device with computing functions, such as devices like terminals and servers.

[0052] This method is mainly used to train a question-and-answer model for question consultation. Based on this, consultation cases can be obtained as training materials in this step. In this method, a consultation case is a record of a real consultation event on record, including the consultation record between the consultation service provider and the consultation user. Among them, the consultation event involved in the consultation case is a question-and-answer between people, and neither of the two consulting parties includes AI.

[0053] In the consulting industry, the consultation events between the consultation service provider and the consultation user will be recorded and archived as consultation cases. Among them, the consultation user is the party that raises the question, and the consultation service provider is the party that answers the question. The consultation case will at least include the content of each round of conversation between the consultation user and the consultation service provider, the actions and emotions of both parties during the conversation, and the consultation result of whether the consultation is ultimately successful, etc.

[0054] Compared with the dialogue samples between humans and AI, the consultation cases between humans have higher learning value. This method first obtains the consultation cases for use in training the question-and-answer model in subsequent steps.

[0055] S102: Semantically disassemble the consultation case to obtain a problem element, an object element, and a consultation element. Among them, the problem element is used to represent the problem information involved in the consultation case, the object element is used to represent the information of the consultation user, and the consultation element is used to represent the information of the consultation service provider.

[0056] In traditional question-and-answer model training methods, generally, the corpus such as the conversation content is directly input into a basic model such as a large language model to complete the training of the question-and-answer model. However, in this traditional method, the trained model can often only learn a small amount of content, the training effect is very little, and the capabilities shown in actual applications are also very difficult to be satisfactory.

[0057] In this method, the way of directly inputting the training corpus into the large language model without processing in the traditional method is abandoned. Instead, the consultation case is semantically disassembled first before training. The semantic disassembly of this method disassembles each consultation case into three parts, namely, the problem element, the object element, and the consultation element.

[0058] The problem element is used to characterize the information related to the core problem that the consulting user consults the consulting service provider in the consulting case. Specifically, it can at least include elements such as problem content, problem-related parties, problem time, and problem environment. Among them, the problem content is the specific content of the core problem that the consulting user consults the consulting service provider in this consultation; the problem-related parties are the people, things, and objects involved in the core problem in this consultation; the problem time is the time when this consultation occurs and other times involved in the consultation; the problem environment is all background factors when the consulting user consults the consulting service provider, such as location, weather, social background, etc.

[0059] The object element is used to characterize the information related to the consulting user in the consulting case. Specifically, it can at least include the basic information of the consulting user and the first text, first action, and first emotion of the consulting user in the consulting record. Among them, the basic information of the consulting user is the user portrait information, which can include, but is not limited to, information such as gender, age, and job; the first text is the text of the content stated by the consulting user in each round of Q&A in the consulting process; the first action is the action made by the consulting user in each round of Q&A; the first emotion is the emotion of the consulting user in each round of Q&A.

[0060] The consulting element is used to characterize the information related to the consulting service provider in the consulting case. Specifically, it can at least include the second text and second action of the consulting service provider in each round of Q&A in the consulting process. Among them, the second text is the text of the content stated by the consulting service provider in each round of Q&A, and the second action is the action made by the consulting service provider in each round of Q&A.

[0061] In this step, each consulting case is semantically split in the above manner, and the obtained problem elements, object elements, and consulting elements can be used to train the Q&A model in subsequent steps.

[0062] The above semantic splitting operation can be completed in a variety of different ways. For example, a general large language model can be used for splitting, etc. This specification does not make specific restrictions on this.

[0063] Additionally, among the contents included in each element obtained after semantic splitting, there may very likely be a part of entities with the same meaning but different expressions. For this part of the content, embedding and clustering processing can be performed to standardize the semantic expression, merge entities with the same semantics, label entities with the same meaning, and assign the same label, so that there is only one corresponding expression for one semantics, reducing the amount of data to be processed in subsequent training. Among them, the embedding and clustering process can be implemented through methods such as the SBERT model, and the annotation process can be implemented through, for example, manual annotation or annotation by a large model.

[0064] S104: Construct a triple containing a state, an action, and a reward, where the state is determined based on the problem element and the object element, the action is determined based on the consultation element, and the reward is determined based on the consultation result of the consultation case and the object element.

[0065] After the semantic decomposition of the consultation case is completed in step S102, the elements obtained from the decomposition can be further processed in this step to obtain a triple of state, action, and reward for reinforcement learning training of the question-and-answer model.

[0066] Figure 2 This is a schematic diagram showing the relationship between the elements obtained from semantic decomposition and the triple in this specification. As Figure 2 shown, in this method, the state is determined based on the problem element and the object element; the action is determined based on the consultation element; the reward is determined based on the consultation result of the consultation case itself and the object element. Among them, each triple contains a state, an action, and a reward, and each triple can be used as a training sample in reinforcement learning. A corresponding triple can be determined for each round of conversation in the consultation case. Among them, a round of conversation is defined as a question from the consultation user plus the answer to this question by the consultation service provider.

[0067] When determining the state and action in a triple, specifically, for each round of conversation in the consultation case, the problem element and the object element of this round of conversation are determined as the state of this round of conversation, and the consultation element of this round of conversation is determined as the action of this round of conversation.

[0068] When determining the reward in a triple, it can be jointly determined by combining the result of the consultation case and the object element. Specifically, for each round of conversation in the consultation case, the pending reward of this round of conversation can be determined according to the first action and the first emotion of the consultation user in this round of conversation, and the first action and the first emotion of the consultation user in the next round of conversation of this round of conversation; according to the consultation result of the consultation case, the pending rewards of each round of conversation are adjusted to obtain the rewards of each round of conversation.

[0069] In the above method, the design idea of the reward is to determine the reward according to the state change of the consulting user in each round of conversation. During the consultation process, the state of the consulting user will change with the conversation, mainly manifested as different degrees of satisfaction. When the consulting user has a high degree of satisfaction, the consulting user may have positive emotions such as happiness in the conversation and may make corresponding actions; similarly, when the consulting user has a low degree of satisfaction, the consulting user may have negative emotions such as anger and confusion in the conversation and may make corresponding actions. Based on this, the effect of this round of conversation can be judged according to the state difference between this round of conversation and the next round of conversation of the consulting user, that is, the change of the first action and the first emotion of the consulting user, and then the pending reward of the triple corresponding to this round of conversation can be determined.

[0070] Furthermore, this method believes that when determining the reward for each round of conversation, the consultation result of the consultation case to which the conversation belongs should also be taken into consideration. Based on this, in this method, after determining the pending reward for each round of conversation, the pending reward for each round of conversation will be adjusted according to the consultation result of the consultation case to which the conversation belongs to obtain the final reward for each round of conversation.

[0071] For example, in a specific embodiment, the pending rewards can be divided into three levels, sorted from low to high as getting worse, remaining the same, and getting better. When the satisfaction level of the consulting user in the next round of conversation is lower than that in this round of conversation, the pending reward for this round of conversation is getting worse; when the satisfaction level of the consulting user in the next round of conversation remains the same as that in this round of conversation, the pending reward for this round of conversation is remaining the same; when the satisfaction level of the consulting user in the next round of conversation is higher than that in this round of conversation, the pending reward for this round of conversation is getting better.

[0072] At the same time, the consultation results can be divided into two types: successful consultation and failed consultation. When the consultation result of a consultation case is a successful consultation, the pending rewards of all rounds of conversations included in this consultation case are increased by one level; conversely, when the consultation result of a consultation case is a failed consultation, the pending rewards of all rounds of conversations included in this consultation case are decreased by one level. When the pending reward is getting worse, it becomes very bad after being decreased by one level; when the pending reward is getting better, it becomes very good after being increased by one level. In other words, the finally determined rewards can be divided into five levels, sorted from low to high as very bad, getting worse, remaining the same, getting better, and very good.

[0073] The above embodiments are only one of the optional specific embodiments when determining the rewards, and there may be other forms for the specific settings of the rewards, which are not specifically limited in this specification.

[0074] S106: Perform reinforcement learning on the pre-trained first question-and-answer model according to the state, the action, and the reward to obtain the target question-and-answer model.

[0075] After determining the triples for reinforcement learning in step S104, reinforcement learning can finally be performed on the pre-trained first question-answering model in this step to obtain the target question-answering model. Among them, the first question-answering model can be any model with basic question-answering capabilities after pre-training, such as a general large language model, etc. This specification does not make specific restrictions on this.

[0076] Furthermore, in the process of obtaining the target question-answering model, a method combining reinforcement learning and joint training can be adopted to obtain a target question-answering model with better performance. Specifically, for each round of conversation in the consultation case, the state and action of this round of conversation are determined as text corpus and input into the pre-trained first question-answering model to adjust the parameters of the first question-answering model to obtain a second question-answering model; for each round of conversation in the consultation case, the state and action of this round of conversation are input into the second question-answering model to obtain the predicted reward and predicted object elements output by the second question-answering model; a loss function is determined according to the predicted reward and the predicted object elements, and the parameters of the second question-answering model are adjusted using the loss function to obtain an evaluation model; the second question-answering model and the evaluation model are jointly trained to obtain the target question-answering model.

[0077] In the above joint training method, two models for joint training need to be constructed first. One is the model for generating consultation strategies, that is, the model that can implement the question-answering function, and it is also the main training object, namely the second question-answering model. The second question-answering model can be obtained by fine-tuning the first question-answering model, and the corpus used for fine-tuning is the state and action obtained by disassembling the consultation case. Among them, the corpus for fine-tuning can be set in the form of questions and answers, the question is the state, that is, the combination of the question element and the object element; the answer is the action, that is, the consultation element. By means of, for example, the LoRA framework, the first question-answering model is fine-tuned using the corpus in the above form to obtain the second question-answering model.

[0078] The other is the evaluation model for evaluating the model that generates consultation strategies. The evaluation model can be obtained by performing supervised learning on the second question-answering model. Specifically, the state and action in the triple, or the conversation content of a round of conversation, can be used as the training samples input to the second question-answering model, and the corresponding reward is used as the label for supervised learning. In the above training, the output of the second question-answering model is the estimated advantage of the input, and the possible emotional changes of the consulting user, that is, the predicted reward and the predicted object elements.

[0079] The predicted rewards and predicted object elements output by the second Q&A model can both be used to train the second Q&A model. Specifically, a first difference between the predicted reward and the reward of this round of conversation can be determined, as well as a second difference between the predicted object element and the object element of the next round of conversation of this round of conversation; a loss function is determined according to the first difference and the second difference.

[0080] Among them, the first difference is the difference between the predicted reward output by the model and the true reward of this round of conversation. The optional items of the predicted reward output by the model are the same as those of the reward of each round of conversation when set. For example, assume that the optional gears of the reward when set are getting worse, remaining unchanged, and getting better, then the possible outputs of the second Q&A model are also getting worse, remaining unchanged, and getting better. When calculating, the above optional rewards can be quantified. For example, getting worse is quantified as -1, remaining unchanged is quantified as 0, and getting better is quantified as 1. Thus, the first difference can be determined.

[0081] The second difference is the difference between the predicted object element output by the model and the object element of the next round of conversation. Similarly, the optional items of the predicted object element output by the second Q&A model are the same as the content that the object element itself can contain. The determination method of the second difference is similar to that of the first difference, and it can also be determined in a quantified form, which will not be elaborated in this specification.

[0082] Furthermore, there can be multiple ways to determine the loss function according to the first difference and the second difference. For example, the mean square error between the predicted reward and the actual reward and the cross-entropy loss between the predicted object element and the actual object element can be jointly determined as the loss function, and this specification does not make specific restrictions on this.

[0083] The second Q&A model is trained and adjusted through the above loss function, and finally an evaluation model can be obtained.

[0084] Finally, the second Q&A model and the evaluation model can be jointly trained. When conducting joint training, specifically, sample question elements and sample object elements can be obtained; the sample question elements and the sample object elements are input into the second Q&A model to obtain the predicted consultation elements generated by the second Q&A model; the sample question elements, sample object elements, and the predicted consultation elements are input into the evaluation model to obtain the evaluation rewards output by the evaluation model; the parameters of the second Q&A model are adjusted according to the evaluation rewards to obtain the target Q&A model.

[0085] The specific idea of the above joint training is that the second question-answering model is responsible for generating corresponding answers according to questions, and the evaluation model is used to evaluate the answers generated by the second question-answering model. Subsequently, the second question-answering model can fine-tune its parameters based on the evaluation rewards given by the evaluation model to complete the training. Finally, after the above joint training, the second question-answering model can obtain a target question-answering model with better consultation effect.

[0086] Figure 3 This is a schematic diagram showing the relationship between the models involved in a joint training process provided in this specification. As Figure 3 shown, the first question-answering model is a basic general large language model, which can be directly obtained by obtaining an open-source large model; the second question-answering model is obtained by fine-tuning the first question-answering model through methods such as the LoRA framework; the evaluation model is obtained by performing supervised learning on the second question-answering model; finally, after the joint training of reinforcement learning between the second question-answering model and the evaluation model, a target question-answering model evolved from the second question-answering model is obtained.

[0087] It is worth mentioning that after obtaining the target question-answering model, the joint training between the target question-answering model and the evaluation model can also be continuously carried out with the records of the consultation process with users as samples during subsequent application processes, and the target question-answering model and the evaluation model can be continuously optimized and iterated to ensure that the target question-answering model maintains good performance and timeliness.

[0088] When training the target question-answering model using the question-answering model training method provided in this specification, the question elements, object elements, and consultation elements can be obtained by semantic disassembling of consultation cases, and triples can be constructed based on this; and the first question-answering model can be reinforced using the triples to obtain the final target question-answering model.

[0089] Most of the consultation systems obtained by traditional training methods rely on simple keyword matching or predefined templates, with weak semantic understanding ability and unable to handle complex and multi-dimensional consultation scenarios. In contrast, this method can effectively extract and standardize a large amount of consultation data by disassembling the semantics of existing consultation cases into question elements, object elements, and consultation elements, and performing clustering and standardization, laying a good foundation for the further development and optimization of subsequent related models. At the same time, this method constructs a generative consultation advice assistance method based on reinforcement learning, which can provide personalized advice for each consultation process. By differentially combining the reward function with the state of the consultation object, it helps to achieve dynamic adjustment for each consultation scenario, thereby improving the fineness and practicality of the consultation effect. In addition, when training the reward function, this method increases the prediction of the feedback of the consultation object, enabling the simultaneously generated evaluation model to be effectively applied in consultation teaching and practice.

[0090] The above is the Q&A model training method provided in this specification. Based on the same idea, this specification also provides a corresponding Q&A model training device, as Figure 4 shown.

[0091] Figure 4 is a schematic diagram of a Q&A model training device provided in this specification, specifically including:

[0092] An acquisition module 200, configured to acquire consultation cases, where the consultation cases include consultation records between a consultation service provider and a consultation user;

[0093] A disassembling module 202, configured to semantically disassemble the consultation case to obtain problem elements, object elements, and consultation elements. Among them, the problem elements are used to represent the problem information involved in the consultation case, the object elements are used to represent the information of the consultation user, and the consultation elements are used to represent the information of the consultation service provider;

[0094] A construction module 204, configured to construct a triple including a state, an action, and a reward. Among them, the state is determined according to the problem elements and the object elements, the action is determined according to the consultation elements, and the reward is determined according to the consultation result of the consultation case and the object elements;

[0095] A training module 206, configured to perform reinforcement learning on a pre-trained first Q&A model according to the state, the action, and the reward to obtain a target Q&A model.

[0096] Optionally, the problem elements at least include problem content, problem related parties, problem time, and problem environment, the object elements at least include the basic information of the consultation user and the first text, first action, and first emotion of the consultation user in the consultation record; the consultation elements at least include the second text and second action of the consultation service provider in the consultation record.

[0097] Optionally, the construction module 204 is specifically configured to, for each round of conversation in the consultation case, determine the problem elements and object elements of this round of conversation as the state of this round of conversation, and determine the consultation elements of this round of conversation as the action of this round of conversation.

[0098] Optionally, the construction module 204 is specifically configured to, for each round of conversation in the consultation case, determine the pending reward of this round of conversation according to the first action and first emotion of the consultation user in this round of conversation, and the first action and first emotion of the consultation user in the next round of conversation of this round of conversation; adjust the pending rewards of each round of conversation according to the consultation result of the consultation case to obtain the rewards of each round of conversation.

[0099] Optionally, the training module 206 is specifically configured to, for each round of conversation in the consultation case, determine the state and action of this round of conversation as text corpus and input it into a pre-trained first question-and-answer model to adjust the parameters of the first question-and-answer model to obtain a second question-and-answer model; for each round of conversation in the consultation case, input the state and action of this round of conversation into the second question-and-answer model to obtain a predicted reward and a predicted object element output by the second question-and-answer model; determine a loss function according to the predicted reward and the predicted object element, and use the loss function to adjust the parameters of the second question-and-answer model to obtain an evaluation model; perform joint training on the second question-and-answer model and the evaluation model to obtain a target question-and-answer model.

[0100] Optionally, the training module 206 is specifically configured to determine a first difference between the predicted reward and the reward of this round of conversation, and a second difference between the predicted object element and the object element of the next round of conversation of this round of conversation; determine a loss function according to the first difference and the second difference.

[0101] Optionally, the training module 206 is specifically configured to obtain sample question elements and sample object elements; input the sample question elements and the sample object elements into the second question-and-answer model to obtain predicted consultation elements generated by the second question-and-answer model; input the sample question elements, the sample object elements, and the predicted consultation elements into the evaluation model to obtain an evaluation reward output by the evaluation model; adjust the parameters of the second question-and-answer model according to the evaluation reward to obtain a target question-and-answer model.

[0102] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above Figure 1 provided question-and-answer model training method.

[0103] This specification also provides Figure 5 a schematic structural diagram of the electronic device shown. As Figure 5 described, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, there may also be other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 described question-and-answer model training method. Of course, in addition to the software implementation method, this specification does not exclude other implementation methods, such as logical devices or a combination of software and hardware. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or logical devices.

[0104] For an improvement in a technology, it can be clearly distinguished whether it is a hardware improvement (e.g., improvement in circuit structures such as diodes, transistors, switches, etc.) or a software improvement (improvement in method processes). However, with the development of technology, many improvements in method processes today can be regarded as direct improvements in hardware circuit structures. Almost all designers obtain the corresponding hardware circuit structure by programming the improved method process into the hardware circuit. Therefore, it cannot be said that an improvement in a method process cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can program themselves to "integrate" a digital system on a single PLD without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. Currently, the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method process with the above-mentioned several hardware description languages and programming it into the integrated circuit, it is easy to obtain the hardware circuit that implements the logical method process.

[0105] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or structures within the hardware component.

[0106] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0107] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0108] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code.

[0109] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and combinations of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce means for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0110] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including instruction means that implement the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one Figure 1 flow or multiple flows and / or blocks Figure 1 or means for implementing the functions specified in one block or multiple blocks.

[0112] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0113] The memory may include non-permanent memory in the form of computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. The memory is an example of computer-readable media.

[0114] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0116] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0118] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the corresponding description in the method embodiment.

[0119] The above description is only for the embodiments of this specification and is not intended to limit this specification. For those skilled in the art, various modifications and changes can be made to this specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included within the scope of the claims of this application.

Claims

1. A question-answering model training method, characterized in that: include: Obtaining consulting cases, wherein the consulting cases include consulting records between consulting service providers and consulting users; Semantically decomposing the consulting case to obtain a question element, an object element, and a consulting element, wherein the question element is used to represent the question information involved in the consulting case, the object element is used to represent the information of the consulting user, and the consulting element is used to represent the information of the consulting service provider; Constructing a triple containing a state, an action, and a reward, wherein the state is determined according to the question element and the object element, the action is determined according to the consultation element, and the reward is determined according to the consultation result of the consultation case and the object element; According to the state, the action, and the reward, reinforcement learning is performed on the pre-trained first question-answering model to obtain a target question-answering model.

2. The method according to claim 1, characterized in that The question elements include at least the question content, question-related parties, question time, and question environment; the object elements include at least the basic information of the consulting user and the first text, first action, and first emotion of the consulting user in the consultation record; the consulting elements include at least the second text and second action of the consulting service provider in the consultation record.

3. The method according to claim 1, characterized in that Determining a state according to the question element and the object element, and determining an action according to the consultation element, specifically includes: For each round of dialogue in the consultation case, the question element and the object element of the dialogue are determined as the state of the dialogue, and the consultation element of the dialogue is determined as the action of the dialogue.

4. The method according to claim 2, characterized in that The reward is determined based on the consulting results of the consulting case and the object elements, including: For each round of dialogue in the consultation case, determine the pending reward for the round of dialogue according to the first action and the first emotion of the consulting user in the round of dialogue, and the first action and the first emotion of the consulting user in the next round of dialogue after the round of dialogue; According to the consulting results of the consulting case, the pending rewards for each round of dialogue are adjusted to obtain the rewards for each round of dialogue.

5. The method according to claim 1, characterized in that According to the state, the action, and the reward, reinforcement learning is performed on the pre-trained first question-answering model to obtain a target question-answering model, specifically including: For each round of dialogue of the consulting case, the state and action of the round of dialogue are determined as text corpus, and the pre-trained first question-answering model is input to adjust the parameters of the first question-answering model to obtain a second question-answering model; For each round of dialogue of the consulting case, the state and action of the round of dialogue are input into the second question-answering model to obtain the predicted reward and predicted object element output by the second question-answering model; Determining a loss function according to the predicted reward and the predicted object element, and using the loss function to adjust parameters of the second question-answering model to obtain an evaluation model; The second question-answering model is jointly trained with the evaluation model to obtain a target question-answering model.

6. The method according to claim 5, characterized in that Determining a loss function according to the predicted reward and the predicted object element specifically includes: Determine a first difference between the predicted reward and the reward for the current round of dialogue, and a second difference between the predicted object element and the object element for the next round of dialogue after the current round of dialogue; A loss function is determined according to the first difference and the second difference.

7. The method according to claim 5, characterized in that The second question-answering model is used to perform joint training with the evaluation model, specifically including: Get sample question element and sample object element; Inputting the sample question element and the sample object element into the second question-answering model to obtain a prediction consultation element generated by the second question-answering model; Inputting the sample question element, the sample object element and the prediction consultation element into the evaluation model to obtain the evaluation reward output by the evaluation model; The parameters of the second question-answering model are adjusted according to the evaluation reward to obtain a target question-answering model.

8. A question-answering model training device, characterized in that: include: An acquisition module, used to acquire consulting cases, wherein the consulting cases include consulting records between consulting service providers and consulting users; A disassembly module, used for semantically disassembling the consulting case to obtain a problem element, an object element and a consulting element, wherein the problem element is used to represent the problem information involved in the consulting case, the object element is used to represent the information of the consulting user, and the consulting element is used to represent the information of the consulting service provider; A construction module, used to construct a triple containing a state, an action, and a reward, wherein the state is determined according to the question element and the object element, the action is determined according to the consultation element, and the reward is determined according to the consultation result of the consultation case and the object element; A training module is used to perform reinforcement learning on the pre-trained first question-answering model according to the state, the action and the reward to obtain a target question-answering model.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images