Customer service question and answer model optimization method and device, electronic equipment and storage medium

Abstract

The invention provides a customer service question and answer model optimization method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a first data set which comprises an expected standard question and at least one corresponding first similar question; inputting the first similar question sentences into a customer service question and answer model to obtain first prediction standard question sentences corresponding to the first similar question sentences; determining a first similar question, different from the expected standard question, of the first predicted standard question as a second similar question; and determining a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question and at least one first similar question corresponding to the expected standard question of the second similar question, and optimizing the customer service question and answer model based on the second data set. Therefore, the second data set which is not learned by the model can be obtained, and the model is optimized based on the second data set, so that the performance of the customer service question and answer model is improved, and the prediction accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent question-answering technology, and in particular to an optimization method, device, electronic device, and storage medium for a customer service question-answering model. Background Art

[0002] Intelligent customer service robots are usually composed of modules such as automatic speech recognition (ASR), intent recognition, question-answering module, knowledge base management, knowledge graph, dialogue management, text generation, and text to speech (TTS). Among them, the frequently asked questions (FAQ) based on the knowledge base and the answers to the corresponding questions are the most important modules in intelligent customer service robots. They provide users with satisfactory answers by searching the knowledge base for standard questions that match the user's questions. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose a method for optimizing a customer service question and answer model, which can obtain a second data set that has not been learned by the customer service question and answer model, and then optimize the customer service question and answer model based on the second data set, thereby improving the performance of the customer service question and answer model and thereby improving the prediction accuracy.

[0005] The second objective of this application is to propose an optimization device for a customer service question and answer model.

[0006] The third objective of the present application is to provide an electronic device.

[0007] A fourth objective of the present application is to provide a computer-readable storage medium.

[0008] A fifth object of the present application is to provide a computer program product.

[0009] To achieve the above objectives, the first embodiment of the present application proposes a method for optimizing a customer service question-and-answer model, including:

[0010] Acquire a first data set, wherein the first data set includes an expected standard question sentence and at least one corresponding first similar question sentence;

[0011] Inputting the first similar question into a customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question;

[0012] determining a first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as a second similar question sentence;

[0013] Determine a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes a first target standard question and at least one first target similar question corresponding to the first target standard question;

[0014] Based on the second data set, the customer service question and answer model is optimized.

[0015] To achieve the above-mentioned purpose, the second embodiment of the present application proposes an optimization device for a customer service question-and-answer model, comprising:

[0016] An acquisition module, configured to acquire a first data set, wherein the first data set includes an expected standard question sentence and at least one corresponding first similar question sentence;

[0017] A processing module, configured to input the first similar question into a customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question;

[0018] A first determination module is used to determine a first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as a second similar question sentence;

[0019] A second determination module is configured to determine a second data set according to a first predicted standard question of the second similar question, an expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes a first target standard question and at least one first target similar question corresponding to the first target standard question;

[0020] An optimization module is used to optimize the customer service question and answer model based on the second data set.

[0021] To achieve the above-mentioned purpose, the third aspect of the present application provides an electronic device, including:

[0022] A processor, and a memory communicatively connected to the processor;

[0023] The memory stores computer-executable instructions;

[0024] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the embodiment of the first aspect.

[0025] To achieve the above-mentioned purpose, the fourth aspect embodiment of the present application proposes a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processor, they are used to implement the method described in the first aspect embodiment.

[0026] To achieve the above-mentioned purpose, the fifth aspect of the present application proposes a computer program product, a computer program, which implements the method described in the first aspect when executed by a processor.

[0027] The optimization method, device, electronic device and storage medium of the customer service question and answer model provided by the present application obtain a first data set, wherein the first data set includes an expected standard question and at least one corresponding first similar question, input the first similar question into the customer service question and answer model, obtain a first predicted standard question corresponding to the first similar question, determine the first predicted standard question different from the expected standard question as a second similar question, determine the second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question, and optimize the customer service question and answer model based on the second data set. Thus, the first similar question predicted incorrectly in the first data set, the corresponding first predicted standard question and the first expected standard question can be processed according to the customer service question and answer model to obtain a second data set that has not been learned by the customer service question and answer model, and then the customer service question and answer model is optimized based on the second data set, so that the performance of the customer service question and answer model can be improved, and the prediction accuracy can be improved.

[0028] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A flowchart of a method for optimizing a customer service question-and-answer model provided in an embodiment of the present application;

[0031] Figure 2 A flowchart of another method for optimizing a customer service question-and-answer model provided in an embodiment of the present application;

[0032] Figure 3 A flowchart of another method for optimizing a customer service question-and-answer model provided in an embodiment of the present application; and

[0033] Figure 4 A schematic diagram of the structure of an optimization device for a customer service question and answer model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0035] In some embodiments, the customer service question and answer model can be applied to insurance business handling scenarios, can be used for questions related to insurance business handling, and train the customer service question and answer model. Alternatively, it can also be applied to other question and answer scenarios. This disclosure does not limit this.

[0036] The following describes the optimization method and device of the customer service question and answer model of the embodiment of the present application with reference to the accompanying drawings.

[0037] Figure 1 A flowchart of a method for optimizing a customer service question and answer model provided in an embodiment of the present application.

[0038] like Figure 1 As shown, the optimization method of the customer service question and answer model includes the following steps:

[0039] Step 101: Acquire a first data set, wherein the first data set includes an expected standard question and at least one corresponding first similar question.

[0040] The first similar question corresponding to the expected standard question is a question with the same semantics as the expected standard question.

[0041] The first data set may include a plurality of expected standard questions, and one expected standard question may correspond to one or more first similar questions.

[0042] Among them, the first data set can be a training data set of the customer service question and answer model.

[0043] Step 102: input the first similar question into the customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question.

[0044] It should be noted that the knowledge base stores expected standard questions and the answers to the expected standard questions. After obtaining the user's question, it is necessary to match the user's question with the expected standard question, and then determine the answer to the successfully matched expected standard question as the answer to the user's question. Therefore, it is necessary to train a customer service question-answering model based on standard questions and corresponding similar questions.

[0045] The prediction accuracy of the customer service question-and-answer model is greater than the accuracy threshold. For example, the accuracy threshold may be 80%, 85%, etc. This disclosure does not limit this.

[0046] Step 103: determine the first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as the second similar question sentence.

[0047] Among them, the first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence is the first similar question sentence that is incorrectly predicted by the customer service question-answering model.

[0048] Step 104 , determining a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question.

[0049] The second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question.

[0050] In some embodiments, when the expected standard question is the first expected standard question and the first predicted standard question is the second expected standard question and the first number of second similar questions corresponding to the question is greater than a quantity threshold, the first similarity between the first expected standard question and the second expected standard question is determined; when the first similarity is greater than a first threshold, the first expected standard question and the second expected standard question are merged to obtain a first target standard question; all first similar questions associated with the first expected standard question and all first similar questions associated with the second expected standard question are determined as first target similar questions corresponding to the first target standard question.

[0051] For example, a plurality of first similar questions and corresponding expected standard questions and first predicted standard questions are shown in Table 1:

[0052] Table 1

[0053]

[0054]

[0055] Among them, similar question 4 and similar question 5 correspond to expected standard question A, and the corresponding first predicted standard question is expected standard question B. If the quantity threshold is 2, the expected standard question B is determined as the first expected labeled question, and the expected standard question A is determined as the second expected labeled question.

[0056] It should be noted that if the first similarity between the first expected standard question and the second expected standard question is greater than the first threshold, it means that the first expected standard question and the second expected standard question have the same semantics and can be merged into the same expected standard question. Therefore, in the disclosed embodiment, the first expected standard question and the second expected standard question are merged to obtain the first target standard question, and all the first similar questions associated with the first expected standard question and all the first similar questions associated with the second expected standard question are determined as the first target similar questions corresponding to the first target standard question.

[0057] In some embodiments, the first expected standard question and the second expected standard question may be input into a large model to determine a first similarity between the first expected standard question and the second expected standard question.

[0058] In some embodiments, the target similarity between the first expected standard question sentence and the second expected standard question sentence may also be determined by the following steps:

[0059] (1) determining a second number of second similar questions corresponding to the expected standard question being the second expected standard question and the first predicted standard question being the first expected standard question;

[0060] (2) determining the total number of second similar questions, the third number corresponding to all first similar questions of the first expected standard question, the fourth number corresponding to all first similar questions of the second expected standard question, the fifth number corresponding to the second similar questions whose expected standard question is the first expected standard question, and the sixth number corresponding to the second similar questions whose expected standard question is the second expected standard question;

[0061] Among them, the total number of second similar questions is the total number of first similar questions that are incorrectly predicted by the customer service question and answer model.

[0062] The third number is the number of first similar questions having the same semantics as the first expected standard question.

[0063] The fourth number is the number of first similar questions having the same semantics as the second expected standard question.

[0064] The fifth number is the number of first similar questions corresponding to the first expected standard question that are predicted incorrectly by the customer service question answering model. For example, if the number of first similar questions corresponding to the first expected standard question is 10, and 5 of the first similar questions are predicted incorrectly by the customer service question answering model, then the fifth number is determined to be .

[0065] The fifth number is the number of first similar questions corresponding to the second expected standard question that are predicted incorrectly by the customer service question answering model. For example, if the number of first similar questions corresponding to the second expected standard question is 12, and 4 of them are predicted incorrectly by the customer service question answering model, then the sixth number is determined to be 4.

[0066] (3) determining a first ratio between the first quantity and the third quantity, a second ratio between the second quantity and the fourth quantity, a third ratio between the sum of the first quantity and the second quantity and the total quantity, a fourth ratio between the fifth quantity and the total quantity, and a fifth ratio between the sixth quantity and the total quantity;

[0067] The first ratio represents the probability that the first similar question corresponding to the first expected standard question is recognized as the second expected standard question by the customer service question and answer model.

[0068] The second ratio represents the probability that the first similar question corresponding to the second expected standard question is recognized as the first expected standard question by the customer service question and answer model.

[0069] Among them, the third ratio represents the probability of being recognized as the first expected standard question and the second expected standard question by the customer service question and answer model among the total number of second similar questions.

[0070] Among them, the fourth ratio represents the probability that the customer service question and answer model should recognize the first expected standard question among the total number of second similar questions.

[0071] Among them, the fifth ratio represents the probability that the customer service question and answer model should recognize the second expected standard question among the total number of second similar questions.

[0072] (4) determining the ratio between the sum of the first ratio and the second ratio and the third ratio as the semantic confusion probability between the first expected standard question and the second expected standard question;

[0073] Among them, the formula for semantic confusion probability is as follows:

[0074]

[0075] Wherein, Q_a is the second expected standard question, Q_b is the first expected standard question, S_i is the i-th first similar question corresponding to the second expected standard question, S_j is the j-th first similar question corresponding to the first expected standard question, P(Q_a|{S_i,Q_b}) is the second ratio. P(Q_b|{S_j,Q_a}) is the first ratio. P(Q_a,Q_b) is the third ratio. P({Q_a,Q_b}) is the semantic confusion probability between the first expected standard question and the second expected standard question.

[0076] (5) Determine the ratio of the semantic confusion probability to the sum of the fourth ratio and the fifth ratio as the first similarity between the first expected standard question and the second expected standard question.

[0077] Therefore, the first similarity between the first expected standard question and the second expected standard question can be accurately determined based on the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question.

[0078] In some embodiments, the first expected standard question may be determined as the first target standard question; or, the second expected standard question may be determined as the first target standard question; or, the first expected standard question may be merged with the second expected standard question to obtain the first target standard question.

[0079] Step 105: Optimize the customer service question and answer model based on the second data set.

[0080] It should be noted that the first similar question sentence that is different from the expected standard question sentence is the first similar question sentence that is incorrectly predicted by the customer service question-answering model. In the disclosed embodiment, the first similar question sentence that is incorrectly predicted by the initial prediction model, and the corresponding first predicted standard question sentence and the first expected standard question sentence are processed to obtain a new first target standard question sentence and the corresponding first target similar question sentence, and the customer service question-answering model is continuously trained, thereby improving the performance of the customer service question-answering model and improving the prediction accuracy.

[0081] In this embodiment, a first data set is obtained, wherein the first data set includes an expected standard question and at least one corresponding first similar question, the first similar question is input into the customer service question and answer model, a first predicted standard question corresponding to the first similar question is obtained, the first predicted standard question is different from the expected standard question, and the first similar question is determined as a second similar question, and a second data set is determined according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question, and the customer service question and answer model is optimized based on the second data set. Thus, the first similar question predicted incorrectly in the first data set, and the corresponding first predicted standard question and first expected standard question can be processed according to the customer service question and answer model to obtain a second data set that has not been learned by the customer service question and answer model, and then the customer service question and answer model is optimized based on the second data set, so that the performance of the customer service question and answer model can be improved, and then the prediction accuracy can be improved.

[0082] Figure 2The following is a flow chart of a method for optimizing a customer service question-and-answer model provided in an embodiment of the present application. Figure 2 As shown, the optimization method of the customer service question and answer model may include the following steps:

[0083] Step 201 : Acquire a first data set, wherein the first data set includes an expected standard question and at least one corresponding first similar question.

[0084] Step 202: input the first similar question into the customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question.

[0085] Step 203: determine the first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as the second similar question sentence.

[0086] Step 204 , determining a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question.

[0087] The second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question.

[0088] The specific implementation forms of steps 201 to 204 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0089] Step 205: Obtain the user's query statement.

[0090] Step 206: input the query statement into the customer service question-answering model to obtain a second predicted standard question sentence corresponding to the query statement.

[0091] Step 207 : determining a second target standard question and a corresponding second target similar question according to the query sentence, the second predicted standard question and the first data set.

[0092] In some embodiments, when the second predicted standard question is the same as the third expected standard question in the first data set, and the query statement is different from the first similar question corresponding to the third expected standard question, the query statement and the third expected standard question are input into the large model to obtain a second similarity between the query statement and the third expected standard question. When the second similarity is greater than a second threshold, the third expected standard question is determined as the second target standard question, and the query statement is determined as the second target similar question corresponding to the third expected standard question.

[0093] The third expected standard question is the expected standard question in the first data set that is the same as the second predicted standard question.

[0094] It should be noted that the second predicted standard question is the same as the third expected standard question in the first data set, and the query statement is not the same as the first similar question corresponding to the third expected standard question, indicating that in the data set of the customer service question and answer model, the query statement is not the first similar question corresponding to the third expected standard question. In the disclosed embodiment, the second similarity between the query statement and the third expected standard question is further determined, and when the second similarity is greater than the second threshold, the third expected standard question and the corresponding query statement are added to the second data set, thereby increasing the training data of the customer service question and answer model and improving the performance of the customer service question and answer model.

[0095] In some embodiments, determining the second target standard question and the corresponding second target similar question according to the query sentence, the second predicted standard question and the first data set further includes:

[0096] (1) A query sentence in which the second predicted standard question sentence is different from all expected standard questions in the first data set is determined as a target query sentence.

[0097] (2) Clustering all target query statements to obtain at least one cluster set, wherein one cluster set contains multiple target query statements.

[0098] In some embodiments, the target query statement may be semantically clustered to obtain at least one set of clusters with the same semantics. For example, a K-means algorithm may be used to semantically cluster the target query statement. This disclosure does not limit this.

[0099] Among them, the semantics of multiple target query statements in a clustering set are the same.

[0100] (3) Multiple target query statements in each cluster set are input into the large model to obtain the fusion statement corresponding to each cluster set.

[0101] In some embodiments, any target query statement in the cluster set may also be determined as a fusion statement corresponding to the cluster set.

[0102] (4) Match the fused sentence with the expected standard questions in the first dataset.

[0103] (5) When the fused sentence is the same as the fourth expected standard question, the fourth expected standard question is determined as the second target standard question, and the multiple target query sentences in the cluster set corresponding to the fused sentence are the second target similar questions corresponding to the fourth expected standard question.

[0104] The fourth expected standard question is an expected standard question in the first data set that has the same semantics as the fused sentence.

[0105] (6) Alternatively, when the fused sentence is different from all expected standard questions in the first data set, the fused sentence is determined as the second target standard question, and the multiple target query sentences in the cluster set corresponding to the fused sentence are the second target similar questions corresponding to the fused sentence.

[0106] Step 208: Add the second target standard question and the corresponding second target similar question into the second data set.

[0107] Step 209: Optimize the customer service question and answer model based on the second data set.

[0108] In the disclosed embodiment, based on the prediction results of the customer service question and answer model for the first similar question in the first data set and the prediction results for the user's query, the first data set is combined to obtain a second data set, and the customer service question and answer model is optimized based on the second data set. Thus, the data diversity of the second data set is improved, thereby further improving the performance of the customer service question and answer model, and further improving the prediction accuracy.

[0109] Figure 3 The following is a flow chart of a method for optimizing a customer service question-and-answer model provided in an embodiment of the present application. Figure 3 As shown, the optimization method of the customer service question and answer model may include the following steps:

[0110] Step 301: Acquire a first data set, wherein the first data set includes an expected standard question and at least one corresponding first similar question.

[0111] Step 302: input the first similar question into the customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question.

[0112] Step 303: determine the first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as the second similar question sentence.

[0113] Step 304 : determining a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question.

[0114] The second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question.

[0115] The specific implementation forms of steps 201 to 204 can refer to the detailed descriptions in other embodiments of the present disclosure, and will not be described in detail here.

[0116] Step 305: Input the user's query statement into the customer service question-answering model to obtain a second predicted standard question statement corresponding to the query statement.

[0117] Step 306, when the second predicted standard question is the same as any expected standard question in the first data set, obtain a target answer corresponding to the second predicted standard question from a knowledge base, wherein the knowledge base contains each expected standard question and a corresponding answer.

[0118] Step 307: Obtain user feedback data on the target answer.

[0119] The feedback data may be data indicating whether the user likes or dislikes the target answer.

[0120] Step 308: Determine a third target standard question and a corresponding third target similar question according to the query sentence, the feedback data and the first data set.

[0121] In some embodiments, the target answer and the query statement are input into the large model to obtain a third similarity between the query statement and the target answer. When the feedback data is positive feedback, the third similarity is greater than a third threshold, and the query statement is different from the first similar question corresponding to any expected standard question, any expected standard question is determined as the third target standard question, and the query statement is determined as the third target similar question corresponding to any expected standard question.

[0122] Among them, positive feedback is user likes. It should be noted that when the user likes, it is necessary to further determine whether the target answer and the query sentence are really highly similar. Therefore, when the third similarity is greater than the third threshold, and the query sentence is different from the first similar question corresponding to any expected standard question, any expected standard question is determined as the third target standard question, and the query sentence is determined as the third target similar question corresponding to any expected standard question.

[0123] In some embodiments, the target answer and the query statement are input into the large model to obtain a third similarity between the query statement and the target answer. When the feedback data is negative feedback, the third similarity is less than a fourth threshold, and the query statement is the same as any first similar question corresponding to any expected standard question, any expected standard question is determined as the third target standard question, and the first similar question corresponding to any expected standard question in the first data set, except the query statement, is determined as the third target similar question corresponding to any expected standard question.

[0124] Among them, positive feedback refers to user clicks and dislikes.

[0125] The fourth threshold is smaller than the third threshold.

[0126] Step 309: Add the third target standard question and the corresponding third target similar question into the second data set.

[0127] Step 310: Optimize the customer service question and answer model based on the second data set.

[0128] In the disclosed embodiment, based on the prediction result of the customer service question and answer model for the first similar question in the first data set and the feedback data of the user on the processing result of the customer service question and answer model for the query, the first data set is combined to obtain a second data set, and the customer service question and answer model is optimized based on the second data set. Thus, the data diversity of the second data set is improved, thereby further improving the performance of the customer service question and answer model, and then improving the prediction accuracy.

[0129] In some embodiments, a second data set may be generated based on the first target standard question and the corresponding first target similar question, the second target standard question and the corresponding second target similar question, and the third target standard question and the corresponding third target similar question, and then the customer service question and answer model may be optimized based on the second data set. This may obtain more comprehensive training data, optimize the customer service question and answer model, and further improve the performance and prediction accuracy of the customer service question and answer model.

[0130] In order to implement the above embodiment, the present application also proposes an optimization device for a customer service question and answer model.

[0131] Figure 4 This is a schematic diagram of the structure of a customer service question and answer model optimization device provided in an embodiment of the present application. Figure 4 As shown, the optimization device of the customer service question and answer model includes:

[0132] The acquisition module 401 is used to acquire a first data set, wherein the first data set includes an expected standard question sentence and at least one corresponding first similar question sentence;

[0133] Processing module 402, for inputting the first similar question into the customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question;

[0134] A first determination module 403 is used to determine a first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as a second similar question sentence;

[0135] A second determination module 404 is used to determine a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question;

[0136] The optimization module 405 is used to optimize the customer service question and answer model based on the second data set.

[0137] In some embodiments, the second determining module 404 is configured to:

[0138] When the expected standard question is a first expected standard question and the first predicted standard question is a second expected standard question and the first number of second similar questions corresponding to the second expected standard question is greater than a number threshold, determining a first similarity between the first expected standard question and the second expected standard question;

[0139] When the first similarity is greater than the first threshold, merging the first expected standard question with the second expected standard question to obtain a first target standard question;

[0140] All first similar questions associated with the first expected standard question and all first similar questions associated with the second expected standard question are determined as first target similar questions corresponding to the first target standard question.

[0141] In some embodiments, the second determining module 404 is configured to:

[0142] Determine a second number of second similar questions corresponding to the expected standard question being the second expected standard question and the first predicted standard question being the first expected standard question;

[0143] Determine the total number of second similar questions, the third number corresponding to all first similar questions of the first expected standard question, the fourth number corresponding to all first similar questions of the second expected standard question, the fifth number corresponding to the second similar questions whose expected standard question is the first expected standard question, and the sixth number corresponding to the second similar questions whose expected standard question is the second expected standard question;

[0144] determining a first ratio between the first quantity and the third quantity, a second ratio between the second quantity and the fourth quantity, a third ratio between the sum of the first quantity and the second quantity and the total quantity, a fourth ratio between the fifth quantity and the total quantity, and a fifth ratio between the sixth quantity and the total quantity;

[0145] Determine the ratio between the sum of the first ratio and the second ratio and the third ratio as the semantic confusion probability between the first expected standard question and the second expected standard question;

[0146] The ratio of the semantic confusion probability to the sum of the fourth ratio and the fifth ratio is determined as the first similarity between the first expected standard question and the second expected standard question.

[0147] In some embodiments, a third determining module is further included, which is used to:

[0148] Get the user's query statement;

[0149] Input the query statement into the customer service question-answering model to obtain the second predicted standard question statement corresponding to the query statement;

[0150] Determine a second target standard question and a corresponding second target similar question according to the query sentence, the second predicted standard question and the first data set;

[0151] The second target standard question and the corresponding second target similar question are added to the second data set.

[0152] In some embodiments, the third determining module is used to:

[0153] When the second predicted standard question is the same as the third expected standard question in the first data set, and the query statement and the first similar question corresponding to the third expected standard question are different, inputting the query statement and the third expected standard question into the large model to obtain a second similarity between the query statement and the third expected standard question;

[0154] When the second similarity is greater than the second threshold, the third expected standard question is determined as the second target standard question, and the query sentence is determined as the second target similar question corresponding to the third expected standard question.

[0155] In some embodiments, the third determining module is used to:

[0156] Determine a query sentence in which the second predicted standard question sentence is different from all expected standard question sentences in the first data set as a target query sentence;

[0157] Clustering all target query statements to obtain at least one cluster set, wherein one cluster set includes multiple target query statements;

[0158] Input multiple target query statements in each cluster set into the large model to obtain the fusion statement corresponding to each cluster set;

[0159] Matching the fused sentences with expected standard questions in the first data set;

[0160] When the fused sentence is the same as the fourth expected standard question, the fourth expected standard question is determined as the second target standard question, and the multiple target query sentences in the cluster set corresponding to the fused sentence are second target similar questions corresponding to the fourth expected standard question;

[0161] Alternatively, when the fused sentence is different from all expected standard questions in the first data set, the fused sentence is determined as the second target standard question, and the multiple target query sentences in the cluster set corresponding to the fused sentence are the second target similar questions corresponding to the fused sentence.

[0162] In some embodiments, a fourth determining module is further included, configured to:

[0163] Input the user's query statement into the customer service question-answering model to obtain a second predicted standard question statement corresponding to the query statement;

[0164] When the second predicted standard question is the same as any expected standard question in the first data set, obtaining a target answer corresponding to the second predicted standard question from a knowledge base, wherein the knowledge base contains each expected standard question and a corresponding answer;

[0165] Obtain user feedback data on the target answer;

[0166] Determine a third target standard question and a corresponding third target similar question according to the query sentence, the feedback data and the first data set;

[0167] The third target standard question and the corresponding third target similar question are added to the second data set.

[0168] In some embodiments, a fourth determining module is included, configured to:

[0169] Input the target answer and the query statement into the large model to obtain the third similarity between the query statement and the target answer;

[0170] When the feedback data is positive feedback, the third similarity is greater than the third threshold, and the query sentence is different from the first similar question sentence corresponding to any expected standard question, any expected standard question sentence is determined as the third target standard question sentence, and the query sentence is determined as the third target similar question sentence corresponding to any expected standard question sentence; or,

[0171] When the feedback data is negative feedback, the third similarity is less than the fourth threshold, and the query statement is the same as any first similar question corresponding to any expected standard question, any expected standard question is determined as the third target standard question, and the first similar question corresponding to any expected standard question in the first data set except the query statement is determined as the third target similar question corresponding to any expected standard question.

[0172] It should be noted that the aforementioned explanation of the embodiment of the optimization method of the customer service question and answer model is also applicable to the optimization device of the customer service question and answer model of this embodiment, and will not be repeated here.

[0173] The optimization method, device, electronic device and storage medium of the customer service question and answer model provided by the present application obtain a first data set, wherein the first data set includes an expected standard question and at least one corresponding first similar question, input the first similar question into the customer service question and answer model, obtain a first predicted standard question corresponding to the first similar question, determine the first predicted standard question different from the expected standard question as a second similar question, determine the second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes the first target standard question and at least one first target similar question corresponding to the first target standard question, and optimize the customer service question and answer model based on the second data set. Thus, the first similar question predicted incorrectly in the first data set, the corresponding first predicted standard question and the first expected standard question can be processed according to the customer service question and answer model to obtain a second data set that has not been learned by the customer service question and answer model, and then the customer service question and answer model is optimized based on the second data set, so that the performance of the customer service question and answer model can be improved, and the prediction accuracy can be improved.

[0174] In order to implement the above embodiments, the present application also proposes an electronic device, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided by the above embodiments.

[0175] In order to implement the above embodiments, the present application also proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the methods provided by the above embodiments.

[0176] In order to implement the above embodiments, the present application also proposes a computer program product, including a computer program, which implements the methods provided by the above embodiments when executed by a processor.

[0177] The collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0178] It should be noted that personal information from users should be collected for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. In addition, such collection / sharing should be carried out after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign the agreement / authorization including authorization of relevant user information before the user uses the function. In addition, any necessary steps should be taken to protect and safeguard access to such personal information data and ensure that others who have access to personal information data comply with its privacy policy and procedures.

[0179] The present application is expected to provide an implementation scheme for users to selectively block the use or access of personal information data. That is, the present disclosure is expected to provide hardware and / or software to prevent or block access to such personal information data. Once the personal information data is no longer needed, the risk can be minimized by limiting data collection and deleting the data. In addition, when applicable, such personal information is de-identified to protect the privacy of the user.

[0180] In the description of the aforementioned embodiments, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0181] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0182] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0183] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0184] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0185] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0187] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A method for optimizing a customer service question-answering model, characterized in that: The following steps are involved: Acquire a first data set, wherein the first data set includes an expected standard question sentence and at least one corresponding first similar question sentence; Inputting the first similar question into a customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question; determining a first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as a second similar question sentence; Determine a second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes a first target standard question and at least one first target similar question corresponding to the first target standard question; Based on the second data set, the customer service question and answer model is optimized.

2. The method according to claim 1, characterized in that The determining of the second data set according to the first predicted standard question of the second similar question, the expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question includes: When the expected standard question is a first expected standard question and the first predicted standard question is a second expected standard question and the first number of second similar questions corresponding to the second expected standard question is greater than a number threshold, determining a first similarity between the first expected standard question and the second expected standard question; When the first similarity is greater than a first threshold, merging the first expected standard question with the second expected standard question to obtain the first target standard question; All first similar questions associated with the first expected standard question and all first similar questions associated with the second expected standard question are determined as first target similar questions corresponding to the first target standard question.

3. The method according to claim 2, characterized in that The determining a first similarity between the first expected standard question sentence and the second expected standard question sentence includes: Determine a second number of second similar questions whose expected standard question is the second expected standard question and whose first predicted standard question is the first expected standard question; Determine the total number of the second similar questions, the third number corresponding to all the first similar questions of the first expected standard question, the fourth number corresponding to all the first similar questions of the second expected standard question, the fifth number corresponding to the second similar questions whose expected standard question is the first expected standard question, and the sixth number corresponding to the second similar questions whose expected standard question is the second expected standard question; determining a first ratio between the first quantity and the third quantity, a second ratio between the second quantity and the fourth quantity, a third ratio between the sum of the first quantity and the second quantity and the total quantity, a fourth ratio between the fifth quantity and the total quantity, and a fifth ratio between the sixth quantity and the total quantity; determining the ratio between the sum of the first ratio and the second ratio and the third ratio as the semantic confusion probability between the first expected standard question and the second expected standard question; The ratio of the semantic confusion probability to the sum of the fourth ratio and the fifth ratio is determined as the first similarity between the first expected standard question and the second expected standard question.

4. The method according to claim 1, characterized in that Before optimizing the customer service question and answer model based on the second data set, the method further includes: Get the user's query statement; Input the query statement into the customer service question-answering model to obtain a second predicted standard question sentence corresponding to the query statement; Determine the second target standard question and the corresponding second target similar question according to the query sentence, the second predicted standard question and the first data set; The second target standard question and the corresponding second target similar question are added to the second data set.

5. The method according to claim 4, characterized in that The determining, according to the query sentence, the second predicted standard question sentence and the first data set, the second target standard question sentence and the corresponding second target similar question sentence comprises: When the second predicted standard question is the same as the third expected standard question in the first data set, and the query statement and the first similar question corresponding to the third expected standard question are different, inputting the query statement and the third expected standard question into the macro model to obtain a second similarity between the query statement and the third expected standard question; When the second similarity is greater than a second threshold, the third expected standard question is determined as the second target standard question, and the query sentence is determined as the second target similar question corresponding to the third expected standard question.

6. The method according to claim 4, characterized in that The determining, according to the query sentence, the second predicted standard question sentence and the first data set, the second target standard question sentence and the corresponding second target similar question sentence comprises: Determine a query sentence where the second predicted standard question sentence is different from all expected standard question sentences in the first data set as a target query sentence; Clustering all the target query statements to obtain at least one cluster set, wherein one cluster set contains multiple target query statements; Inputting the multiple target query statements in each cluster set into the large model to obtain a fusion statement corresponding to each cluster set; Matching the fused sentence with expected standard questions in the first data set; In the case where the fused sentence is the same as the fourth expected standard question, the fourth expected standard question is determined as the second target standard question, and the multiple target query sentences in the cluster set corresponding to the fused sentence are second target similar questions corresponding to the fourth expected standard question; Alternatively, when the fused statement is different from all expected standard questions in the first data set, the fused statement is determined as the second target standard question, and multiple target query statements in the cluster set corresponding to the fused statement are second target similar questions corresponding to the fused statement.

7. The method according to claim 1, characterized in that Before optimizing the customer service question and answer model based on the second data set, the method further includes: Inputting the user's query statement into the customer service question-answering model to obtain a second predicted standard question statement corresponding to the query statement; When the second predicted standard question is the same as any expected standard question in the first data set, obtaining a target answer corresponding to the second predicted standard question from a knowledge base, wherein the knowledge base contains each of the expected standard questions and a corresponding answer; Obtaining user feedback data on the target answer; Determining the third target standard question and the corresponding third target similar question according to the query sentence, the feedback data and the first data set; The third target standard question and the corresponding third target similar question are added to the second data set.

8. The method according to claim 7, characterized in that The determining, according to the query statement, the feedback data and the first data set, the third target standard question and the corresponding third target similar question comprises: Inputting the target answer and the query statement into the large model to obtain a third similarity between the query statement and the target answer; When the feedback data is positive feedback, the third similarity is greater than a third threshold, and the query sentence is different from the first similar question sentence corresponding to any expected standard question, determining any expected standard question sentence as the third target standard question sentence, and determining the query sentence as the third target similar question sentence corresponding to any expected standard question sentence; or, When the feedback data is negative feedback, the third similarity is less than a fourth threshold, and the query statement is the same as any first similar question corresponding to any expected standard question, the any expected standard question is determined as the third target standard question, and the first similar question corresponding to any expected standard question in the first data set except the query statement is determined as the third target similar question corresponding to the any expected standard question.

9. An optimization device for a customer service question-answering model, characterized in that: include: An acquisition module, configured to acquire a first data set, wherein the first data set includes an expected standard question sentence and at least one corresponding first similar question sentence; A processing module, configured to input the first similar question into a customer service question-answering model to obtain a first predicted standard question corresponding to the first similar question; A first determination module is used to determine a first similar question sentence that is different from the first predicted standard question sentence and the expected standard question sentence as a second similar question sentence; A second determination module is configured to determine a second data set according to a first predicted standard question of the second similar question, an expected standard question of the second similar question, and at least one first similar question corresponding to the expected standard question of the second similar question, wherein the second data set includes a first target standard question and at least one first target similar question corresponding to the first target standard question; An optimization module is used to optimize the customer service question and answer model based on the second data set.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images