Data processing method and device and model training method and device for question-answering system

Abstract

The invention provides a data processing and model training method and device for a question answering system, and relates to the technical field of artificial intelligence, in particular to the technical field of intelligent customer service. The data processing method for the question and answer system comprises the steps of obtaining a user question; determining the meaning ambiguity of the user question through a machine learning model; under the condition that the meaning ambiguity is larger than a threshold value, a guiding problem is generated according to the user problem, and the guiding problem is a problem for guiding the user to execute at least one operation of interpreting ambiguity or supplementing missing information; and feeding back a guide problem.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of intelligent customer service technology, and more particularly to a data processing and model training method and device for a question-and-answer system. Background Art

[0002] In e-commerce question-and-answer scenarios, there are a large number of original user questions with repeated meanings and known answers. By identifying original user questions and reusing existing answer data, it is possible to automatically answer original user questions and improve customer service efficiency.

[0003] In the related art, vector similarity is calculated based on the user's original question and the local question-and-answer knowledge base. If a question equivalent to the user's original question can be determined in the knowledge base, the answer to the equivalent question is used as the response to the user's original question. Summary of the invention

[0004] One purpose of the present disclosure is to improve the accuracy and success rate of automatic responses to user's original voice questions.

[0005] According to one aspect of some embodiments of the present disclosure, a data processing method for a question-answering system is proposed, including: obtaining a user question; determining the ambiguity of the meaning of the user question through a machine learning model; generating a guiding question based on the user question when the ambiguity of the meaning is greater than a threshold, the guiding question being a question that guides the user to perform at least one operation of explaining the ambiguity or supplementing the missing information; and feeding back the guiding question.

[0006] In some embodiments, the ambiguity is an evaluation value of at least one degree of ambiguity or information missing in the user's question.

[0007] In some embodiments, generating guiding questions based on user questions includes: generating guiding question generation instructions based on a question instruction template and context according to a sentence analysis result of the user question; and generating a first number of guiding questions through a large language model according to the guiding question generation instructions.

[0008] In some embodiments, according to the sentence analysis results of the user's question, based on the question instruction template and the context, generating a guide question generation instruction includes: determining whether the user's question lacks a subject, an object, and whether there is ambiguity through sentence analysis of the user's question as the sentence analysis result; filling the sentence analysis result, the user's question, the context, and the question scenario data into the question instruction template to generate a sentence generation instruction.

[0009] In some embodiments, the method also includes: when the ambiguity of meaning is greater than a threshold, determining a recommended question associated with the user question based on the user question, including: obtaining a hypothetical answer based on the user question; obtaining a second number of answers from the knowledge base as similar answers based on the semantic similarity between the answers in the knowledge base and the hypothetical answer; obtaining questions corresponding to the similar answers in the knowledge base as recommended questions; and, feeding back the recommended question.

[0010] In some embodiments, the method further includes: when the uncertainty is less than or equal to a threshold, determining an equivalent question in the knowledge base based on the similarity between the user question and the question in the knowledge base; and feeding back an answer corresponding to the equivalent question.

[0011] In some embodiments, the method further includes: obtaining supplementary questions from the user, wherein the supplementary questions are generated based on information provided by the user based on the guiding questions, or are recommended questions selected by the user; using the supplementary questions as user questions, and performing an operation of obtaining user questions until an answer is fed back.

[0012] In some embodiments, the method further includes: obtaining response data of the user's answer to the feedback, wherein the response data includes at least one of service satisfaction, user purchase of corresponding items, and user provision of a definitive answer; when it is determined based on the response data that the answer to the feedback matches the user's question, adding an association relationship between the user's question and the answer to the feedback to the knowledge base.

[0013] In some embodiments, determining an equivalent question in the knowledge base according to the similarity between the user question and the question in the knowledge base includes: determining a third number of candidate questions in the knowledge base according to the similarity between the user question and the question in the knowledge base; determining an equivalent question of the user question from the candidate questions using a large language model;

[0014] In some embodiments, the method further includes: when the uncertainty is less than or equal to a threshold and when an equivalent question of the user question fails to be determined among the candidate questions using the large language model, initiating a supplementary reply operation.

[0015] In some embodiments, the machine learning model is generated by training based on user question samples, where the user question samples include user question data and meaning ambiguity tags.

[0016] In some embodiments, the ambiguity mark of the meaning of the user question sample is determined by processing the user question data through a large language model, and the machine learning model is a lightweight discriminant model.

[0017] According to one aspect of some embodiments of the present disclosure, a model training method is proposed, comprising: obtaining a set of first sample data based on historical data of user questions; determining the ambiguity of the meaning of the first sample data through a language analysis model, and obtaining a set of second sample data, wherein the second sample data includes the first sample data and a mark of the ambiguity of the meaning; training a machine learning model through the set of second sample data until the training is completed, wherein the machine learning model is used to determine the ambiguity of the meaning of the user questions in any of the above data processing methods for a question-and-answer system.

[0018] In some embodiments, the method also includes: determining the type of the second sample data based on the ambiguity of the meaning, the type including a difficult sample type and a simple sample type; training the machine learning model through a set of second sample data includes: in a first training stage, gradually increasing the proportion of samples of the difficult sample type in the set of second sample data used to train the machine learning model until a proportion threshold is reached; in a second training stage, gradually reducing the proportion of samples of the difficult sample type in the set of second sample data used to train the machine learning model from the proportion threshold until the training is completed.

[0019] In some embodiments, training a machine learning model using a set of second sample data includes: replaying the second sample data used in iterative rounds during model training.

[0020] In some embodiments, determining the meaning ambiguity of the first sample data through a language analysis model includes: repeatedly reasoning the first sample data through the language analysis model to obtain multiple reasoning results of the first sample data; determining the variance of the multiple reasoning results; and determining the uncertainty of the first sample data based on the variance.

[0021] According to one aspect of some embodiments of the present disclosure, a data processing device for a question-answering system is proposed, comprising: a question acquisition unit, configured to acquire user questions; a question analysis unit, configured to determine the meaning ambiguity of the user questions through a machine learning model; a feedback content generation unit, configured to generate a guiding question based on the user questions when the meaning ambiguity is greater than a threshold, the guiding question being a question that guides the user to perform at least one operation of interpreting ambiguity or supplementing missing information; and a feedback unit, configured to feedback the guiding question.

[0022] According to one aspect of some embodiments of the present disclosure, a model training device is proposed, comprising: a sample collection unit, configured to obtain a set of first sample data based on user questions; a sample labeling unit, configured to determine the meaning ambiguity of the first sample data through a language analysis model, and obtain a set of second sample data, wherein the second sample data includes the first sample data and a label of the meaning ambiguity; a training unit, configured to train a machine learning model through the set of second sample data until the training is completed, wherein the machine learning model is used to determine the meaning ambiguity of the user question in any of the above data processing methods for a question-and-answer system.

[0023] According to one aspect of some embodiments of the present disclosure, a data processing device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the above-mentioned methods based on instructions stored in the memory.

[0024] According to one aspect of some embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, any one of the methods mentioned above is implemented.

[0025] According to one aspect of some embodiments of the present disclosure, a computer program product is provided, including a computer program or instructions, and when the computer program or instructions are executed by a processor, any one of the methods mentioned above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute improper limitations on the present disclosure.

[0027] Figure 1 The present invention is a flowchart of some embodiments of the data processing method for the question answering system of the present invention.

[0028] Figure 2 The present invention is a flowchart of some embodiments of generating guiding questions and recommended questions in the data processing method for the question-answering system of the present invention.

[0029] Figure 3 Flowcharts of other embodiments of the data processing method for a question-answering system disclosed in the present invention.

[0030] Figure 4 Flowcharts of some embodiments of the model training method disclosed herein.

[0031] Figure 5 Schematic diagram of some embodiments of the data processing device for the question-answering system disclosed in the present invention.

[0032] Figure 6 Schematic diagrams of some embodiments of the model training device disclosed in the present invention.

[0033] Figure 7 Schematic diagrams of some embodiments of the data processing device disclosed herein.

[0034] Figure 8 Schematic diagrams of other embodiments of the data processing device disclosed in the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present disclosure is further described in detail below through the accompanying drawings and embodiments.

[0036] The inventors found that there are a large number of user questions with incomplete input and ambiguous intentions, which are difficult to match with candidate questions, resulting in a low answer rate for user questions.

[0037] In response to the above problems, the present disclosure proposes a data processing, model training method and device for a question-answering system, which can screen out questions with high ambiguity from user questions, and guide users to clarify their questions in a targeted manner, thereby improving the accuracy and success rate of automatic answers to user questions.

[0038] The flowcharts of some embodiments of the data processing method for the question answering system disclosed in the present invention are as follows: Figure 1 As shown, steps S11-S15 are included.

[0039] In step S11, the user question is obtained. The user question mentioned in the present disclosure refers to the original user question, that is, the question directly asked by the user without content processing. In some embodiments, the question data provided by the user through text or voice can be directly extracted as the user question. In some embodiments, the question data provided by the user through the customer service function interface can be obtained as the user question.

[0040] In step S12, the ambiguity of the meaning of the user question is determined by a machine learning model. The machine learning model is a pre-trained model that can output the ambiguity of the meaning for the input user question. In some embodiments, the ambiguity of the meaning is an evaluation value of at least one defect degree of ambiguity or missing information in the user question.

[0041] For example, the ambiguity of meaning can be a numerical value related to the clarity of the question and the probability of getting an accurate answer. The clearer the meaning of the question and the higher the probability of getting an accurate answer directly, the lower the ambiguity of the meaning; the more ambiguous the meaning and the lower the probability of getting an accurate answer directly, the higher the ambiguity of the meaning.

[0042] For example, the meaning ambiguity of the question output by the machine learning model can be a first value or a second value, where the first value is less than a threshold and the second value is greater than the threshold. For example, a 0 / 1 value can be output, and the threshold is any value between 0-1.

[0043] In some embodiments, the machine learning model is a lightweight discriminant model, such as a tinyBert model, thereby improving data processing efficiency and reducing resource usage of a system performing data processing.

[0044] In some embodiments, the machine learning model can be generated in advance based on user question sample training. The user question sample includes user question data and meaning ambiguity tags. The meaning ambiguity tags can be manually calibrated or determined through large language model processing.

[0045] In some embodiments, the machine learning model is generated by training using any one of the model training methods mentioned below.

[0046] In step S13, it is determined whether the ambiguity of the meaning of the user's question is greater than a threshold. If the ambiguity of the meaning is greater than the threshold, step S14 is executed. In some embodiments, the threshold is preset and can be adjusted according to actual needs or execution effects.

[0047] In step S14, guiding questions are generated according to the user questions.

[0048] The above-mentioned guiding questions are targeted questions for the vague and missing information in the user's questions, guiding the user to supplement the information, thereby clarifying the user's questions. In some embodiments, the information that guides the user to supplement includes at least one of explaining ambiguity or supplementing missing information. Therefore, guiding questions can be generated from the perspective of guiding the user to perform at least one operation of explaining ambiguity or supplementing missing information.

[0049] In some embodiments, the missing information type (such as subject or object, etc.) can be found by analyzing the user's question, and then a guiding question can be generated for the missing information type. In addition, the scenario in which the user asks the question can be considered, such as the type of item or item model targeted by the user's question, and targeted guiding questions can be generated, which is conducive to increasing the amount of effective information obtained.

[0050] In some embodiments, a large language model LLM can be used to generate guiding questions. Figure 2 As shown in , when it is determined in step S13 that the meaning ambiguity of the user's question is greater than the threshold, the guiding question is determined through steps 411 and 412.

[0051] In step 411, according to the sentence analysis result of the user's question, based on the question instruction template and the context, a guide question generation instruction is generated. In some embodiments, the process of generating the guide question generation instruction can also refer to the scene in which the user asks the question and the information of the corresponding items in the scene. In some embodiments, the required information can be determined by searching the knowledge base. In some embodiments, the question instruction template includes the location of various required information, and the guide question generation instruction is generated by filling the sentence analysis result, context information, etc. into the question instruction template.

[0052] In some embodiments, the above-mentioned knowledge base may be a question-answering knowledge base provided by a brand or an official, such as: question (How to update device X?) and answer (Click on the video to learn how to update device X).

[0053] In some embodiments, by analyzing the sentence of the user's question, it is determined whether the user's question lacks a subject or an object, and whether the user's question is ambiguous, and the above judgment result is used as the sentence analysis result. Considering that the subject and object of the question generally contain key information of the question, by analyzing whether the subject and object are missing, the pertinence and accuracy of guiding the user to supplement the information can be improved. Further, the sentence analysis results, user questions, context, and data in the questioning scenario are filled into the question instruction template to generate a guide question generation instruction.

[0054] For example, the user's question is: What does 10 cores mean? This question has both a subject and an object, but it is ambiguous. 10 cores may refer to a central processing unit or a graphics processing unit. The process of the counter-question module is to check the user's previous text and the knowledge base of the product that the user is currently questioning, and generate a guide question generation instruction based on the item X (of product type / brand Y) that the user's question is about. The instruction can be as follows.

[0055] You are an expert on product Y. For item X, the user asked [What does 10 cores mean], the user did not have the above question, the attributes of the item are {"Chip CPU core count": "8-core CPU, with 4 performance cores and 4 energy efficiency cores"}, {"Chip GPU core count": "10-core GPU"}, {"Chip neural network engine core count": "16-core neural network engine"}. This user question is missing and ambiguous. List three possible user questions without explanation.

[0056] In step 412, according to the guiding question generation instruction, a first number (eg, 3) guiding questions are generated by a large language model. The large language model may be an LLM.

[0057] Taking the situation mentioned in step 411 as an example, the above instruction is sent to LLM, and three questions are returned by LLM:

[0058] 1. What does 10 cores of this item X mean?

[0059] 2. Does the 10 cores in this product description refer to the CPU or the GPU?

[0060] 3.10 cores refers to which part of the core number specifically?

[0061] Similarly, for situations where the subject or object is missing, a corresponding question instruction template may be used. This template is slightly different from the template in the above instruction. For example, "This user's question is missing or ambiguous" is modified to "This user's question has a missing subject / object."

[0062] Based on the method in the embodiment shown above, targeted counter-questions can be generated for unclear user questions, thereby improving the efficiency of users supplementing effective information; templates and related data are used to provide rich information for generating guided questions, and large language models are used to generate guided questions, thereby improving the efficiency and accuracy of generating guided questions.

[0063] In some embodiments, in the current step S14, recommended questions associated with the user's question may also be determined. The recommended questions associated with the user's question may be determined by using the similarity between the existing questions in the knowledge base and the user's question, providing the user with questions that can be clearly answered for selection without the user having to provide further supplementary question information, thereby improving the efficiency of answering user questions; avoiding misleading the user by directly providing the recommended answer to the user, thereby improving the accuracy of the answer.

[0064] In some embodiments, the idea of ​​“using hypothetical document embedding to improve retrieval” can also be used to determine the recommendation problem, for example, Figure 2 As shown in , after determining in step S13 that the meaning ambiguity of the user's question is greater than a threshold, steps 421-423 are used to determine the recommended question.

[0065] In step 421, a hypothetical answer is obtained based on the user's question. In some embodiments, an instruction template can be designed to input the user's question, generate an instruction requiring the model to provide a hypothetical answer, input the instruction into the large language model, and obtain a hypothetical answer output by the model. The above hypothetical answer is: assuming the exact meaning of the user's question, and using the exact meaning as the answer to the question.

[0066] In step 422, according to the semantic similarity between the answers in the knowledge base and the hypothetical answers, a second number of answers are obtained from the knowledge base as similar answers. The knowledge base stores existing questions and their corresponding answers. In some embodiments, any semantic similarity checking method in the relevant technology can be used to analyze the similarity between the hypothetical answer and the answer in the knowledge base, and a second number of answers (for example, 3) are obtained in descending order of similarity as similar answers.

[0067] In step 423, questions corresponding to similar answers are obtained in the knowledge base as recommended questions. For example, based on the correspondence between existing questions and answers stored in the knowledge base, questions corresponding to similar answers are obtained as recommended questions.

[0068] Through the method in this embodiment, the recommended questions are inferred from the perspective of answer matching, which can avoid the defects of inaccurate question matching caused by the limitations of question scenarios in the knowledge base, improve the effective utilization of information in the knowledge base, and improve the accuracy of recommended questions.

[0069] In step S15, feedback guidance questions are given.

[0070] In some embodiments, recommended questions associated with the user's question can also be fed back. For example, on the user's question interface, guiding questions and recommended questions can be displayed in different areas. For example, it can be displayed: "Please answer the following questions regarding your question:" + guiding question; "What do you want to ask?" + recommended question.

[0071] Through the method in the embodiment shown in the above text, it is possible to screen out questions that are not clear and definite enough and difficult to get accurate answers directly through ambiguity analysis, and guide users to clarify such questions, thereby improving the effective utilization of existing questions and answers in the knowledge base, and improving the accuracy and success rate of automatic responses to user questions.

[0072] In some embodiments, Figure 1 As shown, the data processing method for the question-answering system disclosed in the present invention also includes steps S16 and S17.

[0073] In the above step S13, if the ambiguity of the meaning of the user's question is less than or equal to the threshold, it means that the user's question is a relatively clear question and can be directly answered by using the existing knowledge base, and step S16 is executed.

[0074] In step S16, equivalent questions are determined in the knowledge base based on the similarity between the user question and the questions in the knowledge base.

[0075] In some embodiments, a third number (e.g., 3) of candidate questions may be determined in the knowledge base based on the similarity between the user question and the questions in the knowledge base. For example, the candidate questions may be determined in the knowledge base by calculating the cosine similarity. Further, the candidate questions and the user question are input into the large language model, and the large language model is required to determine the equivalent questions of the user question in the candidate questions. If the large language model successfully determines the equivalent questions of the user question in the candidate questions, step S17 is executed.

[0076] For example, the cosine similarity between the user question q and the vector corresponding to the local knowledge base is calculated, and the top three candidate questions with the highest similarity are returned. The user question q and the candidate questions q1, q2, and q3 are filled into the preset instruction template, and LLM determines whether there are equivalent questions in the top 3 candidate questions. If so, the answer corresponding to the equivalent question in the knowledge base is output.

[0077] In step S17, the answer corresponding to the equivalent question is fed back.

[0078] Through the method in the above embodiment, for questions with clear meanings, equivalent questions and their corresponding answers can be determined by question matching, thereby improving answering efficiency.

[0079] Flow charts of other embodiments of the data processing method for the question-answering system disclosed in the present invention are as follows: Figure 3 shown.

[0080] Steps S11 to S15 are the same as or similar to the corresponding steps in the above embodiment.

[0081] After executing step S15, the user may provide supplementary information or select a recommended question for the provided guiding question or recommended question. The supplementary information provided by the user or the recommended question selected by the user is obtained, and step 301 is executed.

[0082] In step 301, a supplementary question of the user is obtained. The supplementary question can be generated based on the information provided by the user based on the guiding question. In some embodiments, if the user provides information based on the guiding question, the provided information can be combined with the previous user question or the corresponding guiding question, and the sentence combination can be performed through semantic analysis to obtain a supplementary question. The supplementary question is clearer than the user question.

[0083] When recommended questions are provided to a user and the user selects a recommended question, the recommended question selected by the user is used as a supplementary question.

[0084] In step 302, the supplementary question is taken as the user question, and step S11 is executed to trigger the next cycle.

[0085] In the next cycle, if the ambiguity of the user's question is less than or equal to the threshold, the above-mentioned step S16 can be executed, or the step S20 can be executed. Figure 3 Step 303 shown in ; if the ambiguity of the meaning of the user's question is greater than the threshold, the guiding question and the recommended question are fed back again to guide the user to supplement the information, and the next cycle is triggered again.

[0086] Based on the method in the embodiment shown above, the problem can be gradually clarified through one or more cycles, thereby improving the success rate of answering user questions through existing questions and answers in the knowledge base, improving the accuracy of automatic responses to user questions, and improving customer service efficiency.

[0087] In some embodiments, Figure 3 As shown, the data processing method for the question-answering system disclosed in the present invention includes steps 303 to 305. In the above step S13, if the meaning ambiguity of the user's question is less than or equal to the threshold, step 303 is triggered.

[0088] In step 303, it is determined whether the equivalent problem of the user problem is successfully determined in the candidate problems. In some embodiments, the candidate problems can be determined first by the method in the embodiment shown in step S16 above, and then the equivalent problem of the user problem is determined in the candidate problems using the large language model. If the large language model successfully determines the equivalent problem of the user problem in the candidate problems, step 304 is executed; if the large language model fails to determine the equivalent problem of the user problem in the candidate problems, step 305 is executed.

[0089] In step 304, the answer corresponding to the equivalent question is fed back.

[0090] In step 305, a supplementary reply operation is initiated, such as contacting a human customer service representative so that the human customer service representative can answer the user's question.

[0091] Through the method in the embodiment shown above, when the user's question is clear enough but there are no suitable questions and answers in the knowledge base, manual customer service can be used to answer it, ensuring that the user's question can be answered correctly, thereby improving the user experience.

[0092] In some embodiments, Figure 3 As shown, the data processing method for the question-answering system disclosed in the present invention also includes steps 306 to 308.

[0093] In step 306, the response data of the user's answer to the feedback is obtained. In some embodiments, the response data may be generated when the user decides to end the question-and-answer service, or may be generated based on the user's message in the customer service interface, or may be generated when the user makes a purchase. The triggering of step 306 above after step 304 or 305 is only an example and does not constitute an undue limitation on the present disclosure. In some embodiments, the above response data includes at least one of service satisfaction, user purchase of corresponding items, and user provision of a definite answer. Service satisfaction may be obtained through user service evaluation; user purchase of corresponding items may be obtained through user order data; user provision of a definite answer may be a user reply or a message in the customer service interface, such as thank you, understand, etc.

[0094] In step 307, it is determined whether the answer to the feedback determined based on the response data matches the user question. If the answer to the feedback matches the user question, step 308 is executed. If the answer to the feedback does not match the user question, it will not be used as data for updating the knowledge base.

[0095] In step 308, the association relationship between the user question and the feedback answer is added to the knowledge base. In some embodiments, the knowledge base can be updated in real time or regularly. In some embodiments, in order to avoid adding an incorrect question-answer relationship into the knowledge base, it can be provided to the business personnel for review and added to the knowledge base after the review is passed.

[0096] Taking into account that user questions are more life-oriented and colloquial, the method in the embodiment shown in the present disclosure can improve the answer rate by updating the knowledge base, which is conducive to gradually further improving customer service efficiency.

[0097] The flowcharts of some embodiments of the model training method disclosed in the present invention are as follows Figure 4 The machine learning model generated by the model training method of the present disclosure can be applied to any of the data processing methods for question-answering systems mentioned above, and the user's question is input into the machine learning model to obtain the meaning ambiguity of the user's question.

[0098] In step 401, a set of first sample data is obtained according to the user question. In some embodiments, user question history data may be collected, and the user question in each piece of history data is used as a piece of first sample data. The first sample data are aggregated to obtain a set of first sample data.

[0099] In step 402, the ambiguity of the meaning of the first sample data is determined by the language analysis model, and the first sample data is associated with the corresponding ambiguity mark to form the second sample data, that is, the second sample data includes the first sample data and the ambiguity mark. The second sample data is aggregated to obtain a set of the second sample data.

[0100] The ambiguity of the meaning of the first sample data can be determined by analyzing the large language model. The first sample data is repeatedly inferred multiple times by the large language model, and an inference result is obtained each time, thereby obtaining multiple inference results of the first sample data; further, the variance of the multiple inference results is calculated, and the ambiguity of the meaning of the first sample data is determined according to the variance. Based on the method in this embodiment, there is no need to manually label the sample data, which reduces the burden of manual operation and improves processing efficiency.

[0101] For example, design an instruction template: You are an expert in the field of M. You are given a user question Q. Please judge whether the question Q is clear and unambiguous. You only need to answer yes or no. Set the temperature parameter of LLM to 0.5. Each question is inferred n times by the large language model LLM. The variance of the output result is calculated. When the variance is greater than the threshold, it is marked as a question with ambiguous meaning. Based on this, a set of second sample data is obtained. Each piece of data is a question-label pair. The question is the first sample data, and the label is obtained by the large language model LLM.

[0102] In step 403, a machine learning model is trained by using a set of second sample data. The machine learning model is a lightweight model.

[0103] For example, the lightweight discriminant model tinyBert (extremely small knowledge distillation model) is used as the machine learning model, and its loss function is:

[0104]

[0105] Among them, ∑ represents the sum of all samples, y is the true label vector after onehot encoding, p is the probability of each category predicted by the model, and log(p) represents the logarithmic probability of each correct classification. The loss obtained in this way is the degree of prediction error.

[0106] In some embodiments, the second sample data used in the iterative rounds can be replayed during the model training process to prevent the model from forgetting old knowledge and avoid catastrophic forgetting during training.

[0107] Based on the method in the embodiment shown above, it is possible to use sample data to train and generate a machine learning model, providing a model tool foundation for data processing used in question-answering systems; by training a lightweight model for subsequent data processing, it is possible to improve data processing efficiency and reduce the resource usage of the system performing data processing.

[0108] In some embodiments, the model training method of the present disclosure further includes: determining the type of the second sample data according to the ambiguity of the meaning, and the type includes a difficult sample type and a simple sample type. For example, a classification threshold is set. If the ambiguity of the meaning in the annotation is greater than the classification threshold, it belongs to the difficult sample type; if the ambiguity of the meaning of the annotation is less than or equal to the classification threshold, the data is a simple sample type. The above classification threshold may be the same as or different from the threshold mentioned in step S13. In some embodiments, the classification threshold can be set or adjusted as needed.

[0109] Furthermore, in the above step 403, the training of the machine learning model is divided into two stages. In the first training stage, the proportion of samples of the difficult sample type in the set of the second sample data used to train the machine learning model is gradually increased until the proportion threshold is reached; in the second training stage, the proportion of samples of the difficult sample type in the set of the second sample data used to train the machine learning model is gradually reduced from the proportion threshold until the training is completed. For example, in the first n rounds, the proportion of difficult samples increases from 0 to 0.5, and the proportion of simple samples decreases from 1 to 0.5. The change can be a linear change; from n rounds until the training is completed, the proportion of difficult samples changes in accordance with the cosine ratio, decaying from 0.5 to close to 0.

[0110] Based on the method in this embodiment, it is possible to absorb the advantages of course learning, organize the ratio of difficult data to simple data according to the strategy of starting with the easy and then the difficult, improve the efficiency of model training, and increase the model's absorption ratio of effective information in the sample, which is conducive to improving the accuracy of the trained machine learning model.

[0111] Schematic diagrams of some embodiments of the data processing device for the question-answering system disclosed in the present invention are as follows Figure 5 shown.

[0112] The question acquisition unit 51 can acquire user questions. In some embodiments, the question data provided by the user through text or voice can be directly extracted as the user question. In some embodiments, the question data provided by the user through the customer service function interface can be acquired as the user question.

[0113] The question analysis unit 52 can determine the meaning ambiguity of the user question through a machine learning model. The machine learning model is a pre-trained machine learning model that can output the meaning ambiguity for the input user question. In some embodiments, the meaning ambiguity analysis unit 52 can perform the method in any embodiment of step S12 above.

[0114] The feedback content generating unit 53 can generate a guiding question according to the user question when the ambiguity of the meaning of the user question is greater than a threshold. The guiding question is a question that guides the user to perform at least one of explaining the ambiguity or supplementing the missing information. In some embodiments, the feedback content generating unit 53 can perform the method in any embodiment of steps S13 and S14 above.

[0115] The feedback unit 54 can feedback guidance issues.

[0116] Based on the device in the embodiment shown above, it is possible to screen out questions that are not clear and definite enough and difficult to get accurate answers directly through ambiguity analysis, and guide users to clarify such questions, thereby improving the effective utilization of existing questions and answers in the knowledge base, and improving the accuracy and success rate of automatic responses to user questions.

[0117] In some embodiments, the feedback content generation unit 53 can also determine the recommended questions associated with the user's question when the ambiguity of the meaning of the user's question is greater than a threshold. The feedback unit 54 can also feedback recommended questions. For example, it can display guiding questions and recommended questions in different areas on the interface where the user asks questions. For example, it can display: "For your question, please answer the following questions:" + guiding questions; "What do you want to ask is" + recommended questions. Such a device can guide users to clarify their questions from the two perspectives of counter-questions and recommended questions, thereby improving the efficiency of question-answering.

[0118] In some embodiments, the feedback content generation unit 53 can also determine an equivalent question in the knowledge base based on the similarity between the user question and the question in the knowledge base when the meaning ambiguity of the user question is less than or equal to a threshold. A third number (e.g., 3) of candidate questions can be determined in the knowledge base based on the similarity between the user question and the question in the knowledge base. For example, the candidate questions can be determined in the knowledge base by cosine similarity calculation. Further, the candidate questions and the user questions are input into the large language model, and the large language model is required to determine the equivalent questions of the user questions in the candidate questions. The feedback unit 54 can also feedback the answers corresponding to the equivalent questions.

[0119] Based on the device in the embodiment shown above, for questions with clear meanings, equivalent questions and their corresponding answers can be determined by question matching, thereby improving answering efficiency.

[0120] In some embodiments, if the feedback content generation unit 53 fails to determine the equivalent question of the user question when the ambiguity of the meaning of the user question is less than or equal to the threshold, the manual customer service is triggered so that the user question can be answered by the manual customer service. Such a device can answer the user question through the manual customer service when the user question is clear enough but there is no suitable question and answer in the knowledge base, ensuring that the user question can be answered correctly and improving the user experience.

[0121] In some embodiments, the question acquisition unit 51 can also acquire the user's supplementary questions. The supplementary questions can be generated based on the information provided by the user based on the guiding question. In some embodiments, if the user provides information based on the guiding question, the provided information can be combined with the previous user question or the corresponding guiding question, and the sentences can be combined through semantic analysis to obtain the supplementary question. The supplementary question is clearer than the user question. When the user selects a recommended question, the recommended question selected by the user is used as the supplementary question. The question acquisition unit 51 uses the supplementary question as the user question and triggers the meaning ambiguity analysis unit 52 to execute the next loop. In the next loop, if the meaning ambiguity of the user question is less than or equal to the threshold, the step S16 mentioned above can be executed, or the step S20 can be executed. Figure 3 Step 303 shown in ; if the ambiguity of the meaning of the user's question is greater than the threshold, the guiding question and the recommended question are fed back again to guide the user to supplement the information, and the next cycle is triggered again.

[0122] Such a device can gradually clarify the questions through one or more cycles, improve the success rate of answering user questions through existing questions and answers in the knowledge base, improve the accuracy of automatic responses to user questions, and improve customer service efficiency.

[0123] In some embodiments, the data processing device for the question-answering system also includes a knowledge base updating unit capable of executing the operations in the embodiments shown in steps 306 to 308 above to add an association relationship between user questions and feedback answers in the knowledge base.

[0124] Such a device can take into account the more lifelike and colloquial characteristics of user questions, and improve the answer rate by updating the knowledge base, which is conducive to gradually further improving customer service efficiency.

[0125] Schematic diagrams of some embodiments of the model training device disclosed in the present invention are as follows Figure 6 The machine learning model generated by the model training device of the present disclosure can be applied to any of the data processing methods and devices for question-answering systems mentioned above, and the user's question is input into the machine learning model to obtain the meaning ambiguity of the user's question.

[0126] The sample collection unit 61 can obtain a set of first sample data according to the user question. In some embodiments, user question history data can be collected, and the user question in each historical data is used as a first sample data. The first sample data are aggregated to obtain a set of first sample data.

[0127] The sample labeling unit 62 can determine the ambiguity of the meaning of the first sample data through the language analysis model, associate the first sample data with the label of the corresponding ambiguity of the meaning, and form the second sample data, that is, the second sample data includes the first sample data and the label of the ambiguity of the meaning. The second sample data is aggregated to obtain a set of second sample data. The ambiguity of the meaning of the first sample data can be determined by analyzing the large language model. The first sample data is repeatedly inferred multiple times by the large language model, and an inference result is obtained each time, thereby obtaining multiple inference results of the first sample data; further, the variance of the multiple inference results is calculated, and the ambiguity of the meaning of the first sample data is determined according to the variance, so that there is no need to manually label the sample data, which reduces the burden of manual operation and improves processing efficiency.

[0128] The training unit 63 can train the machine learning model through the set of the second sample data until the training is completed. In some embodiments, the training unit 63 can execute the method in any embodiment of step 403 above.

[0129] Based on the model training device shown above, it is possible to use sample data to train and generate a machine learning model, providing a model tool foundation for data processing used in question-answering systems; by training a lightweight model for subsequent data processing, it is possible to improve data processing efficiency and reduce the resource usage of the system that performs data processing.

[0130] In some embodiments, the sample labeling unit 62 can also determine the type of the second sample data according to the ambiguity of the meaning, and the type includes a difficult sample type and a simple sample type. For example, a classification threshold is set. If the ambiguity of the meaning in the annotation is greater than the classification threshold, it belongs to the difficult sample type; if the ambiguity of the meaning of the annotation is less than or equal to the classification threshold, the data is a simple sample type. The above classification threshold may be the same as or different from the threshold mentioned in step S13. In some embodiments, the classification threshold can be set or adjusted as required. Further, the training unit 63 can train the machine learning model through two stages. In the first training stage, the proportion of samples of the difficult sample type in the set of the second sample data used to train the machine learning model is gradually increased until the proportion threshold is reached; in the second training stage, the proportion of samples of the difficult sample type in the set of the second sample data used to train the machine learning model is gradually reduced from the proportion threshold until the training is completed. For example, in the first n rounds, the proportion of difficult samples increases from 0 to 0.5 in sequence, and the proportion of simple samples decreases from 1 to 0.5 in sequence. The change can be a linear change; from n rounds until the training is completed, the proportion of difficult samples changes in accordance with the cosine ratio, decaying from 0.5 to close to 0.

[0131] Based on the model training device shown above, it is possible to absorb the advantages of course learning, organize the ratio of difficult data to simple data according to the strategy of starting with the easy and then the difficult, improve the efficiency of model training, and increase the model's absorption ratio of effective information in the sample, which is conducive to improving the accuracy of the trained machine learning model.

[0132] A structural diagram of an embodiment of the data processing device disclosed in the present invention is shown in FIG. Figure 7 As shown. The data processing device includes a memory 701 and a processor 702. Among them: the memory 701 can be a disk, a flash memory or any other non-volatile storage medium. The memory is used to store the instructions in the corresponding embodiments of the data processing method or model training method for the question-answering system described above. The processor 702 is coupled to the memory 701 and can be implemented as one or more integrated circuits, such as a microprocessor or a microcontroller. The processor 702 is used to execute the instructions stored in the memory, which can improve the accuracy and success rate of automatic answers to user questions.

[0133] In one embodiment, it is also possible to Figure 8 As shown, the data processing device 800 includes a memory 801 and a processor 802. The processor 802 is coupled to the memory 801 via a BUS bus 803. The data processing device 800 can also be connected to an external storage device 805 via a storage interface 804 to call external data, and can also be connected to a network or another computer system (not shown) via a network interface 806. No further detailed description will be given here.

[0134] In this embodiment, the accuracy and success rate of automatically answering user questions can be improved by storing data instructions in a memory and processing the instructions through a processor.

[0135] In another embodiment, a computer-readable storage medium stores computer program instructions thereon, which, when executed by a processor, implement the steps of the method in the corresponding embodiment of the data processing method or model training method for a question-answering system. It should be understood by those skilled in the art that the embodiments of the present disclosure may be provided as methods, devices, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] So far, the present disclosure has been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

[0140] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0141] It should be noted that the terms "first", "second", etc. in the specification, claims, and drawings of the present disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present disclosure rather than to limit it. Although the present disclosure has been described in detail with reference to the preferred embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present disclosure can still be modified or some technical features can be replaced by equivalents without departing from the spirit of the technical solution of the present disclosure, which should be included in the scope of the technical solution for protection of the present disclosure.

Claims

1. A data processing method for a question-answering system, comprising: Get user questions; Determine the ambiguity of the meaning of the user question through a machine learning model; When the ambiguity of the meaning is greater than a threshold, generating a guiding question according to the user question, wherein the guiding question is a question for guiding the user to perform at least one of explaining the ambiguity or supplementing the missing information; Feedback on the guide issues mentioned.

2. The data processing method according to claim 1, wherein: The ambiguity of meaning is an evaluation value of at least one defect degree of ambiguity or missing information in the user question.

3. The data processing method according to claim 1, wherein: Generating a guiding question according to the user question comprises: Generate a guiding question generation instruction based on the sentence analysis result of the user question and the question instruction template and context; According to the guiding question generating instruction, a first number of guiding questions are generated by using a large language model.

4. The data processing method according to claim 3, wherein: The generating of the guiding question generating instruction according to the sentence analysis result of the user question and based on the questioning instruction template and context includes: By analyzing the sentence of the user question, determining whether the user question lacks a subject and an object, and whether there is ambiguity, as the sentence analysis result; The sentence analysis result, the user question, the context, and the question scenario data are filled into the question instruction template to generate the sentence generation instruction.

5. The data processing method according to claim 1, further comprising: When the meaning ambiguity is greater than a threshold, determining a recommended question associated with the user question according to the user question includes: Obtaining hypothetical answers based on the user questions; According to the semantic similarity between the answer in the knowledge base and the hypothetical answer, obtaining a second number of answers in the knowledge base as similar answers; Obtaining questions corresponding to the similar answers in the knowledge base as the recommended questions; and, Feedback on the recommended issues.

6. The data processing method according to claim 1, further comprising: When the uncertainty is less than or equal to the threshold, Determining equivalent questions in the knowledge base according to the similarity between the user question and the questions in the knowledge base; Feedback the corresponding answer to the equivalent question.

7. The data processing method according to claim 1 or 6, further comprising: Acquire a supplementary question of the user, wherein the supplementary question is generated according to information provided by the user based on the guiding question, or is the recommended question selected by the user; The supplementary question is taken as the user question, and the operation of obtaining the user question is performed until an answer is fed back.

8. The data processing method according to claim 6, further comprising: Acquire response data of the user's answer to the feedback, wherein the response data includes at least one of service satisfaction, user purchase of corresponding items, and user providing a definitive answer; When it is determined according to the response data that the answer to the feedback matches the user question, an association relationship between the user question and the answer to the feedback is added to the knowledge base.

9. The data processing method according to claim 6, wherein: Determining equivalent questions in the knowledge base according to the similarity between the user question and the questions in the knowledge base includes: Determining a third number of candidate questions in the knowledge base according to the similarity between the user question and the questions in the knowledge base; By using a large language model, equivalent questions of the user question are determined among the candidate questions.

10. The data processing method according to claim 9, further comprising: When the uncertainty is less than or equal to a threshold and when an equivalent question of the user question fails to be determined from among the candidate questions using a large language model, a supplementary reply operation is initiated.

11. The data processing method according to claim 1, wherein: The machine learning model is generated by training based on user question samples, and the user question samples include user question data and meaning ambiguity tags.

12. The data processing method according to claim 11, wherein: The meaning ambiguity mark of the user question sample is determined by processing the user question data through a large language model, and the machine learning model is a lightweight discriminant model.

13. A model training method, comprising: According to the historical data of the user's questions, a set of first sample data is obtained; Determine the ambiguity of the meaning of the first sample data through a language analysis model, and obtain a set of second sample data, wherein the second sample data includes the first sample data and a mark of the ambiguity of the meaning; Training a machine learning model using the second sample data set, The machine learning model is used to determine the ambiguity of the meaning of user questions in a data processing method for a question-answering system.

14. The model training method according to claim 13, further comprising: Determine the type of the second sample data according to the meaning ambiguity, the type including a difficult sample type and a simple sample type; The training of the machine learning model by the set of the second sample data comprises: In a first training stage, gradually increasing the proportion of samples of difficult sample types in the set of the second sample data used to train the machine learning model until a proportion threshold is reached; In the second training stage, the proportion of samples of the difficult sample type in the set of the second sample data used to train the machine learning model is gradually reduced from the proportion threshold until the training is completed.

15. The model training method according to claim 13, wherein: The training of the machine learning model using the set of the second sample data includes: replaying the second sample data used in iterative rounds during the model training process.

16. The model training method according to claim 13, wherein: Determining the meaning ambiguity of the first sample data by using a language analysis model includes: Repeatedly reasoning the first sample data by using the language analysis model to obtain multiple reasoning results of the first sample data; determining a variance of the plurality of inference results; The uncertainty of the first sample data is determined according to the variance.

17. A data processing device for a question-answering system, comprising: A question acquisition unit, configured to acquire user questions; A question analysis unit, configured to determine the ambiguity of the meaning of the user question through a machine learning model; A feedback content generating unit is configured to generate a guiding question according to the user question when the meaning ambiguity is greater than a threshold, wherein the guiding question is a question for guiding the user to perform at least one of explaining the ambiguity or supplementing the missing information; The feedback unit is configured to feedback the guidance problem.

18. A model training device, comprising: A sample collection unit, configured to obtain a set of first sample data according to a user question; A sample marking unit is configured to determine the ambiguity of the meaning of the first sample data through a language analysis model, and obtain a set of second sample data, wherein the second sample data includes the first sample data and a mark of the ambiguity of the meaning; a training unit, configured to train the machine learning model using the set of the second sample data until the training is completed, Wherein, the machine learning model is used to determine the meaning ambiguity of user questions in the data processing method for a question-answering system described in any one of claims 1 to 12.

19. A data processing device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to execute the method according to any one of claims 1 to 16 based on instructions stored in the memory.

20. A computer-readable storage medium having computer instructions stored thereon, wherein the instructions, when executed by a processor, implement the method according to any one of claims 1 to 16.

21. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the method according to any one of claims 1 to 16 is implemented.

Images