Statistical method, device and electronic equipment for multi-turn dialogue, and computer storage medium

Abstract

The present disclosure provides a statistical method, device, electronic equipment and storage medium for multi-turn dialogue, the method comprising: predicting dialogue data between a first object and a second object by using a machine learning model to obtain a plurality of predicted sequences; wherein the predicted sequence comprises one or more keywords between the first object and the second object; and performing keyword statistics on the plurality of predicted sequences to obtain a statistical result; wherein the statistical result is used to determine at least a dialogue topic, a dialogue-related question, a dialogue satisfaction of any one of the dialogue parties, and an intention of any one of the dialogue parties. Even for a small amount of data samples, a good statistical result can be achieved.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to statistical methods, apparatus, electronic devices, and computer storage media for multi-turn dialogues. Background Technology

[0002] To achieve commercial viability and obtain stable and good test metrics, existing predictions trained on multi-turn dialogue data often require large datasets. However, for small to medium-sized datasets, it is impossible to obtain satisfactory test metrics.

[0003] Therefore, there is a need for a device that can handle large datasets while also achieving good statistical results for multi-turn dialogues on small to medium datasets. Summary of the Invention

[0004] This disclosure provides a statistical method, apparatus, electronic device, and computer storage medium for multi-turn dialogues.

[0005] According to a first aspect of the present disclosure, a statistical method for multi-turn dialogue is provided, the method comprising:

[0006] A machine learning model is used to predict the dialogue data between a first object and a second object, resulting in multiple prediction sequences; wherein, the prediction sequence includes one or more keywords from the dialogue between the first object and the second object;

[0007] Keyword statistics are performed on the multiple predicted sequences to obtain statistical results; wherein, the statistical results are used at least to determine the dialogue topic, the issues involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data.

[0008] Optionally, the step of using a machine learning model to predict the dialogue data between the first object and the second object yields multiple prediction sequences, including:

[0009] The first model in the machine learning model is used to predict the first dialogue dataset corresponding to the first object to obtain a first prediction sequence; wherein, the first model is trained based on the dialogue dataset of the same object as the first type of object and / or the historical dialogue data of the first object;

[0010] A second prediction sequence is obtained by using the second model in the machine learning model to predict the second dialogue dataset corresponding to the second object; wherein, the second model is trained based on the dialogue dataset of the same object as the second type of object and / or the historical dialogue data of the second object.

[0011] Optionally, the step of performing keyword statistics on the plurality of predicted sequences to obtain statistical results includes:

[0012] Keyword statistics are performed on the first predicted sequence to obtain the first statistical result;

[0013] Keyword statistics were performed on the second predicted sequence to obtain the second statistical result;

[0014] Based on the product of the first statistical result and the first weight, and the product of the second statistical result and the second weight, a statistical result is obtained for determining the topic of the dialogue.

[0015] Optionally, before using a machine learning model to predict the dialogue between the first object and the second object to obtain multiple prediction sequences, the method further includes:

[0016] The first dialogue dataset is obtained by collecting dialogues in which the speaker is the first object.

[0017] The second dialogue dataset is obtained by collecting dialogues in which the speaker is the second object.

[0018] Optionally, the method further includes:

[0019] Remove the nth keyword from the predicted sequence whose relevance to the mth keyword is lower than a first threshold; wherein the mth keyword is any keyword in the predicted sequence other than the nth keyword;

[0020] The step of performing keyword statistics on the plurality of predicted sequences to obtain statistical results includes:

[0021] Keyword statistics are performed on the multiple predicted sequences after the nth keyword has been removed, and the statistical results are obtained.

[0022] Optionally, before using a machine learning model to predict the dialogue between the first object and the second object to obtain multiple prediction sequences, the method further includes:

[0023] The historical dialogue between the first object and the second object is divided into a first historical dialogue dataset corresponding to the first object and a second historical dialogue dataset corresponding to the second object.

[0024] The first historical dialogue dataset is subjected to classification training and completeness training in machine learning to obtain the first model;

[0025] The second historical dialogue dataset is trained using classification in machine learning to obtain a second model; wherein the dialogue in the process of using the machine learning model to predict the dialogue between the first object and the second object occurs after the historical dialogue.

[0026] Optionally, the statistical results are specifically used to determine at least one of the following:

[0027] Product type;

[0028] Pre-sales issues related to products;

[0029] Product after-sales issues.

[0030] Optionally, the statistical results are also used to determine at least one of the following:

[0031] The degree of satisfaction of the first object with the second object's answer;

[0032] The first individual's intention to purchase the product;

[0033] The first object's evaluation tendency towards the product.

[0034] A second aspect of this disclosure provides a statistical apparatus for multi-turn dialogues, the apparatus comprising:

[0035] The prediction module is used to predict the dialogue data between a first object and a second object using a machine learning model to obtain multiple prediction sequences; wherein, the prediction sequence includes one or more keywords from the dialogue between the first object and the second object;

[0036] The statistics module is used to perform keyword statistics on the multiple predicted sequences to obtain statistical results; wherein, the statistical results are used at least to determine the dialogue topic, the questions involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data.

[0037] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0038] Memory;

[0039] A processor, connected to the memory, is configured to execute computer instructions stored in the memory, enabling the implementation of the steps in the statistical method for multi-turn dialogue provided in the first aspect above.

[0040] According to a fourth aspect of the present disclosure, a computer storage medium is provided, wherein executable instructions are stored therein; when executed by a processor, the computer executable instructions are able to implement the steps in the statistical method for multi-turn dialogue provided in the first aspect.

[0041] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0042] This embodiment utilizes a machine learning model to predict the dialogue data between a first object and a second object. Compared to existing technologies that train and predict only the input content of one object, this approach simultaneously considers the inputs of both the first and second objects in multi-turn dialogues, resulting in more comprehensive and accurate statistical results. Furthermore, after predicting multiple prediction sequences using the machine learning model, keyword statistics are performed on these multiple prediction sequences to obtain statistical results. This approach combines machine learning and statistical methods. Compared to existing technologies that rely solely on machine learning for training and prediction, requiring large amounts of labeled datasets, this embodiment performs keyword statistics on the multiple prediction sequences after machine learning training and prediction. This method is more accurate and user-friendly for small to medium-sized datasets, yielding more precise statistical results.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a statistical method for multi-turn dialogue as an exemplary embodiment;

[0045] Figure 2 A flowchart illustrating a statistical method for multi-turn dialogue as an exemplary embodiment;

[0046] Figure 3 A flowchart illustrating a statistical method for multi-turn dialogue as an exemplary embodiment;

[0047] Figure 4 A flowchart illustrating a statistical method for multi-turn dialogue as an exemplary embodiment;

[0048] Figure 5 A flowchart illustrating a statistical method for multi-turn dialogue as an exemplary embodiment;

[0049] Figure 6 A schematic diagram illustrating a statistical device for multi-turn dialogue as an exemplary embodiment;

[0050] Figure 7 A schematic diagram illustrating a statistical device for multi-turn dialogue as an exemplary embodiment;

[0051] Figure 8 A schematic diagram illustrating a statistical device for multi-turn dialogue as an exemplary embodiment;

[0052] Figure 9 A schematic diagram illustrating a statistical device for multi-turn dialogue as an exemplary embodiment;

[0053] Figure 10 A schematic diagram illustrating a statistical device for multi-turn dialogue as an exemplary embodiment;

[0054] Figure 11 This is a schematic diagram of a statistical device for multi-turn dialogue, as shown in an exemplary embodiment. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this disclosure as detailed in the appended claims.

[0056] like Figure 1 As shown in the embodiments of this disclosure, a statistical method for multi-turn dialogue is provided, the method comprising:

[0057] Step S110: Use a machine learning model to predict the dialogue data between the first object and the second object to obtain multiple prediction sequences; wherein, the prediction sequence includes one or more keywords in the dialogue between the first object and the second object;

[0058] Step S120: Perform keyword statistics on the plurality of predicted sequences to obtain statistical results; wherein, the statistical results are used at least to determine the dialogue topic, the issues involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data.

[0059] In this embodiment of the disclosure, the multi-turn dialogue may occur between two language and / or voice input parties, or between three or more language and / or voice input parties.

[0060] In some embodiments, the language and / or voice input parties in the multi-turn dialogue may be replaced by newly joined input parties that have left the conversation, provided that both are guaranteed. For example, if the chatbot cannot meet the user's needs, it may leave, and a human customer service representative may replace the chatbot in answering the user's questions.

[0061] In this embodiment of the disclosure, the first object and the second object are different dialogue input parties. The first object can be user input, and the second object can be input from a human customer service representative or a chatbot. In some embodiments, when the answer from a chatbot cannot resolve the first object's problem, the second object can be replaced by a human customer service representative.

[0062] In some embodiments, the keyword may be: the word with the highest information entropy in the dialogue data.

[0063] In other embodiments, the keywords may be verbs and / or nouns.

[0064] In this embodiment of the disclosure, the machine learning model can be a training model obtained by training a supervised learning model on a dialogue dataset. For example, it can include, but is not limited to, any classification model such as linear classifiers like LR (logistic regression), support vector machines (SVM), or deep neural networks (DNN) trained on the dialogue dataset.

[0065] In this embodiment of the disclosure, the machine learning model includes a completeness detection model and a classification model. The completeness detection model is used to detect the completeness of keywords, and the classification model is used to classify the keywords or the sentences containing the keywords. In some embodiments, classifying the keywords or the sentences containing the keywords includes, but is not limited to: classifying the content of the sentences, classifying the content of the sentences to obtain the themes and emotions involved in the sentences, and then classifying the themes involved in the sentences and / or the emotions involved in the sentences.

[0066] In this embodiment of the disclosure, the topic types involved in the keywords or the sentences containing the keywords include, but are not limited to, product types, pre-sales issues, and after-sales issues. The emotions involved in the sentences include, but are not limited to, positive or negative emotions. For example, the emotions in sentences involving evaluations can be classified, with evaluations containing positive emotions being classified as positive evaluations, and evaluations containing negative emotions being classified as negative evaluations.

[0067] In this embodiment of the disclosure, natural language vectorization technology, one-hot encoding, or word vectorization techniques can be used to preprocess and vectorize the dialogue data between the first object and the second object to obtain K-dimensional / H-dimensional vectors.

[0068] In this embodiment of the disclosure, the keywords are related to the dialogue topic, the questions involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data. Therefore, the statistical results obtained by performing keyword statistics on the multiple prediction sequences have high accuracy.

[0069] In this embodiment of the disclosure, the dialogue topic of the dialogue data can be multiple relatively discrete topics of different categories. For example, a product belongs to one topic, and a problem belongs to another. Products can include items such as electronic products, daily necessities, food, and clothing. Problems can include issues such as broken, damaged, expired, or slow logistics. The discrete product topics plus the problem topics can be combined into a final comprehensive topic. In one embodiment, the product can be "computer," and the product problem can be "broken." These two are discrete topics, but they can be combined to obtain a comprehensive topic such as "the computer is broken."

[0070] In this embodiment of the disclosure, the method of performing keyword statistics on multiple predicted sequences to obtain statistical results integrates the dialogue input by the first object and the second object, without wasting the dialogue content input by either object, which can improve the accuracy of the statistical results.

[0071] In this embodiment of the disclosure, the method of performing keyword statistics on multiple predicted sequences includes count statistics and probability statistics. The count statistics and probability statistics of multiple predicted sequences are fused according to weights, and then the count statistics and probability statistics of multiple predicted sequences are fused again according to preset parameters. Compared with machine learning, which may lead to underfitting or overfitting on small sample datasets and thus fail to obtain accurate results, the statistical results obtained by multiple statistical methods and multiple fusions are more accurate.

[0072] In this embodiment of the disclosure, statistical results are obtained by combining machine learning with statistical methods. While being compatible with large datasets, it can also perform machine learning and statistical processing on small and medium datasets to obtain more accurate statistical results.

[0073] like Figure 2 As shown, in a statistical method for multi-turn dialogue in this embodiment of the present disclosure, step S110 involves using a machine learning model to predict the dialogue data between a first object and a second object, obtaining multiple prediction sequences, including:

[0074] Step S1101: Use the first model in the machine learning model to predict the first dialogue dataset corresponding to the first object to obtain a first prediction sequence; wherein, the first model is trained based on the dialogue dataset of the same object as the first type of object and / or the historical dialogue data of the first object.

[0075] Step S1102: Use the second model in the machine learning model to predict the second dialogue dataset corresponding to the second object to obtain a second prediction sequence; wherein, the second model is trained based on the dialogue dataset of the same object as the second type of object and / or the historical dialogue data of the second object.

[0076] In this embodiment of the disclosure, when the first object is a user, the first model includes a first classification model and a completeness detection model. The completeness detection model is used to detect the completeness of individual sentences in the first dialogue dataset; the first classification model is used to classify the individual sentences in the first dialogue dataset. In some embodiments, classifying the individual sentences in the first dialogue dataset includes, but is not limited to: classifying the content of the individual sentences, classifying them according to the content of the individual sentences to obtain the topics involved in the individual sentences and the emotions involved in the individual sentences. Then, classifying the topics involved in the individual sentences and / or classifying the emotions involved in the individual sentences.

[0077] In some embodiments, the topic types involved in the single sentences include, but are not limited to, product type, pre-sales product issues, and after-sales product issues. The emotions involved in the single sentences include, but are not limited to, positive or negative emotions. For example, the emotions in single sentences involving evaluations can be classified, with evaluations including positive emotions being classified as positive evaluations, and evaluations including negative emotions being classified as negative evaluations. In this embodiment of the disclosure, when the second object is a skilled chatbot customer service representative, the second model includes a classification model for classifying single sentences in the second dialogue dataset.

[0078] In this embodiment of the disclosure, when the second object is a human customer service representative, the second model may also include: a completeness detection model and a classification model.

[0079] In this embodiment of the disclosure, when the second object is a robot customer service representative, but the robot customer service representative is not proficient enough and may also have incomplete sentences, the second model may also include: a completeness detection model and a classification model.

[0080] In this embodiment of the disclosure, the completeness detection model is used to detect whether a sentence is complete. If the sentence is complete, it is marked as complete; if the sentence is incomplete, it is marked as incomplete. In one embodiment, a complete sentence is marked as 1, and an incomplete sentence is marked as 0.

[0081] In this embodiment of the disclosure, incomplete sentences are merged. For example, incomplete sentences are merged based on their category and the continuity of the context. For instance, in some embodiments, if the previous sentence is incomplete and relates to a product, and the next sentence is also incomplete and relates to a product issue, and the previous and next sentences are entered consecutively, then the incomplete sentence from the previous sentence can be merged with the incomplete sentence from the next sentence.

[0082] In this embodiment of the disclosure, the dialogue dataset of the same object in the first type of object is a dialogue dataset of different users belonging to the same type. The historical dialogue data of the first object is the historical dialogue data belonging to the same user. The historical dialogue data refers to dialogue data that occurred before the current moment. In this way, a first model combining different user data and historical dialogue data of the same user can be obtained, which makes the utilization rate of the dataset higher and can also improve the accuracy of the final result.

[0083] In this embodiment of the disclosure, the dialogue dataset of the second type of object is a dialogue dataset input by different customer service representatives of the same type (e.g., robot customer service and human customer service). The historical dialogue data of the second object is a dialogue dataset input by the same customer service representative, i.e., the same robot customer service representative or the same human customer service representative. In this way, a second model can be obtained by combining different customer service data and historical dialogue data of the same customer service representative, which makes the utilization rate of the dataset higher and can also improve the accuracy of the final result.

[0084] In this embodiment of the disclosure, the first dialogue dataset corresponding to the first object is predicted to obtain a first prediction sequence, or the second dialogue dataset corresponding to the second object is predicted to obtain a second prediction sequence. The dialogue content input by the first object and the second object is separated, preprocessed and vectorized, which helps to improve the processing efficiency.

[0085] like Figure 3 As shown, in a statistical method for multi-turn dialogue in this embodiment of the present disclosure, step S120, which involves performing keyword statistics on the plurality of predicted sequences to obtain statistical results, includes:

[0086] Step S1201: Perform keyword statistics on the first predicted sequence to obtain the first statistical result;

[0087] Step S1202: Perform keyword statistics on the second predicted sequence to obtain the second statistical result;

[0088] Step S1203: Based on the product of the first statistical result and the first weight, and the product of the second statistical result and the second weight, obtain the statistical result used to determine the topic of the dialogue.

[0089] In this embodiment of the disclosure, the keyword statistics for the first predicted sequence include: performing frequency statistics and probability statistics on the topics reflected by the top N most frequent keywords in the first predicted sequence, and then summing the frequency statistics and normalizing the probability statistics to obtain a first frequency normalized value L1 and a first probability normalized value PL1. The first statistical result includes: the first frequency normalized value L1 and the first probability normalized value PL1.

[0090] In this embodiment of the disclosure, keywords themselves can reflect the topic of the conversation, and keyword combinations can also reflect the topic of the conversation.

[0091] Similarly, in this embodiment of the disclosure, the keyword statistics for the second predicted sequence include: performing frequency statistics and probability statistics on the topics containing the top N most frequent keywords in the second predicted sequence, and then summing the frequency statistics and normalizing the probability statistics to obtain a second frequency normalized value L2 and a second probability normalized value PL2. The second statistical result includes: the second frequency normalized value L2 and the second probability normalized value PL2.

[0092] In this embodiment of the disclosure, based on the product of the first statistical result and the first weight w, and the product of the second statistical result and the second weight 1-w, a count statistical result TL1 and a probability statistical result TL2 are obtained. Based on the product of the first preset parameter w1 and the count statistical result TL1, and the product of the second preset parameter 1-w1 and the probability statistical result TL2, a statistical result for determining the dialogue topic is obtained. The values ​​of the first weight w and the second weight 1-w range from 0 to 1. w=1 represents complete emphasis on the statistics of the first object, and w=0 represents complete emphasis on the statistics of customer service.

[0093] In this embodiment of the disclosure, the preset parameter w1 ranges from 0 to 1. w1 = 1 indicates that the result is completely biased towards the count statistics, and w1 = 0 indicates that the result is completely biased towards the probability statistics.

[0094] In this embodiment of the disclosure, the first weight w is calculated based on the mean, variance, and effective dialogue turns (1 / n, o1, n), (1 / m, o2, m) of the correlation coefficient sequence group distribution index of the first prediction sequence and the second prediction sequence, using optimized Gaussian fusion: the second prediction sequence weight coefficient w = sqrt(n*o1) / (sqrt(n*o1)+sqrt(m*o2)), where sqrt means square root.

[0095] In this embodiment of the disclosure, the probability cumulative statistics and the count cumulative statistics method, which integrates the probability cumulative statistics and count cumulative statistics of the first object and the second object, can adapt to datasets of various sizes, especially small and medium-sized datasets. Furthermore, it can obtain relatively accurate statistical results.

[0096] like Figure 4 As shown, in a statistical method for multi-turn dialogue in this embodiment of the present disclosure, before performing step S110, which uses a machine learning model to predict the dialogue between a first object and a second object to obtain multiple prediction sequences, the method further includes:

[0097] Step S104: Collect the dialogues in the dialogue data where the speaker is the first object to obtain the first dialogue dataset;

[0098] Step S105: Collect the dialogues in the dialogue data where the speaker is the second object to obtain the second dialogue dataset.

[0099] In this embodiment of the disclosure, the datasets of speakers from different objects in the dialogue data are collected separately and independently predicted and statistically analyzed in subsequent processes before being merged. This allows for the fusion of dialogue datasets input from different objects, supplementing user input, obtaining complete multi-turn dialogue information, and achieving better accuracy.

[0100] In this embodiment of the disclosure, before performing keyword statistics on the plurality of predicted sequences to obtain statistical results, the statistical method for the multi-turn dialogue further includes:

[0101] Obtain the relevance of the keywords in the predicted sequence, and adjust the keywords in the predicted sequence according to the relevance; wherein, obtaining the relevance of the keywords in the predicted sequence includes: obtaining the relevance of the keywords according to the element values ​​and the number of elements in the predicted sequence;

[0102] The nth keyword in the predicted sequence whose relevance to the mth keyword is lower than the first threshold is moved from the predicted sequence into the first relevant session set.

[0103] In this embodiment of the disclosure, the first relevance session set is a low relevance session set.

[0104] In the embodiments of the present disclosure, the design of the relevance includes: performing relevance design based on the obtained first prediction sequence and second prediction sequence. For example, in one embodiment, after training the first dialogue dataset and the second dialogue dataset using a machine learning model, a prediction result is obtained, and the prediction result includes a first prediction sequence P1 and a second prediction sequence P2. In one embodiment, a probability prediction result is obtained using the softmax function, where 0 < pij < 1, pi1 + pi2 +... + piW = 1, pij is the predicted probability value of the element in the i-th row and j-th column, piW is the predicted probability value of the element in the i-th row and w-th column, i, j, W = 1, 2, 3 ··· n, and n is a positive integer), and the first prediction sequence is obtained as P1 = [[p11, p12,..., p1W], [p21, p22,..., p2W], [p31, p32,..., p3W], [p41, p42,..., p4W]]; the second prediction sequence is obtained as P2 = [[p11, p12,..., p1W], [p21, p22,..., p2W]]. Relevance design is performed on the first prediction sequence P1 and the second prediction sequence P2:

[0105]

[0106] Here, p

[0110] is the predicted probability, W (W >= 2) is the number of prediction types, the value range of R is [0, the higher the relevance of the single sentence where the keyword is located, 1 represents complete relevance, and 0 represents complete irrelevance.

[0107] In the embodiments of the present disclosure, it is assumed that the relevance sequence of the obtained first prediction sequence P1 is [0.6, 0.52, 0.2, 0.7], and the second prediction sequence is [0.61, 0.76]. The relevance threshold of the first prediction sequence P1 is set as R1 = 0.5, and the relevance threshold of the second prediction sequence P2 is set as R2 = 0.5. Then, the relevance in the first prediction sequence P1 and the second prediction sequence P2 is compared with the relevance threshold. If it is greater than the threshold, subsequent statistics are performed. If it is lower than the threshold, the single sentence where the n-th keyword is located is put into the low-relevance conversation set. For example, the ellipsis "···" sentence is put into the low-relevance conversation set.

[0108] In the embodiments of the present disclosure, performing relevance detection on the keywords in the prediction sequence can improve the accuracy of the statistical results of the multi-round dialogue and enhance the controllability between steps in the multi-round dialogue statistical method.

[0109] A statistical method for multi-round dialogue in the embodiments of the present disclosure further includes:

[0110] Step S1103: Remove the nth keyword from the predicted sequence whose relevance to the mth keyword is lower than a first threshold; wherein the mth keyword is any keyword in the predicted sequence other than the nth keyword;

[0111] Step S120, which involves performing keyword statistics on the plurality of predicted sequences to obtain statistical results, includes:

[0112] Step S1204: Perform keyword statistics on the multiple predicted sequences after removing the nth keyword to obtain the statistical results.

[0113] In this embodiment of the disclosure, the m-th keyword and the n-th keyword belong to different sentences.

[0114] In this embodiment of the present disclosure, step S1103 further includes removing the sentence containing the nth keyword in the predicted sequence whose relevance to the sentence containing the mth keyword is lower than a first threshold.

[0115] In this embodiment of the disclosure, step S1204 further includes: performing keyword statistics on the multiple predicted sequences of the single sentence in which the nth keyword is removed, and obtaining the statistical results.

[0116] In this embodiment of the disclosure, keyword statistics are performed on the plurality of predicted sequences after the nth keyword has been removed to obtain the statistical results, including:

[0117] Calculate the sequence distribution index (1 / n, o1(variance), n) for the first predicted sequence P1 and the second predicted sequence P2. For example, the sequence distribution index of P1 is (1 / 4, 0.1403, 4), and the sequence distribution index of P2 is (1 / 2, 0.076, 2). Based on the sequence distribution index, the first weight w is calculated as w = sqrt(4*0.1403) / (sqrt(4*0.1403)+sqrt(2*0.076)) = 0.664.

[0118] In this embodiment of the disclosure, after obtaining the first weight w, steps S1201, S1202, and S1203 can be executed to obtain statistical results for determining the dialogue topic.

[0119] In this embodiment, removing keywords with low relevance or the sentences containing those keywords can improve the execution efficiency of subsequent steps and reduce unnecessary calculations. It can also obtain more accurate weighting coefficients, improving the accuracy of the statistical results of fusing the first and second prediction sequences representing the user. For small to medium-sized datasets, this can improve the accuracy of the statistical results, thereby enhancing the controllability of the statistical method for the multi-turn dialogue.

[0120] Combination Figure 5 As shown, in a statistical method for multi-turn dialogue according to an embodiment of this disclosure, before performing step S110, which uses a machine learning model to predict the dialogue data between a first object and a second object to obtain multiple prediction sequences, the method further includes:

[0121] Step S101: The historical dialogue between the first object and the second object is divided into a first historical dialogue dataset corresponding to the first object and a second historical dialogue dataset corresponding to the second object.

[0122] Step S102: Perform classification training and completeness training in machine learning on the first historical dialogue dataset to obtain the first model;

[0123] Step S103: Perform classification training in machine learning on the second historical dialogue dataset to obtain a second model; wherein, the dialogue in the process of using the machine learning model to predict the dialogue between the first object and the second object occurs after the historical dialogue.

[0124] In this embodiment of the disclosure, before performing step S110, steps S101, S102 and S103 can be performed offline.

[0125] In this embodiment, the first object is the user, and the second object is customer service. Customer service can be a chatbot or a human customer service representative. By using chatbots and / or human customer service representatives as additional data sources, and by independently training and learning them, and using the corresponding predicted statistical results as supplementary information to the user input, more complete multi-turn dialogue information can be obtained, thus improving the accuracy of the conversation summary.

[0126] In this embodiment of the disclosure, the first historical dialogue dataset is a user historical dialogue dataset, and the second historical dialogue dataset is a customer service historical dialogue dataset. The first and second historical dialogue datasets can be small to medium-sized datasets or large datasets.

[0127] In this embodiment of the disclosure, if the historical dialogue occurs only once, the historical dialogue dataset generated by this historical dialogue is collected. If the historical dialogue generated by this historical dialogue is small, the collected historical dialogue dataset is a small dataset. In this case, machine learning training can also be performed on the historical dialogue that has only occurred once. For example, classification training and completeness training in machine learning can be performed on the first historical dialogue dataset, and classification training in machine learning can be performed on the second historical dialogue dataset.

[0128] In this embodiment of the disclosure, after training on the first historical dialogue dataset and the second historical dataset, subsequent dialogues can be directly predicted. For example, after the first historical dialogue occurs, the system can be trained directly on the first historical dialogue to predict the second dialogue. Thus, the solution provided by this embodiment is more user-friendly for multi-turn dialogues with few historical dialogues at startup, and has wider applicability compared to existing methods that require large datasets for training. For example, in one embodiment, after a new user has their first historical dialogue with the application's customer service, when the new user has another dialogue with the application's customer service, the machine learning model obtained from training on the previous historical dialogue can be used to predict the dialogue that the new user will have with customer service again.

[0129] In this embodiment of the disclosure, based on machine learning and combined with statistics, the summary of multi-turn dialogues with fewer initial historical dialogues at startup is more user-friendly and can also obtain more accurate statistical results for better prediction.

[0130] In this embodiment of the disclosure, the statistical results are specifically used to determine at least one of the following:

[0131] Product type;

[0132] Pre-sales issues related to products;

[0133] Product after-sales issues.

[0134] In this embodiment of the disclosure, the product types may include, but are not limited to: electronic products, daily necessities, food, and clothing. Electronic products may include, but are not limited to: computers, mobile phones, headphones, smart bracelets, and hard drives. Pre-sales issues may include, but are not limited to: product quality issues, warranty status, and return / exchange policies. After-sales issues may include, but are not limited to: how to use the product, product quality issues, and whether the product needs to be returned / exchanged.

[0135] In this embodiment of the disclosure, the dialogue is statistically analyzed, and the statistical results are used to determine the product type and pre-sales or after-sales issues of the product. This can provide users with smarter and more accurate services, improve user satisfaction, and also improve service efficiency.

[0136] In this embodiment of the disclosure, a first model in the machine learning model is used to predict the first dialogue dataset corresponding to the first object, thereby obtaining a first prediction sequence. The first model is obtained by performing classification training and completeness training in machine learning on the first historical dialogue dataset. The first model includes: a first classification model and a first completeness detection model.

[0137] In this embodiment of the disclosure, when predicting a dialogue that occurs after a historical dialogue, i.e., the current dialogue, a first completeness detection model is used to detect the completeness of the current dialogue and obtain a completeness sequence for each sentence. When an incomplete sentence is detected, the position corresponding to that sentence in the sequence is set to 0; when a complete sentence is detected, the position corresponding to that sentence in the sequence is set to 1.

[0138] In this embodiment of the disclosure, incomplete statements can be merged based on the category, completeness, and contextual continuity of the user input statements. For example, in one embodiment, based on the following five sentences input by the user: "My computer", "Broken", "Um", "Yes", "Thank you, I have no problem, / ::)", the following single-sentence completeness sequence [0,0,1,1,1] is obtained, and it is detected that "My computer" and "Broken" are two consecutively input incomplete statements. "My computer" belongs to the topic of product type, and "Broken" belongs to the topic of product problem. Thus, "My computer" and "Broken" can be merged into "My computer is broken".

[0139] In this embodiment of the disclosure, incomplete sentences entered by the user can also be merged based on the relevance between keywords. For example, in some embodiments, product type and product problem are set to have a high degree of relevance. In this case, "my computer" and "broken" are highly relevant phrases, and "my computer" belongs to the topic of product type and "broken" belongs to the topic of product problem. Thus, "my computer" and "broken" can be merged into "my computer is broken".

[0140] In this embodiment of the disclosure, merging incomplete but related statements together can yield more accurate topic categories and improve the accuracy of statistical results.

[0141] In this embodiment of the disclosure, the statistical results are also used to determine at least one of the following:

[0142] The degree of satisfaction of the first object with the second object's answer;

[0143] The first individual's intention to purchase the product;

[0144] The first object's evaluation tendency towards the product.

[0145] In this embodiment of the disclosure, the degree of satisfaction of the first object with the second object's answer includes: very dissatisfied, dissatisfied, neutral, satisfied, and very satisfied.

[0146] In this embodiment of the disclosure, the customer service representative's response attitude can be improved and the service competitiveness enhanced based on the satisfaction level of the first object with the second object's response, that is, the user's satisfaction level with the robot customer service and / or human customer service.

[0147] In this embodiment of the disclosure, the purchase intention of the first object towards the product includes: strong purchase intention, general purchase intention, weak purchase intention, and no purchase intention.

[0148] In this embodiment of the disclosure, based on the first object's purchase intention for the product, that is, the user's purchase intention for the product after the user asks pre-sales questions, the application can subsequently provide product recommendations to the user to increase the likelihood of the user making a purchase.

[0149] In this embodiment of the disclosure, the first object's evaluation tendency towards the product includes: a positive evaluation tendency and a negative evaluation tendency. A positive evaluation tendency indicates a positive evaluation of the product, while a negative evaluation tendency indicates a critical and dissatisfied evaluation of the product.

[0150] In this embodiment of the disclosure, the performance of the product can be improved, the competitiveness of the product can be enhanced, and the user market of the product can be expanded based on the first object, namely the user's evaluation tendency of the product.

[0151] Combination Figure 6 As shown in the present embodiment, a statistical device 200 for multi-turn dialogue is provided, characterized in that the device includes:

[0152] The prediction module 210 is used to predict the dialogue data between the first object and the second object using a machine learning model to obtain multiple prediction sequences; wherein, the prediction sequence includes one or more keywords of the dialogue between the first object and the second object;

[0153] The statistics module 220 is used to perform keyword statistics on the plurality of predicted sequences to obtain statistical results; wherein, the statistical results are used at least to determine the dialogue topic, the questions involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data.

[0154] In this embodiment of the disclosure, the prediction module 210 is further configured to:

[0155] The first model in the machine learning model is used to predict the first dialogue dataset corresponding to the first object to obtain a first prediction sequence; wherein, the first model is trained based on the dialogue dataset of the same object as the first type of object and / or the historical dialogue data of the first object;

[0156] A second prediction sequence is obtained by using the second model in the machine learning model to predict the second dialogue dataset corresponding to the second object; wherein, the second model is trained based on the dialogue dataset of the same object as the second type of object and / or the historical dialogue data of the second object.

[0157] In this embodiment of the disclosure, the statistics module 220 is further configured to:

[0158] Keyword statistics are performed on the first predicted sequence to obtain the first statistical result;

[0159] Keyword statistics were performed on the second predicted sequence to obtain the second statistical result;

[0160] Based on the product of the first statistical result and the first weight, and the product of the second statistical result and the second weight, a statistical result is obtained for determining the topic of the dialogue.

[0161] Combination Figure 7 As shown in this embodiment, the multi-turn dialogue statistics device 200 further includes:

[0162] The first collection module 230 is used to collect dialogues in which the speaker is the first object to obtain the first dialogue dataset.

[0163] The second collection module 240 is used to collect dialogues in the dialogue data in which the speaker is the second object to obtain the second dialogue dataset.

[0164] In this embodiment of the disclosure, the statistical device 200 for multi-turn dialogue further includes:

[0165] The relevance acquisition module is used to acquire the relevance of the keywords in the predicted sequence;

[0166] An adjustment module is used to adjust the keywords in the predicted sequence according to the relevance; wherein, the relevance acquisition module is used to obtain the relevance of the keywords according to the element values ​​and the number of elements in the predicted sequence;

[0167] The module is used to add the nth keyword in the predicted sequence whose relevance to the mth keyword is lower than a first threshold to the first relevance session set.

[0168] Combination Figure 8 As shown in this embodiment, the multi-turn dialogue statistics device 200 further includes:

[0169] The removal module 250 is used to remove the nth keyword in the predicted sequence whose relevance to the mth keyword is lower than a first threshold; wherein the mth keyword is any keyword in the predicted sequence other than the nth keyword;

[0170] The statistics module 220 is used to perform keyword statistics on the multiple prediction sequences after the nth keyword has been removed, and to obtain the statistical results.

[0171] Combination Figure 9 As shown in this embodiment, the multi-turn dialogue statistics device 200 further includes:

[0172] The segmentation module 260 is used to segment the historical dialogue between the first object and the second object into a first historical dialogue dataset corresponding to the first object and a second historical dialogue dataset corresponding to the second object.

[0173] The first training module 270 is used to perform classification training and completeness training in machine learning on the first historical dialogue dataset to obtain the first model.

[0174] The second training module 280 is used to perform classification training in machine learning on the second historical dialogue dataset to obtain a second model; wherein, the dialogue in the process of using the machine learning model to predict the dialogue between the first object and the second object occurs after the historical dialogue.

[0175] In conjunction with the above embodiments, the following examples are provided:

[0176] Example 1: Provide a multi-turn dialogue statistics device.

[0177] Combination Figure 10 An overall architecture diagram for summarizing multi-turn dialogues; Figure 11 This is a detailed flowchart of the statistics module.

[0178] The multi-turn dialogue statistical device includes a machine learning training module and a prediction module, and also incorporates a single-sentence classification model and a statistical module.

[0179] Machine Learning Training Module: Used to perform the following steps:

[0180] Step S301: Process the logs of the labeled multi-turn dialogues and divide the dialogue logs into user input set A and customer service (bot) input set B.

[0181] Step S302: The dataset A (which has two labels, one is a classification label and the other is a completeness label of 0 and 1) is preprocessed (filtered) and quantized, and then input into the machine learning training module 1 for training and evaluation to obtain the user single sentence classification model M1 and the completeness detection model X1.

[0182] Step S303 involves preprocessing (filtering) the dataset B and quantizing it, then inputting it into the machine learning training module for training and evaluation to obtain the customer service (robot) single-sentence classification model M2.

[0183] Prediction module: Used to perform the following steps:

[0184] Step S304: Process a complete customer service record and divide it into user input sequence A1 and customer service (robot) input sequence B1;

[0185] Step S305: The user input sequence A1 is preprocessed (filtered) and quantized (same as training X1), then input into the X1 model. The completeness of a single sentence in the model is detected to obtain the completeness sequence detection W1. Based on the completeness detection W1, the sequence A1 is relevance-merged to obtain A2, which is then preprocessed (filtered) and quantized (same as training M1), then input into the model M1 for prediction, thus obtaining the predicted sequence P1.

[0186] Step S306: Preprocess (filter) the sequence B1 and quantize it (same as training), then input it into the model M2 for prediction, and the predicted sequence P2 can be obtained.

[0187] Step S307: Input the sequence results P1 and P2 obtained in steps S305 and S306 into the statistics module for statistical analysis and related post-processing to obtain the statistical result T. The statistical result T is then used as the summary of the multi-turn dialogue.

[0188] Step S307 specifically includes the following steps:

[0189] Step S3071: Input the predicted sequences P1 and P2 into the statistics module, determine whether the statistics are complete, if complete, end and output the results; if not complete, proceed to step S3072.

[0190] Step S3072: Evaluate the relevance of sequence P1 to obtain a relevance sequence (r11, r12...r1n), and set a relevance threshold R1; if it is greater than the threshold R1, proceed to the statistics module in step S3073; otherwise, proceed to the low relevance session set.

[0191] The P2 sequence is relevance evaluated to obtain a relevance sequence (r21, r22...r2m), and a relevance threshold R2 is set. If the relevance is greater than the threshold R2, the process proceeds to the statistics module in step S3073; otherwise, the process proceeds to the low relevance session set.

[0192] Step S3073: Calculate the weight coefficient w of different data sources, calculate the mean, variance and effective dialogue rounds (1 / n, o1, n) and (1 / m, o2, m) of the distribution index of the correlation coefficient sequence group of P1 and P2, and use the optimized Gaussian fusion: P1 weight coefficient w = sqrt(n*o1) / (sqrt(n*o1)+sqrt(m*o2)), where sqrt means square root;

[0193] Step S3074: The filtered P1 and P2 sequences obtained in step S3072 are subjected to cumulative counting (L1, L2) and probability cumulative statistics (PL1, PL2).

[0194] In step S3075, the cumulative counts (L1, L2) and cumulative probability statistics (PL1, PL2) obtained in step S3074 are weighted and accumulated, and the weight coefficient w in 4.23 is introduced. The final statistical result (TL1, TL2) is obtained by w*(L1, L2)+(1-w)*(PL1, PL2). The result (TL1, TL2) can be used as the result of the multi-turn dialogue summary.

[0195] Example 2: Provide a multi-turn dialogue statistics method.

[0196] Step S401: The existing e-commerce scenario customer service multi-turn dialogue dataset D and its labels are shown in the table below, with some examples:

[0197]

[0198]

[0199] Table 1 - Customer Service Dataset D

[0200] Step S402, Training

[0201] Step S4021: Divide the entire e-commerce customer service dataset D from step S401 into user dataset A and customer service (bot) data B, as shown in Tables 2 and 3 below:

[0202]

[0203] Table 2 - User Dataset A

[0204]

[0205] Table 3 - Customer Service (Robot) Dataset B

[0206] Step S4022: Take datasets A and B and perform various preprocessing steps to filter out irrelevant information such as special characters of related emojis, links, and sensitive information such as ID cards and phone numbers. Then, perform routine desensitization processing to obtain datasets a1 and b1.

[0207]

[0208] Table 4 - User Dataset a1

[0209]

[0210]

[0211] Table 5 - Customer Service (Robot) Dataset b1

[0212] Step S4023: Using natural language vectorization techniques, such as one-hot encoding or word vectorization, the a1 / b1 data is vectorized to obtain K-dimensional / H-dimensional vectors, as follows:

[0213] a1=[((x11,x12,...x1K),y11),...,((xn1,xn2,...xnK),y1n)]

[0214] c1=[((x11,x12,...x1K),yc1),...,((xn1,xn2,...xnK),ycn)]

[0215] b1=[((x11,x12,...x1H),y21),...((xm1,xm2,...xmH),y2m)]

[0216] There are W categories in total for y;

[0217] Step S4024: Use any classification model such as machine learning model, LR (logistic regression), SVM (support vector machines), DNN (deep Neural Networks), etc., to train a1, b1, and c1 to obtain user data classification model M1, customer service (robot) classification model M2, and user data integrity detection model X1.

[0218] Step S403, prediction

[0219] Step S4031, predict a complete conversation as shown in Table 6.

[0220]

[0221]

[0222] Table 6 - Predicted Sessions

[0223] Step S4032: Divide the data to obtain user sequence A1 as shown in Table 7 and customer service sequence B1 as shown in Table 8.

[0224]

[0225] Table 7 - User Sequence A1

[0226]

[0227] Table 8 - Customer Service (Robot) Sequence B1

[0228] Step S4033: Perform preprocessing - vectorization (same as training) to obtain the following vector sequence:

[0229] A1=[[(x11,x12,...x1K)],[x21,x22,...x2K],[x31,x32,...x3K],[x41,x42,...x4K],[x51,x52,...x5K]]

[0230] B1=[[(x11,x12,...x1H)],[x21,x22,...x2H]]

[0231] Step S4034: Feed A1 into the X1 model for completeness prediction, resulting in the following completeness sequence [0,0,1,1,1], where 0 indicates the sentence is incomplete and 1 indicates complete. Here, it means that the first sentence "My computer" and the second sentence "It's broken" are both incomplete, so they are merged into "My computer + It's broken". Here, "+" indicates the concatenation of special characters in the sentence. The A1 dataset is as follows:

[0232]

[0233]

[0234] Table 9-A1 Dataset

[0235] The vector features of dataset A1' are as follows:

[0236] A1'=[[(x'11,x'12,...x'1K)],[x'21,x'22,...x'2K],[x'31,x'32,...x'3K],[x'41,x'42,...x'4K]]

[0237] Step S404, according to Figure 11 The process involves feeding A1' / B1 into the M1 and M2 models obtained during training, respectively, to obtain the following prediction results P1 / P2:

[0238] P1=[[p11,p12,...,p1W],[p21,p22,...,p2W],[p31,p32,...,p3W],[p41,p42,...,p4W]]

[0239] P2=[[p11,p12,...,p1W],[p21,p22,...,p2W]]

[0240] softmax probability prediction, where 0 <pij<1,pi1+pi2+...+piW=1。

[0241] Step S405, Calculation of related index

[0242] Step S4051, Relevance Design

[0243]

[0244] Here, pi is the prediction probability, W (W>=2) is the number of prediction categories, and R ranges from [0,1]. The larger R is, the higher the relevance of the sentence containing the keyword. 1 represents complete relevance, and 0 represents complete irrelevance.

[0245] Step S4052, in this example, sets the correlation thresholds for sequences P1 and P2 to R1 = 0.5 and R2 = 0.5, respectively;

[0246] Step S4053: Calculate the relevance of sequences P1 and P2 obtained in step S404 according to the relevance defined in step S4051.

[0247] Assume the correlation sequence of P1 is [0.6, 0.52, 0.2, 0.7];

[0248] Assume the P2 correlation sequence is [0.61, 0.76];

[0249] The relevant statement is compared with the relevance threshold at step S4052. If it is greater than the relevance threshold, proceed to step S406; otherwise, the relevant statement is obtained and transferred to the low-relevance session set O. Here, the "..." statement will enter the low-relevance O.

[0250] In step S4054, the distribution indices (1 / n, o1(variance), n) of sequences P1 and P2 are (1 / 4, 0.1403, 4) and (1 / 2, 0.076, 2), respectively, and w = sqrt(4*0.1403) / (sqrt(4*0.1403)+sqrt(2*0.076)) = 0.664.

[0251] Step S406, the statistics module performs the following steps:

[0252] The relevant data obtained from the probability predictions of top N (N=3 here) and top M (M=3 here) in step S405 are statistically analyzed as shown in Table 10 below. N and M refer to the different topics that occur most frequently in the first N or first M occurrences. For example, in the table below, y1 can be the first type of topic, such as "My computer is broken"; y2 can be the second type of topic, such as "My phone is broken"; y3 can be the third type of topic, such as "My headphones are broken". In {y1, y2, y3}, y1 corresponds to the number of times the topic category of y1 occurs, and so on; in {y1: p11, y2: p12, y3: p13}, y1: p11 corresponds to the probability of the topic category of y1 occurring, and so on.

[0253]

[0254]

[0255] Table 10 - Statistical Examples

[0256] Step S4061: By summing P1 and P2, we can obtain...

[0257] Step S40611, P1 top 3 cumulative count and normalization to L1:

[0258] {y2:3, y3:3, y4:2, y1:1} / max([3,3,2,1])

[0259] Step S40612, P1 top 3 cumulative count statistics and normalization to L1:

[0260] {y2: (p12+p21+p41), y3: (p13+p22+p42), y4: (p24+p44), y1: p11} / max([(p12+p21+p41), (p13+p22+p42), (p24+p44), p11])

[0261] Similarly, in step S40613, we can obtain the cumulative count statistics of P2 top3 and normalize them to PL1, and the cumulative probability statistics of P2 top3 and normalize them to PL2.

[0262] Step S4062, integrate user and customer service (bot) statistical results:

[0263] Based on step S4054, the robot's effective information weight w = 0.664 is obtained. Substitute this weight into the following formula:

[0264] (TL1, TL2)=w*(L1, PL1)+(1-w)*(L2, PL2) (1.3)

[0265] The weighting coefficient w here ranges from [0,1]. w=1 means that user statistics are given full consideration; w=0 means that customer service statistics are given full consideration.

[0266] TL1 and TL2 represent the results of the count-based fusion statistics and the results of the probability fusion statistics, respectively.

[0267] Step S4063, fusion probability and count statistics:

[0268] TL=w1*TL1+(1-w1)*TL2 (1.4)

[0269] This TL statistical summary combines probability statistics and count statistics results; the value of w1 ranges from [0,1], w1=1 completely biases towards probability statistics results, w1=0 completely biases towards count statistics results; TL can be used as the final result of a multi-turn dialogue summary. Adjustable parameters are available.

[0270] Combination Figure 11 The secondary sorting module is used to execute steps S4062 and S4063. Based on machine learning, the secondary sorting module in the statistics module combines probability cumulative statistics, count cumulative statistics methods, and multi-source statistical result fusion techniques, making it suitable for datasets of various sizes, especially small and medium-sized datasets.

[0271] Based on the above embodiments and examples, the solution disclosed herein, in order to address the practical problems of summarizing multi-turn dialogues in the field of intelligent customer service, applies mature machine learning or cutting-edge deep learning, and proposes a completely new technical approach to solve multi-turn dialogues through process control and mathematical statistics methods.

[0272] From a module perspective, it includes an offline training module and an online prediction module; from a technology stack perspective, it not only integrates machine learning (deep learning), a single sentence integrity detection model, and an integrity corpus preprocessing module, but also includes a discrete statistical analysis module for machine learning results, machine learning result flow control, adaptive fusion of multiple data sources, and self-controlled fusion of multiple indicator methods.

[0273] In the prediction module, the complete corpus preprocessing algorithm, single-sentence relevance algorithm, dependency relevance index control method and their algorithm carrier program are as follows: First, based on the machine learning or deep learning model obtained through training, the single-sentence prediction result is used as the input of the single-sentence relevance algorithm to obtain the relevance index result; according to the threshold set according to the actual situation, if it is less than the threshold, the single-sentence prediction result is truncated; conversely, if it is greater than the threshold, it will enter the statistics module.

[0274] In the secondary sorting module, the probability cumulative statistics method, the count cumulative statistics method, and the multi-source statistical result fusion technology method and their final algorithm carrier program: the probability cumulative statistics method and the count cumulative statistics method are two different technical methods that reflect the summary of multi-round dialogue from different perspectives. By using the fusion technology, a final statistical result can be obtained, which can improve the accuracy of the statistical results.

[0275] Compared with existing technologies, the present application has the following technical advantages:

[0276] like Figure 10 and Figure 11 As shown, based on the pipeline control flow architecture and incorporating supervised machine learning, it can be applied to datasets of various sizes, especially small and medium-sized datasets. It is more user-friendly for zero-startup multi-turn dialogue summaries (for example, by performing machine learning and training on the first-turn historical dialogue, the second-turn historical dialogue can be predicted and statistically analyzed). Compared with existing technologies that require large-scale datasets, it has wider applicability.

[0277] Because it adopts an accumulation-based technical solution and a filtering-based multi-source adaptive weighting algorithm, it has high interpretability. Furthermore, it employs self-controlled multi-scheme fusion and process control technology, achieving a certain degree of controllability compared to the uncontrollability and uninterpretability of the end-to-end training and prediction model, which is essentially a black box. Therefore, it has achieved a breakthrough in controllability and interpretability.

[0278] This technical solution integrates various methods. On one hand, it uses partial data from robots (human-computer dialogue) or customer service (human-to-human) as supplementary features to complement user input and obtain complete multi-turn dialogue information. On the other hand, it combines statistical results from different technical solutions to summarize the multi-turn dialogue. In addition, the single-sentence integrity detection module obtains effective contextual data, all of which benefit the algorithm's performance and thus achieve better accuracy.

[0279] In this embodiment of the disclosure, an electronic device is provided, the electronic device comprising:

[0280] processor;

[0281] Memory used to store processor-executable instructions;

[0282] When the processor is used to run the computer service, it implements the steps in the feedback method described above.

[0283] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0284] In this embodiment of the disclosure, a storage medium is provided, the storage medium having computer-executable instructions, which are executed by a processor to implement the steps in the feedback method described above.

[0285] Alternatively, if the integrated units described above in the embodiments of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0286] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A statistical method for multi-turn dialogues, characterized in that, The method includes: A machine learning model is used to predict the dialogue data between a first object and a second object, resulting in multiple prediction sequences; wherein, the prediction sequence includes one or more keywords from the dialogue between the first object and the second object; The relevance of the keyword is obtained based on the element values ​​and the number of elements in the predicted sequence; The nth keyword in the predicted sequence whose relevance to the mth keyword is lower than a first threshold is removed from the predicted sequence and placed into the first relevance session set; wherein, the mth keyword is any keyword in the predicted sequence other than the nth keyword; Remove the nth keyword from the predicted sequence whose relevance to the mth keyword is lower than a first threshold. Keyword statistics are performed on the plurality of predicted sequences after the nth keyword has been removed to obtain statistical results; wherein, the statistical results are used at least to determine the dialogue topic, the issues involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data.

2. The method according to claim 1, characterized in that, The method of using a machine learning model to predict the dialogue data between the first and second objects yields multiple prediction sequences, including: The first model in the machine learning model is used to predict the first dialogue dataset corresponding to the first object to obtain a first prediction sequence; wherein, the first model is trained based on the dialogue dataset of the same object as the first type of object and / or the historical dialogue data of the first object; A second prediction sequence is obtained by using the second model in the machine learning model to predict the second dialogue dataset corresponding to the second object; wherein, the second model is trained based on the dialogue dataset of the same object as the second type of object and / or the historical dialogue data of the second object.

3. The statistical method for multi-turn dialogue according to claim 2, characterized in that, The step of performing keyword statistics on the plurality of predicted sequences to obtain statistical results includes: Keyword statistics are performed on the first predicted sequence to obtain the first statistical result; Keyword statistics were performed on the second predicted sequence to obtain the second statistical result; Based on the product of the first statistical result and the first weight, and the product of the second statistical result and the second weight, a statistical result is obtained for determining the topic of the dialogue.

4. The statistical method for multi-turn dialogue according to claim 2 or 3, characterized in that, Before using a machine learning model to predict the dialogue between the first object and the second object to obtain multiple predicted sequences, the method further includes: The first dialogue dataset is obtained by collecting dialogues in which the speaker is the first object. The second dialogue dataset is obtained by collecting dialogues in which the speaker is the second object.

5. The statistical method for multi-turn dialogue according to claim 2, characterized in that, Before using a machine learning model to predict the dialogue between the first object and the second object to obtain multiple predicted sequences, the method further includes: The historical dialogue between the first object and the second object is divided into a first historical dialogue dataset corresponding to the first object and a second historical dialogue dataset corresponding to the second object. The first historical dialogue dataset is subjected to classification training and completeness training in machine learning to obtain the first model; The second historical dialogue dataset is trained using machine learning classification to obtain a second model; wherein the dialogue in the process of using the machine learning model to predict the dialogue between the first object and the second object occurs after the historical dialogue.

6. The statistical method for multi-turn dialogue according to claim 1, characterized in that, The statistical results are specifically used to determine at least one of the following: Product type; Pre-sales issues related to products; Product after-sales issues.

7. The statistical method for multi-turn dialogue according to claim 1 or 6, characterized in that, The statistical results are also used to determine at least one of the following: The degree of satisfaction of the first object with the second object's answer; The first individual's intention to purchase the product; The first object's evaluation tendency towards the product.

8. A statistical device for multi-turn dialogue, characterized in that, The device includes: The prediction module is used to predict the dialogue data between a first object and a second object using a machine learning model to obtain multiple prediction sequences; wherein, the prediction sequence includes one or more keywords from the dialogue between the first object and the second object; The device is also used to obtain the relevance of the keyword based on the element values ​​and the number of elements in the predicted sequence; The nth keyword in the predicted sequence whose relevance to the mth keyword is lower than a first threshold is removed from the predicted sequence and placed into the first relevance session set; wherein, the mth keyword is any keyword in the predicted sequence other than the nth keyword; Remove the nth keyword from the predicted sequence whose relevance to the mth keyword is lower than a first threshold. The statistics module is used to perform keyword statistics on the plurality of predicted sequences after removing the nth keyword, and obtain statistical results; wherein, the statistical results are used at least to determine the dialogue topic, the issues involved in the dialogue, the dialogue satisfaction of either party, and the intention of either party in the dialogue data.

9. An electronic device, characterized in that, The electronic device includes: Memory; A processor, connected to the memory, is configured to execute computer instructions stored in the memory, enabling it to implement the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium stores the computer-executable instructions; when the computer-executable instructions are executed by the processor, they can implement the method described in any one of claims 1 to 7.

Images