Dialogue quality inspection method and device and storage medium

Abstract

The invention discloses a dialogue quality inspection method and device and a storage medium, and the method comprises the steps: obtaining a complete dialogue text between a plurality of users and an AI robot, and the complete dialogue text at least comprises a sub-dialogue text; analyzing each sub dialogue text in the complete dialogue text to obtain a multi-dimensional index analysis result; performing distribution statistics on each complete dialogue text to obtain a multi-dimensional distribution statistics result; and generating a dialogue quality inspection result based on the multi-dimensional index analysis result and the multi-dimensional distribution statistical result. According to the method, analysis is performed from the sub-dialogue text and the complete dialogue text during quality inspection, so that quality inspection is performed according to the sentence granularity and the whole dialogue granularity, a multi-dimensional analysis result is obtained, and the quality inspection efficiency and accuracy are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a dialogue quality inspection method, device and storage medium. Background Art

[0002] In voice conversation services, it is necessary to conduct quality inspection on the conversation content of AI robots, so as to urge customer service to standardize the language used in the process of communicating with customers and ensure the quality of service. At present, this quality inspection is mostly carried out by quality inspectors who manually read the entire conversation and manually score the AI ​​robots. However, if the number of conversations is large, a large amount of quality inspection manpower is required, which is time-consuming and inefficient. Summary of the invention

[0003] Based on this, it is necessary to provide a dialogue quality inspection method, device and storage medium for the above-mentioned technical problems to solve at least one of the above-mentioned technical problems.

[0004] The present invention provides a dialogue quality inspection method, comprising:

[0005] Obtaining complete conversation texts between a number of users and the AI ​​robot, wherein the complete conversation text includes at least one sub-conversation text;

[0006] Analyze each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result;

[0007] Performing distribution statistics on each of the complete dialogue texts to obtain multi-dimensional distribution statistics results;

[0008] Based on the multi-dimensional indicator analysis results and the multi-dimensional distribution statistical results, a dialogue quality inspection result is generated.

[0009] Optionally, according to a conversation quality inspection method provided by the present invention, the multi-dimensional indicator analysis result includes the accuracy of intention recognition and the accuracy of word slot recognition;

[0010] The analysis of each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result includes:

[0011] For any round of dialogue text:

[0012] Matching the user consultation text in the sub-dialogue text with a preset intention database to obtain a plurality of historical consultation texts and text intentions corresponding to the historical consultation texts; and matching the user consultation text in the sub-dialogue text with a preset word slot database to obtain a plurality of historical consultation texts and word slot information corresponding to the historical consultation texts;

[0013] Input the user consultation text, each of the historical consultation texts and their textual intentions into a preset target macro model to obtain the intention recognition result output by the target macro model; and input the user consultation text, each of the historical consultation texts and their word slot information into a preset target macro model to obtain the word slot recognition result output by the target macro model;

[0014] The intent recognition accuracy is calculated based on the intent recognition results corresponding to all sub-dialogue texts and the user intent recognized by the AI ​​robot, and the intent recognition accuracy is calculated based on the word slot recognition results corresponding to all sub-dialogue texts and the word slot information recognized by the AI ​​robot;

[0015] Among them, the user intention is obtained by the AI ​​robot performing intention recognition on the user consultation text during the conversation process, and the word slot information is obtained by the AI ​​robot performing word slot recognition on the user consultation text during the conversation process.

[0016] Optionally, according to a conversation quality inspection method provided by the present invention, the multi-dimensional indicator analysis result includes a single-round conversation accuracy rate and a complete conversation accuracy rate;

[0017] The analysis of each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result includes:

[0018] For any round of dialogue text:

[0019] Compare the intent recognition result corresponding to the sub-dialogue text with the user intent recognized by the AI ​​robot;

[0020] Comparing the word slot recognition result corresponding to the sub-dialogue text with the word slot recognized by the AI ​​robot;

[0021] The single-round conversation accuracy and the complete conversation accuracy are determined based on the intent comparison results and word slot comparison results corresponding to each round of conversation text.

[0022] Optionally, according to a conversation quality inspection method provided by the present invention, the multi-dimensional distribution statistical results include intention distribution statistical results;

[0023] The distribution statistics of each complete dialogue text are performed to obtain multi-dimensional distribution statistics results, including:

[0024] Matching any of the complete dialogue texts with a preset intention database to obtain a plurality of historical complete dialogue texts and text intentions corresponding to the historical complete dialogue texts;

[0025] Input any of the complete dialogue texts, each of the historical complete dialogue texts and their textual intentions into a preset target macro model, and obtain the intention recognition result output by the target macro model;

[0026] The intention recognition results corresponding to all complete dialogue texts are counted to obtain the intention distribution statistical results.

[0027] Optionally, according to a conversation quality inspection method provided by the present invention, the multi-dimensional distribution statistical results include intention distribution statistical results;

[0028] The distribution statistics of each complete dialogue text are performed to obtain multi-dimensional distribution statistics results, including:

[0029] Matching any of the complete dialogue texts with a preset complete dialogue intention database to obtain a plurality of historical complete dialogue texts and text intentions corresponding to the historical complete dialogue texts;

[0030] Input any of the complete dialogue texts, each of the historical complete dialogue texts and their textual intentions into a preset target large model, and obtain a complete dialogue intention recognition result output by the target large model;

[0031] The complete conversation intention recognition results corresponding to all complete conversation texts are counted to obtain the intention distribution statistical results.

[0032] Optionally, according to a conversation quality inspection method provided by the present invention, the multi-dimensional distribution statistical results include call duration distribution and call end node distribution;

[0033] The distribution statistics of each complete dialogue text are performed to obtain multi-dimensional distribution statistics results, including:

[0034] Determining the call duration and call end node of each complete conversation text;

[0035] The call duration and the call end node are counted respectively to obtain the call duration distribution and the call end node distribution.

[0036] Optionally, according to a conversation quality inspection method provided by the present invention, generating a conversation quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result includes:

[0037] According to the intent recognition results and word slot recognition results corresponding to each round of conversation text, determine the number of intents corresponding to each intent type and the number of word slots corresponding to each word slot type;

[0038] According to the number of intents, the intent recognition results corresponding to each intent type are sorted, and according to the number of word slots, the word slot recognition results corresponding to each word slot type are sorted;

[0039] The dimensional index analysis results, the multi-dimensional distribution statistics results, the intent sorting results and the word slot sorting results are analyzed to obtain the dialogue quality inspection results.

[0040] Optionally, according to a conversation quality inspection method provided by the present invention, after generating the conversation quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result, the method further includes:

[0041] Generate optimization suggestions based on the dialogue quality inspection results;

[0042] Based on the optimization suggestions, the AI ​​robot is adjusted.

[0043] The present invention also provides a dialogue quality inspection device, comprising:

[0044] An acquisition module, used to acquire complete conversation texts between a number of users and the AI ​​robot, wherein the complete conversation text includes at least one sub-conversation text;

[0045] An analysis module, used to analyze each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result;

[0046] A statistical module, used to perform distribution statistics on each of the complete dialogue texts to obtain multi-dimensional distribution statistical results;

[0047] A generation module is used to generate a dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result.

[0048] The present invention also provides a computer device, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, wherein the processor implements the above-mentioned dialogue quality inspection method when executing the computer-readable instructions.

[0049] The present invention also provides one or more readable storage media storing computer-readable instructions, and the computer-readable instructions implement the above-mentioned dialogue quality inspection method when executed by a processor.

[0050] The above-mentioned dialogue quality inspection method, device and storage medium include: obtaining a complete dialogue text between a number of users and an AI robot, wherein the complete dialogue text includes at least one sub-dialogue text; analyzing each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result; performing distribution statistics on each of the complete dialogue texts to obtain a multi-dimensional distribution statistical result; and generating a dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result. The present invention implements quality inspection according to the sentence granularity and the entire conversation granularity by analyzing the sub-dialogue text and the complete dialogue text respectively during quality inspection, thereby obtaining a multi-dimensional analysis result, and effectively improving the efficiency and accuracy of quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0052] Figure 1 is a flow chart of a conversation quality inspection method in one embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of a flow chart of calculating the accuracy of intent recognition provided by an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of a flow chart of calculating the accuracy of word slot recognition provided by an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of a flow chart of calculating the accuracy of a single-round dialogue provided by an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of a flow chart of calculating the accuracy of a single-round dialogue provided by an embodiment of the present invention;

[0057] Figure 6 It is a schematic diagram of a flow chart of calculating the statistical results of intention distribution provided by an embodiment of the present invention;

[0058] Figure 7 It is a flowchart of calculating the statistical results of intention distribution provided by an embodiment of the present invention;

[0059] Figure 8 is a structural schematic diagram of a conversation quality inspection device in one embodiment of the present invention;

[0060] Fig. 9 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The terms used in one or more embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present invention. The singular forms of "a", "said" and "the" used in one or more embodiments of the present invention are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0063] Specifically, in one embodiment, if Figure 1 As shown, Figure 1 : is a flow chart of a conversation quality inspection method in an embodiment of the present invention. The embodiment of the present invention provides a conversation quality inspection method, comprising the following steps:

[0064] Step S11, obtaining complete conversation texts between several users and the AI ​​robot;

[0065] It should be noted that the embodiments of the present invention are applied in scenarios such as invitation scenarios and customer service scenarios. Among them, the invitation scenario generally refers to the dialogue scenario in which the AI ​​robot initiates an appointment or invitation to the user, and the customer service scenario refers to the dialogue scenario in which the AI ​​customer service solves the user's problems or needs.

[0066] Specifically, call voice records of interactions between several users and AI robots are obtained, and then text recognition is performed on the call voice records to obtain a complete conversation text, wherein the complete conversation text includes at least one sub-conversation text, that is, each round of conversation between user consultation and AI response is regarded as a round of sub-conversation text.

[0067] Step S12, analyzing each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result;

[0068] It should be noted that the multi-dimensional indicator analysis results include intent recognition accuracy, word slot recognition accuracy, single-round dialogue accuracy, and complete conversation accuracy, among which the single-round dialogue accuracy refers to the proportion of each independent dialogue round in a dialogue system that the system can correctly understand and respond to. The complete conversation accuracy refers to the proportion of the entire dialogue session that the system can correctly understand and respond to from beginning to end. A complete conversation refers to the entire process from the user starting the conversation to the end of the conversation. In other embodiments, the multi-dimensional indicator analysis results may also include other indicator analysis results such as the opening statement hang-up rate. In addition, it should be noted that in natural language processing (NLP) and dialogue systems, word slots refer to specific information units in a dialogue or text, which usually represent specific details in the user's intention or a specific part of a task that needs to be completed. Word slots are specific types of information that users want to express or that the system needs to collect. They can be dates, times, places, quantities, amounts, names, etc.

[0069] Specifically, for any round of sub-dialogue texts: calculate the similarity between the user consultation text in the sub-dialogue text and each historical consultation text in the preset intention database, and then select multiple historical consultation texts with high similarity, and determine the text intent corresponding to the selected historical consultation text in the preset intention database; then input the user consultation text, each historical consultation text and its text intent into the preset target large model to obtain the intent recognition result output by the target large model. Determine the user intent obtained by the AI ​​robot through intent recognition of the user consultation text during the conversation, and then based on the intent recognition results corresponding to all sub-dialogue texts and the user intent recognized by the AI ​​robot, count the number of times the intent recognition results in all sub-dialogue texts match the user intent; further, based on the number of times and the total number of conversations of all sub-dialogue texts, calculate the intent recognition accuracy.

[0070] In addition, the similarity between the user consultation text in the sub-dialogue text and each historical consultation text in the preset word slot database is calculated, and then multiple historical consultation texts with high similarity are selected, and the word slot information corresponding to the selected historical consultation text is determined in the preset word slot database; then the user consultation text, each historical consultation text and its word slot information are input into the preset target large model to obtain the word slot recognition result output by the target large model. The word slot information obtained by the AI ​​robot during the conversation by word slot recognition of the user consultation text is determined, and then based on the word slot recognition results corresponding to all sub-dialogue texts and the word slot information recognized by the AI ​​robot, the number of times the word slot recognition results in all sub-dialogue texts match the word slot information is statistically obtained; further, based on the number of word slot matches and the total number of conversations of all sub-dialogue texts, the word slot recognition accuracy is calculated.

[0071] In addition, in each round of sub-dialogue text, it is necessary to compare the user intent recognized by the AI ​​robot with the intent recognition result corresponding to the sub-dialogue text output by the target large model, and to compare the word slot value recognized by the AI ​​robot with the word slot recognition result corresponding to the sub-dialogue text output by the target large model. If the intent comparison is successful and the word slot comparison is successful, the sub-dialogue text of this round is determined to be accurate. If the intent comparison fails or the word slot comparison fails, the sub-dialogue text of this round is determined to be incorrect. Then the number of accurate sub-dialogue texts is determined, and further, based on the number and the total number of dialogues for all sub-dialogue texts, the single-round dialogue accuracy is calculated.

[0072] In addition, the complete conversation text is sequentially split into sub-conversation texts for each round of back-and-forth conversations between the user and the AI ​​robot, and then the accuracy of each round of sub-conversation texts is judged in sequence. The process of judging whether the sub-conversation texts are accurate is specifically described in the above embodiment and will not be repeated here. If any round of sub-conversation texts is inaccurate, the complete conversation text is judged to be inaccurate. If all sub-conversation texts of the complete conversation text are accurate, the complete conversation text is judged to be accurate. Further, the number of accurate complete conversation texts and the total number of complete conversation texts are determined, and then the accuracy rate of the complete conversation is calculated based on the number of complete conversation texts and the total number of complete conversation texts.

[0073] Step S13, performing distribution statistics on each complete conversation text to obtain multi-dimensional distribution statistics results;

[0074] It should be noted that the multi-dimensional distribution statistics include intention distribution statistics, intent distribution statistics, call duration distribution, call end node distribution, etc. Intent distribution statistics usually refer to the distribution of the purpose or goal expressed by the user during the conversation or interaction. Intent distribution statistics focus on the specific intention or purpose expressed by the user in the interaction.

[0075] Specifically, the complete conversation text is matched with a preset complete conversation intention database to query multiple historical complete conversation texts with high similarity to the current conversation text, as well as the text intentions corresponding to these historical complete conversation texts. For example, multiple historical complete conversation texts with high similarity to the current conversation text are determined by calculating cosine similarity, edit distance, etc. Then, the complete conversation text, the matched historical complete conversation text, and its corresponding text intention are merged and input into a pre-built target large model, so as to use the target large model to perform intention recognition on the complete conversation text and obtain more accurate conversation intention recognition results. Furthermore, the intention recognition results corresponding to all complete conversation texts are counted to calculate the frequency of occurrence, proportion, distribution trend, etc. of each intention, and obtain intention distribution statistics.

[0076] In addition, the complete conversation text is matched with a preset complete conversation intention database to query multiple historical complete conversation texts with high similarity to the current conversation text, as well as the text intentions corresponding to these historical complete conversation texts. For example, the historical complete conversation text that is most similar to the current conversation text is determined by calculating cosine similarity, edit distance, etc. Then, the complete conversation text, the matched historical complete conversation text, and its corresponding text intention are merged and input into the pre-built target large model, so as to use the target large model to perform intent recognition on the conversation text and obtain more accurate conversation intention recognition results. Furthermore, the intent recognition results corresponding to all complete conversation texts will be counted to calculate the frequency of occurrence, proportion, distribution trend, etc. of each intention, and obtain the intention distribution statistics.

[0077] In addition, the call duration and call end node of each complete conversation text are determined: then, based on the call duration of each complete conversation text, the number of conversations with different call durations in different duration intervals is counted, and the call duration distribution is determined based on the number of conversations in different duration intervals; in addition, all call end nodes are classified, and the number of occurrences of each category is counted, and the call end node distribution is determined based on the number of occurrences of each category.

[0078] Step S14, generating dialogue quality inspection results based on the multi-dimensional indicator analysis results and multi-dimensional distribution statistics results.

[0079] In one embodiment, each analysis result in the multidimensional indicator analysis result is compared with the corresponding numerical threshold, and each statistical result in the multidimensional distribution statistical result is compared with the corresponding numerical threshold. According to each comparison result, the conversation quality inspection result is determined. For example, if the word slot recognition accuracy is less than 80%, the word slot instance needs to be optimized; if the intent recognition accuracy is less than 80%, the model corpus needs to be optimized; if the average call duration is less than 10s, the opening remarks need to be optimized, etc.

[0080] In one embodiment, according to the intention recognition results and word slot recognition results corresponding to each round of dialogue text, the number of intentions corresponding to each intention type and the number of word slots corresponding to each word slot type are counted respectively. Then, according to the number of intentions, the intention recognition results corresponding to each intention type are sorted, so as to select a plurality of intention recognition results with a large number according to the intention sorting results, and the word slot recognition results corresponding to each word slot type are sorted according to the number of word slots, so as to select a large number of word slot recognition results according to the word slot sorting results. Optionally, the selected intention recognition results and word slot recognition results can also be displayed. Further, the dimensional index analysis results, the multi-dimensional distribution statistics results, and the selected intention recognition results and word slot recognition results are compared with the corresponding numerical thresholds respectively, and then the dialogue quality inspection results are generated according to the comparison results. For example, the average call duration is less than 10s, the opening speech is optimized, the same word slot recognition result appears frequently, the word slot instance needs to be optimized, the same intention appears frequently, and the intention speech needs to be optimized.

[0081] In one embodiment of the present invention, after generating the dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result, the method further includes:

[0082] Generate optimization suggestions based on the conversation quality inspection results; adjust the AI ​​robot based on the optimization suggestions.

[0083] Specifically, based on the dialogue quality inspection results, the problems of the AI ​​robot are determined, and then optimization suggestions are generated based on the existing problems. For example, if the accuracy of intent recognition is low, the model corpus corresponding to the AI ​​robot needs to be optimized, and if the accuracy of word slot recognition is low, the word slot examples corresponding to the AI ​​robot need to be optimized; based on the optimization suggestions, the AI ​​robot is adjusted. In this way, targeted optimization suggestions are given for the problems of the existing AI robot, and a high-performance AI robot model is built.

[0084] The embodiment of the present invention, through the above scheme, includes: obtaining the complete dialogue text between several users and the AI ​​robot, wherein the complete dialogue text includes at least one sub-dialogue text; analyzing each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result; performing distribution statistics on each complete dialogue text to obtain a multi-dimensional distribution statistical result; and generating a dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result. The embodiment of the present invention analyzes the sub-dialogue text and the complete dialogue text separately during the quality inspection, thereby achieving quality inspection according to the sentence granularity and the entire conversation granularity, obtaining a multi-dimensional analysis result, and effectively improving the efficiency and accuracy of quality inspection.

[0085] In one embodiment of the present invention, each sub-dialogue text in the complete dialogue text is analyzed to obtain a multi-dimensional index analysis result, including:

[0086] For any round of sub-dialogue texts: match the user consultation text in the sub-dialogue text with the preset intention database to obtain multiple historical consultation texts and the text intentions corresponding to the historical consultation texts; and match the user consultation text in the sub-dialogue text with the preset word slot database to obtain multiple historical consultation texts and the word slot information corresponding to the historical consultation texts; input the user consultation text, each historical consultation text and its text intention into the preset target big model to obtain the intention recognition result output by the target big model; and input the user consultation text, each historical consultation text and its word slot information into the preset target big model to obtain the word slot recognition result output by the target big model; based on the intention recognition results corresponding to all sub-dialogue texts and the user intentions recognized by the AI ​​robot, calculate the intention recognition accuracy, and based on the word slot recognition results corresponding to all sub-dialogue texts and the word slot information recognized by the AI ​​robot, calculate the intention recognition accuracy.

[0087] It should be noted that user intent is obtained by the AI ​​robot through intent recognition of the user consultation text during the conversation process, and word slot information is obtained by the AI ​​robot through word slot recognition of the user consultation text during the conversation process. That is, during the conversation between the user and the AI ​​robot, intent recognition and word slot recognition will be performed on the user consultation text of each conversation process.

[0088] It should be noted that the preset intention database includes a plurality of historical consultation texts collected in advance and the text intentions corresponding to each historical consultation text; the preset word slot database includes a plurality of historical consultation texts collected in advance and the word slot information corresponding to the historical consultation texts.

[0089] Specifically, the following steps are performed for any round of dialogue text:

[0090] For intent recognition accuracy: refer to Figure 2 , Figure 2It is a flow chart of calculating the accuracy of intention recognition provided by an embodiment of the present invention. Specifically, the user consultation text in the sub-dialogue text is matched with the preset intention database. For example, the similarity between the user consultation text and each historical consultation text in the preset intention database is calculated to select multiple historical consultation texts with higher similarity; then the user consultation text, each historical consultation text and its text intention are merged and input into the preset target large model to obtain the intention recognition result corresponding to the user consultation text output by the target large model, and then the intention recognition result corresponding to the sub-dialogue text and the user intention recognized by the AI ​​robot are matched, and the number of intention matches is determined according to the intention matching result, and then the intention recognition accuracy is calculated according to the number of intention matches and the total number of dialogues of the sub-dialogue text, wherein, if the intention matching result is a successful match, it is determined that the user intention recognized by the AI ​​robot is accurate; if the intention matching result is a failed match, it is determined that the user intention recognized by the AI ​​robot is wrong.

[0091] For word slot recognition accuracy: refer to Figure 3 , Figure 3 It is a flow chart of calculating the accuracy of word slot recognition provided by an embodiment of the present invention. Specifically, the user consultation text in the sub-dialogue text is matched with a preset word slot database. For example, the similarity between the user consultation text and each historical consultation text in the preset word slot database is calculated to select multiple historical consultation texts with higher similarity; then the user consultation text, each historical consultation text and its word slot information are merged and input into a preset target large model to obtain the word slot recognition result corresponding to the user consultation text output by the target large model, and then the word slot recognition result corresponding to the sub-dialogue text and the word slot information recognized by the AI ​​robot are matched. According to the word slot matching result, the number of word slot matches is determined, and then the word slot recognition accuracy is calculated according to the number of word slot matches and the total number of dialogues in the sub-dialogue text, wherein, if the word slot matching result is a successful match, it is determined that the word slot information recognized by the AI ​​robot is accurate; if the word slot matching result is a failed match, it is determined that the word slot information recognized by the AI ​​robot is incorrect.

[0092] The embodiment of the present invention performs intent recognition and word slot recognition on the dialogue text, and then combines the intent recognition accuracy and the word slot recognition accuracy to accurately determine the quality inspection result of the AI ​​robot, thereby effectively improving the quality inspection accuracy.

[0093] In one embodiment of the present invention, each sub-dialogue text in the complete dialogue text is analyzed to obtain a multi-dimensional index analysis result, including:

[0094] For any round of sub-dialogue text: compare the intent recognition result corresponding to the sub-dialogue text with the user intent recognized by the AI ​​robot; compare the word slot recognition result corresponding to the sub-dialogue text with the word slot recognized by the AI ​​robot; determine the single-round dialogue accuracy and the complete conversation accuracy based on the intent comparison results and word slot comparison results corresponding to each round of sub-dialogue text.

[0095] Specifically, the following steps are performed for any round of dialogue text:

[0096] For single-round dialogue accuracy: refer to Figure 4 , Figure 4 It is a flow chart of calculating the accuracy of a single-round dialogue provided by an embodiment of the present invention. Specifically, the intent recognition result corresponding to the sub-dialogue text is compared with the user intent recognized by the AI ​​robot; the word slot recognition result corresponding to the sub-dialogue text is compared with the word slot recognized by the AI ​​robot. If the intent comparison result is that the intent comparison is successful, and the word slot comparison result is that the word slot comparison is successful, then it is determined that the sub-dialogue text of this round is accurate. If the intent comparison result is that the intent comparison fails, or the word slot comparison result is that the word slot comparison fails, then it is proved that the sub-dialogue text of this round is wrong, so as to determine the number of times the sub-dialogue text is accurate according to the intent comparison result and the word slot comparison result corresponding to each round of sub-dialogue text. Further, based on the number of times and the total number of dialogues of all sub-dialogue texts, the single-round dialogue accuracy is calculated.

[0097] For complete session accuracy: refer to Figure 5 , Figure 5 It is a flow chart of calculating the accuracy of a single round of dialogue provided by an embodiment of the present invention. Specifically, the complete dialogue text is sequentially divided into sub-dialogue texts for each round of dialogue between the user and the AI ​​robot, and each round of sub-dialogue texts is judged to be accurate according to the intention comparison results and word slot comparison results of each round of sub-dialogue texts. If there is an inaccurate sub-dialogue text, the complete dialogue text is judged to be inaccurate; if each round of sub-dialogue text of the complete dialogue text is accurate, the complete dialogue text is accurate, and the number of accurate complete dialogue texts is determined according to the judgment results of each complete dialogue text, and then the complete dialogue accuracy is calculated based on the number of complete dialogue texts and the total number of complete dialogue texts.

[0098] Through the above scheme, the present invention realizes quality inspection according to the sentence granularity and the entire conversation granularity, obtains multi-dimensional analysis results, and effectively improves the quality inspection efficiency and accuracy.

[0099] In one embodiment of the present invention, distribution statistics are performed on each complete dialogue text to obtain multi-dimensional distribution statistics results, including:

[0100] Match any complete dialogue text with the preset intention database to obtain multiple historical complete dialogue texts and the text intentions corresponding to the historical complete dialogue texts; input any complete dialogue text, each historical complete dialogue text and its text intention into the preset target big model to obtain the intention recognition result output by the target big model; count the intention recognition results corresponding to all complete dialogue texts to obtain the intention distribution statistical results.

[0101] It should be noted that the preset intention database stores a plurality of pre-collected historical complete dialogue texts and the text intentions corresponding to the historical complete dialogue texts. Figure 6 , Figure 6 It is a flowchart of calculating the statistical results of intention distribution provided by an embodiment of the present invention. Specifically, any complete dialogue text is matched with a preset intention database. For example, the similarity between any complete dialogue text and the historical complete dialogue texts in the preset intention database is calculated, and then multiple historical complete dialogue texts with higher similarity are selected. Further, any complete dialogue text, each historical complete dialogue text and its text intention are input into a preset target large model to obtain the intention recognition result output by the target large model; then the intention recognition results corresponding to all complete dialogue texts are counted to obtain the intention distribution statistical results. Optionally, the frequency of occurrence of each intention type is calculated based on the intention recognition results corresponding to all complete dialogue texts; the intention distribution statistical results are determined based on the frequency of occurrence of each intention type, wherein pie charts, bar charts or other statistical charts can be used to visualize the intention distribution.

[0102] The embodiment of the present invention obtains intention distribution statistics according to the intention of each complete conversation text, performs conversation quality inspection from the granularity of the entire conversation, and effectively improves the accuracy of quality inspection.

[0103] In one embodiment of the present invention, distribution statistics are performed on each complete dialogue text to obtain multi-dimensional distribution statistics results, including:

[0104] Match any complete dialogue text with the preset complete dialogue intention database to obtain multiple historical complete dialogue texts and the text intentions corresponding to the historical complete dialogue texts; input any complete dialogue text, each historical complete dialogue text and its text intention into the preset target big model to obtain the complete dialogue intention recognition result output by the target big model; count the complete dialogue intention recognition results corresponding to all complete dialogue texts to obtain the intention distribution statistical results.

[0105] It should be noted that the preset intention database stores a plurality of historical complete dialogue texts collected in advance and the text intentions corresponding to the historical complete dialogue texts. Figure 7 , Figure 7It is a flow chart of calculating the statistical results of intention distribution provided by an embodiment of the present invention. Specifically, any complete dialogue text is matched with a preset intention database. For example, the similarity between any complete dialogue text and the historical complete dialogue texts in the preset intention database is calculated, and then multiple historical complete dialogue texts with higher similarity are selected. Further, any complete dialogue text, each historical complete dialogue text and its textual intention are input into a preset target large model to obtain the intention recognition result output by the target large model; then, the intention recognition results corresponding to all complete dialogue texts are typed and statistical results of intention distribution are obtained. Optionally, the frequency of occurrence of each intention type is calculated based on the intention distribution statistical results, and the intention distribution statistical results are determined based on the frequency of occurrence of each intention type. Pie charts, bar charts or other statistical charts can be used to visualize the intention distribution.

[0106] The embodiment of the present invention obtains intent distribution statistics according to the intent of each complete conversation text, performs conversation quality inspection at the granularity of the entire conversation, and effectively improves the accuracy of quality inspection.

[0107] In one embodiment of the present invention, distribution statistics are performed on each complete dialogue text to obtain multi-dimensional distribution statistics results, including:

[0108] Determine the call duration and call end node of each complete conversation text; and count the call duration and call end node respectively to obtain the call duration distribution and call end node distribution.

[0109] It should be noted that the call end node refers to a specific node or event that marks the end of the conversation during the conversation process. For example, the opening node, the closing node (using specific language or phrases to end the conversation, such as "goodbye", "thank you", "good night", etc.), the user rejection node (such as "busy", "not interested", "inconvenient", etc.) and other nodes.

[0110] Specifically, the call duration and call end node of each complete conversation text are determined, and then the number of conversations with different call durations in different time intervals is counted according to the call duration of each complete conversation text. Statistical charts such as histograms and box plots can be used to display the distribution of call durations. In addition, according to the call end node, the number of occurrences of each call end node type is counted. Pie charts, bar charts, and other charts can be used to display the proportions of different end node types.

[0111] In one embodiment of the present invention, based on the multi-dimensional indicator analysis results and the multi-dimensional distribution statistics results, a dialogue quality inspection result is generated, including:

[0112] According to the intent recognition results and word slot recognition results corresponding to each round of dialogue text, determine the number of intents corresponding to each intent type and the number of word slots corresponding to each word slot type; sort the intent recognition results corresponding to each intent type according to the number of intents, and sort the word slot recognition results corresponding to each word slot type according to the number of word slots; analyze the dimensional indicator analysis results, multi-dimensional distribution statistics results, intent sorting results and word slot sorting results to obtain the dialogue quality inspection results.

[0113] Specifically, according to the intention recognition results and word slot recognition results corresponding to each round of dialogue text, the number of intentions corresponding to each intention type and the number of word slots corresponding to each word slot type are counted respectively. Then, according to the number of intentions, the intention recognition results corresponding to each intention type are sorted, so as to select a plurality of intention recognition results with a large number according to the intention sorting results. In addition, according to the number of word slots, the word slot recognition results corresponding to each word slot type are sorted, so as to select a large number of word slot recognition results according to the word slot sorting results. Further, the dimensional indicator analysis results, the multi-dimensional distribution statistics results, and the selected intention recognition results and word slot recognition results are compared with the corresponding numerical thresholds, and the dialogue quality inspection results are generated according to the comparison results. For example, the average call duration is less than 10s, the opening speech is optimized, the same word slot recognition result appears frequently, the word slot instance needs to be optimized, the same intention appears frequently, and the intention speech needs to be optimized.

[0114] The embodiment of the present invention performs quality inspection according to the sentence granularity and the entire conversation granularity to obtain multi-dimensional analysis results, thereby effectively improving the efficiency and accuracy of quality inspection.

[0115] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0116] In one embodiment, a conversation quality inspection device is provided, which corresponds one-to-one to the conversation quality inspection method in the above embodiment. Figure 8 As shown, Figure 8 : is a structural schematic diagram of a conversation quality inspection device in one embodiment of the present invention, the conversation quality inspection device comprises:

[0117] An acquisition module 21 is used to acquire complete conversation texts between a number of users and the AI ​​robot, wherein the complete conversation text includes at least one sub-conversation text;

[0118] An analysis module 22 is used to analyze each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result;

[0119] A statistical module 23 is used to perform distribution statistics on each complete dialogue text to obtain multi-dimensional distribution statistical results;

[0120] The generation module 24 is used to generate dialogue quality inspection results based on the multi-dimensional indicator analysis results and the multi-dimensional distribution statistics results.

[0121] The analysis module 22 is also used for:

[0122] For any round of dialogue text:

[0123] Matching the user consultation text in the sub-dialogue text with the preset intention database to obtain multiple historical consultation texts and text intentions corresponding to the historical consultation texts; and matching the user consultation text in the sub-dialogue text with the preset word slot database to obtain multiple historical consultation texts and word slot information corresponding to the historical consultation texts;

[0124] Input the user consultation text, each historical consultation text and its textual intention into the preset target large model, and obtain the intention recognition result output by the target large model; and input the user consultation text, each historical consultation text and its word slot information into the preset target large model, and obtain the word slot recognition result output by the target large model;

[0125] Based on the intent recognition results corresponding to all sub-dialogue texts and the user intent recognized by the AI ​​robot, the intent recognition accuracy is calculated, and based on the word slot recognition results corresponding to all sub-dialogue texts and the word slot information recognized by the AI ​​robot, the intent recognition accuracy is calculated;

[0126] Among them, user intent is obtained by the AI ​​robot through intent recognition of user consultation text during the conversation process, and word slot information is obtained by the AI ​​robot through word slot recognition of user consultation text during the conversation process.

[0127] The analysis module 22 is also used for:

[0128] For any round of dialogue text:

[0129] Compare the intent recognition result corresponding to the sub-conversation text with the user intent recognized by the AI ​​robot;

[0130] Compare the word slot recognition results corresponding to the sub-dialogue text with the word slots recognized by the AI ​​robot;

[0131] The single-round conversation accuracy and the complete conversation accuracy are determined based on the intent comparison results and word slot comparison results corresponding to each round of conversation text.

[0132] The statistics module 23 is also used for:

[0133] Match any complete dialogue text with a preset intention database to obtain multiple historical complete dialogue texts and text intentions corresponding to the historical complete dialogue texts;

[0134] Input any complete dialogue text, each historical complete dialogue text and its textual intention into the preset target large model, and obtain the intention recognition result output by the target large model;

[0135] The intention recognition results corresponding to all complete conversation texts are counted to obtain the intention distribution statistics.

[0136] The statistics module 23 is also used for:

[0137] Match any complete dialogue text with a preset complete dialogue intention database to obtain multiple historical complete dialogue texts and text intentions corresponding to the historical complete dialogue texts;

[0138] Input any complete conversation text, each historical complete conversation text and its textual intention into the preset target large model, and obtain the complete conversation intention recognition result output by the target large model;

[0139] The complete conversation intention recognition results corresponding to all complete conversation texts are counted to obtain the intention distribution statistics results.

[0140] The statistics module 23 is also used for:

[0141] Determine the call duration and call end point of each complete conversation text;

[0142] The call duration and call end node are counted separately to obtain the call duration distribution and call end node distribution.

[0143] The generation module 24 is also used for:

[0144] According to the intent recognition results and word slot recognition results corresponding to each round of conversation text, determine the number of intents corresponding to each intent type and the number of word slots corresponding to each word slot type;

[0145] Sort the intent recognition results corresponding to each intent type according to the number of intents, and sort the word slot recognition results corresponding to each word slot type according to the number of word slots;

[0146] The dimensional indicator analysis results, multi-dimensional distribution statistics results, intent sorting results, and word slot sorting results are analyzed to obtain the dialogue quality inspection results.

[0147] The dialogue quality inspection device also includes:

[0148] Generate optimization suggestions based on the dialogue quality inspection results;

[0149] Adjust the AI ​​robot based on optimization suggestions.

[0150] For the specific definition of the conversation quality inspection device, please refer to the definition of the conversation quality inspection method above, which will not be repeated here. Each module in the above-mentioned conversation quality inspection device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0151] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown, Fig. 9 : is a schematic diagram of a computer device in an embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected by a device bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating device, a computer-readable instruction and a database. The internal memory provides an environment for the operation of the operating device and the computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the dialogue quality inspection method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instruction is executed by the processor, a dialogue quality inspection method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0152] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, and a network interface connected through a device bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer-readable instructions are executed by the processor, a dialogue quality inspection method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0153] In one embodiment, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned conversation quality inspection method when executing the computer-readable instructions.

[0154] In one embodiment, a readable storage medium is provided, the readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by the processor to implement the above-mentioned dialogue quality inspection method steps. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0155] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0156] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A conversation quality inspection method, characterized in that: include: Obtaining complete conversation texts between a number of users and the AI ​​robot, wherein the complete conversation text includes at least one sub-conversation text; Analyze each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result; Performing distribution statistics on each of the complete dialogue texts to obtain multi-dimensional distribution statistics results; Based on the multi-dimensional indicator analysis results and the multi-dimensional distribution statistical results, a dialogue quality inspection result is generated.

2. The conversation quality inspection method according to claim 1, characterized in that: The multi-dimensional indicator analysis results include intent recognition accuracy and word slot recognition accuracy; The analysis of each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result includes: For any round of dialogue text: Matching the user consultation text in the sub-dialogue text with a preset intention database to obtain a plurality of historical consultation texts and text intentions corresponding to the historical consultation texts; and matching the user consultation text in the sub-dialogue text with a preset word slot database to obtain a plurality of historical consultation texts and word slot information corresponding to the historical consultation texts; Input the user consultation text, each of the historical consultation texts and their textual intentions into a preset target macro model to obtain the intention recognition result output by the target macro model; and input the user consultation text, each of the historical consultation texts and their word slot information into a preset target macro model to obtain the word slot recognition result output by the target macro model; The intent recognition accuracy is calculated based on the intent recognition results corresponding to all sub-dialogue texts and the user intent recognized by the AI ​​robot, and the intent recognition accuracy is calculated based on the word slot recognition results corresponding to all sub-dialogue texts and the word slot information recognized by the AI ​​robot; Among them, the user intention is obtained by the AI ​​robot performing intention recognition on the user consultation text during the conversation process, and the word slot information is obtained by the AI ​​robot performing word slot recognition on the user consultation text during the conversation process.

3. The conversation quality inspection method according to claim 2, characterized in that: The multi-dimensional indicator analysis results include single-round dialogue accuracy and complete conversation accuracy; The analysis of each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result includes: For any round of dialogue text: Compare the intent recognition result corresponding to the sub-dialogue text with the user intent recognized by the AI ​​robot; Comparing the word slot recognition result corresponding to the sub-dialogue text with the word slot recognized by the AI ​​robot; The single-round conversation accuracy and the complete conversation accuracy are determined based on the intent comparison results and word slot comparison results corresponding to each round of conversation text.

4. The conversation quality inspection method according to claim 1, characterized in that: The multi-dimensional distribution statistical results include intention distribution statistical results; The distribution statistics of each complete dialogue text are performed to obtain multi-dimensional distribution statistics results, including: Matching any of the complete dialogue texts with a preset intention database to obtain a plurality of historical complete dialogue texts and text intentions corresponding to the historical complete dialogue texts; Input any of the complete dialogue texts, each of the historical complete dialogue texts and their textual intentions into a preset target macro model, and obtain the intention recognition result output by the target macro model; The intention recognition results corresponding to all complete dialogue texts are counted to obtain the intention distribution statistical results.

5. The conversation quality inspection method according to claim 1, characterized in that: The multi-dimensional distribution statistical results include intention distribution statistical results; The distribution statistics of each complete dialogue text are performed to obtain multi-dimensional distribution statistics results, including: Matching any of the complete dialogue texts with a preset complete dialogue intention database to obtain a plurality of historical complete dialogue texts and text intentions corresponding to the historical complete dialogue texts; Input any of the complete dialogue texts, each of the historical complete dialogue texts and their textual intentions into a preset target large model, and obtain a complete dialogue intention recognition result output by the target large model; The complete conversation intention recognition results corresponding to all complete conversation texts are counted to obtain the intention distribution statistical results.

6. The conversation quality inspection method according to claim 1, characterized in that: The multi-dimensional distribution statistical results include call duration distribution and call end node distribution; The distribution statistics of each complete dialogue text are performed to obtain multi-dimensional distribution statistics results, including: Determining the call duration and call end node of each complete conversation text; The call duration and the call end node are counted respectively to obtain the call duration distribution and the call end node distribution.

7. The conversation quality inspection method according to claim 2, characterized in that: The generating of the dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result includes: According to the intent recognition results and word slot recognition results corresponding to each round of conversation text, determine the number of intents corresponding to each intent type and the number of word slots corresponding to each word slot type; According to the number of intents, the intent recognition results corresponding to each intent type are sorted, and according to the number of word slots, the word slot recognition results corresponding to each word slot type are sorted; The dimensional index analysis results, the multi-dimensional distribution statistics results, the intent sorting results and the word slot sorting results are analyzed to obtain the dialogue quality inspection results.

8. The conversation quality inspection method according to claim 1, characterized in that: After generating the dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result, the method further includes: Generate optimization suggestions based on the dialogue quality inspection results; Based on the optimization suggestions, the AI ​​robot is adjusted.

9. A conversation quality inspection device, characterized in that: include: An acquisition module, used to acquire complete conversation texts between a number of users and the AI ​​robot, wherein the complete conversation text includes at least one sub-conversation text; An analysis module, used to analyze each sub-dialogue text in the complete dialogue text to obtain a multi-dimensional indicator analysis result; A statistical module, used to perform distribution statistics on each of the complete dialogue texts to obtain multi-dimensional distribution statistical results; A generation module is used to generate a dialogue quality inspection result based on the multi-dimensional indicator analysis result and the multi-dimensional distribution statistical result.

10. A readable storage medium having computer readable instructions stored thereon, characterized in that: When the computer-readable instructions are executed by a processor, the conversation quality inspection method according to any one of claims 1 to 8 is implemented.

Images