Customer service verbal skill quality inspection result determination method and device, equipment, medium and product

Abstract

The invention provides a customer service talk skill quality inspection result determination method and device, equipment, a medium and a product, and belongs to the technical field of artificial intelligence. Determining customer service verbal skills, user verbal skills and target standard verbal skills corresponding to the customer service verbal skills according to the conversation interaction voice data, and inputting the customer service verbal skills, the user verbal skills and the target standard verbal skills into a large language model to obtain a quality inspection result; the large language model can determine an integrity index score, a key point index score, a logicality index score and an emotion index score corresponding to a single sentence verbal skill in the customer service verbal skill based on a preset instruction and a target standard verbal skill so as to output a quality inspection result. According to the method, the integrity, the key points, the logicality and the emotional expression of the dialogue are deeply analyzed by utilizing a large language model, the method is not limited to literal matching any more, and higher flexibility and accuracy are realized in customer service dialogue quality inspection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, medium and product for determining the quality inspection results of customer service speech. Background Art

[0002] In the field of modern customer service, technologies such as Natural Language Processing (NLP), machine learning models, deep learning models, and big data analysis have been widely used to improve the quality and efficiency of customer service and marketing. Through the above technologies, the conversation content in the service and marketing process can be automatically analyzed and processed, such as real-time quality inspection in the process of external manual service by customer service agents, marketing personnel, account managers, etc., or offline quality inspection after the service, to determine whether the speech meets the predetermined specification requirements. For example, during the skill training process, personnel who use dialogue interaction as a service method can improve their service skills by practicing dialogue with machines. During the skill training process, it is necessary to evaluate the dialogue skills of personnel, and automatically output the non-compliant parts of the practice to achieve the purpose of learning, evaluation, and assessment.

[0003] The core of existing customer service speech quality inspection methods include keyword matching, template matching and edit distance. The above methods cannot effectively identify semantic similarities and can only match on the surface. The accuracy of quality inspection needs to be improved. Summary of the invention

[0004] The present application provides a method, device, equipment, medium and product for determining the quality inspection results of customer service speech, so as to solve the defects in the prior art that the semantic similarity cannot be effectively identified, and matching can only be performed on the surface, and the accuracy of quality inspection needs to be improved.

[0005] This application provides a method for determining customer service speech quality inspection results, including: Acquire the conversational interaction voice data generated by the preset personnel during the external service or training process, and determine the customer service speech, user speech, and target standard speech corresponding to the customer service speech according to the conversational interaction voice data; Input the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; Among them, the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service script based on preset instructions and the target standard script, and determine the comprehensive score of the customer service script according to the integrity index score, key point index score, logic index score and emotional index score corresponding to the single sentence of customer service script, and use the comprehensive score as the quality inspection result.

[0006] As an embodiment, the preset instructions include a first preset instruction determined based on a thought chain and a few-sample learning method, and the large language model is used to determine the semantic similarity between a single-sentence customer service speech and a corresponding single-sentence standard speech based on the first preset instruction, and determine the integrity index score according to the semantic similarity, and the semantic similarity and the integrity index score are positively correlated.

[0007] As an embodiment, the preset instructions include a second preset instruction determined based on a thought chain and a one-time learning method, and the large language model is used to determine whether a single sentence of customer service script contains all preset key points based on the second preset instruction, and determine the key point indicator score according to the determination result.

[0008] As an embodiment, the large language model is used to identify the key point positions contained in a single sentence of customer service dialogue and generate a key point sequence, and the logic index score is determined according to the longest matching sequence between the key point sequence and a preset key point sequence.

[0009] As an embodiment, the preset instructions include a third preset instruction determined based on a thought chain, and the large language model is used to determine the emotion category of a single sentence of customer service dialogue based on the third preset instruction, and determine the emotion index score according to the emotion category.

[0010] As an embodiment, determining a comprehensive score of the customer service speech according to the completeness index score, the key point index score, the logic index score, and the emotion index score corresponding to a single sentence of customer service speech includes: Substituting the integrity index score, the key point index score, the logic index score, and the emotion index score corresponding to a single sentence of customer service speech into a first comprehensive calculation formula to determine a single sentence comprehensive score; Substituting the comprehensive scores of each single sentence into a second comprehensive calculation formula to determine the comprehensive score of the customer service speech; Among them, the first comprehensive calculation formula is determined based on weighted sum, weighted multiplication or machine learning, and the second comprehensive calculation formula is determined based on weighted average, weighted multiplication or machine learning.

[0011] As an embodiment, determining the customer service speech, the user speech, and the target standard speech corresponding to the customer service speech according to the conversation interaction voice data includes: Performing voice transcription on the dialogue interaction voice data based on at least one of an accent variable, a speech speed variable, and an intonation variable to obtain dialogue interaction text data; Performing text cleaning on the conversation interaction text data to complete data optimization; Sentence the cleaned dialogue interaction text data, and determine the customer service speech and the user speech according to the sentence segmentation result; The user speech is matched with a library of standard speech, and the target standard speech is determined according to the matching result.

[0012] The present application also provides a customer service speech quality inspection result determination device, comprising: A determination module is used to obtain the conversation interaction voice data generated by the preset personnel during the external service or training process, and determine the customer service speech, user speech and target standard speech corresponding to the customer service speech according to the conversation interaction voice data; A quality inspection module, used to input the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; Among them, the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service script based on preset instructions and the target standard script, and determine the comprehensive score of the customer service script according to the integrity index score, key point index score, logic index score and emotional index score corresponding to the single sentence of customer service script, and use the comprehensive score as the quality inspection result.

[0013] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, a method for determining a customer service speech quality inspection result as described in any one of the above-mentioned methods is implemented.

[0014] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for determining the customer service speech quality inspection result as described in any one of the above-mentioned methods is implemented.

[0015] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for determining customer service speech quality inspection results.

[0016] The customer service speech quality inspection result determination method, device, equipment, medium and product provided in this application determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence in the customer service speech based on a large language model, and can deeply analyze the integrity, key points, logic and emotional expression of the conversation, and is no longer limited to literal matching, achieving higher flexibility and accuracy in customer service conversation quality inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 This is one of the flow charts of the method for determining the customer service speech quality inspection results provided by this application.

[0019] Figure 2 This is the second flow chart of the method for determining the customer service speech quality inspection results provided by this application.

[0020] Figure 3 This is a schematic diagram of the quality inspection process for a single sentence of customer service script provided by this application.

[0021] Figure 4 It is a structural diagram of the device for determining the customer service speech quality inspection results provided in this application.

[0022] Figure 5 It is a schematic diagram of the structure of the electronic device provided by this application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be noted that all actions of acquiring signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0025] Existing customer service speech quality inspection methods usually use preset standard speech or preset keywords and key sentences, and use methods such as keyword matching, template matching and edit distance to match the actual customer service speech with the preset content. The implementation process of existing methods usually includes data preprocessing, pattern matching and quality assessment.

[0026] Data preprocessing: First, the customer service conversation data is cleaned and preprocessed, including noise removal, speech recognition, sentence segmentation, word segmentation, and part-of-speech tagging, to prepare standardized text data for subsequent analysis.

[0027] Pattern matching: By pre-defining a set of keywords or using simple regular expressions, the system attempts to identify preset patterns or key information points in the conversation. In addition, the edit distance algorithm may also be used to evaluate the similarity between the customer service speech and the pre-defined speech.

[0028] Quality Assessment: Finally, the quality of the conversation is assessed based on the number of matched keywords or the accuracy of pattern matching. The system may also combine simple sentiment analysis tools to assess the emotional tendency of the conversation, thereby giving an overall score for the quality of the conversation.

[0029] The above methods mainly rely on static, predefined rules and often lack the understanding of complex context and implicit meaning. When dealing with diverse language expressions and complex dialogue structures, these technical solutions usually have low recall and accuracy.

[0030] In addition, methods such as keyword matching, template matching, and edit distance also have some defects.

[0031] Keyword matching methods rely on a predefined list of keywords, which limits their flexibility and adaptability. When the content of the conversation changes or non-standard language is used, this method can easily miss key information, leading to misunderstandings or missed detections. In addition, it cannot effectively handle synonyms and context changes, so it is not effective in understanding the deep meaning and complex context of the conversation.

[0032] Template matching relies on fixed response templates, and its efficiency and accuracy will drop significantly when faced with non-standard queries or slightly different user inputs. Template matching is not suitable for handling the flexibility and creative expression of language, so it may not accurately capture the real needs and emotions of users in practical applications.

[0033] Edit distance evaluates similarity by calculating the literal modification cost between two texts, which may lead to high false positive rates when dealing with texts with grammatical errors or using non-standard language. Edit distance cannot effectively identify semantic similarity and can only match on the surface.

[0034] Although existing technical solutions have certain effects in specific environments, they show obvious limitations in dealing with complex and diverse language expressions and highly flexible dialogue scenarios. This application aims to overcome these shortcomings by introducing deep analysis and understanding based on large language models, and provide a comprehensive and accurate customer service speech quality inspection solution.

[0035] Figure 1 This is one of the flow charts of the method for determining the customer service speech quality inspection results provided by this application, such as Figure 1 As shown, the present application provides a method for determining customer service speech quality inspection results, including steps S100 to S200.

[0036] Step S100, obtain the conversation interaction voice data generated by the preset personnel in the external service or training process, and determine the customer service speech, user speech and the target standard speech corresponding to the customer service speech according to the conversation interaction voice data; the preset personnel refers to customer service, marketing personnel, account managers and other personnel, the customer service speech refers to the words spoken by the preset personnel, the user speech refers to the words spoken by the user, and the target standard speech refers to the standard speech that the preset personnel should feed back to the user after obtaining the user speech in the specific scenario of the external service or training process.

[0037] Step S200, input the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; after inputting the customer service speech, the user speech and the target standard speech into the large language model, the large language model outputs the quality inspection result.

[0038] Among them, the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service script based on preset instructions and the target standard script, and determine the comprehensive score of the customer service script according to the integrity index score, key point index score, logic index score and emotional index score corresponding to the single sentence of customer service script, and use the comprehensive score as the quality inspection result.

[0039] The quality inspection and evaluation indicators of customer service dialogues by large-scale language models include but are not limited to completeness indicators, key point indicators, logic indicators and sentiment indicators.

[0040] Completeness indicator: Evaluate the completeness of the answer by comparing the semantic similarity between the customer service script and the standard script.

[0041] Key Points: Identify and confirm whether the customer service response contains all the necessary key points. The identification process relies on the semantic understanding and pattern recognition capabilities of the Large Language Model (LLM) to automatically identify the key elements in the conversation.

[0042] Logical indicators: Check whether the order of key points in the customer service’s answers is reasonable and logical, to ensure that the conversation process is reasonable and can effectively answer customer questions.

[0043] Emotional indicators: Analyze emotional changes in the conversation and check whether the customer service staff effectively matches the customer's emotions when answering, ensuring that their answers can appropriately respond to the customer's emotions.

[0044] The completeness index score, key point index score, logic index score and sentiment index score are respectively provided with corresponding preset instruction prompts. In the design process of preset instruction prompts, the prompt design can be optimized by using techniques such as chain of thought (CoT), in-context learning or few-shot learning according to the capabilities of specific large language models. The preset instruction prompt will be customized according to the specific needs of each indicator to ensure that the required information can be accurately captured, and guide the large language model to directly output the corresponding score (with a value range of 0-1 or other predetermined ranges), helping the large language model to better understand the task requirements and improve the accuracy and reliability of the output results.

[0045] It is understandable that this application uses a large language model to perform quality inspection of customer service speech. There is no need to train multiple models separately. By using the powerful capabilities of the LLM model, the integrity, key points, logic and emotional expression of the conversation content can be deeply analyzed. This application has significant advantages in understanding and processing complex customer service conversations, surpassing traditional technologies that rely on static rules and simple pattern matching. In addition, this application not only calculates the matching degree of key points, but also comprehensively evaluates multiple levels of the conversation to ensure the comprehensiveness and accuracy of quality inspection. The enhanced analytical capabilities make the quality inspection of customer service speech more comprehensive and accurate, greatly improving the recall rate and accuracy, while significantly reducing the reliance on expert rules and the need for data annotation. This application does not need to be retrained when performing similar tasks in customer service scenarios in different processing industries and fields, reflecting good migration adaptability, and can accurately understand and process synonyms, phrase variants and meanings in context in natural language, more effectively adapt to changing language expressions and complex conversation scenarios, and provide an innovative quality control solution for the customer service field.

[0046] Figure 2 This is the second flow chart of the method for determining the customer service speech quality inspection results provided by this application, such as Figure 2 As shown, based on the above embodiment, as an optional embodiment, the customer service speech, user speech and target standard speech corresponding to the customer service speech are determined according to the conversation interaction voice data, including steps S110-S140.

[0047] Step S110, performing voice transcription on the dialogue interaction voice data based on at least one of an accent variable, a speech speed variable, and an intonation variable to obtain dialogue interaction text data.

[0048] For the conversational interaction voice data generated in telephone or voice customer service channels or during training, speech transcription is required. This step includes signal processing, feature extraction and model inference to capture the language content and details of the speech, including variables such as accent, speaking speed and intonation, to improve the accuracy of speech transcription.

[0049] In one embodiment, high-precision Automatic Speech Recognition (ASR) technology can be used to convert the conversational interaction voice data into text data. ASR technology generally uses deep learning algorithms, including but not limited to convolutional neural networks (CNN) or recurrent neural networks (RNN), and the latest Transformer model to improve the accuracy and robustness of speech transcription.

[0050] Step S120, performing text cleaning on the conversation interaction text data to complete data optimization; to ensure that the ASR-transcribed text has a certain clarity and readability, the present application includes a text cleaning step, which includes but is not limited to removing meaningless filler words, repeated sentences, and mistranslations introduced by background noise, so as to further improve the accuracy and efficiency of large-scale language model analysis.

[0051] Step S130, segment the cleaned conversation interaction text data into sentences, and determine the customer service speech and the user speech according to the sentence segmentation results; segment the entire conversation according to the roles in the transcribed conversation interaction text data, and especially mark the user speech and customer service speech.

[0052] Step S140, matching the user speech with the standard speech library, and determining the target standard speech according to the matching result. The user speech can be matched to the preset standard customer service speech, and the matching method includes but is not limited to traditional NLP technology, vector retrieval technology, neural network classification technology, and large language model technology.

[0053] It is understandable that before using a large language model to conduct quality inspection of customer service scripts, sufficient preprocessing of the entire conversation interaction voice data can ensure the accuracy of the data and the effectiveness of the analysis, provide high-quality text input for the quality inspection of single-sentence customer service scripts, and greatly reduce the complexity and time consumption of traditional data preprocessing. It not only speeds up the entire quality inspection process, but also enables large language models to better adapt to the rapidly changing customer service environment.

[0054] Figure 3 This is a schematic diagram of the quality inspection process for a single sentence of customer service provided by this application, such as Figure 3 As shown, the single-sentence customer service speech scoring is a detailed analysis and scoring of multiple dimensions of a single-sentence customer service speech. The following uses several embodiments to explain the quality inspection of single-sentence customer service speech in detail.

[0055] Based on the above embodiment, as an optional embodiment, the preset instructions include a first preset instruction determined based on a thought chain and a few-sample learning method, and the large language model is used to determine the semantic similarity between a single-sentence customer service speech and a corresponding single-sentence standard speech based on the first preset instruction, and determine the integrity index score according to the semantic similarity, and the semantic similarity is positively correlated with the integrity index score.

[0056] In the present application, the integrity assessment specifically evaluates the coverage of customer speech to standard speech. Usually the standard speech is preset, and the large language model directly outputs the similarity score between a single sentence of customer service speech and a single sentence of standard speech through the designed first preset instruction.

[0057] The first preset instruction combines the instruction design of Chain of Thought (CoT) reasoning and few-shot learning methods to evaluate the semantic similarity of sentences. Through CoT reasoning, large language models first perform detailed sentence analysis and summary before providing the final similarity score. The transparent reasoning process improves the interpretability and accuracy of decisions. At the same time, through few-shot learning, large language models quickly adapt to new tasks by observing a small number of examples, effectively improving performance in data-scarce environments. In addition, the first preset instruction also includes a structured output format that makes the results easy to interpret and subsequently process, which not only optimizes the learning efficiency of the model, but also enhances its performance in complex semantic evaluation tasks. It should be noted that the above design is not a necessary process and the method is not unique.

[0058] The evaluation goal of completeness quality inspection is to evaluate the semantic similarity between customer service answers and preset standard answers.

[0059] Implementation method of integrity quality inspection: The process of implementing integrity assessment involves taking the customer service's actual answer and the preset standard answer as input, and evaluating the semantic similarity between the two through a large language model. The implementation process is automatically guided by the first preset instruction, and the similarity score is directly obtained from the large language model. The integrity score is directly output by the large language model, and the score range is from 0 to 1 or other preset score ranges. The higher the score, the greater the similarity between the answer and the standard answer, that is, the better the integrity of the answer, which can be recorded as .

[0060] The first preset instruction may be "Please evaluate the similarity of two sentences and output them in the form of structured information. Before giving a similarity score, please first summarize and analyze the two sentences. The scoring standard is in the range of X to Y, where X represents complete similarity and Y represents complete dissimilarity."

[0061] Specifically, the first preset instruction is exemplified as follows.

[0062] Please evaluate the similarity of two sentences based on the following call service scenario example and output it in the form of structured information. Before giving a similarity score, please first summarize and analyze the two sentences. The scoring standard is a range of 0 to 1, where 1 means complete similarity and 0 means complete dissimilarity.

[0063] # Example

[0064] ## Input

[0065] Sentence 1: "I would like to know the details of the home broadband package."

[0066] Sentence 2: "Can you tell me what home broadband options are available?"

[0067] ## Output

[0068] ```json

[0069] {

[0070] "analysis": "Both sentences are asking about home broadband packages, and although the wording is different, the subject is the same, so they are very similar.",

[0071] "Similarity": 0.9

[0072] }

[0073] ```

[0074] ## Input

[0075] Sentence 1: "Why is my cell phone network so slow today?"

[0076] Sentence 2: "I want to change my cell phone plan."

[0077] ## Output

[0078] ```json

[0079] {

[0080] "Analysis": "The first sentence expresses dissatisfaction with the current mobile network speed, while the second sentence is about the need to change the package. The two questions have different topics and are less relevant.",

[0081] "Similarity": 0.2

[0082] }

[0083] ```

[0084] # Task

[0085] ## Input

[0086] Sentence 1: "What specific problem do you need help with?"

[0087] Sentence 2: "Can you describe the problem that needs to be solved in detail?"

[0088] ## Output format

[0089] ```json

[0090] {

[0091] "Analysis": "Please summarize and analyze the two sentences",

[0092] "Similarity": "Probability value"

[0093] }

[0094] ```

[0095] The above can be recorded as Prompt 1: Integrity Assessment.

[0096] It is understandable that the scoring process of the completeness index score is entirely achieved through the built-in understanding and generation capabilities of large language models, without the need for an external similarity calculation step.

[0097] Based on the above embodiment, as an optional embodiment, the preset instruction includes a second preset instruction determined based on a thought chain and a one-time learning method, and the large language model is used to determine whether a single sentence of customer service script contains all preset key points based on the second preset instruction, and determine the key point indicator score according to the determination result.

[0098] Key point identification is a key step to ensure that customer service answers contain all necessary information (i.e., preset key points). During the key point evaluation process, preset key points are not required to appear literally in the customer service script, but semantic coverage is sufficient.

[0099] The evaluation goal of key point quality inspection is to evaluate whether the customer service script contains all the necessary key information points and score the performance of each key point.

[0100] How to implement key point quality inspection: First, a set of key points (i.e. preset key points) are predefined according to business needs and conversation scenarios. The preset key points are the basis for evaluating the quality of customer service answers. Then, a second preset instruction is designed to guide the large language model to identify key points in the conversation and score the performance of each key point. This process relies entirely on the model's built-in understanding and generation capabilities, without the need for external additional calculation steps.

[0101] Score calculation for key point quality inspection: When scoring a single key point, the large language model can directly output the score of each key point, ranging from 0 to 1 or other preset score ranges. The higher the score, the more accurate and complete the expression of the key point. For the total score calculation of all key points in a sentence, the weighted average of these key point scores can be taken as the final score of the sentence, which ensures the fairness of the scoring and reflects the importance of each key point in the speech.

[0102] The formula for the weighted average is:

[0103] in, is the key point index score, K is the total number of key points, is the score of the i-th key point in a single customer service speech.

[0104] The second preset instruction requires the large language model to perform a detailed analysis of the conversation content, evaluate the degree of match between each key point and the conversation content based on the provided list of key points, and give a score between the score ranges. Through CoT reasoning, the large language model displays its analysis process, enhancing the transparency and interpretability of the results. At the same time, through one-shot learning, the model can quickly adapt to this specific scoring task after observing an example. The above design is not a necessary process, and the method is not unique.

[0105] The second preset instruction may be "Please analyze the customer service conversation and give a matching score for each key point based on the key point list. The score should be between X and Y, where Y indicates a complete match to the intent of the key point and X indicates no match. The score should be based on the semantic analysis of the conversation content and the coverage of the key point intent."

[0106] Specifically, an example of the second preset instruction is as follows.

[0107] Please analyze the following customer service conversation and give a matching score for each key point based on the key point list. The score should be between 0 and 1, where 1 indicates a complete match to the intent of the key point and 0 indicates no match. The score should be based on the semantic analysis of the conversation content and the coverage of the key point intent.

[0108] # Example

[0109] ## Input

[0110] ### Keypoints list

[0111] - Key point 1: Refund processing

[0112] - Key point 2: Check bank account

[0113] - Key point 3: Contact us

[0114] ### Customer Service Dialogue

[0115] ```

[0116] “Customer Service: Hello, your refund has been processed. You can check your bank account within the next 24 hours to see the corresponding changes. If you have any questions or need more help, please contact us through our official channels.”

[0117] ```

[0118] ## Output

[0119] ```json

[0120] {

[0121] "Summary and Analysis": "The customer service conversation content matches the key points list to a high degree, covering the completion of refund processing, guidance on checking bank accounts, and providing contact information.",

[0122] "Key Point Scoring": [

[0123] {

[0124] "Key Points": "Refund Processing",

[0125] "Rating": 0.9,

[0126] "Original value": "Hello, your refund has been processed."

[0127] },

[0128] {

[0129] "Key Points": "Check Bank Accounts",

[0130] "Rating": 0.9,

[0131] "Original value": "You can check your bank account within the next 24 hours to see the corresponding changes."

[0132] },

[0133] {

[0134] "Key Points": "Contact Us",

[0135] "Rating": 0.9,

[0136] "Original value": "Please contact us through our official channels."

[0137] } ]

[0139] }

[0140] ```

[0141] The above can be recorded as Prompt 2: Key point evaluation.

[0142] Specifically, the second example of the second preset instruction is as follows.

[0143] # Task

[0144] ## Input

[0145] ### Keypoints list

[0146] - Key point 1: Data usage query

[0147] - Key point 2: Network coverage problem solving

[0148] - Key point 3: Upgrade package suggestions

[0149] ### Customer Service Dialogue

[0150] ```

[0151] “Customer Service: Hello, thank you for contacting us. Regarding your data usage, you have used 80% of your data so far. If you often encounter network coverage issues, we recommend that you consider upgrading to a higher package for better service. If you need further help, please call our customer support.”

[0152] ```

[0153] ## Output

[0154] ```json

[0155] {

[0156] "Summary and Analysis": "",

[0157] "Key point scoring": []

[0158] }

[0159] ```

[0160] The above can also be recorded as Prompt 2: Key point evaluation.

[0161] It can be understood that in this embodiment, a large language model is used to automatically detect and confirm whether all necessary key points are covered in the customer service conversation, and then evaluate the completeness and effectiveness of the answer without the need for external calculation steps.

[0162] Based on the above embodiment, as an optional embodiment, the large language model is used to identify the key point positions contained in a single sentence of customer service dialogue and generate a key point sequence, and the logic index score is determined according to the longest matching sequence between the key point sequence and a preset key point sequence.

[0163] The evaluation goal of the logical quality inspection is to evaluate whether the order and logic of the key information in the customer service response are in line with expectations.

[0164] Implementation method of logical property inspection: define the order in which key points should appear in the ideal situation under business logic, and form a template key point sequence (i.e. preset key point sequence). Use the key point recognition results generated by the key point evaluation part, which includes the recognition status of each key point and its specific position in the text. Compare the key point sequence in the customer service answer with the preset key point sequence to find the longest matching sequence between the two.

[0165] The calculation formula for the longest matching sequence is as follows:

[0166] in, Represents the score weight, which is pre-set according to the importance of the key point in the business logic; is the length of the longest matching sequence; is the total length of the preset key point sequence.

[0167] It can be understood that the embodiments of the present application provide a technical solution for key point indicator scoring, which provides flexible and highly customized scoring results by being able to adjust weights according to different business needs.

[0168] Based on the above embodiment, as an optional embodiment, the preset instruction includes a third preset instruction determined based on the thought chain, and the large language model is used to determine the emotion category of a single sentence of customer service script based on the third preset instruction, and determine the emotion index score according to the emotion category.

[0169] Emotional quality inspection is an important part of customer service script quality inspection. It is used to evaluate whether the emotional expression in customer service answers is appropriate and to ensure that the emotions match the customer's situation and expectations.

[0170] The evaluation goal of emotional quality inspection is to ensure that the emotional attitude used by customer service when answering is consistent with the customer's needs and the content of the conversation. Especially when dealing with customer questions or complaints, correct emotional expression can significantly improve customer satisfaction.

[0171] Implementation method of emotional quality inspection: define the emotional categories that should appear in customer service answers. Emotional categories usually include positive, neutral, and negative. Note that the number of categories is not unique, but the technical methods are consistent. Design a third preset instruction to guide the large language model to analyze the emotional expression in the conversation and score it. The score reflects the appropriateness and effectiveness of the emotional expression.

[0172] Calculation of sentiment index scores: The customer service representative's actual response is used as input, and a large language model is used to directly evaluate the appropriateness of sentiment expression based on the prompt instruction. Scores from X to Y are provided for "positive" and "neutral" sentiments, while no score is given for "negative" sentiments. The scoring method is not unique.

[0173] The formula for sentiment index scoring is as follows: .

[0174] The third preset instruction may be "Please perform sentiment analysis and output it in the form of structured information. Before classification, please summarize and analyze the customer service answers, and then classify and score the sentences. The scoring criteria provide scores from X to Y for "positive" and "neutral" emotions, and 0 for "negative" emotions."

[0175] Specifically, the third preset instruction is exemplified as follows.

[0176] Please perform sentiment analysis on the customer service words in the hotline service scenario and output them in the form of structured information. Before classification, please summarize and analyze the customer service answers, and then classify and score the sentences. The scoring criteria provide scores from 0 to 1 for "positive" and "neutral" emotions, and 0 for "negative" emotions.

[0177] # Example

[0178] ## Input

[0179] Customer Service: We understand your urgency very well. I will help you speed up the process now.

[0180] ## Output

[0181] ```json

[0182] {

[0183] "Summary and Analysis": "Customer service expressed understanding of the customer's situation and willingness to take immediate action.",

[0184] "Emotion": "Positive",

[0185] "Rating": 0.85

[0186] }

[0187] ```

[0188] ## Input

[0189] Customer Service: Please wait a little longer. I can’t solve it quickly.

[0190] ## Output

[0191] ```json

[0192] {

[0193] "Summary and Analysis": "Customer service expressed helplessness and lack of willingness to resolve the issue.",

[0194] "Emotion": "Negative",

[0195] "Rating": 0.2

[0196] }

[0197] ```

[0198] # Task

[0199] ## Input

[0200] Customer Service: Please wait, I am checking your account details and will have an answer soon.

[0201] ## Output

[0202] ```json

[0203] {

[0204] "Summary and Analysis": "Please summarize and analyze the customer service sentences",

[0205] "Sentiment": "Select one from 'Positive', 'Neutral' and 'Negative'",

[0206] "Rating": The score corresponding to the emotion

[0207] }

[0208] ```

[0209] The above can be recorded as Prompt 3: Emotional Assessment.

[0210] It can be understood that the embodiments of the present application provide a technical solution for scoring emotional indicators, which can evaluate whether the emotional expression in customer service answers is appropriate and ensure that the emotions match the customer's context and expectations.

[0211] Based on the above embodiment, as an optional embodiment, the comprehensive score of the customer service script is determined according to the completeness index score, the key point index score, the logic index score and the emotion index score corresponding to a single sentence of customer service script, including the following steps.

[0212] The integrity index score, the key point index score, the logic index score and the emotion index score corresponding to a single sentence of customer service speech are substituted into the first comprehensive calculation formula to determine the comprehensive score of the single sentence.

[0213] Substitute the comprehensive scores of each single sentence into the second comprehensive calculation formula to determine the comprehensive score of the customer service speech.

[0214] Among them, the first comprehensive calculation formula is determined based on weighted sum, weighted multiplication or machine learning, and the second comprehensive calculation formula is determined based on weighted average, weighted multiplication or machine learning.

[0215] Optionally, multiple dimensions of a single sentence evaluation can be combined into a single sentence comprehensive score. There are many ways to combine the scores, including weighted sum, weighted multiplication, or machine learning methods (such as neural networks). For example, the weighted sum calculation formula is as follows:

[0216] in, For a comprehensive score of a single sentence, is the completeness index weight, is the key point indicator weight, is the logical indicator weight, is the sentiment indicator weight, , , and They are completeness index score, key point index score, logic index score and sentiment index score.

[0217] Optionally, a comprehensive score is obtained based on the comprehensive scores of each single sentence. There are many methods for comprehensive scoring, which are not unique, such as weighted average method, weighted multiplication, or machine learning method (such as neural network). The calculation formula of the weighted average method is as follows:

[0218] in, For the comprehensive rating, is the comprehensive score of the single sentence corresponding to the i-th single sentence customer service script, and n is the total number of single sentence customer service scripts.

[0219] The large language model uses the comprehensive score as the final output, which can be used as the comprehensive score of a single customer service or as an evaluation result of the customer service in skill training.

[0220] It is understandable that the present application can adjust weights according to different business needs, provide flexible and highly customized scoring results, and through a carefully designed scoring algorithm, the weights of each dimension can be freely adjusted according to different business needs to achieve highly customized quality assessment.

[0221] The following is a description of the customer service speech quality inspection result determination device provided in the present application. The customer service speech quality inspection result determination device described below and the customer service speech quality inspection result determination method described above can be referenced to each other.

[0222] Figure 4 It is a timing diagram of the device for determining the customer service speech quality inspection result provided by this application, such as Figure 4 As shown, the present application also provides a device for determining the results of customer service speech quality inspection, including the following modules.

[0223] The determination module 410 is used to obtain the dialogue interaction voice data generated by the preset personnel during the external service or training process, and determine the customer service speech, user speech and the target standard speech corresponding to the customer service speech according to the dialogue interaction voice data.

[0224] The quality inspection module 420 is used to input the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result.

[0225] Among them, the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service script based on preset instructions and the target standard script, and determine the comprehensive score of the customer service script according to the integrity index score, key point index score, logic index score and emotional index score corresponding to the single sentence of customer service script, and use the comprehensive score as the quality inspection result.

[0226] As an embodiment, the preset instructions include a first preset instruction determined based on a thought chain and a few-sample learning method, and the large language model is used to determine the semantic similarity between a single-sentence customer service speech and a corresponding single-sentence standard speech based on the first preset instruction, and determine the integrity index score according to the semantic similarity, and the semantic similarity and the integrity index score are positively correlated.

[0227] As an embodiment, the preset instructions include a second preset instruction determined based on a thought chain and a one-time learning method, and the large language model is used to determine whether a single sentence of customer service script contains all preset key points based on the second preset instruction, and determine the key point indicator score according to the determination result.

[0228] As an embodiment, the large language model is used to identify the key point positions contained in a single sentence of customer service dialogue and generate a key point sequence, and the logic index score is determined according to the longest matching sequence between the key point sequence and a preset key point sequence.

[0229] As an embodiment, the preset instructions include a third preset instruction determined based on a thought chain, and the large language model is used to determine the emotion category of a single sentence of customer service dialogue based on the third preset instruction, and determine the emotion index score according to the emotion category.

[0230] As an embodiment, the large language model is also used for: Substituting the integrity index score, the key point index score, the logic index score, and the emotion index score corresponding to a single sentence of customer service speech into a first comprehensive calculation formula to determine a single sentence comprehensive score; Substituting the comprehensive scores of each single sentence into a second comprehensive calculation formula to determine the comprehensive score of the customer service speech; Among them, the first comprehensive calculation formula is determined based on weighted sum, weighted multiplication or machine learning, and the second comprehensive calculation formula is determined based on weighted average, weighted multiplication or machine learning.

[0231] As an embodiment, the determining module is further used for: Performing voice transcription on the dialogue interaction voice data based on at least one of an accent variable, a speech speed variable, and an intonation variable to obtain dialogue interaction text data; Performing text cleaning on the conversation interaction text data to complete data optimization; Sentence the cleaned dialogue interaction text data, and determine the customer service speech and the user speech according to the sentence segmentation result; The user speech is matched with a library of standard speech, and the target standard speech is determined according to the matching result.

[0232] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a method for determining the quality inspection result of customer service speech, which includes: obtaining the conversation interaction voice data generated by the preset personnel during the external service or training process, and determining the customer service speech, user speech and the target standard speech corresponding to the customer service speech according to the conversation interaction voice data; inputting the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; wherein the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service speech based on preset instructions and the target standard speech, and determine the comprehensive score of the customer service speech according to the integrity index score, the key point index score, the logic index score and the emotional index score corresponding to the single sentence of customer service speech, and use the comprehensive score as the quality inspection result.

[0233] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0234] On the other hand, the present application also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the customer service speech quality inspection result determination method provided by the above methods, the method including: obtaining conversation interaction voice data generated by preset personnel during external service or training, and determining customer service speech, user speech and target standard speech corresponding to the customer service speech according to the conversation interaction voice data; inputting the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; wherein the large language model is used to determine the integrity index score, key point index score, logic index score and emotion index score corresponding to a single sentence of customer service speech based on preset instructions and the target standard speech, and determine the comprehensive score of the customer service speech according to the integrity index score, the key point index score, the logic index score and the emotion index score corresponding to the single sentence of customer service speech, and use the comprehensive score as the quality inspection result.

[0235] On the other hand, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the customer service speech quality inspection result determination method provided by the above-mentioned methods, the method comprising: obtaining conversation interaction voice data generated by preset personnel during external service or training, and determining customer service speech, user speech and target standard speech corresponding to the customer service speech according to the conversation interaction voice data; inputting the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; wherein the large language model is used to determine the integrity index score, key point index score, logic index score and emotion index score corresponding to a single sentence of customer service speech based on preset instructions and the target standard speech, and determining a comprehensive score of the customer service speech according to the integrity index score, the key point index score, the logic index score and the emotion index score corresponding to the single sentence of customer service speech, and using the comprehensive score as the quality inspection result.

[0236] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0237] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining customer service speech quality inspection results, characterized in that: include: Acquire the conversational interaction voice data generated by the preset personnel during the external service or training process, and determine the customer service speech, user speech, and target standard speech corresponding to the customer service speech according to the conversational interaction voice data; Input the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; Among them, the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service script based on preset instructions and the target standard script, and determine the comprehensive score of the customer service script according to the integrity index score, key point index score, logic index score and emotional index score corresponding to the single sentence of customer service script, and use the comprehensive score as the quality inspection result.

2. The method for determining the customer service speech quality inspection result according to claim 1, characterized in that: The preset instructions include a first preset instruction determined based on a thought chain and a few-sample learning method. The large language model is used to determine the semantic similarity between a single-sentence customer service speech and a corresponding single-sentence standard speech based on the first preset instruction, and the integrity index score is determined according to the semantic similarity. The semantic similarity is positively correlated with the integrity index score.

3. The method for determining the customer service speech quality inspection result according to claim 1 or 2, characterized in that: The preset instructions include a second preset instruction determined based on a thought chain and a one-time learning method. The large language model is used to determine whether a single sentence of customer service script contains all preset key points based on the second preset instruction, and determine the key point indicator score according to the determination result.

4. The method for determining the customer service speech quality inspection result according to claim 3, characterized in that: The large language model is used to identify the key point positions contained in a single sentence of customer service dialogue and generate a key point sequence, and the logic index score is determined according to the longest matching sequence between the key point sequence and a preset key point sequence.

5. The method for determining the customer service speech quality inspection result according to claim 1, 2 or 4, characterized in that: The preset instructions include a third preset instruction determined based on a thought chain, and the large language model is used to determine an emotion category of a single sentence of customer service speech based on the third preset instruction, and determine the emotion index score according to the emotion category.

6. The method for determining the customer service speech quality inspection result according to claim 1, characterized in that: The step of determining a comprehensive score of the customer service script according to the integrity index score, the key point index score, the logic index score, and the emotion index score corresponding to a single sentence of customer service script comprises: Substituting the integrity index score, the key point index score, the logic index score, and the emotion index score corresponding to a single sentence of customer service speech into a first comprehensive calculation formula to determine a single sentence comprehensive score; Substituting the comprehensive scores of each single sentence into a second comprehensive calculation formula to determine the comprehensive score of the customer service speech; Among them, the first comprehensive calculation formula is determined based on weighted sum, weighted multiplication or machine learning, and the second comprehensive calculation formula is determined based on weighted average, weighted multiplication or machine learning.

7. The method for determining the customer service speech quality inspection result according to claim 1, characterized in that: The determining, according to the conversation interaction voice data, a customer service speech, a user speech, and a target standard speech corresponding to the customer service speech includes: Performing voice transcription on the dialogue interaction voice data based on at least one of an accent variable, a speech speed variable, and an intonation variable to obtain dialogue interaction text data; Performing text cleaning on the conversation interaction text data to complete data optimization; Sentence the cleaned dialogue interaction text data, and determine the customer service speech and the user speech according to the sentence segmentation result; The user speech is matched with a library of standard speech, and the target standard speech is determined according to the matching result.

8. A device for determining customer service speech quality inspection results, characterized in that: include: A determination module is used to obtain the conversation interaction voice data generated by the preset personnel during the external service or training process, and determine the customer service speech, user speech and target standard speech corresponding to the customer service speech according to the conversation interaction voice data; A quality inspection module, used to input the customer service speech, the user speech and the target standard speech into a pre-trained large language model to determine the quality inspection result; Among them, the large language model is used to determine the integrity index score, key point index score, logic index score and emotional index score corresponding to a single sentence of customer service script based on preset instructions and the target standard script, and determine the comprehensive score of the customer service script according to the integrity index score, key point index score, logic index score and emotional index score corresponding to the single sentence of customer service script, and use the comprehensive score as the quality inspection result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, it implements the method for determining the customer service speech quality inspection result as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for determining the customer service speech quality inspection result as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the method for determining the customer service speech quality inspection result as described in any one of claims 1 to 7.

Images