Quality inspection method and system for dialogue content of customer service staff based on large language model

Abstract

The invention discloses a customer service staff dialogue content quality inspection method and system based on a large language model, and relates to the technical field of e-commerce, and the method comprises the steps: obtaining a plurality of dialogue contents between customer service staff and a customer, and carrying out the data cleaning; and carrying out quality inspection on all the dialogue contents after data cleaning by utilizing a preset large language model, and generating improvement suggestions. According to the method, the reply content is subjected to quality inspection by using the preset large language model, the quality inspection result can be objectively given, the improvement suggestion can be generated, the quality inspection cost can be effectively reduced, all dialogue content can be comprehensively inspected, the important problem of omission is avoided, the flexibility is relatively high, and the construction process and the maintenance process are simple.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of electronic commerce, and particularly relates to a quality inspection method and system for customer service personnel dialogue content based on a large language model. BACKGROUND

[0002] Currently, in order to ensure the professionalism of customer service personnel, the dialogue content between customer service personnel and customers is often quality inspected, which is realized by a manual sampling inspection and manual evaluation method, a dialogue text similarity analysis quality inspection method, and a knowledge graph quality inspection method. Specifically:

[0003] 1) The manual sampling inspection and manual evaluation method: quality inspectors randomly select some dialogues from a large number of customer service dialogues for auditing, or the quality inspectors score and evaluate the selected dialogues according to preset standards, and then provide feedback to the customer service personnel according to the evaluation results, so that the customer service personnel can improve later. However, the disadvantage is that:

[0004] ① Low efficiency: Since human intervention is required, the quality inspection process is time-consuming and labor-intensive.

[0005] ② Subjectivity: Different quality inspectors may have different judgments on the same dialogue, resulting in inconsistent evaluation results.

[0006] ③ Limited coverage: Due to resource constraints, it is not possible to conduct a comprehensive inspection of all dialogue content, which may miss important issues.

[0007] 2) The dialogue text similarity analysis quality inspection method: usually uses natural language processing (NLP) technology to analyze, compare and audit dialogue content. Its main features and implementation methods include:

[0008] ① Text vectorization: converting customer service dialogue text into vector representation, commonly used methods are word embedding methods (Word Embeddings), such as Word2Vec, etc.

[0009] ② Similarity calculation: calculate the similarity (such as cosine similarity, Euclidean distance, etc.) between dialogue text vectors to determine the degree of similarity between new dialogue content and historical dialogue content.

[0010] ③ Automatic quality inspection: according to the similarity result, automatically match the quality inspection result or label in the historical dialogue, thereby realizing the automatic quality inspection of the new dialogue content.

[0011] The dialogue text similarity analysis quality inspection method has the advantage of being able to quickly process a large number of dialogues and maintaining consistency to some extent. However, the disadvantage is that:

[0012] ①Dependence on historical data: Requires a large amount of labeled data to train the model, and the model's effectiveness is highly dependent on data quality.

[0013] ②Lack of deep understanding: Judgments based on similarity may not accurately capture complex semantics and subtle differences in conversations.

[0014] 3) Knowledge graph-based quality inspection method: Understands and processes customer service conversations by constructing and utilizing a knowledge graph. Its main features and implementation methods include:

[0015] ①Knowledge graph construction: Structurize domain knowledge into a graph form, with nodes representing entities and edges representing relationships between entities.

[0016] ②Semantic understanding: Use the knowledge graph to analyze the semantics of the conversation, understand the entities and relationships in the conversation, and more accurately capture the semantics of the conversation.

[0017] ③Intelligent quality inspection: Through the semantic information provided by the knowledge graph, conduct a deeper analysis and quality inspection of the conversation, which can identify complex semantic errors and inconsistencies.

[0018] The advantage of the knowledge graph-based quality inspection method is that it can provide deeper semantic understanding and analysis. However, the disadvantage is:

[0019] ①Complex construction: The construction and maintenance of the knowledge graph require a large amount of domain knowledge and technical investment.

[0020] ②Limited flexibility: The updating and expansion of the knowledge graph may be difficult, especially in rapidly changing fields. SUMMARY

[0021] The technical problem to be solved by the present application is to overcome the shortcomings of the prior art, and specifically provides a quality inspection method and system for customer service personnel conversation content based on a large language model, as follows:

[0022] 1) In the first aspect, the present application provides a quality inspection method for customer service personnel conversation content based on a large language model, and the specific technical solution is as follows:

[0023] Obtain multiple segments of conversation content between customer service personnel and customers, and perform data cleaning;

[0024] Use a pre-set large language model to inspect all data-cleaning conversation content and generate improvement suggestions.

[0025] The quality inspection method for customer service personnel conversation content based on a large language model provided by the present application has the following beneficial effects:

[0026] The preset large language model is used for quality inspection of reply content, which can objectively give quality inspection results, generate improvement suggestions, effectively reduce quality inspection cost, comprehensively check all dialogue content, avoid missing important problems, and has high flexibility, simple construction process and maintenance process.

[0027] Based on the above scheme, the quality inspection method of the customer service personnel dialogue content based on the large language model can be further improved as follows.

[0028] Further, it further comprises:

[0029] Classify the dialogue content of each piece of data after data cleaning according to business scenarios to determine the business scenario category corresponding to the dialogue content of each piece of data after data cleaning;

[0030] Receive the target customer service personnel and the target business scenario category determined by the user, query the dialogue content after data cleaning of the target customer service personnel in the target business scenario category from all dialogue content after data cleaning, and score the target customer service personnel in multiple dimensions according to the dialogue content after data cleaning of the target customer service personnel in the target business scenario category to obtain the multi-dimensional score result of the target customer service personnel in the target business scenario category.

[0031] Further, it further comprises:

[0032] Obtain and determine the comprehensive performance score of the target customer service personnel in the preset period according to the multiple multi-dimensional score results of the target customer service personnel in the preset period.

[0033] Further, it further comprises:

[0034] Obtain and generate the comparative analysis result between different customer service personnel according to the dialogue content after data cleaning and the multi-dimensional score result of different customer service personnel.

[0035] Further, it further comprises:

[0036] Generate a standardized case library according to the dialogue content after data cleaning of each business scenario category.

[0037] Further, it further comprises:

[0038] Based on the standardized case library, a training mode for the customer service personnel to simulate reply is constructed.

[0039] 2) In a second aspect, the present application also provides a quality inspection system for dialogue content of customer service personnel based on a large language model, and the specific technical scheme is as follows:

[0040] The system comprises a data cleaning module and a quality inspection module;

[0041] The data cleaning module is configured to: acquire conversation content between multiple customer service personnel and customers, and perform data cleaning.

[0042] The quality inspection module is configured to: perform quality inspection on all the cleaned conversation content by using a preset large language model, and generate improvement suggestions.

[0043] Based on the above scheme, the quality inspection system for conversation content of customer service personnel based on a large language model can be further improved as follows.

[0044] Further, the system further comprises a business scenario classification module and a multi-dimensional scoring module.

[0045] The business scenario classification module is configured to: classify the cleaned conversation content of each segment of data according to business scenarios, and determine the business scenario category corresponding to each segment of cleaned conversation content.

[0046] The multi-dimensional scoring module is configured to: receive a target customer service personnel and a target business scenario category determined by a user, query the cleaned conversation content of the target customer service personnel in the target business scenario category from all the cleaned conversation content, and perform multi-dimensional scoring on the target customer service personnel according to the cleaned conversation content of the target customer service personnel in the target business scenario category, to obtain a multi-dimensional scoring result of the target customer service personnel in the target business scenario category.

[0047] Further, the system further comprises a comprehensive performance evaluation module, and the comprehensive performance evaluation module is configured to: acquire and determine a comprehensive performance score of the target customer service personnel in a preset period according to multiple multi-dimensional scoring results of the target customer service personnel in the preset period.

[0048] Further, the system further comprises a comparative analysis module, and the comparative analysis module is configured to:

[0049] Acquire and generate a comparative analysis result between different customer service personnel according to the cleaned conversation content and the multi-dimensional scoring results of the different customer service personnel.

[0050] Further, the system further comprises a standardized case library generation module, and the standardized case library generation module is configured to: generate a standardized case library according to each segment of cleaned conversation content under each business scenario category.

[0051] Further, the system further comprises a training mode construction module, and the training mode construction module is configured to:

[0052] Construct a training mode for the customer service personnel to perform simulated replies based on the standardized case library.

[0053] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor coupled to a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor, so that the electronic device can implement any of the above-mentioned quality inspection methods for customer service personnel conversation content based on a large language model.

[0054] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned quality inspection methods for customer service personnel conversation content based on a large language model.

[0055] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments of the present invention:

[0057] Figure 1 This is a flow chart of a method for quality inspection of customer service personnel conversation content based on a large language model according to an embodiment of the present invention;

[0058] Figure 2 This is a schematic structural diagram of a system for quality inspection of customer service personnel conversation content based on a large language model according to an embodiment of the present invention;

[0059] Figure 3 The figure is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0061] The following describes in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems using specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following embodiments of the present invention are described in conjunction with the accompanying drawings.

[0062] like Figure 1 As shown, a method for quality inspection of customer service personnel conversation content based on a large language model according to an embodiment of the present invention includes the following steps:

[0063] S1. Obtain multiple conversations between customer service staff and customers and perform data cleaning.

[0064] Obtain desensitized conversations between multiple customer service representatives and customers and perform data cleaning to ensure the high quality and consistency of the cleaned conversation data. The specific process of data cleaning is as follows:

[0065] S10. Standardize the format of the conversation content to ensure that all conversation content is converted into a unified text format, thereby obtaining standardized conversation content. Specifically, the format of the conversation content can be standardized using UTF-8 encoding to ensure readability and consistency of the standardized conversation content. The standardization process can be implemented using the Python pandas library.

[0066] S11. Use Python's re library to perform regular expression matching and replacement on the standardized conversation content to achieve noise filtering and obtain noise-filtered conversation content. This step aims to identify and remove noise characters in the standardized conversation content, such as extra spaces and special symbols.

[0067] S12. Perform a data integrity check on the noise-filtered conversation content to ensure that each conversation record contains necessary fields, such as the conversation time, customer service ID, and customer ID. This results in cleaned conversation data. Specifically, a Python script can be written to automatically detect missing values ​​and outliers and perform appropriate processing, such as filling in missing values ​​or deleting outlier records.

[0068] S2. Use the preset large language model to perform quality inspection on all conversation content after data cleaning and generate improvement suggestions.

[0069] Among them, the preset large language model can be GPT-3, GPT-4, Deepseek V3 or Doubao, etc., which can be selected according to actual conditions.

[0070] In another embodiment, S2 includes:

[0071] S21, extract multi-dimensional dialogue features from all data cleaned dialogue content, including sentiment tendency features, customer intention features, service specification keyword features and conversation logic coherence features; wherein, based on the matching method of the dictionary, the dialogue text is compared with the sentiment vocabulary dictionary, the positive and negative sentiment word appearance frequency is counted, for example, for the dialogue text "I am very dissatisfied with this service, your processing effect is too low!", negative sentiment words such as "dissatisfied" and "too low" can be matched, and the appearance frequency is counted to judge the sentiment tendency of the text. Or, a deep learning model (such as Bert) is used to directly predict the sentiment polarity (positive, negative, neutral) and intensity of the text. For the same text "I am very dissatisfied with this service, your processing effect is too low!", the Bert model can directly output the sentiment polarity as "negative" and the sentiment intensity as "high", realizing accurate judgment of the text emotion. First, train the convolutional neural network (CNN) or recurrent neural network (RNN) based on a large number of dialogue data marked with different intentions, and then use the trained model to extract the key semantic information in the text, and classify the dialogue into the corresponding customer intention category. For example, for the dialogue text "What is my express delivery number? When can it be delivered?", the trained model can extract the key semantic information "express delivery number" and "delivery time" in the text, and then classify it into the "logistics query" intention category. Set the service specification keyword dictionary, and use the text matching algorithm to find out whether these keywords exist in the dialogue and the appearance frequency, such as using regular expressions to accurately locate keywords, such as polite expressions or business-related terms like "sorry" "thank you" "as soon as possible". Analyze the semantic continuity between dialogue turns, use text similarity calculation methods (such as cosine similarity) to judge the coherence of adjacent rounds of topics, and consider the use of logical connection words such as cause and effect, transition, etc. between sentences to evaluate the fluency of the entire dialogue logic.

[0072] S22, input the multi-dimensional dialogue features into the preset large language model, generate an evaluation vector for evaluating the dialogue quality through the knowledge reasoning ability of the preset large language model, the evaluation vector includes service specification score, intention matching degree score and emotion coordination degree score;

[0073] The extracted multi-dimensional conversation features are integrated into a unified feature vector and input into a preset large language model. The model scores the service standardization based on predefined rules and knowledge graphs. Specifically, the preset large language model calculates the frequency of service standard keywords, identifies the appropriate use of polite language and business terms, and evaluates the compliance of service processes based on the context. Intent matching is scored: The model determines the degree of consistency between the customer's actual intent and the system's understanding of the intent, comparing the customer's expression with the standard intent template using a semantic similarity algorithm. Emotional coordination is scored: The model analyzes the reasonable shift in emotional tendencies during the conversation to ensure that the customer service response is positive and matches the customer's emotions. Ultimately, an evaluation vector containing three scores is generated, providing a quantitative assessment of conversation quality.

[0074] S23. Based on the comparison results of the evaluation vector and the preset business standards, locate the conversation rounds that do not meet the quality requirements;

[0075] The generated evaluation vectors are compared item by item with the preset business standards. Specifically, if the service standardization score falls below the threshold required by the business standards, it is marked as non-standard; if the intention matching score falls below the set matching threshold, it is judged as unclear or misunderstood; if the emotion coordination score does not meet the preset emotional standard range, it is identified as improper emotional processing. By identifying these non-standard conversation turns, a specific problem list is generated, and conversation turns that do not meet quality requirements are marked and feedback is provided.

[0076] S24. By pre-setting a large language model, combining a library of historical high-quality conversation samples and a business knowledge graph, we generate improvement suggestions that include correction strategies, standardized speech templates, and situational response plans.

[0077] Using a pre-set large language model, the model analyzes conversation turns that don't meet requirements and identifies the underlying issues, such as deficiencies in service standardization, intent alignment, or emotional coordination. The model then retrieves high-quality conversation examples similar to the issues from a library of historical high-quality conversation samples, extracting standard dialogue templates and effective response strategies. Simultaneously, incorporating the business knowledge graph, the model completes the business details in the conversation and integrates them with the standard dialogue to generate corrective strategies for specific issues. Finally, the model generates situational response plans based on the business scenario, ensuring that the improved conversation meets both business standards and customer needs, thereby generating comprehensive improvement recommendations. This process involves feature analysis, sample retrieval, knowledge fusion, and strategy generation, leveraging the model's knowledge reasoning and the reference function of the sample library to continuously optimize conversation quality.

[0078] Optionally, in the above technical solution, the following is further included:

[0079] S3, classifying the cleaned conversation content of each piece of data according to a business scene, and determining the business scene category corresponding to the cleaned conversation content of each piece of data;

[0080] The cleaned conversation content of each piece of data is combined with a classification template based on the BROKE framework to generate a classification prompt instance respectively. The classification template defines five main business scene categories in detail: a commodity-related business scene category, an activity discount business scene category, a purchase operation business scene category, a logistics delivery problem business scene category, and a post-sale problem business scene category. Under each business scene category, it is further subdivided into more specific scenes to more accurately determine the business scene category corresponding to the cleaned conversation content of each piece of data. By explicitly setting the task background, setting the role of the large model, setting the final goal, supplementing the key results, and self-checking settings in the classification template, the classification prompt instance is input into the preset large language model, which can guide the preset large language model to output the expected results, i.e., the business scene category corresponding to the cleaned conversation content of each piece of data.

[0081] The cleaned conversation content is integrated into the classification template to generate a classification prompt instance. Specifically, in the classification prompt instance, the task background is clearly stated (such as this conversation is a communication between the customer service and the customer on the e-commerce platform, aiming to determine the specific business scene category of the customer's inquiry), the role of the large language model is set (such as an intelligent classification assistant), the final goal is clarified (accurately determining the business scene category to which the conversation content belongs), the key results are supplemented (such as listing each business scene category and the subdivided scenes under each business scene category), and the self-checking settings are set (requiring the model to evaluate the rationality of its classification results, etc.). The generated classification prompt instance is input into the preset large language model, which will analyze and process the content in the classification prompt instance according to its own training rules and knowledge, and output the corresponding results according to the set role, task goal, etc., i.e., determine the business scene category corresponding to the cleaned conversation content of each piece of data.

[0082] S4, receiving a target customer service personnel and a target business scene category determined by a user, querying the cleaned conversation content of the target customer service personnel in the target business scene category from all cleaned conversation contents, and scoring the target customer service personnel in multiple dimensions according to the cleaned conversation content of the target customer service personnel in the target business scene category, to obtain a multi-dimensional score result of the target customer service personnel in the target business scene category.

[0083] Among them, the target customer service personnel and the target business scene category can be specified by the user.

[0084] The data cleaning dialogue content of the target customer service personnel in the target business scenario category obtained by the query is combined with the preset scoring template to generate a scoring prompt instance. The scoring template provides detailed scoring rules covering information in multiple dimensions. The scoring system is divided into three dimensions: soft skills, technical skills, and professional quality. Each dimension can be further subdivided into specific categories to ensure comprehensive evaluation. The scoring prompt instance can be input into a preset large language model to obtain multi-dimensional scoring results, including soft skill scores, technical skill scores, and professional quality scores. Through the multi-dimensional scoring method, the consistency and fairness of the scoring standard are ensured. In addition, by regularly calibrating the scoring standards used in the scoring template, the actual business needs are continuously met.

[0085] When generating the scoring prompt instance, the task background needs to be clearly stated in the scoring template, and the purpose of the scoring is to comprehensively evaluate the performance of the customer service personnel in the dialogue. The generated scoring prompt instance is input into the preset large language model. The preset large language model analyzes and evaluates the data cleaning dialogue content of the target customer service personnel in the target business scenario category according to the rules and requirements in the scoring prompt instance. Through the calculation and reasoning of the preset large language model, the multi-dimensional scoring results of the target customer service personnel in the target business scenario category are finally output, including the scores of soft skills, technical skills, and professional quality, as well as the corresponding scoring reasons.

[0086] To ensure the continuous effectiveness and adaptability of the scoring standard, the scoring standards used in the scoring template need to be calibrated regularly. According to the development and changes of the business, cases and feedbacks in actual work can be collected to optimize and adjust the scoring rules. For example, if it is found that the scoring standard of a certain dimension is too loose or strict, or some specific categories are not accurate or comprehensive enough, timely modification and improvement can be made to ensure that the scoring standard always meets the actual business needs.

[0087] Optionally, the specific implementation process of S3 and the specific implementation process of S4 can be integrated through the Dify platform to form a complete classification and scoring workflow. The workflow is implemented in the background execution program to classify and score the customer service dialogue data, and the processing results are stored in the database, laying a solid foundation for the subsequent customer service quality inspection system.

[0088] Optionally, in the above technical solution, it further includes:

[0089] S5, obtaining and determining the comprehensive performance score of the customer service personnel in the preset period according to the multi-dimensional scoring results of the target customer service personnel in the preset period, specifically:

[0090] S50, calculate the average of the scores of the soft skills, the average of the scores of the technical skills, and the average of the scores of the professional qualities;

[0091] S51, determine the weight distribution: according to business needs, the soft skills, technical skills, and professional qualities can be assigned different weights, for example, the weight of soft skills is 40%, the weight of technical skills is 30%, and the weight of professional qualities is 30%.

[0092] S52, weight and sum the average of the scores of the soft skills, the average of the scores of the technical skills, and the average of the scores of the professional qualities with the corresponding weights to obtain the comprehensive performance score, for example: comprehensive performance score = (average of the scores of the soft skills x 0.4) + (average of the scores of the technical skills x 0.3) + (average of the scores of the professional qualities x 0.3).

[0093] The comprehensive performance score can be set to 100 points, and 90-100 points are set as excellent, 80-89 points are set as good, and so on, to divide the performance level of the customer service personnel and realize comprehensive performance evaluation.

[0094] Through the comprehensive performance score of the target customer service personnel in the preset period, the degree of professionalism of the target customer service personnel in the preset period can be reflected, and for customer service personnel with a lower comprehensive performance score, one-on-one guidance can be used to improve professional skills.

[0095] By statistically analyzing all the conversation contents of the customer service personnel in the preset period and obtaining the multi-dimensional score results of each conversation content, the comprehensive performance score of the customer service personnel is calculated and generated. The comprehensive score can reflect the overall service performance of the customer service personnel in the evaluation period (i.e., the comprehensive score of the whole scene). In addition, the user can also select a specific service scene to calculate and display the comprehensive performance score of the customer service personnel in this scene. This mechanism provides fine-grained performance insight, enabling users to comprehensively evaluate the performance of customer service personnel in different dimensions and specific business scenarios.

[0096] The preset period can be one week or one month, which can be set according to actual conditions.

[0097] Optionally, in the above technical solution, it further comprises:

[0098] The conversation contents and multi-dimensional score results of different customer service personnel after data cleaning are obtained and analyzed to generate a comparative analysis result between different customer service personnel, and through the comparative analysis result, the professional technical differences between different customer service personnel can be intuitively compared.

[0099] In addition to screening individual customer service personnel, the application introduces a new screening mode. The new screening mode keeps the other screening options unchanged, adds a reference customer service option, and allows users to locate the conversation record list of the two customer service personnel under the same scenario (the same business scenario category) and the same condition (for example: the same customer group) by selecting two customer service personnel. Users can select the conversation content of each of the two customer service personnel to obtain the conversation content and the corresponding multi-dimensional score result. If the user wants to obtain more detailed comparative analysis records, the user can select comparative analysis. Specifically, the conversation records and scores are combined with a pre-set comparative analysis template to form a comparative analysis prompt instance. The comparative analysis prompt instance is input into a pre-set large language model, and the pre-set large language model performs comparative analysis on the two conversation contents. The pre-set large language model can identify the differences between the specified customer service personnel and the reference customer service personnel (the user can specify the customer service personnel and the reference customer service personnel from the two customer service personnel), and the differences are not limited to the score, but also include the response speed of the customer service personnel, the problem solving rate and the customer satisfaction. Finally, a detailed comparative analysis result is obtained, and specific improvement suggestions are made for the specified customer service personnel.

[0100] In another embodiment, the specific implementation process of generating the comparative analysis result between different customer service personnel includes:

[0101] Obtain the dialogue content of different customer service personnel and the corresponding multi-dimensional score results from the database or related storage location. The dialogue content includes the communication records between the customer service and the customer, and the multi-dimensional score results are the scores of the soft skills, technical skills, and professional quality of the customer service personnel. Integrate and standardize the obtained data to ensure the consistency and integrity of the data. For example, uniformly convert different formats of data, handle missing values or abnormal values, etc. to provide a reliable data basis for subsequent comparative analysis. According to business needs and data characteristics, determine the analysis method suitable for comparing customer service personnel, such as descriptive statistical analysis, correlation analysis, etc. At the same time, specific comparison indicators are determined, such as communication ability, service attitude, and emotion management in the soft skill dimension, problem solving ability, operation proficiency, etc. in the technical skill dimension, responsibility, and integrity in the professional quality dimension. These indicators will be used to quantify and compare the performance of different customer service personnel. Use the selected analysis method to compare the scores of different customer service personnel in each dimension. For example, calculate the average score, standard deviation, etc. of different customer service personnel in each indicator to analyze their strengths and weaknesses in different dimensions. Visualization tools such as bar charts and line charts can be used to visually display the score comparison of each customer service personnel in each dimension, helping to better understand the differences between them. The data and charts obtained from the comparative analysis are integrated into a complete comparative analysis report, which includes the score comparison of different customer service personnel in each dimension, the evaluation of their overall performance, and possible gaps and problems, etc. to provide a basis for subsequent customer service personnel training, performance evaluation, etc.

[0102] Optionally, in the above technical solutions, further comprising:

[0103] According to the cleaned dialogue content of each piece of data under each business scenario category, a standardized case library is generated. Specifically:

[0104] The cleaned dialogue content of each piece of data under different business scenario categories is analyzed to identify and filter out representative customer problem instances. Then, in combination with each customer problem instance, the cleaned dialogue content of each piece of data under different business scenario categories is reviewed in detail, and based on the review results, corresponding standardized customer reply templates are developed. The standardized customer reply templates and all cleaned dialogue data are entered into the database according to different business scenario categories to obtain a standardized case library. The standardized case library not only stores and manages data, but also provides a basis and support for subsequent training and skill improvement of customer service personnel.

[0105] Among them, the representative customer problem instances can be filtered out by manual means and the corresponding standardized customer reply templates can be developed, and the following methods can also be used:

[0106] ① Utilize the semantic understanding ability of the preset large language model to analyze the cleaned conversation content of each piece of data, and identify the customer's problems and needs. The preset large language model can understand language expression and semantic information in different business scenarios through training. According to the business scenario category, the identified problems are classified. The large language model can classify the problems into the corresponding business scenarios according to the preset category labels. Extract the features of representative consumer problem instances, such as the frequency, similarity, and impact of the problem. Through statistical analysis and machine learning algorithms, find out the problem instances that appear frequently, have similar characteristics, or have a greater impact on the business in different business scenario categories. According to the preset filtering criteria and weights, sort and filter the extracted problem instances to determine the customer problem instances.

[0107] ② Associate the identified customer problem instances with the corresponding business scenario categories. This step ensures that the model can accurately grasp the core problems and scenario background of the conversation. Use the preset large language model to further analyze the cleaned conversation content of each piece of data, extract key information, including the core points of the problem, the special needs of the customer, and emotional tendencies. These information will be important basis for developing customer service reply templates. Through the review of a large number of similar business scenario conversations, the preset large language model identifies common problem patterns and coping strategies. The preset large language model summarizes successful reply patterns and key phrases to provide reference for developing standardized templates. Based on the extracted key information and identified patterns, the preset large language model generates standardized customer service reply templates. At the same time, combined with business rules and historical conversation data, the preset large language model is optimized to ensure its accuracy, completeness, and compliance with business specifications.

[0108] Through the conversation record filtering mechanism, users can preliminarily filter the massive stored conversation content based on multiple conditions, including but not limited to store ownership, specific date range, specified customer service personnel ID, and pre-set scenario classification.

[0109] To further improve the relevance and quality of the filtering results, the conversation record (conversation content) filtering mechanism introduces a conversation length filtering function, which allows setting minimum / maximum length thresholds for conversations, automatically excluding conversation records with abnormal length or insufficient information. After completing the preliminary filtering, further apply keyword matching techniques, such as regular expressions (which can be implemented using Python's re library), to perform secondary accurate matching on the filtered conversation records, thereby accurately retrieving and extracting target conversation records containing specific keywords. This multi-stage, multi-condition joint filtering method ensures the efficiency and accuracy of conversation record positioning.

[0110] Wherein, through the dialogue record screening mechanism, the user filters the stored massive dialogue content and / or standardized case library based on multiple conditions, and the specific implementation process is as follows:

[0111] ① Metadata annotation is performed on each dialogue record to generate a dialogue feature matrix containing store code, timestamp, customer service identity label and scene classification label, specifically:

[0112] For each dialogue record, key information is first extracted from its storage format (such as a text file, a database record, etc.) through data parsing technology. This includes metadata such as store code, timestamp, customer service identity label and scene classification label. The extracted metadata is structured, for example, the store code, timestamp, customer service identity label and scene classification label are taken as feature dimensions. The specific value of each dialogue record in these four dimensions constitutes a feature vector. The feature vectors corresponding to all dialogue records are arranged in order to form a two-dimensional table, i.e. a dialogue feature matrix. Each row represents a dialogue record, and each column corresponds to a metadata feature. In this way, the dialogue feature matrix containing store code, timestamp, customer service identity label and scene classification label is generated.

[0113] ② The dialogue record screening mechanism receives the multiple filtering conditions input by the user, decomposes the conditions into store attribution constraints, time window parameters, ID white list of customer service personnel and scene classification tree structure, and binds the conditions with the dialogue feature matrix, specifically:

[0114] The dialogue record screening mechanism receives the multiple filtering conditions input by the user. Then, these conditions are decomposed into store attribution constraints, time window parameters, ID white list of customer service personnel and scene classification tree structure, and then the conditions are bound with the dialogue feature matrix. Specifically, the store attribution constraints are matched with the store code feature in the dialogue feature matrix to filter out dialogue records that meet the specific store attribution; the time window parameters are compared with the timestamp feature in the dialogue feature matrix to determine dialogue records within the specified time range; the ID white list of customer service personnel is used to filter the customer service identity label feature to retain customer service dialogue records within the white list; and the scene classification tree structure is used to hierarchically filter the scene classification label feature to accurately locate dialogue records that meet the specific scene classification path. Through this condition parameter binding method, dialogue records that meet the user's multiple filtering conditions are filtered from the dialogue feature matrix.

[0115] ③ Distributed preprocessing is performed on the massive dialogue content, an inverted index and a Bloom filter are used to construct a store-time-customer three-dimensional joint index structure, and a candidate dialogue subset that meets the store attribution constraints and time window parameters is generated, specifically:

[0116] Firstly, distributed preprocessing is performed to distribute massive dialogue content to multiple computing nodes for parallel processing. On each node, an inverted index is constructed with store code, timestamp, and customer service personnel's ID as index keys to record the location of each key corresponding to the dialogue record. Meanwhile, a Bloom filter is used for fast matching of store attribution constraints and customer service personnel's ID whitelist to filter out possible qualified dialogue records. Then, the inverted index is combined with the Bloom filter to construct a store-time-customer three-dimensional joint index structure. Specifically, the dialogue record range that meets the store attribution constraints and customer service personnel's ID conditions is first quickly screened out by the Bloom filter, and then the inverted index is used to accurately find the dialogue records that meet the time window parameters within these ranges. Finally, the dialogue content that meets all conditions is extracted from these records to generate a candidate dialogue subset.

[0117] ④A scene classification semantic index is established on the candidate dialogue subset, and a hybrid retrieval algorithm is used to perform similarity matching between the scene classification tree structure and the semantic vector of the dialogue text, and the classification threshold is dynamically adjusted to optimize the retrieval recall rate, specifically:

[0118] Firstly, on the candidate dialogue subset, each dialogue text is encoded into a semantic vector using a pre-set large language model to construct a scene classification semantic index for efficient storage and retrieval of semantic information. At the same time, the semantic labels of various scenes are defined according to the scene classification tree structure and converted into corresponding semantic vectors as matching references. Then, a hybrid retrieval algorithm is used to combine cosine similarity calculation based on semantic vectors and exact matching based on keywords. The semantic vector of the dialogue text is matched with the semantic labels of each node in the scene classification tree structure to calculate the similarity scores of each dialogue with various scenes, and the classification threshold is dynamically adjusted, such as through machine learning algorithms (such as Bayesian optimization or genetic algorithms) to optimize the threshold according to historical matching results and business requirements to balance the accuracy and recall rate of retrieval, ensuring that the dialogue can be accurately matched to the appropriate scene classification.

[0119] ⑤Based on reinforcement learning, a multi-condition dynamic weight matching model is constructed to automatically optimize the matching weight of customer service personnel's ID and the priority of scene classification according to user historical screening behavior data, and output the final dialogue record set that meets the specified customer service personnel's ID whitelist and scene classification conditions, specifically:

[0120] First, collect user historical screening behavior data, including user past selection preferences for customer service personnel IDs and scenario classification usage frequency, etc. Based on these data, a multi-condition dynamic weight matching model is constructed, using reinforcement learning algorithms such as Q-learning or DQN (Deep Q Network) in deep reinforcement learning. During model training, the customer service personnel ID matching weight and scenario classification priority are used as adjustable parameters. The input of the model is the current screening conditions (such as customer service personnel ID whitelist and scenario classification conditions), and the output is the matched conversation record set. By defining a reward function, for example, when the model output conversation record set is highly matched with user actual needs (such as high user satisfaction with the result, high usage frequency, etc.), a positive reward is given, and vice versa, a negative reward is given. The model continuously adjusts the weight and priority parameters according to the reward signal. In practical application, when there is a new screening request, the model uses the optimized weight and priority to reorder and screen the candidate conversation subset, and finally outputs the conversation record set that meets both the customer service personnel ID whitelist and the scenario classification conditions. At the same time, the model continuously receives user feedback on new results for further iterative optimization to adapt to changes in user needs.

[0121] (6) The screening results are reorganized according to the time dimension of the conversation flow to generate a visual conversation graph with multi-level classification labels, and the user is fed back through an interactive interface for secondary condition fine screening, to achieve preliminary filtering of the massive stored conversation content, specifically:

[0122] First, the conversation records after preliminary screening are reorganized according to the time dimension of the conversation flow. Based on the timestamp, the conversation records are sorted, and the multiple records in the same conversation thread are reorganized in chronological order to form a complete conversation flow. According to the information of store attribution, customer service personnel ID and scenario classification, multi-level classification labels are added to the reorganized conversation flow. For example, the first level classification can be the store code, the second level classification can be the customer service personnel ID, and the third level classification can be the scenario classification label, etc. The conversation flow with classification labels is converted into an intuitive conversation graph using data visualization technology (such as D3.js or ECharts). In the graph, nodes represent conversation records, and lines between nodes represent the order of conversation flow. Different colors or shapes of nodes distinguish different classification levels. The visual conversation graph is displayed to the user through an interactive interface (such as a web front-end interface). The interface provides the function of secondary condition fine screening, such as text boxes, drop-down menus, etc. The user can input new screening conditions, and the front-end will pass the conditions to the back-end, which will call the screening model to recalculate, update the conversation flow and graph and feed back to the user, to achieve preliminary filtering and fine viewing of massive conversation content.

[0123] Optionally, in the above technical solution, further comprising: when the target dialogue record is screened out, the multi-dimensional score result of the dialogue content corresponding to the target dialogue record can also be displayed to the user. If the user wants to conduct a more in-depth analysis, the dialogue analysis can be selected, specifically, the dialogue content corresponding to the target dialogue record and the multi-dimensional score result are combined with the pre-set analysis template to form an analysis prompt instance. The analysis prompt instance is input into the pre-set large language model, so that the pre-set large language model analyzes to obtain the corresponding score reason, and points out which reply content can be further optimized and adjusted, while providing a reference modification example.

[0124] Optionally, in the above technical solution, further comprising:

[0125] Based on the standardized case library, a training mode for the customer service personnel to simulate reply is constructed.

[0126] In the training mode, by selecting an exercise scene, relevant cases are extracted from the pre-prepared standard reply case library for the customer service personnel to simulate reply. The simulated reply content of the customer service personnel, the standardized customer service reply template and all the dialogue content of the simulated reply of the customer service personnel are combined, a pre-designed practical analysis template is applied, and a practical analysis prompt instance is generated. Subsequently, the pre-set large language model is used to process the practical analysis prompt instance, and the effect difference between the answer (simulated reply content) of the customer service personnel and the standard reply (standardized customer service reply template) is evaluated. Based on the evaluation result, the system will provide specific improvement suggestions and point out the key points that the customer service personnel should pay more attention to in the answering process, specifically:

[0127] ①When the customer service personnel selects an exercise scene in the training mode, relevant standard reply cases are extracted from the pre-prepared standard reply case library through label matching and semantic search.

[0128] ②When the customer service personnel simulates reply to the extracted cases, the simulated reply content is integrated with the standard reply template and combined with the entire dialogue content to form a complete dialogue data set.

[0129] ③The pre-designed practical analysis template is applied to structurally analyze the integrated dialogue content and extract key elements such as question identification accuracy, information integrity, and emotional coordination.

[0130] ④The analysis result is converted into a practical analysis prompt instance, which clearly contains the requirements for comparing the simulated reply and the standard reply.

[0131] ⑤The pre-set large language model is used to process the practical analysis prompt instance to compare the simulated reply and the standard reply, evaluate the effect difference, such as semantic similarity, information coverage, and standardization.

[0132] ⑥According to the evaluation result, the system generates specific improvement suggestions, pointing out the key problems of the customer service personnel in the problem understanding depth, information organization logic, emotional expression, etc.

[0133] By simulating real customer service scenarios and combining the analysis capabilities of the preset large language model, targeted training and improvement suggestions are provided for customer service personnel, effectively improving customer service skills and reply quality.

[0134] Optionally, in the above technical solution, it further includes:

[0135] S101, obtaining the current conversation content between the customer service personnel and the customer, the current conversation content including the latest question raised by the customer;

[0136] S102, using a preset large language model to analyze the current conversation content between the customer service personnel and the customer, determining the answering technique for the latest question raised by the customer, and providing it to the customer service personnel, so that the customer service personnel communicates with the customer using the answering technique.

[0137] In this embodiment, the preset large language model can generate the answering technique for the customer's question in real time, so that the customer service personnel communicates with the customer using the answering technique, which not only ensures the professionalism of the customer service personnel, but also effectively reduces the time cost and cost of training the customer personnel.

[0138] In another embodiment, the specific implementation process of determining the answering technique for the latest question raised by the customer includes:

[0139] S1021, real-time acquisition of multi-round conversation content between the customer service personnel and the customer, structural processing of the current conversation content, generation of a conversation sequence containing time markers, specifically:

[0140] The multi-round conversation content between the customer service and the customer is acquired in real time through the customer service system or the communication interface. Then, the time stamp acquisition technology is used to add accurate time markers on each conversation information, recording the specific time of the conversation. Then, the conversation content with time markers is structured and processed, and is organized into a conversation sequence in chronological order. Data structures such as lists or arrays can be used to store each conversation and its corresponding time stamp. Finally, the generated conversation sequence containing time markers is output or stored for subsequent analysis and use, for example, using the conversation sequence for conversation quality evaluation and customer intent analysis, etc.

[0141] S1022, inputting the conversation sequence into the preset large language model for multi-dimensional semantic analysis, identifying the customer request entity through the entity extraction layer, determining the customer demand type through the intent classification layer, and evaluating the customer emotional state through the sentiment analysis layer, specifically:

[0142] First, the dialogue sequence with time sequence label is input into the preset large language model, an entity extraction layer is set in the preset large language model, and the related entity information of the customer request, such as product name and service type, is accurately identified by using the named entity recognition technology. Then, through the intent classification layer, the content expressed by the customer is classified according to the preset intent category label, the demand type such as consultation and complaint is determined, and the emotion state of the customer is evaluated according to the words and tone in the dialogue in the emotion analysis layer, and it is judged whether it is positive, negative or neutral, and finally the results of the customer request entity, demand type and emotion state are output.

[0143] S1023, the latest problem expression of the customer is extracted based on the time sequence label of the dialogue sequence, and a three-dimensional feature vector is generated combining the customer request entity, the customer demand type and the customer emotion state, specifically:

[0144] From the structured dialogue sequence, the latest problem expression of the customer is extracted according to the time stamp. Then, the customer request entity, demand type and emotion state information obtained by the large language model are combined. Then, for the customer request entity, a pre-trained entity embedding model (such as Word2Vec or BERT) is used to generate an entity feature vector to reflect the semantic meaning of the entity; for the customer demand type, it is converted into One-Hot Encoding form to form a demand type feature vector; and the customer emotion state is converted into a numerical representation (such as positive emotion for 1, negative emotion for -1, and neutral emotion for 0) according to the emotion analysis result. Finally, the feature vectors of the three dimensions are spliced to form a complete three-dimensional feature vector to comprehensively represent the latest problem of the customer and its related features.

[0145] S1024, the similar case data in the historical dialogue knowledge base is called, the three-dimensional feature vector is matched with the historical case features, and the generation parameters of the preset large language model are dynamically adjusted based on the matching result, specifically:

[0146] Firstly, a large number of similar case data is retrieved from the historical dialogue knowledge base, and a feature vector containing the request entity, demand type and emotional state is extracted for each case to construct a historical case feature library. Then, the vector dot product or cosine similarity algorithm is used to calculate the similarity of the three-dimensional feature vector of the current customer and the historical case feature vector, and the top N cases with a similarity higher than a threshold (such as 0.7) are selected. Next, the model generation parameters (such as temperature, maximum length, etc.) corresponding to these cases are analyzed, and the recommended parameter values are calculated based on the weighted average of the similarity. Finally, the generation parameters of the large language model are dynamically adjusted to the recommended values to make the model output more suitable for the current customer's problem and the solution based on historical experience. This process needs to be noted that the knowledge base needs to be updated regularly to ensure the timeliness of the cases, and the similarity algorithm needs to be optimized to improve the matching accuracy.

[0147] S1025, by performing compliance verification and emotional adaptation degree evaluation on the generated candidate answer scripts, output the answer scripts that meet the customer service specifications and adapt to the current emotional state, specifically:

[0148] From the set of candidate answer scripts generated by the preset large language model, compliance verification is performed one by one. Through regular expression matching, keyword filtering and rule matching, etc., it is checked whether the candidate answer contains sensitive words, whether it violates the customer service specifications, etc., and the candidate answers that do not meet the specifications are excluded. Then, for the candidate answers that pass the compliance verification, emotional adaptation degree evaluation is performed. By using the emotional corpus, the emotional tendency (positive, negative, neutral) of the candidate answer is analyzed, and the emotional adaptation degree score is calculated by combining the current emotional state of the customer, and the candidate answers with high emotional adaptation degree are selected as the answer scripts.

[0149] Optionally, in the above technical solution, it further comprises:

[0150] S103, when the customer service personnel replies with the answer script for the latest question raised by the customer, the reply content is analyzed again by using the preset large language model to obtain optimization adjustment suggestions and provide them to the customer service personnel.

[0151] The present application provides a comprehensive, efficient and intelligent customer service quality inspection and capability improvement solution, which can significantly improve the customer service quality and the overall efficiency of the team, and the beneficial effects are as follows:

[0152] 1) An automatic, multi-dimensional and scenario-based customer service dialogue evaluation method is provided: by integrating data cleaning, automatic classification based on scenario templates and a multi-dimensional scoring system covering soft skills, technical skills and professional quality, objective and detailed evaluation of customer service dialogues is realized, and performance in specific scenarios can be viewed, improving the representativeness and usability of the evaluation results.

[0153] 2) Implement a standardized scenario-based cross-customer comparison analysis function: Provide a mechanism that allows for horizontal comparison of different customer service (including excellent customer service) dialogue performance under the same scenario classification. Use AI technology to analyze their behavior differences in key indicators (such as response speed, problem solving path, communication skills, customer satisfaction influence, etc.), generate structured comparison reports and targeted improvement suggestions, and provide data support for precise benchmarking learning and best practice promotion.

[0154] 3) Establish a complete closed-loop process integrating evaluation, analysis, suggestions and practice training: Combine automated evaluation, in-depth analysis and feedback, scenario-based comparative learning, and simulation training based on a structured case library. Through AI evaluation and feedback on simulation training results, form a continuous optimization cycle of "finding problems - analyzing reasons - obtaining suggestions - simulation practice - verifying effects", significantly accelerating the conversion process from theoretical cognition to skill proficiency of customer service personnel.

[0155] 4) Significantly improve customer service quality inspection efficiency and empower customer service personnel to improve their skills independently: Through process automation and AI intelligent processing, significantly reduce the workload of manual quality inspection and training, and reduce operating costs. At the same time, provide customer service personnel with powerful self-learning and training tools, enabling them to actively and efficiently enhance their skills based on individualized evaluation and analysis results, thereby improving the overall team's work efficiency and professional level.

[0156] 5) Automatically classify and score customer service conversations in multiple dimensions. Combined with the deep analysis of large language models (LLM), customer service personnel can clearly understand their performance scores in different scenarios, the reasons for the scores, and obtain targeted and actionable optimization suggestions and reference modification examples. Further, through the introduction of comparative analysis, customer service personnel can compare their performance with excellent cases or reference objects to identify their differences. The standardized case library provides best practice examples for learning, while the simulation training module transforms theoretical knowledge into practical operational skills. Through LLM's immediate feedback and improvement suggestions on simulation performance, a closed loop of "evaluation - analysis - learning - practice - feedback" is formed, enabling precise, personalized and continuous improvement of customer service personnel's skills.

[0157] 6) By automating the cleaning, classification and preliminary scoring of customer service dialogues, the tedious and repetitive workload in traditional manual quality inspection is significantly reduced, effectively reducing the workload of business personnel (such as quality inspectors, trainers), so that they can focus on more complex judgments and strategy planning. The standardized case library and automated analysis capabilities established by the system provide a powerful tool for self-learning and self-improvement for customer service personnel, promoting the balanced development of the overall skill level of the team. At the same time, the automation and data-driven characteristics of the system ensure the consistency and scalability of quality inspection standards, enabling efficient handling of the growing volume of dialogue data, ultimately improving the service efficiency and management effectiveness of the entire customer service team.

[0158] 7) All dialogue data is classified and scored, and multiple filtering is supported. Combined with individual and comparative analysis functions, the system can provide management with performance data and trend analysis for different customer service personnel and different scenarios, identify the strengths and weaknesses of the team as a whole, and provide objective and quantitative data support for management decisions such as training plans, service process optimization, and personnel allocation.

[0159] In the above embodiments, although the steps are numbered S1, S2, etc., this is only a specific embodiment given by the present application, and those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is within the protection scope of the present application. It can be understood that in some embodiments, some or all of the above embodiments can be included.

[0160] As shown in Figure 2 The quality inspection system 200 for dialogue content of customer service personnel based on a large language model according to an embodiment of the present application includes a data cleaning module 201 and a quality inspection module 202.

[0161] The data cleaning module 201 is configured to obtain dialogue content between customer service personnel and customers and perform data cleaning.

[0162] The quality inspection module 202 is configured to perform quality inspection on all data cleaned dialogue content using a preset large language model and generate improvement suggestions.

[0163] Optionally, in the above technical solution, a business scenario classification module and a multi-dimensional scoring module are further included.

[0164] The business scenario classification module is configured to classify the business scenarios of each data cleaned dialogue content and determine the business scenario category corresponding to each data cleaned dialogue content.

[0165] The multi-dimensional scoring module is configured to receive a target service personnel determined by a user and a target business scenario category, query data cleaned conversation content of the target service personnel in the target business scenario category from all data cleaned conversation content, and score the target service personnel in multiple dimensions according to the data cleaned conversation content of the target service personnel in the target business scenario category, to obtain a multi-dimensional scoring result of the target service personnel in the target business scenario category.

[0166] Optionally, in the technical solution above, the system further comprises a comprehensive performance evaluation module, which is configured to obtain and determine a comprehensive performance score of the target service personnel in a preset period according to multiple multi-dimensional scoring results of the target service personnel in the preset period.

[0167] Optionally, in the technical solution above, the system further comprises a comparative analysis module, which is configured to:

[0168] obtain and generate a comparative analysis result between different service personnel according to data cleaned conversation content and multi-dimensional scoring results of the different service personnel.

[0169] Optionally, in the technical solution above, the system further comprises a standardized case library generation module, which is configured to generate a standardized case library according to each piece of data cleaned conversation content under each business scenario category.

[0170] Optionally, in the technical solution above, the system further comprises a training mode construction module, which is configured to:

[0171] construct a training mode for the service personnel to simulate a reply based on the standardized case library.

[0172] It should be noted that the beneficial effects of the system 200 for quality inspection of service personnel conversation content based on a large language model provided in the above embodiments are the same as those of the method for quality inspection of service personnel conversation content based on a large language model, which will not be repeated here. In addition, when the system provided in the above embodiments implements its functions, only the division of the above functional modules is exemplified, and in actual applications, the above functions can be completed by different functional modules according to needs, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the above described functions. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process is described in the method embodiments, which will not be repeated here.

[0173] In the present application, the quality inspection system for customer service personnel dialogue content based on large language model can be a computer program (including program code) running in a computer device. For example, the quality inspection system for customer service personnel dialogue content based on large language model is an application software, which can be used to execute the corresponding steps in the quality inspection method for customer service personnel dialogue content based on large language model.

[0174] In some embodiments, the quality inspection system for customer service personnel dialogue content based on large language model can be implemented in a combination of software and hardware. For example, the quality inspection system for customer service personnel dialogue content based on large language model can be a hardware decoding processor programmed to execute the quality inspection method for customer service personnel dialogue content based on large language model. For example, the hardware decoding processor can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements.

[0175] In the embodiments of the present application, the modules involved in the description can be implemented in the form of software or hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0176] An electronic device according to an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the quality inspection methods for customer service personnel dialogue content based on large language model described above. That is, the electronic device according to an embodiment of the present application can include, but is not limited to, a processor and a memory. The memory is used to store a computer program. The processor is used to execute the quality inspection method for customer service personnel dialogue content based on large language model according to any embodiment of the present application by calling the computer program.

[0177] In an optional embodiment, an electronic device is provided, as shown in Figure 3 , and Figure 3The electronic device 4000 shown includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction, such as data transmission and / or data reception, between the electronic device and other electronic devices. It should be noted that the transceiver 4004 is not limited to one in actual applications, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0178] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in conjunction with the present disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0179] The bus 4002 can include a path for transmitting information between the above-mentioned components. The bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used to represent the bus 4002, but it does not mean that there is only one bus or only one type of bus.

[0180] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0181] The memory 4003 is used to store application program code (computer program) for implementing the scheme of the present application, and is controlled by the processor 4001 to execute. The processor 4001 is used to execute the application program code stored in the memory 4003 to realize the content shown in the foregoing method embodiments.

[0182] The electronic device can also be a terminal device, which can be any device that can install an application, including at least one of a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a smart television, and a smart vehicle device.

[0183] It should be noted that, Figure 3 The electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0184] The computer readable storage medium of the embodiments of the present application, the computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize any one of the above-mentioned quality inspection methods of the customer service personnel dialogue content based on the large language model.

[0185] Optionally, the computer readable storage medium can be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0186] In an example embodiment, a computer program product or computer program is also provided, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the electronic device to perform any of the above-described methods for quality inspection of customer service personnel dialogue content based on a large language model.

[0187] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0188] It should be understood that the flowchart and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of various embodiments of the present application. In this regard, each block in the flowchart and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the flowchart or block diagrams can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0189] The computer readable storage medium provided by the embodiments of the present application can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0190] The computer readable storage medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the embodiments described above.

[0191] The above description is merely the preferred embodiments of the present application and the explanation of the technical principles used. It should be understood by those skilled in the art that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by any combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by replacing the above features with the technical features disclosed in the present application (but not limited to) having similar functions.

[0192] It should be noted that the terms "first", "second" and the like in the specification and claims of the present application are used to distinguish similar objects, and represent no specific order or sequence. The order of use of similar objects can be interchanged in appropriate cases, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0193] Those skilled in the art know that the present application can be implemented as a system, a method or a computer program product, so the present application can be specifically implemented as follows: it can be a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" in this paper. In addition, in some embodiments, the present application can also be implemented as a computer program product in one or more computer readable media, which contains computer readable program code.

[0194] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A quality inspection method for customer service staff conversation content based on a large language model, characterized in that: include: Obtain multiple conversations between customer service staff and customers and perform data cleaning; Use the preset large language model to perform quality inspection on all conversation content after data cleaning and generate improvement suggestions.

2. A method for quality inspection of customer service personnel conversation content based on a large language model according to claim 1, characterized in that: Also includes: Classify the business scenarios of the conversation content after each data cleansing segment and determine the business scenario category corresponding to the conversation content after each data cleansing segment; Receive the target customer service personnel and target business scenario category determined by the user, query the conversation content of the target customer service personnel after data cleaning in the target business scenario category from all conversation contents after data cleaning, and perform a multi-dimensional score on the target customer service personnel based on the conversation content of the target customer service personnel after data cleaning in the target business scenario category to obtain the multi-dimensional score result of the target customer service personnel under the target business scenario category.

3. The method for quality inspection of customer service personnel conversation content based on a large language model according to claim 2, characterized in that: Also includes: Obtain and determine the comprehensive performance score of the target customer service personnel within the preset period based on multiple multi-dimensional scoring results of the target customer service personnel within the preset period.

4. A method for quality inspection of customer service personnel conversation content based on a large language model according to claim 3, characterized in that: Also includes: Obtain and generate comparative analysis results between different customer service personnel based on the conversation content and multi-dimensional scoring results after data cleaning of different customer service personnel.

5. A method for quality inspection of customer service personnel conversation content based on a large language model according to any one of claims 1 to 4, characterized in that: Also includes: A standardized case library is generated based on the cleaned conversation content of each data segment under each business scenario category.

6. A method for quality inspection of customer service personnel conversation content based on a large language model according to claim 5, characterized in that: Also includes: Based on the standardized case library, a training model is constructed to enable customer service personnel to perform simulated responses.

7. A quality inspection system for customer service staff conversation content based on a large language model, characterized by: Including data cleaning module and quality inspection module; The data cleaning module is used to obtain multiple conversations between customer service personnel and customers and perform data cleaning; The quality inspection module is used to use a preset large language model to perform quality inspection on all conversation contents after data cleaning and generate improvement suggestions.

8. A quality inspection system for customer service staff conversation content based on a large language model according to claim 7, characterized in that: It also includes a business scenario classification module and a multi-dimensional scoring module; The business scenario classification module is used to classify the conversation content after each data segment is cleaned, and determine the business scenario category corresponding to the conversation content after each data segment is cleaned; The multi-dimensional scoring module is used to: receive the target customer service personnel and target business scenario category determined by the user, query the data-cleaned conversation content of the target customer service personnel in the target business scenario category from all data-cleaned conversation content, and perform a multi-dimensional scoring on the target customer service personnel based on the data-cleaned conversation content of the target customer service personnel in the target business scenario category to obtain a multi-dimensional scoring result of the target customer service personnel under the target business scenario category.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for quality inspection of customer service personnel conversation content based on a large language model as described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a quality inspection method for customer service personnel conversation content based on a large language model as described in any one of claims 1 to 6.

Images