Service quality evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a service quality evaluation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of service quality management. The evaluation method comprises the steps of obtaining a semantic vector of a current session; according to the semantic vector of the current session, determining a session type of the current session and a multi-dimensional quality inspection model corresponding to the session type; and inputting the semantic vector of the current session into a multi-dimensional quality inspection model corresponding to the session type, and evaluating a multi-dimensional service quality index of the current session, the multi-dimensional service quality index comprising a combination of at least two of the following: compliance, emotion, efficiency, service attitude and potential risk. The problems that a large number of potential problems are omitted, the coverage rate is low, subjectivity is high, the quality inspection dimension is single, depth is lacked, accuracy is poor, the inference cost is extremely high, response is slow, and the method is not suitable for high-concurrency scenes in the prior art are at least solved. The method is suitable for service quality detection and management scenes.

Description

Technical Field

[0001] This invention relates to the field of service quality management technology, and in particular to a service quality assessment method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] Traditional quality inspection methods for service ticket content mainly rely on manual sampling and keyword retrieval. Manual sampling involves quality inspectors randomly selecting a small number of service ticket contents for manual review, resulting in low coverage, high subjectivity, low efficiency, and high labor costs. Keyword retrieval methods, by setting rules for "prohibited words" and "sensitive words" to screen massive amounts of chat logs, improve efficiency but cannot understand contextual semantics. For example, keyword methods struggle to identify user dissatisfaction expressed by vague terms like "hehe," leading to quality inspection failures, high rates of missed and false positives, and difficulty in deeply evaluating the complex dimensions of service attitude, communication skills, and problem-solving effectiveness.

[0003] With the development of artificial intelligence technology, existing solutions attempt to introduce machine learning models for quality inspection. However, most of these rely on large amounts of pre-labeled data, resulting in high labeling costs and limited model generalization capabilities, making it difficult to cope with constantly changing business scenarios and communication styles. While quality inspection methods based on Large Language Models (LLMs) offer strong analytical capabilities, directly applying them to full-scale real-time quality inspection faces issues such as excessively high inference costs and slow response times, making it difficult to meet the requirements of large-scale production.

[0004] In summary, existing service order quality inspection solutions have the following main drawbacks: 1. Manual sampling relies on the inspector's personal experience, making it difficult to standardize and impossible to inspect all service order content. Many potential problems are overlooked, resulting in low coverage and strong subjectivity. 2. Keyword-based methods fail to understand the true intent and emotion of the dialogue, leading to poor accuracy. 3. Directly applying large language models for full-scale quality inspection incurs extremely high inference costs and slow response times, making it unsuitable for high-concurrency scenarios; while traditional small models are limited by high data annotation costs and weak generalization ability. 4. Existing methods struggle to assess deep service quality, resulting in a single inspection dimension and a lack of depth. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a service quality assessment method, apparatus, electronic device and computer-readable storage medium, which can achieve efficient and accurate service quality assessment.

[0006] In a first aspect, the present invention provides a service quality assessment method, comprising: obtaining the semantic vector of the current session; determining the session type of the current session and its corresponding multidimensional quality inspection model based on the semantic vector of the current session; inputting the semantic vector of the current session into the multidimensional quality inspection model corresponding to the session type to assess the multidimensional service quality indicators of the current session, wherein the multidimensional service quality indicators include a combination of at least two of the following: compliance, sentiment, efficiency, service attitude and potential risk.

[0007] Preferably, before determining the session type and its corresponding multidimensional quality inspection model based on the semantic vector of the current session, the service quality assessment method further includes: obtaining the semantic vector of historical sessions and performing cluster analysis on them to obtain several historical clusters; identifying the session types of several historical clusters, wherein the session types include mainstream types and at least one specialized type; and training neural network models based on the historical sessions in all historical clusters belonging to each session type to obtain the multidimensional quality inspection models corresponding to the mainstream types and at least one specialized type.

[0008] Preferably, identifying the session types of several historical clusters specifically includes: calculating the data distribution characteristics of several historical clusters, wherein the data distribution characteristics include one or more combinations of the following: density, outlier degree, and semantic volatility; and identifying the session types of several historical clusters based on the data distribution characteristics and a preset mapping table, wherein the preset mapping table includes the data distribution characteristics, the session types, and the mapping relationship between the two.

[0009] Preferably, a neural network model is trained based on historical sessions in all historical clusters belonging to each conversation type to obtain a multi-dimensional quality inspection model corresponding to the mainstream type and at least one specialized type. Specifically, this includes: extracting representative samples corresponding to the mainstream type and at least one specialized type from historical sessions in all historical clusters belonging to each conversation type; inputting the representative samples into a large language model to evaluate the multi-dimensional service quality indicators of the representative samples; and training a neural network model based on the representative samples and their multi-dimensional service quality indicators to obtain a multi-dimensional quality inspection model corresponding to the mainstream type and at least one specialized type.

[0010] Preferably, representative samples corresponding to the mainstream type and at least one special type are extracted from all historical sessions belonging to each session type. Specifically, this includes: evaluating the priority of each historical cluster under the mainstream type and the special type according to the data distribution characteristics; sorting each historical cluster under the mainstream type and the special type according to the priority, and extracting representative samples of the top N historical clusters under the mainstream type and the special type respectively. The representative samples include one or more combinations of cluster centers, boundary points and outliers.

[0011] Preferably, the session type and its corresponding multidimensional quality inspection model of the current session are determined based on the semantic vector of the current session. Specifically, this includes: calculating the distance between the semantic vector of the current session and several historical clusters according to the nearest neighbor search algorithm, and determining the historical clusters with a distance less than a preset value as the target clusters of the current session; and determining the session type of the target cluster and its corresponding multidimensional quality inspection model as the session type and its corresponding multidimensional quality inspection model of the current session.

[0012] Preferably, after calculating the distances between the semantic vector of the current session and several historical clusters, and before determining the session type of the target cluster and its corresponding multidimensional quality inspection model as the session type of the current session and its corresponding multidimensional quality inspection model, the service quality assessment method further includes: determining whether all distances are greater than preset values; and in response to the fact that all distances are greater than preset values, issuing warnings and conducting reviews to determine the target cluster of the current session.

[0013] Preferably, the semantic vector of the current session is input into the multidimensional quality inspection model corresponding to the session type to evaluate the multidimensional service quality indicators of the current session. The service quality evaluation method further includes: merging the semantic vectors of the current session and the historical session, and performing cluster analysis on the merged semantic vectors to obtain several updated clusters; calculating the similarity between the updated clusters and the historical clusters, and determining whether the similarity is less than a preset threshold; in response to the similarity being less than the preset threshold, identifying the changed keywords between the updated clusters and the historical clusters, and performing early warning and root cause analysis based on the changed keywords.

[0014] Secondly, the present invention also provides a service quality assessment device, comprising a first acquisition module, a determination module, and an assessment module. The first acquisition module is used to acquire the semantic vector of the current session. The determination module is connected to the first acquisition module and is used to determine the session type of the current session and its corresponding multidimensional quality inspection model based on the semantic vector of the current session. The assessment module is connected to the determination module and is used to input the semantic vector of the current session into the multidimensional quality inspection model corresponding to the session type to assess the multidimensional service quality indicators of the current session. The multidimensional service quality indicators include a combination of at least two of the following: compliance, sentiment, efficiency, service attitude, and potential risk.

[0015] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the service quality assessment method provided in the first aspect above.

[0016] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the service quality assessment method provided in the first aspect above.

[0017] This invention provides a service quality assessment method, apparatus, electronic device, and computer-readable storage medium. By matching corresponding multi-dimensional quality inspection models to different types of sessions, it can more accurately assess the service quality of different sessions, reduce false positives and false negatives, improve the accuracy of quality inspection results, and better cope with constantly changing business scenarios and communication styles. Different types of sessions use specialized quality inspection models, allowing the models to more flexibly adapt to diverse session content and improve their generalization ability. Through multi-dimensional quality inspection models that can evaluate a wealth of service quality indicators, a more comprehensive and in-depth assessment of service quality is achieved, helping enterprises to better understand the service status and improve service levels. Therefore, this invention can achieve efficient and accurate service quality assessment. Attached Figure Description

[0018] Figure 1 This is a flowchart of a service quality assessment method according to Embodiment 1 of the present invention;

[0019] Figure 2 This is an example diagram of the multi-dimensional quality inspection model constructed and trained for the session type in Embodiment 1 of the present invention;

[0020] Figure 3 This is an example diagram illustrating the multidimensional service quality indicators used to evaluate representative samples in Embodiment 1 of the present invention.

[0021] Figure 4 This is an example diagram of the multi-dimensional quality inspection models trained for the mainstream type and at least one special type in Embodiment 1 of the present invention.

[0022] Figure 5 This is a flowchart of a service quality assessment method according to Embodiment 2 of the present invention;

[0023] Figure 6 This is a flowchart of another service quality assessment method according to Embodiment 2 of the present invention;

[0024] Figure 7 This is a schematic diagram of a service quality assessment device according to Embodiment 3 of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0026] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0027] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0028] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0029] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0030] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0031] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0032] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0033] Example 1:

[0034] like Figure 1 As shown, this embodiment provides a method for evaluating service quality. The method for evaluating service quality includes:

[0035] S101, obtain the semantic vector of the current session.

[0036] In this embodiment, unstructured text data of the current session (such as online customer service dialogues, Wo operation work orders, and speech-to-text transcription) is obtained from multiple sources. The unstructured text data undergoes data preprocessing and semantic vectorization to obtain the semantic vector of the current session. Data preprocessing includes, but is not limited to: noise removal (such as order serial numbers, addresses, and phone numbers), format standardization (unified timestamps and role tags), and dialogue turn segmentation. Semantic vectorization of the unstructured text data specifically includes: using a Chinese enhanced text embedding model (such as bge-large-zh-v1.5) to map each complete session in the unstructured text data into a 768-dimensional semantic vector, preserving contextual semantic information.

[0037] S102, Based on the semantic vector of the current session, determine the session type of the current session and its corresponding multidimensional quality inspection model.

[0038] In this embodiment, the conversation type is a classification of conversations based on various characteristics such as the topic, purpose, participating roles, and communication scenario. This helps to differentiate and manage conversations of different natures and purposes, enabling targeted processing methods and evaluation standards. The multi-dimensional quality inspection model is used to evaluate and quantify multi-dimensional service quality indicators for conversations. These indicators include combinations of at least two of the following: compliance, emotion, efficiency, service attitude, and potential risk. Compliance reflects whether the conversation follows standard scripts (such as greetings and closing remarks); emotion reflects the emotional trajectory of both parties (such as a user's shift from anger to satisfaction); efficiency reflects whether the conversation closes the loop within a reasonable number of rounds; service attitude reflects whether there are negative behaviors such as shirking responsibility, questioning, or indifference; and potential risk reflects whether the conversation involves sensitive words or shows a tendency to escalate complaints.

[0039] Optionally, before S102: determining the session type and its corresponding multidimensional quality inspection model based on the semantic vector of the current session, the service quality assessment method further includes:

[0040] S104. Obtain the semantic vectors of historical sessions and perform cluster analysis on them to obtain several historical clusters.

[0041] In this embodiment, before determining the session type and its corresponding multidimensional quality inspection model based on the semantic vector of the current session, the service quality assessment method further includes: constructing and training a multidimensional quality inspection model corresponding to the session type. For example... Figure 2 As shown, constructing and training a multi-dimensional quality inspection model corresponding to each session type includes: obtaining the semantic vector of historical sessions is similar to obtaining the semantic vector of the current session (i.e., Figure 2 The process involves inputting original chat logs from various channels, cleaning and segmenting the data, and vectorizing the text. Clustering algorithms are used to perform unsupervised clustering of the semantic vectors of historical conversations, automatically identifying semantically similar vectors to obtain several historical clusters. These clustering algorithms include, but are not limited to, HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) and DBSCAN (Density-Based Spatial Clustering of Applications with Noise).

[0042] It should be noted that the number of historical clusters obtained by clustering in this embodiment does not need to be preset, that is, the number of historical clusters is a positive integer greater than 0.

[0043] S105, identify several historical clusters of session types, wherein the session types include a mainstream type and at least one specialized type.

[0044] In this embodiment, based on the semantic intent, business scenario, or interaction complexity of the session, it is divided into two categories: mainstream types and specialized types. Mainstream types refer to high-frequency, universally applicable basic session types that cover most business scenarios, such as resolving user operation or technical issues like "The router cannot connect to the network, how to reset it?" Specialized types refer to low-frequency scenarios that require special handling, typically involving sensitive issues, high-value businesses, or complex processes, such as "number portability disputes."

[0045] Specifically, S105: Identify the session types of several historical clusters, including steps S1051-S1052:

[0046] S1051, calculate the data distribution characteristics of several historical clusters, wherein the data distribution characteristics include one or more combinations of the following: density, outlier degree and semantic volatility.

[0047] In this embodiment, data distribution features are used to reflect the spatial distribution and characteristics of several historical clusters, helping to assess the quality, stability, and similarity and difference of data within a cluster. Density refers to the degree of clustering of data points within a cluster. High-density clusters mean that data points are relatively concentrated within the cluster, with relatively small distances between data points, indicating high similarity among data points within the cluster. Low-density clusters, on the other hand, indicate that data points are relatively dispersed. Outlier degree measures the degree to which a data point deviates from its cluster or the overall data distribution. Semantic volatility measures the degree of semantic change in data within historical clusters. If semantic volatility is low, it indicates that the semantics of the data within the cluster are relatively consistent, and the semantic differences between data points are not significant. Conversely, if semantic volatility is high, it indicates that the semantic changes of the data within the cluster are frequent, and the semantic differences between data points are significant.

[0048] This embodiment calculates the nearest neighbor distance between each sample data point and the average value within several historical clusters to obtain the density of these clusters. The outlier scores of these historical clusters are calculated using HDBSCAN's built-in outlier_scores or the LOF (Local Outlier Factor) algorithm. The semantic volatility of these historical clusters is obtained by calculating the BERT (Bidirectional Encoder Representation from Transformers) score or semantic vector variance within each cluster.

[0049] S1052, Based on the data distribution characteristics and the preset mapping table, identify the session types of several historical clusters, wherein the preset mapping table includes the data distribution characteristics, session types, and the mapping relationship between the two.

[0050] In this embodiment, high-density large clusters (i.e., high density, low outlier, and low semantic volatility) represent high-frequency standard scenarios, typically corresponding to mainstream session types. Low-density edge clusters (i.e., low density, high outlier, and variable semantic volatility) may represent emerging issues or service blind spots, marked as "potential novel defects," typically corresponding to specialized session types. High semantic volatility indicates inconsistent service responses and potential execution deviations. The preset mapping table is shown in Table 1, where mainstream types include basic equipment fault diagnosis and package change consultation, and specialized types include number portability disputes, exorbitant international roaming bills, and government / enterprise dedicated line failures. This embodiment achieves efficient and automated session type identification by quantifying and statistically analyzing the data distribution characteristics of historical clusters, combining classification accuracy, business interpretability, and anomaly detection capabilities.

[0051] Table 1 Preset Mapping Table

[0052]

[0053] S106, Train neural network models based on historical sessions in all historical clusters belonging to each session type to obtain multi-dimensional quality inspection models corresponding to the mainstream type and at least one special type.

[0054] In this embodiment, considering that large-scale historical clusters can lead to reduced efficiency and increased cost in building and training multidimensional quality inspection models, the construction and training of multidimensional quality inspection models corresponding to session types further includes: identifying the session types of several historical clusters (i.e., Figure 2 The clustering results analysis shows that the mainstream session clusters are all historical clusters belonging to the mainstream type, and the abnormal / marginal clusters are all historical clusters belonging to the specialized type. Representative samples are extracted from the historical sessions belonging to each session type to construct and train the multidimensional quality inspection model (i.e., Figure 2 The process involves generating a quality inspection priority map, extracting representative samples from high-priority clusters, using Qwen3 for intelligent annotation, and building lightweight small models to improve the efficiency and reduce the cost of building and training multidimensional quality inspection models. Neural network models include, but are not limited to, MT5 (MetaTrader 5).

[0055] Specifically, S106: Train neural network models based on historical sessions in all historical clusters belonging to each session type to obtain multi-dimensional quality inspection models corresponding to the mainstream type and at least one specialized type, including steps S1061-S1063:

[0056] S1061, extract representative samples corresponding to the mainstream type and at least one special type from the historical sessions in all historical clusters belonging to each session type.

[0057] Specifically, S1061: Extract representative samples corresponding to the mainstream type and at least one special type from all historical sessions belonging to each session type, including: evaluating the priority of each historical cluster under the mainstream type and at least one special type according to the data distribution characteristics; sorting each historical cluster under the mainstream type and at least one special type according to the priority, and extracting representative samples of the top N historical clusters under the mainstream type and at least one special type respectively, wherein the representative samples include one or more combinations of cluster centers, boundary points and outliers.

[0058] In this embodiment, according to the formula The priorities of each historical cluster under the mainstream type and the special type were evaluated separately. Indicates the first one belonging to each session type A historical cluster, , , Representing historical clusters Density, outlier degree, and semantic volatility, , , Representing historical clusters Weights for density, outlier, and semantic volatility, for example: =0.3, =0.4, =0.3, Representing historical clusters The priority of a historical cluster is determined by its density, outlier, and semantic volatility. Higher priority clusters are more likely to contain service quality anomalies or novel issues and should be prioritized for quality inspection. This embodiment calculates the priority of each historical cluster by weighting its density, outlier, and semantic volatility. Then, by sorting historical clusters under mainstream and specialized types and extracting representative samples from the top N clusters, the cluster centers, boundary points, and outliers in these representative samples contain key information about the historical clusters. This reflects the data's central tendency, distribution range, and special cases, allowing for rapid and targeted identification of key areas requiring quality inspection for each session type. This is used to build and train multidimensional quality inspection models, improving the efficiency of multidimensional quality inspection model construction and training while ensuring its accuracy and relevance. This embodiment does not annotate the entire dataset but selects high-value samples based on priority. Subsequent use of dynamic prompt templates allows the large language model to focus on in-depth analysis of key issues, significantly improving annotation efficiency and quality while reducing the cost of calling the large model.

[0059] It's important to note that in conversation type analysis, cluster centers may represent typical characteristics or patterns of a particular type of conversation. For example, in a customer service-related conversation type, the cluster center might be a conversation exhibiting typical customer service behaviors (such as answering frequently asked questions and providing basic services). In conversation type analysis, boundary points may represent the intersection of two or more conversation types, or the starting point of a new trend or change. For example, in a sales promotion-related conversation type, boundary points might be conversations that include both regular sales information and personalized promotional strategies. In conversation type analysis, anomalies may represent abnormal or unexpected conversation behavior, such as malicious attacks or abnormal conversations caused by system malfunctions. For instance, a conversation containing a large amount of bot behavior could be considered an anomaly.

[0060] S1062, input representative samples into the large language model, and evaluate the multi-dimensional service quality indicators of the representative samples respectively.

[0061] In this embodiment, as Figure 3 As shown, multidimensional service quality indicators for representative samples were evaluated, specifically including: generating representative samples based on Prompt Engineering (i.e., Figure 3 The intent task Prompt (i.e., the representative session sample in the sample) corresponds to the representative session sample in the sample. Figure 3 The multi-dimensional quality inspection Prompt of the representative samples is input into the large language model to evaluate the multi-dimensional service quality indicators (i.e., ...) of the representative samples. Figure 3 The data used in this embodiment is high-quality, multi-dimensional labeled data, including, but not limited to, large language models such as Qwen and DeepSeek. The multi-dimensional service quality indicators of representative samples evaluated by the large language models not only possess semantic richness but also category and business representativeness, providing accurate samples for subsequent small model training.

[0062] S1063, based on representative samples and their multidimensional service quality indicators, train neural network models to obtain multidimensional quality inspection models corresponding to mainstream types and at least one special type.

[0063] In this embodiment, as Figure 4 As shown, unlike the traditional approach of unifying the modeling of mainstream and specialized session types, this embodiment supports two flexible modeling paths (i.e., Figure 4The model training method (as described in the text) trains multi-dimensional quality inspection models corresponding to mainstream types and at least one specialized type. Flexible modeling paths include: a multi-task classification model construction path and a multiple specialized quality inspection model construction path. The multi-task classification model can simultaneously evaluate compliance, sentiment, efficiency, service attitude, and potential risks, suitable for multi-dimensional service quality indicator evaluation of mainstream type conversations. Multiple specialized quality inspection models are trained separately for each historical cluster belonging to each specialized type. These models can simultaneously or sequentially evaluate the compliance, sentiment, efficiency, service attitude, and potential risks of their corresponding specialized type conversations. Specifically, there is one multi-dimensional quality inspection model corresponding to the mainstream type, while the number of multi-dimensional quality inspection models corresponding to each specialized type is greater than one. For example, if multi-dimensional service quality indicators include compliance, sentiment, efficiency, service attitude, and potential risks, the multi-dimensional quality inspection models corresponding to each specialized type include a compliance quality inspection model, a sentiment quality inspection model, an efficiency quality inspection model, a service attitude quality inspection model, and a potential risk quality inspection model. This improves the quality inspection efficiency of the multi-dimensional quality inspection models corresponding to the mainstream type and the fine-grained discrimination capability of the multi-dimensional quality inspection models corresponding to the specialized types. This embodiment automatically mines semantic structures and potential risk patterns in historical conversations through unsupervised clustering. This guides a large language model to efficiently label representative samples, thereby accurately building a multi-dimensional quality inspection model and enabling dynamic decision-making and continuous optimization of the multi-dimensional quality inspection model in subsequent real-time quality inspections. Compared to traditional methods that rely on manual rules or full-scale labeling, this embodiment trains multi-dimensional quality inspection models corresponding to mainstream types and at least one specialized type, achieving a paradigm shift from passive inspection to proactive discovery, and from unified processing to categorized governance. This significantly improves quality inspection coverage, accuracy, and interpretability.

[0064] It should be noted that incremental clustering updates are triggered periodically to train and optimize the multidimensional quality inspection model. During the training process, an inter-cluster balancing sampling mechanism is introduced to prevent mainstream cluster types from dominating the training process and to ensure the ability to identify low-frequency but high-risk sessions. After training, the multidimensional quality inspection model is packaged as a microservice component and deployed to the real-time quality inspection engine, supporting high-concurrency, low-latency online inference and achieving the training and deployment of a lightweight multidimensional quality inspection model.

[0065] Specifically, S102: Based on the semantic vector of the current session, determine the session type of the current session and its corresponding multidimensional quality inspection model, including steps S1021-S1022:

[0066] S1021, Based on the nearest neighbor search algorithm, calculate the distance between the semantic vector of the current session and several historical clusters, and determine the historical clusters with a distance less than a preset value as the target clusters of the current session.

[0067] In this embodiment, after the current session ends, the nearest neighbor (ANN) algorithm is used to find the historical clusters whose semantic vectors are closest to the current session. The nearest neighbor search algorithm includes, but is not limited to, the FAISS (Facebook AI Similarity Search, a C++ library for efficient similarity search and clustering that supports fast finding of nearest neighbors in large-scale vector sets) algorithm.

[0068] Optionally, after calculating the distances between the semantic vector of the current session and several historical clusters, and before determining the historical clusters with distances less than a preset value as the target clusters of the current session, the service quality assessment method further includes: determining whether all distances are greater than the preset value; and in response to the fact that all distances are greater than the preset value, issuing an early warning and conducting a review to determine the target clusters of the current session.

[0069] In this embodiment, if the closest historical cluster to the semantic vector of the current session is not found, that is, the distance between the semantic vector of the current session and several historical clusters is greater than a preset value, the semantic vector of the current session is determined to be a new cluster, triggering an alert, and the semantic vector of the current session is marked as a sample to be reviewed for review. The review includes, but is not limited to, pushing it to a human quality inspector for manual review to determine the target cluster of the current session.

[0070] It should be noted that after verifying and confirming the target cluster of the current session, this embodiment also incorporates the semantic vector of the current session into the target cluster of the historical database, and inputs the semantic vector of the current session into a large language model to evaluate the multi-dimensional service quality indicators of the current session for incremental training (fine-tuning) of the multi-dimensional quality inspection model. This embodiment dynamically adjusts the prompting task based on the semantic vector of the current session when determining that it is a new cluster, evaluates the multi-dimensional service quality indicators of the current session for incremental training of the multi-dimensional quality inspection model, enhances the understanding of unknown scenarios, and ensures comprehensive annotation coverage.

[0071] S1022, determine the session type of the target cluster and its corresponding multidimensional quality inspection model as the session type and its corresponding multidimensional quality inspection model of the current session.

[0072] In this embodiment, if the historical cluster whose semantic vector is closest to that of the current session is found to be historical cluster A, and the session type of historical cluster A is mainstream session, then the multidimensional quality inspection model corresponding to the mainstream session is called to perform multidimensional quality inspection.

[0073] It should be noted that, while determining historical clusters with a distance less than a preset value as the target clusters of the current session, this embodiment also includes: calculating the confidence level of the target cluster; determining whether the confidence level is greater than a first preset threshold; if the target cluster is a low-confidence result, that is, the confidence level is less than or equal to the first preset threshold, the semantic vector of the current session is also marked as a sample to be reviewed for review.

[0074] S103, input the semantic vector of the current session into the multi-dimensional quality inspection model corresponding to the session type, and evaluate the multi-dimensional service quality indicators of the current session. The multi-dimensional service quality indicators include a combination of at least two of the following: compliance, sentiment, efficiency, service attitude and potential risks.

[0075] In this embodiment, if the historical cluster whose semantic vector is closest to that of the current session is found to be historical cluster A, and the session type of historical cluster A is mainstream session, then the multidimensional quality inspection model corresponding to the mainstream session is called to perform multidimensional quality inspection.

[0076] It should be noted that, while determining historical clusters with a distance less than a preset value as the target clusters of the current session, this embodiment also includes: calculating the confidence level of the target cluster; determining whether the confidence level is greater than a first preset threshold; if the target cluster is a low-confidence result, that is, the confidence level is less than or equal to the first preset threshold, the semantic vector of the current session is also marked as a sample to be reviewed for review.

[0077] Optionally, in S103: after inputting the semantic vector of the current session into the multi-dimensional quality inspection model corresponding to the session type to evaluate the multi-dimensional service quality indicators of the current session, the service quality evaluation method further includes:

[0078] S107, merge the semantic vectors of the current session and the historical session, and perform cluster analysis on the merged semantic vectors to obtain several updated clusters.

[0079] S108, calculate the similarity between the updated cluster and the historical cluster, and determine whether the similarity is less than a preset threshold.

[0080] S109, in response to a similarity of less than a preset threshold, identify the changed keywords between the updated cluster and the historical cluster, and perform early warning and root cause analysis based on the changed keywords.

[0081] In this embodiment, to address the gradual evolution of business scenarios or sudden events (such as a large number of complaints triggered by the launch of a new package), a clustering structure stability monitoring mechanism is introduced to proactively detect service semantic distribution drift. Specifically, this includes: running a full clustering operation once after each session or each cycle (e.g., weekly) to generate the clustering structure after the current session or the current cycle, i.e., several updated clusters. The clustering structure before the current session ended or in the previous period, i.e., several historical clusters. Cluster alignment analysis is performed, specifically by using the Hungarian Algorithm to calculate the optimal cluster mapping relationship, such as: Including clusters B1, B2, and B3, Including clusters B1, B2', and B3, the optimal cluster mapping relationships are cluster B1-cluster B1, cluster B2-cluster B2', and cluster B3-cluster B3; according to the formula Calculate the similarity between the updated cluster and the historical cluster. ,in, Indicates the first A historical cluster, Indicates the first One updated cluster, That is, in the optimal cluster mapping relationship The updated cluster of the mapping, The function represents the degree of overlap calculation; if Preset Threshold If the value is 0.7, the updated cluster is determined to be "clustering baseline drift". The system extracts the changing keywords (e.g., cluster B2' newly added compared to cluster B2 samples), disappearing clusters (e.g., cluster B2' reduced compared to cluster B2 samples), and mutation clusters (e.g., cluster B2' inconsistent with cluster B2 samples) from the clusters. For example, the frequency of the term "number portability" increases by 300%. Based on these changing keywords, the system performs early warnings and root cause analysis. This embodiment periodically compares the semantic alignment and stability index of the clustering structure. When the semantic alignment and stability index fall below a preset threshold, it is determined to be a business model mutation, automatically triggering root cause analysis and quality inspection strategy upgrades. This enables service quality assessment to have rapid perception and response capabilities to emergencies, significantly improving the level of intelligent operation and maintenance, achieving proactive perception and emergency response to business model mutations, and significantly improving the system's adaptability and risk warning level.

[0082] It should be noted that this embodiment also automatically increases the priority of newly added clusters, disappearing clusters, and mutated clusters, pushes them to the management warning dashboard, and triggers the incremental training process of large language model annotation and multi-dimensional quality inspection model in advance, thereby accelerating the adaptation of service quality assessment.

[0083] This embodiment provides a service quality assessment method that, by matching corresponding multi-dimensional quality inspection models to different types of sessions, can more accurately assess the service quality of different sessions, reduce false positives and false negatives, improve the accuracy of quality inspection results, and better cope with constantly changing business scenarios and dialogue. Different types of sessions use specialized quality inspection models, enabling the models to more flexibly adapt to diverse session content and improve the generalization ability of the models. Through multi-dimensional quality inspection models that can evaluate a wealth of multi-dimensional service quality indicators, a more comprehensive and in-depth assessment of service quality is achieved, helping enterprises to better understand the service status, improve service levels, and achieve efficient and accurate service quality assessment.

[0084] Example 2:

[0085] like Figure 5 or Figure 6 As shown, this embodiment provides a method for evaluating service quality. The method for evaluating service quality includes:

[0086] S201, Obtain the semantic vectors of historical conversations and perform cluster analysis to obtain several historical clusters; calculate the data distribution characteristics of several historical clusters, wherein the data distribution characteristics include one or more combinations of the following: density, outlier, and semantic volatility; identify the conversation types of several historical clusters based on the data distribution characteristics and a preset mapping table, wherein the preset mapping table includes data distribution characteristics, conversation types, and the mapping relationship between the two; evaluate the priority of each historical cluster under the mainstream type and the specialized type respectively based on the data distribution characteristics; rank each historical cluster under the mainstream type and at least one specialized type according to the priority, and extract representative samples of the top N historical clusters under the mainstream type and at least one specialized type respectively, wherein the representative samples include one or more combinations of cluster centers, boundary points, and outliers; input the representative samples into a large language model to evaluate the multidimensional service quality indicators of the representative samples respectively; train neural network models based on the representative samples and their multidimensional service quality indicators to obtain multidimensional quality inspection models corresponding to the mainstream type and at least one specialized type.

[0087] In this embodiment, historical sessions are... Figure 5 The original chat logs, several historical clusters Figure 5 The clustering structure in the data, including historical clusters under the mainstream type, is as follows: Figure 5 The session clusters in the context, and the historical clusters under the specific type, are... Figure 5 The abnormal clusters in the data, and the multidimensional service quality indicators of representative samples, are... Figure 5 The labeled data or Figure 6 The large-scale model annotation data in the middle, the multi-dimensional quality inspection models corresponding to the mainstream types, that is Figure 5 The multi-task model in the text refers to at least one multi-dimensional quality inspection model corresponding to a specific type. Figure 5 The dedicated model in the text refers to the multi-dimensional quality inspection model corresponding to the session type. Figure 5 Small and lightweight models or Figure 6 Lightweight classification models and small models in the field.

[0088] S202, obtain the semantic vector of the current session; calculate the distance between the semantic vector of the current session and several historical clusters according to the nearest neighbor search algorithm; determine whether the distances are all greater than preset values; in response to the distances being greater than preset values, issue an early warning and conduct a review to determine the target cluster of the current session; determine the historical clusters with distances less than preset values ​​as the target clusters of the current session; determine the session type of the target cluster and its corresponding multidimensional quality inspection model as the session type of the current session and its corresponding multidimensional quality inspection model; input the semantic vector of the current session into the multidimensional quality inspection model corresponding to the session type to evaluate the multidimensional service quality indicators of the current session, wherein the multidimensional service quality indicators include a combination of at least two of the following: compliance, sentiment, efficiency, service attitude, and potential risks.

[0089] In this embodiment, the semantic vector of the current session is... Figure 5 The newly added chat history, whose distance is greater than the preset value, is the semantic vector of the current session. Figure 5 New clusters or Figure 6 High-risk / low-confidence samples in the data, whose distance is less than a preset value, are the semantic vectors of the current session. Figure 6 In the standard sample, multidimensional service quality indicators are Figure 6 The quality inspection results.

[0090] S202, merge the semantic vectors of the current session and the historical session, and perform cluster analysis on the merged semantic vectors to obtain several updated clusters; calculate the similarity between the updated clusters and the historical clusters, and determine whether the similarity is less than a preset threshold; in response to the similarity being less than the preset threshold, identify the changed keywords between the updated clusters and the historical clusters, and perform early warning and root cause analysis based on the changed keywords.

[0091] In this embodiment, the similarity is less than a preset threshold, i.e. Figure 5 The drift in the middle, several historical clusters Figure 5 Historical clustering in the data, several updated clusters are... Figure 5 The current cluster is used to calculate the similarity between the updated cluster and the historical clusters, and to determine whether the similarity is less than a preset threshold. Figure 5 Periodic clustering comparison in [the context of the data].

[0092] This embodiment provides a service quality assessment method that, by matching corresponding multi-dimensional quality inspection models to different types of sessions, can more accurately assess the service quality of different sessions, reduce false positives and false negatives, improve the accuracy of quality inspection results, and better cope with constantly changing business scenarios and dialogue. Different types of sessions use specialized quality inspection models, enabling the models to more flexibly adapt to diverse session content and improve the generalization ability of the models. Through multi-dimensional quality inspection models that can evaluate a wealth of multi-dimensional service quality indicators, a more comprehensive and in-depth assessment of service quality is achieved, helping enterprises to better understand the service status, improve service levels, and achieve efficient and accurate service quality assessment.

[0093] Example 3:

[0094] like Figure 7 As shown, this embodiment also provides a service quality assessment device, including a first acquisition module 31, a determination module 32, and an assessment module 33. The first acquisition module 31 is used to acquire the semantic vector of the current session. The determination module 32 is connected to the first acquisition module 31 and is used to determine the session type of the current session and its corresponding multi-dimensional quality inspection model based on the semantic vector of the current session. The assessment module 33 is connected to the determination module 32 and is used to input the semantic vector of the current session into the multi-dimensional quality inspection model corresponding to the session type to assess the multi-dimensional service quality indicators of the current session. The multi-dimensional service quality indicators include a combination of at least two of the following: compliance, sentiment, efficiency, service attitude, and potential risk.

[0095] Optionally, the service quality assessment device further includes: a second acquisition module 34, an identification module 35, and a training module 36. The second acquisition module 34 is used to acquire the semantic vectors of historical sessions and perform cluster analysis on them to obtain several historical clusters. The identification module 35 is used to identify the session types of several historical clusters, wherein the session types include mainstream types and at least one specialized type. The training module 36 is used to train neural network models according to the historical sessions in all historical clusters belonging to each session type, to obtain multi-dimensional quality inspection models corresponding to the mainstream types and at least one specialized type.

[0096] Specifically, the identification module 35 includes a calculation unit 351 and an identification unit 352. The calculation unit 351 is used to calculate the data distribution characteristics of several historical clusters, wherein the data distribution characteristics include one or more combinations of the following: density, outlier degree and semantic volatility. The identification unit 352 is used to identify the session type of several historical clusters according to the data distribution characteristics and a preset mapping table, wherein the preset mapping table includes data distribution characteristics, session type and the mapping relationship between the two.

[0097] Specifically, the training module 36 includes: an extraction unit 361, an evaluation unit 362, and a training unit 363. The extraction unit 361 is used to extract representative samples corresponding to the mainstream type and at least one special type from all historical sessions belonging to each session type. The evaluation unit 362 is used to input the representative samples into the large language model and evaluate the multi-dimensional service quality indicators of the representative samples. The training unit 363 is used to train neural network models based on the representative samples and their multi-dimensional service quality indicators to obtain multi-dimensional quality inspection models corresponding to the mainstream type and at least one special type.

[0098] Specifically, the extraction unit 361 includes an evaluation subunit and an extraction subunit. The evaluation subunit is used to evaluate the priority of each historical cluster under the mainstream type and the special type according to the data distribution characteristics. The extraction subunit is used to sort each historical cluster under the mainstream type and at least one special type according to the priority, and extract representative samples of the top N historical clusters under the mainstream type and at least one special type respectively. The representative samples include one or more combinations of cluster centers, boundary points and outliers.

[0099] Specifically, the determination module 32 includes: a first determination unit 321 and a second determination unit 322. The first determination unit 321 is used to calculate the distance between the semantic vector of the current session and several historical clusters according to the nearest neighbor search algorithm, and determine the historical clusters with a distance less than a preset value as the target clusters of the current session. The second determination unit 322 is used to determine the session type of the target cluster and its corresponding multidimensional quality inspection model as the session type of the current session and its corresponding multidimensional quality inspection model.

[0100] Optionally, the determining module 32 further includes: a judging unit 323 and a verification unit 324. The judging unit 323 is used to judge whether all distances are greater than a preset value. The verification unit 324 is used to issue an early warning and perform a verification in response to the fact that all distances are greater than the preset value, so as to determine the target cluster of the current session.

[0101] Optionally, the service quality assessment device further includes: a merging module 37, a calculation module 38, and a root cause analysis module 39. The merging module 37 is used to merge the semantic vectors of the current session and the historical session, and perform cluster analysis on the merged semantic vectors to obtain several updated clusters. The calculation module 38 is used to calculate the similarity between the updated clusters and the historical clusters, and determine whether the similarity is less than a preset threshold. The root cause analysis module 39 is used to identify the changing keywords between the updated clusters and the historical clusters in response to the similarity being less than the preset threshold, and to perform early warning and root cause analysis based on the changing keywords.

[0102] Understandably, the service quality assessment device provided above performs the service quality assessment method corresponding to Embodiment 1 provided above. Therefore, the beneficial effects it can achieve can be referred to the beneficial effects of the scheme corresponding to the service quality assessment method of Embodiment 1 above, which will not be repeated here.

[0103] Example 4:

[0104] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to implement the service quality assessment method in Embodiment 1 or Embodiment 2 described above.

[0105] Example 5:

[0106] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the service quality evaluation method in Embodiment 1 or Embodiment 2 above.

[0107] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method of evaluating quality of service, characterized by, The method comprises the following steps: obtaining a semantic vector of a current session; determining a session type of the current session and a corresponding multi-dimensional quality inspection model according to the semantic vector of the current session; inputting the semantic vector of the current session into the multi-dimensional quality inspection model corresponding to the session type to evaluate a multi-dimensional service quality index of the current session, wherein the multi-dimensional service quality index comprises a combination of at least two of the following: compliance, emotion, efficiency, service attitude and potential risk.

2. The method of evaluating quality of service according to claim 1, wherein, Before the step of determining the session type of the current session and the corresponding multi-dimensional quality inspection model according to the semantic vector of the current session, the method further comprises the following steps: obtaining semantic vectors of historical sessions and performing cluster analysis on the semantic vectors to obtain a plurality of historical clusters; and identifying the session types of the plurality of historical clusters, wherein the session types include a mainstream type and at least one special type; training neural network models according to historical sessions in all historical clusters belonging to each session type to obtain multi-dimensional quality inspection models corresponding to the mainstream type and the at least one special type.

3. The method of evaluating quality of service according to claim 2, wherein, The step of identifying the session types of the plurality of historical clusters specifically comprises the following steps: calculating data distribution characteristics of the plurality of historical clusters, wherein the data distribution characteristics include a combination of one or more of the following: density, outlying degree and semantic volatility; identifying the session types of the plurality of historical clusters according to the data distribution characteristics and a preset mapping table, wherein the preset mapping table includes data distribution characteristics, session types and a mapping relationship therebetween.

4. The method of evaluating quality of service according to claim 3, wherein, The step of training neural network models according to historical sessions in all historical clusters belonging to each session type to obtain multi-dimensional quality inspection models corresponding to the mainstream type and the at least one special type specifically comprises the following steps: extracting representative samples corresponding to the mainstream type and the at least one special type from the historical sessions in all historical clusters belonging to each session type; inputting the representative samples into a large language model to evaluate multi-dimensional service quality indexes of the representative samples; training neural network models according to the representative samples and the multi-dimensional service quality indexes of the representative samples to obtain multi-dimensional quality inspection models corresponding to the mainstream type and the at least one special type.

5. The method of evaluating quality of service according to claim 4, characterized in that, The step of extracting representative samples corresponding to the mainstream type and the at least one special type from the historical sessions in all historical clusters belonging to each session type specifically comprises the following steps: evaluating priorities of each historical cluster under the mainstream type and the special type according to the data distribution characteristics; sorting each historical cluster under the mainstream type and the special type according to the priorities and extracting representative samples of the first N historical clusters under the mainstream type and the special type, wherein the representative samples include a combination of one or more of cluster centers, boundary points and abnormal points.

6. The method of evaluating quality of service according to claim 2, wherein, The step of determining the session type of the current session and the corresponding multi-dimensional quality inspection model according to the semantic vector of the current session specifically comprises the following steps: calculating distances between the semantic vector of the current session and a plurality of historical clusters according to a nearest neighbor search algorithm, and determining historical clusters with distances less than a preset value as target clusters of the current session; determining the session type of the target clusters and the corresponding multi-dimensional quality inspection model as the session type of the current session and the corresponding multi-dimensional quality inspection model.

7. The method of evaluating quality of service according to claim 6, wherein, After the computing of the distances between the semantic vector of the current session and the historical clusters respectively, and before the determining of the session type of the target cluster and its corresponding multidimensional quality inspection model as the session type of the current session and its corresponding multidimensional quality inspection model, further comprising: judging whether the distances are all greater than a preset value; in response to the distances being all greater than the preset value, performing early warning and review to determine the target cluster of the current session.

8. The method of evaluating quality of service according to claim 2, wherein, After the inputting of the semantic vector of the current session into the multidimensional quality inspection model corresponding to the session type and the evaluating of the multidimensional service quality index of the current session, further comprising: merging the semantic vectors of the current session and the historical sessions and performing clustering analysis on the merged semantic vectors to obtain a plurality of updated clusters; computing the similarity between the updated clusters and the historical clusters and judging whether the similarity is less than a preset threshold; in response to the similarity being less than the preset threshold, identifying the change keywords between the updated clusters and the historical clusters and performing early warning and root cause analysis according to the change keywords.

9. A quality of service evaluation apparatus characterized by comprising: comprising a first acquisition module, a determination module and an evaluation module, the first acquisition module is configured to acquire the semantic vector of the current session, the determination module is connected with the first acquisition module and is configured to determine the session type of the current session and its corresponding multidimensional quality inspection model according to the semantic vector of the current session, the evaluation module is connected with the determination module and is configured to input the semantic vector of the current session into the multidimensional quality inspection model corresponding to the session type to evaluate the multidimensional service quality index of the current session, wherein the multidimensional service quality index comprises a combination of at least two of the following: compliance, emotion, efficiency, service attitude and potential risk.

10. An electronic device, comprising: comprising a memory and a processor, the memory stores a computer program, and the processor is configured to run the computer program to implement the service quality evaluation method according to any one of claims 1 to 8.

11. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the service quality evaluation method according to any one of claims 1 to 8. The computer program is executed by the processor to implement the service quality evaluation method according to any one of claims 1 to 8.

Images