Method for automatically testing voice customer service system

Abstract

The invention relates to the technical field of voice customer service testing, in particular to an automatic testing method for a voice customer service system, which comprises the following steps of: acquiring multi-source data, and performing standardized preprocessing to form a standardized data set; constructing a multi-dimensional quantitative model, determining a core dimension classification rule and a test index standard, and performing iterative optimization through a multi-dimensional test configuration particle swarm intelligent optimization model to generate an optimal test configuration scheme; through a full-link test, collecting test data and extracting a core index, generating a system performance level and an optimization direction through a comprehensive evaluation model, and forming decision support data; constructing an abnormal root cause positioning model to mine an index association link, and positioning a system abnormal core cause; and integrating whole-process data to construct a traceable evaluation system, and outputting a quantitative evaluation result and optimized data. According to the scheme, the testing accuracy and efficiency of the intelligent voice interaction system are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of voice customer service testing technology, and in particular to a method for automated testing of a voice customer service system. Background Technology

[0002] In recent years, with the deep penetration of intelligent voice interaction technology in fields such as customer service, navigation, and smart homes, intelligent voice interaction systems have become a core support tool for enterprises to improve service efficiency and optimize user experience. However, traditional intelligent voice interaction system testing solutions are mostly based on fixed scenarios and static configuration designs. Their testing models rely on preset rules and historical data patterns, which have significant limitations in dynamic business scenarios and complex deployment environments, making it difficult to adapt to the upgraded testing requirements brought about by system iteration and business changes.

[0003] Specifically, traditional testing solutions lack a standardized system for data collection and preprocessing. The fields for collecting historical test data, business scenario characteristic data, and system deployment environment data are unclear and inconsistent in format. Furthermore, the lack of targeted outlier removal and missing value completion strategies leads to low-quality data in subsequent analysis, affecting the accuracy of test results. Moreover, test configurations lack dynamic optimization capabilities. The quantitative rules for core dimensions such as business scenario complexity and system deployment environment are vague. Test indicator weights and threshold settings rely on manual experience and cannot be adaptively adjusted according to business complexity levels and environmental differences, resulting in an imbalance between test accuracy and efficiency. Additionally, the ability to pinpoint the root cause of anomalies is weak, only able to identify surface-level anomalies and struggling to uncover deep correlations between indicators. The lack of a systematic anomaly propagation path analysis mechanism leads to low troubleshooting efficiency and delayed repairs. Fourth, the test evaluation system is incomplete. The system performance level classification standards are subjective, lacking data credibility verification and a full-process traceability mechanism, failing to provide accurate and reliable decision support for system optimization. Summary of the Invention

[0004] This invention effectively improves the accuracy and efficiency of testing intelligent voice interaction systems.

[0005] The technical solution proposed in this invention is: a method for automated testing of a voice customer service system, the method comprising: Collect data from multiple sources and perform standardized preprocessing to form a standardized dataset; Based on the standardized dataset, a multi-dimensional quantitative model is constructed to determine the core dimension classification rules and test indicator standards. The optimal test configuration scheme is generated by iterative optimization of the particle swarm intelligent optimization model through multi-dimensional test configuration. Based on the optimal test configuration scheme, test data is collected and core indicators are extracted through full-link testing. The system performance level and optimization direction are generated through a comprehensive evaluation model, forming decision support data. Based on decision support data, an anomaly root cause localization model is constructed to mine the correlation links of indicators and locate the core causes of system anomalies. Integrate data from the entire process to build a traceable evaluation system, and output quantitative evaluation results and optimization data.

[0006] Preferably, the specific process for obtaining the standardized dataset is as follows: Multi-source data encompasses historical test data, business scenario characteristic data, and system deployment environment data; Abnormal data for numerical indicators and data with non-standard formats or mismatched categories are removed according to preset thresholds; For continuous indicators, missing values ​​are filled using linear interpolation, while for discrete indicators, missing values ​​are filled using the mode. Standardized indicator values ​​are retained to two decimal places to ensure data consistency.

[0007] Preferably, the specific process for determining the core dimension classification rules and the test indicator standards is as follows: The core dimensions include business scenario complexity, system deployment environment, and system iteration version. Business scenario complexity is classified into levels by weighted average of intent complexity, dialogue rounds, and input complexity. System deployment environment is labeled by server architecture, network type, and combination of regional nodes. System iteration version is divided into new feature version, regular optimization version, and bug fix version. The test index standard includes five categories of indicators: technical modules, performance stability, business effectiveness, interactive experience, and user feedback. The total weight is 100%, and the weight of each individual indicator is limited to 5%-40%. Each indicator also has a pre-defined threshold range.

[0008] Preferably, the specific process of iterative optimization of the particle swarm intelligent optimization model is as follows: Particle mapping is a candidate scheme for the combination of core dimension features and the configuration of test index weights. The initial position and velocity are generated within the preset constraint boundary. A linear decreasing strategy is used to dynamically adjust the inertia weight, and a sine function is used to adaptively adjust the self-cognition and social cognition learning factors. The particle velocity and position are updated according to a preset formula. The process stops when the preset maximum number of iterations or the change in the global optimal fitness function value is less than a threshold after 5 consecutive iterations, and the optimal test configuration scheme is output.

[0009] Preferably, the specific process for performing the end-to-end test is as follows: The corresponding test case set is invoked based on the complexity of the business scenario, the system deployment environment, and the system iteration version. Each test case is executed three times to eliminate occasional errors, and the test cycle covers both peak and off-peak business periods. Collect ASR recognition results, NLP intent data, system operation logs, and TTS output audio at a frequency not lower than the specified value.

[0010] Preferably, the specific process of generating the system performance level through the comprehensive evaluation model is as follows: The real-time compliance rate of various indicators is calculated according to preset rules. The minimum value is taken for technical module indicators, and the average value is taken for performance stability, business effect, interaction experience and user feedback indicators. The comprehensive score is calculated based on the weighted average of the indicators, and the scores are divided into four levels: excellent, good, qualified, and unqualified.

[0011] Preferably, the specific process of mining the correlation links of the indicators is as follows: The core metrics are abstracted into five categories: abnormal behavior, technology, performance, business, and environment. The pheromone concentration between nodes was calculated by weighting the Pearson correlation coefficient and the frequency of historical anomalies. The ant colony was initialized and the number of ants and the maximum number of iterations were set. All ants start from the abnormal node and select the next unvisited node by roulette wheel method; After each iteration, the pheromone concentration between all nodes is adjusted according to the global update strategy to enhance the pheromone of high-quality paths and evaporate the pheromone of low-quality paths. After iterating to the preset maximum number of times, the path with the highest cumulative pheromone concentration is selected as the root cause path of the anomaly.

[0012] Preferably, the specific process for outputting the quantitative evaluation results is as follows: Integrate standardized datasets, decision support data, anomaly root cause localization results, and raw data from end-to-end testing; The quantitative evaluation results include data credibility, system performance level, and optimization priority. The optimized data is fed back into the test configuration optimization process.

[0013] The present invention also provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the method for automated testing of a voice customer service system.

[0014] The beneficial effects of this invention are: 1. By establishing a standardized multi-source data processing workflow, combined with precise outlier removal, missing value completion for classification, and unified format rules, data quality and consistency are significantly improved, providing a reliable foundation for test modeling. The particle swarm optimization algorithm is used to map core dimensional features and test indicator weights into particles. By dynamically adjusting the inertia weight and learning factor, global optimization is achieved, breaking through the limitations of traditional manual configuration that relies on experience. This generates optimal test configuration schemes that adapt to different business scenarios and deployment environments, greatly improving test accuracy and efficiency, reducing the proportion of redundant tests, and lowering test resource consumption.

[0015] 2. Execute automated full-link testing based on optimal configuration, covering peak and off-peak business periods, and comprehensively collect core component interaction data and system operation data; construct a multi-indicator weighted comprehensive evaluation model, and realize quantitative judgment and accurate profiling of system performance through classification compliance calculation and level division, replacing the traditional subjective evaluation method, clearly presenting the shortcomings of core indicators and optimization directions, providing data-driven accurate decision support for system optimization, and improving the practicality and guidance of test results.

[0016] 3. The core indicators are abstracted into ant colony algorithm nodes. A pheromone concentration model between nodes is constructed by combining Pearson correlation coefficient and historical anomaly frequency. Starting from the abnormal performance node, the optimal root cause path is explored through multiple rounds of iteration. The mechanism strengthens the high-quality path, evaporates the poor path, and filters the path by dynamically updating the pheromone. It breaks through the limitation of traditional anomaly localization that only stays on the surface indicators, accurately explores the deep causal relationship between "environment-performance-performance", quickly locates the core fault cause, significantly improves the efficiency of fault diagnosis, reduces the system repair cost and time, and ensures the stable operation of the intelligent voice interaction system. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for automating testing of a voice customer service system; Figure 2 This is a flowchart illustrating the testing process of an automated testing method for a voice customer service system. Detailed Implementation

[0018] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0019] It is understood that the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0020] like Figure 1 and Figure 2 As shown, three core data categories are first collected: historical test data, business scenario characteristics, and system deployment environment data. These are then processed through outlier removal, missing value completion, and format standardization to form a standardized dataset. Based on this dataset, three core dimensions—business scenario complexity, system deployment environment, and system iteration version—are defined with quantitative rules and five types of test indicator standards. An optimal "dimensional-indicator" configuration scheme is generated through iterative optimization using a particle swarm optimization algorithm. Subsequently, automated full-link testing is executed according to the optimal configuration, collecting data and extracting core indicators. A comprehensive evaluation model is used to generate system performance levels and decision support data. Next, an ant colony algorithm-based anomaly root cause localization model is constructed to uncover deep causal links from anomaly nodes, accurately locating the cause of the failure. Finally, the entire process data is integrated to build a traceable evaluation system, outputting quantitative evaluation results and optimized data to feed back into the test configuration optimization stage, achieving closed-loop management of system lifecycle test optimization and anomaly repair.

[0021] Furthermore, three core data categories are collected: historical test data, business scenario characteristic data, and system deployment environment data. Invalid data is removed using pre-defined outlier detection rules, and missing values ​​for continuous indicators are filled using linear interpolation, while missing values ​​for discrete indicators are filled using the mode. A unified data format standard is also implemented, ultimately forming a standardized structured dataset. The specific details are as follows: Historical test data (HTD) includes full-scenario metrics and related information for the past 12 months. Key parameters include ASR word error rate (WER) and human referral rate (HTR). ASR word error rate (WER) is the proportion of incorrectly recognized words to the total number of recognized words, calculated using the formula: ,in To identify the number of erroneous characters, integers, ; The total number of characters recognized is a positive integer. value range The Human Resource Transfer Rate (HTR) is the proportion of business volume transferred to human agents out of the total business volume. The formula is: ,in For volume transferred to manual processing, integers. ; Total business volume, a positive integer. value range Simultaneously, it retrieves the system version iteration records (format: Vx.yz, where x / y / z are non-negative integers, obtained from the version management system).

[0022] Business Scenario Feature Data (BSFD) includes Time Period Traffic Distribution (TPD) and High-Frequency Business Type Ratio (BTR). Time Period Traffic Distribution (TPD) represents the traffic fluctuation characteristics of different time periods, covering peak promotion periods (TP1) and regular periods (TP2), including traffic peaks (…). , positive integer, ), average flow rate ( Positive real numbers ), traffic trough ( non-negative integers The percentage of high-frequency service types (BTR) is the proportion of each high-frequency service type to the total service volume, including logistics inquiries (B1), after-sales complaints (B2), etc., and the formula is: ( For the number of business types, ).

[0023] System deployment environment data (SDED) includes: server configuration (SC), network architecture type (NT), and geographic node distribution (RN). Server configuration (SC) includes the number of CPU cores (…). , positive integer, ), memory specifications ( Unit: GB, positive integer ), number of nodes ( , positive integer, The network architecture type (NT) is the network mode in which the system operates, and its value is... (Discrete values, 1 represents 4G, 2 represents 5G, and 3 represents a leased line). Geographic Node Distribution (RN) refers to the geographical area where the system is deployed, with values... (Discrete values, 1 represents North China, 2 represents East China, and 3 represents South China).

[0024] To ensure data quality, a full-process preprocessing operation must be performed after data collection. First, outlier removal is performed. For all numerical indicators (such as WER, HTR, FP, CPUC, etc.), if the actual measured value is less than the preset lower threshold or greater than the preset upper threshold (both upper and lower limits refer to the clearly defined value ranges of the aforementioned parameters; for example, WER lower limit 0%, upper limit 100%, CPUC lower limit 4%, upper limit 64), it is judged as invalid outlier data and directly removed. Records with non-standard formats (e.g., version numbers not in Vx.yz format) or mismatched categories (e.g., business type tags not in the preset high-frequency business list) in the tag-based data are also removed. Second, missing value completion is performed. For continuous indicators (such as WER, FA), linear interpolation is used, with the formula: ,in To fill in missing values, The most recent valid value before it was missing. The most recent valid value after the missing value. (In time series order); discrete indicators (such as NT, RN) are completed using the mode, the formula is as follows: ,in For indicator values The number of times it appears, The value that appears most frequently is selected. Finally, the format is standardized, with timestamps formatted as YYYY-MM-DDHH:MM:SS (string type, accurate to seconds), and all indicator values ​​are rounded to two decimal places. The formula is... (Rounding calculations) ensure the consistency and comparability of data from different sources, ultimately forming a standardized structured dataset, providing accurate and reliable basic data support for subsequent multi-dimensional modeling, algorithm initialization, and test execution.

[0025] Furthermore, based on a standardized structured dataset, the quantitative classification rules for three core dimensions—"business scenario complexity, system deployment environment, and system iteration version"—are clarified. This involves dividing business scenario complexity into 1-5 levels, labeling the system deployment environment with quantifiable tags, and classifying system iteration version types. Simultaneously, the initial weights and threshold ranges for five categories of test indicators—technical modules, interactive experience, performance stability, business effectiveness, and user feedback—are determined. The particle swarm optimization algorithm is initialized by mapping the "combination of features from the three core dimensions + weight configuration of the five categories of test indicators" to particles. Constraints such as a total indicator weight of 100% and individual indicator weights of 5%-40% are set to obtain the multi-dimensional model and the initial configuration scheme for the particle swarm optimization. The specific details are as follows: Based on the obtained standardized structured dataset, quantitative classification rules for business scenario complexity (SCC), system deployment environment (SDE), and system iteration version (SIV) are determined. Business scenario complexity is a quantitative indicator that comprehensively reflects the difficulty of business interactions, with a value range of 1-5 (integers). The specific value is calculated using a multi-factor weighted formula, which is as follows: , in the formula , , The weighting coefficients were determined through regression analysis of the impact of historical business failures. , , ,satisfy ; Intent complexity represents the degree of complexity of the business intent, and is determined by extracting the number of intents and their relationships. 1 represents a single intent, and 2 represents a mixture of multiple intents; The dialogue rounds represent the average number of interaction rounds per business function, determined by statistically analyzing the average number of rounds per business function. 1 represents a simple dialogue (1-2 rounds), and 2 represents a complex dialogue with multiple rounds (≥3 rounds). Input complexity represents the degree of standardization of user input, with limited values. 1 represents standard input, and 2 represents abnormal input (including ambiguous expressions, excessively long speech, and dialectal mixing). Simultaneously, based on... The calculation results are divided into levels, with level 1 corresponding to the calculation results. (Basic scenario of single intent, simple dialogue, and standard input), Level 2 corresponds to the calculation result. (Single intent, complex multi-turn, standard input or multiple intents + simple dialogue, standard input scenario), Level 3 corresponds to the calculation result. (Single intent, complex multi-turn, abnormal input or multiple intents, simple dialogue, abnormal input scenario), the four levels correspond to the calculation results. (Multi-intent, complex multi-turn, standard input scenario), 5 levels correspond to calculation results (High-concurrency complex scenarios involving multiple intents, complex multi-turns, and abnormal inputs).

[0026] After determining the quantification rules for the complexity of the business scenario, the system deployment environment is further quantified and labeled. The System Deployment Environment (SDE) is a comprehensive set of quantified labels for the system's operating hardware and software environment, labeled according to a three-dimensional combination of "Server Architecture (SA) - Network Type (NT) - Regional Node (RN)," with parameters for each dimension extracted from the corresponding records in the dataset. Among these, the server architecture... (1 represents a single node, 2 represents a cluster; network type) 1 represents 4G, 2 represents 5G, and 3 represents a dedicated line; regional nodes 1 represents North China, 2 represents East China, and 3 represents South China. Simultaneously, to fully characterize environmental features, hardware parameters such as the number of server CPU cores, memory capacity, and dedicated line bandwidth are extracted.

[0027] After completing the quantitative labeling of the system deployment environment, the system iteration versions are further classified. System Iteration Versions Category tags for system version update types, with values... 1 represents the new feature version (the version update description contains keywords such as "new" or "new module"), 2 represents the regular optimization version (the version update description contains keywords such as "optimization", "improvement", or "adjustment"), and 3 represents the bug fix version (the version update description contains keywords such as "fix", "bug", or "vulnerability"). The update content of the corresponding functional modules, iteration time nodes (in the format YYYY-MM-DDHH:MM:SS) and related test records are extracted simultaneously for each version.

[0028] After completing the quantitative classification rules for the three core dimensions, and combining the historical performance of the data with business requirements, the initial weights and threshold ranges for five categories of test indicators—technical module indicators, performance stability indicators, business effectiveness indicators, interaction experience indicators, and user feedback indicators—were determined. All indicator weights met the following requirements. ,in Technical module metrics represent a set of performance indicators for the core technical modules of the system, including ASR word error rate (WER) and NLP intent recognition accuracy (NLP-AR), with initial weights. value range (Determined through weighted analysis of business impact based on historical test metrics), ASR word error rate (WER) threshold ≤ 8%, NLP intent recognition accuracy (NLP-AR) threshold ≥ 92%; Performance stability metrics are a set of indicators representing the stability of system operation, including system response latency (SRL), server CPU utilization (CPUU), and initial weights. Limit the range of values System response latency (SRL) is calculated from the system logs in the dataset, and is measured in seconds. Threshold ≤ 2s, CPU utilization (CPUU) threshold ≤ 70%; These are business performance metrics, a set of indicators representing the effectiveness of business processing. Core metrics include First-Time Resolution Rate (FSR) and Human Referral Rate (HTR), with initial weights. value range The formula for First-Time Resolution Rate (FSR) is: ( To address the initial business volume issue, (Total business volume), threshold ≥85%, HTR threshold ≤10%; Interaction experience metrics represent a set of indicators for user interaction experience, including TTSMOS score, user rating score, and initial weights. value range TTSMOS value The threshold is ≥3.5 points, and the user rating score is set as follows: Threshold ≥ 4.0 points; User feedback metrics represent the set of indicators directly reflected by users, including complaint rate (CR), satisfaction rating, and initial weights. value range The complaint rate (CR) formula is: ( To handle complaint volume, (Total business volume), threshold ≤1%, satisfaction rating value Threshold ≥ 4.2 points.

[0029] After determining the initial weights and threshold ranges for the five major test metrics, the particle swarm optimization algorithm is initialized. A particle (P) is the basic unit for optimization, and each particle maps to a group of candidate solutions. The particle dimension corresponds one-to-one with the core dimension and metric weight, and its expression is: In order to ensure the rationality and feasibility of particle configuration, constraint boundaries are set, including total weight constraint (the total weight of all test indicators is fixed at 100%), single weight range constraint (the weight of a single indicator is not less than 5% and not more than 40%), and core dimension value constraint.

[0030] The initial position (X) of the particle is generated within the above-mentioned constraint boundary using a random number generator, generating a value that satisfies... ,in , The initial particle velocity (V) is calculated using a linear formula to ensure reasonable particle motion amplitude. This formula is: , in the formula A random number uniformly distributed in the interval [0,1]. , The upper and lower speed limits are set based on experience. , This limits the velocity range and prevents particle position updates from deviating excessively from a reasonable range.

[0031] Through the above steps, a multi-dimensional model and initial configuration scheme for the particle swarm are finally obtained, which includes three core dimensions of quantization rules (including parameter definition, calculation formula, and value range), initial weights and threshold standards for five categories of test indicators, particle definition and constraint boundaries, and initial position and velocity parameters of particles.

[0032] Furthermore, focusing on test accuracy (matching scenario and deployment environment requirements) and test efficiency (percentage of tests without redundancy), a fitness function is designed to quantify particle performance. The particle inertia weight (balancing global and local optimization) and learning factor (guiding particles towards the optimal solution) are dynamically adjusted. Particle swarm iterations are performed through velocity and position updates, gradually bringing the particle swarm closer to the global optimum. Iteration stops when a preset number of iterations is reached or the fitness value stabilizes, and the optimal "dimension-metric" configuration scheme adapted to the current scenario is output. Details are as follows: Based on the obtained multi-dimensional model and initial particle swarm configuration, a fitness function is designed with testing accuracy and efficiency as the core. By dynamically adjusting algorithm parameters and performing iterative update operations, the global optimum is approximated, and finally, the optimal "dimensional-metric" configuration scheme adapted to the current scenario is output. The specific content is as follows: fitness function It is a comprehensive evaluation index for quantifying the merits of particles, used to measure the testing accuracy and efficiency of candidate solutions. The formula is: ,in: , The weights were determined through verification in historical test scenarios to evaluate their effectiveness. , And satisfy Test accuracy The range of values ​​represents the degree of matching between the scenario and the deployment environment requirements. Through formula Calculated, where , , The first The quantitative values ​​for the business scenario complexity, system deployment environment, and system iteration version corresponding to each test case. , , These are the maximum quantized values ​​for the three main dimensions. Total number of test cases; test efficiency The percentage of tests without redundancy, with a range of values. Through formula Calculated, where For the first One redundant test case (a test case that satisfies "repeated scenario" and "consistent indicator weight configuration"). The total number of redundant test cases, and the fitness function value. The larger the value, the better the candidate solution corresponding to the particle.

[0033] Based on the fitness function, the inertia weight and learning factor of the particle swarm optimization algorithm are dynamically adjusted: Inertia weight The key parameters for balancing global and local optimization are categorized by their value range. The update uses a linear decreasing strategy, and the formula is as follows: ,in , , This represents the current iteration number. Preset maximum number of iterations (range of values) The learning factors are determined based on the complexity of the scenario; learning factors include self-awareness learning factors ( ) and social cognitive learning factors ( The parameters used to guide particles toward convergence to the individual optimal solution and the global optimal solution, and their values ​​are all within the range of... An adaptive adjustment strategy is used for updating, and the formulas are as follows: , ,in , (Initial learning factor, set according to the algorithm's convergence characteristics) Through this dynamic adjustment strategy, the particles focus on global exploration in the early stage of iteration and on local refinement in the later stage.

[0034] During particle iteration, velocity and position updates are performed according to the following formula: Particle velocity ( The updated formula is as follows: ,in For the first particle The velocity vector of the next iteration For the first particle The position vector of the next iteration. This is the individual optimal position vector of the particle (the position with the highest fitness function value during the particle's iteration process). It is the global optimal position vector of the particle swarm (the position with the largest fitness function value during all particle iterations). , The random numbers are uniformly distributed in the interval [0,1], and each component of the velocity vector satisfies the following conditions: ( For dimensional indexing, , (consistent with the upper and lower limits of the particle's initial velocity); particle position ( The updated formula is as follows: After the position vector is updated, constraint verification is required to ensure that each dimension component meets the preset constraint boundaries: business scenario complexity (Integer), the values ​​of discrete dimensions such as server architecture and network type conform to the corresponding quantification rules, and the weights of the five major test indicators meet the constraints. If the updated position exceeds the constraints, it will be truncated to the boundary value according to the nearest principle.

[0035] The iteration process is subject to two termination conditions: one is reaching the preset maximum number of iterations (…). Secondly, the change in the global optimal fitness function value over five consecutive iterations is less than a preset threshold. The iteration stops when any of the following conditions are met. After the iteration terminates, the globally optimal position vector is output. The corresponding "dimension-metric" configuration scheme clarifies the precise matching relationship between "business scenario complexity level + system deployment environment combination + system iteration version type" and "five categories of test metric weights" to ensure that it adapts to the testing needs of the current business scenario and deployment environment.

[0036] Furthermore, automated end-to-end testing is performed according to the optimal "dimension-metric" configuration scheme, collecting all data within the testing period, including ASR recognition results, NLP intent and entity extraction data, system operation logs (server CPU utilization, network latency, etc.), and TTS output audio. Real-time performance of core metrics is extracted from the collected end-to-end data, such as ASR word error rate, NLP intent recognition accuracy, system response latency, and human intervention rate, forming a real-time metric data table. The specific details are as follows: Based on the aforementioned optimal "dimension-metric" configuration scheme, automated full-link testing is first executed. According to the "Business Scenario Complexity (SCC) - System Deployment Environment (SDE) - System Iteration Version (SIV)" combination in the configuration scheme, test case sets for the corresponding scenarios are called from the standardized structured dataset to ensure coverage of the specified complexity level, deployment environment, and version type. The test scripts call the core components of ASR, NLP, and TTS and system interfaces to simulate the real user interaction process. Each test case is executed three times to eliminate occasional errors. The test cycle is set to 2 hours (determined by combining the characteristics of peak / normal time periods extracted from the traffic distribution of the dataset to cover business peak and off-peak periods).

[0037] During the test execution, all data within the period were collected, including ASR recognition results (including the original speech file path, recognized text, and recognition timestamp, in the format YYYY-MM-DDHH:MM:SS), NLP intent and entity extraction data (including input text, intent labels, entity information, and parsing results), system operation logs (including CPU utilization, memory usage, network latency, and interface response time, collected once per second), and TTS output audio (including audio file path, corresponding text, audio duration, and MOS score pre-score).

[0038] Based on the real-time performance of five major test indicators extracted from the collected full data, among the technical module indicators, the ASR character error rate ( According to the formula calculate, This refers to the real-time error count (the number of characters differing between the recognized text and the manually annotated correct text, expressed as an integer) from the ASR recognition results. ), The total number of characters recognized in real time (positive integers) NLP intent recognition accuracy (); According to the formula calculate, The number of times the NLP parsing result is correct in real time (integer). ), The total number of recognitions in real time (positive integer). The data comes from the NLP component processing logs.

[0039] Among the performance stability metrics, the average system response delay ( According to the formula calculate, Single response delay (unit: seconds) ), For real-time counting of positive integers, Average server CPU utilization ( According to the formula calculate, CPU utilization per cycle (in %) ), The total number of data collections (positive integers) (corresponding to 2 hours × 3600 seconds / hour), the data comes from the CPU load records of the server monitoring platform.

[0040] Among the business performance metrics, the first-time resolution rate ( According to the formula calculate, To resolve the first instance of business volume (integer, ), The total real-time traffic volume (positive integer, The headcount rate (HTR) is calculated using the formula... calculate, For real-time conversion of customer service volume to human assistance (integer, ).

[0041] In the interactive experience metrics, TTSMOS score ( Extract the quality detection results corresponding to the TTS output audio, with a value range of... (Positive real number, rounded to one decimal place); User rating score ( ) Simulates user ratings using automated scripts (based on interaction fluency, response speed, and voice quality), with a range of values. (Integer).

[0042] Among user feedback metrics, the complaint rate ( According to the formula calculate, For real-time complaint volume (integer, ); Satisfaction rating ( Extracted from simulated user feedback logs, with a value range of... (Integer).

[0043] After extracting the real-time performance of the five major categories of indicators, a comprehensive judgment process based on core indicators is conducted, using the initial weights of the five major categories of indicators in the optimal configuration scheme. , , , , Based on this, a comprehensive performance scoring model (S) is constructed, which transforms scattered indicators into a unified-dimensional overall system performance evaluation value. The formula is as follows: ; in , , , , Real-time compliance rate (range of values) for various indicators The compliance calculation strictly matches the preset thresholds for each indicator, and the specific rules are as follows: Technical module indicator compliance Take the minimum value between the ASR word error rate and the NLP intent recognition accuracy target. but , but , but ,like but , but , but ,final ; Performance stability index compliance Take the average of the system response latency and the CPU utilization achievement rate. but , but , but ,like but , but , but ,final ; Business performance indicators met Take the average of the initial resolution rate and the rate of manual intervention to meet the target. but , but , but ,like but , but , but ,final ; Interactive experience metrics compliance Take the average of the TTSMOS score and the user rating score to determine the degree of compliance. but , but , but ,like but , but , but ,final ; User feedback metrics met the standards Take the average of the complaint rate and the satisfaction score achievement rate. but , but , but ,like but , but , but ,final .

[0044] Based on overall performance Classify system performance levels: "Excellent" means all indicators have reached the optimal threshold and no optimization is needed. "Good" means the core indicators meet the standards, and non-core indicators can be slightly optimized. The rating is "qualified" (but there are shortcomings in the core indicators, which require specific optimization). The result is marked as "unqualified" (core indicators do not meet the standards, and risks need to be addressed first), and a comprehensive assessment report is generated that includes "overall score - compliance details - level determination - optimization suggestions".

[0045] Finally, the real-time performance of all core indicators and the comprehensive judgment results were compiled into a real-time indicator data table in a unified format. This data table is generated in real time by automated tools, with data integrity ≥99% and accuracy error ≤0.5%. It not only retains the original indicator data to support subsequent ant colony algorithm modeling and anomaly root cause localization, but also clarifies the current status of the system and the direction of optimization through comprehensive judgment results, realizing the seamless connection of the entire process of "data collection - indicator extraction - comprehensive judgment - application support".

[0046] Furthermore, core metrics from modules such as ASR, NLP, and dialogue management are abstracted as "nodes" in the ant colony algorithm. The correlation strength between metrics is calculated by combining historical test data with real-time data of the current metrics, serving as the "pheromone concentration" between nodes. The ant colony is initialized, starting from nodes exhibiting abnormal behavior (such as abnormal system response delays or a sudden increase in complaint rates). Based on pheromone concentration and heuristic factors, the next-hop node is selected to explore the root cause path of the anomaly. After multiple iterations and updating the pheromone (increasing pheromone on high-quality paths and evaporating pheromone on low-quality paths), the path with the highest pheromone concentration is selected as the final root cause path of the anomaly. The specific details are as follows: Based on the generated real-time indicator data table (including comprehensive judgment results) and historical test datasets, an anomaly root cause localization model is constructed using the Ant Colony Algorithm (ACO). First, the core indicators of modules such as ASR, NLP, and dialogue management are abstracted as "nodes" of the algorithm. The node classification strictly matches the five established test indicator categories: anomaly behavior nodes (…). To directly reflect system anomalies, a limited set of values ​​is defined. (like , That is, nodes marked as exhibiting abnormal behavior); technical module nodes ( The corresponding technical module indicator and its value set. Performance module node ( The corresponding performance stability index has a set of values. ( Extracted from system operation logs, unit: ms, threshold ≤100ms); business module node ( The corresponding business performance metrics, and their value set. ; Environment module node ( The corresponding system deployment environment parameters, and their value set. ( Unit: Mbit / s, threshold ≥100 Mbit / s; Unit: %, threshold ≤ 80%), all nodes are assigned a unique identifier (e.g. Represents nodes with abnormal system response delays. The node representing the ASR word error rate is extracted from the real-time data table (current data) and the historical test dataset (data from the past 12 months).

[0047] Next, the pheromone concentration between nodes is calculated ( ), which was first defined as a node With nodes The strength of the correlation between them is calculated by combining the Pearson correlation coefficient with a weighted average of historical anomaly frequencies, as shown in the formula: The weighting coefficient ( The value is 0.6 (determined through historical anomaly root cause review), which satisfies... Pearson correlation coefficient ( Compute nodes With nodes The linear correlation of the indicator series is given by the formula: ( , They are nodes , The numerical sequence of indicators, sample size , Take the absolute value Mapping association strength); percentage of historical co-occurrences ( )middle, Nodes in historical data With nodes The number of times an anomaly occurs simultaneously (integer, ), Nodes in historical data Total number of anomalies (positive integers) ), pheromone concentration initial value .

[0048] Then, ant colony initialization and path exploration were performed. The ant colony initialization parameters were set as follows: number of ants ( 20 (based on the total number of nodes) Sure, ), each ant ( , ) represents a root cause investigation path; maximum number of iterations ( 50 (determined by algorithm convergence testing, path stability ≥ 95% after 50 iterations); heuristic factor ( This heuristic information guides ants in choosing their next hop node and is positively correlated with pheromone concentration, as shown in the formula: ( For nodes With nodes "Distance of abnormal impact", unit: min , ); pheromone volatility coefficient ( 0.1 (Balancing pheromone updates and path diversity). All ants originate from nodes exhibiting abnormal behavior ( Starting from there, select the next node using the roulette wheel method. by ants The probability of selection ( The formula is: ; Among them, the importance factor of pheromones ( The heuristic factor importance factor is 1.5. ) is 1.0, allowing selection of sets ( ) for ants The current set of unvisited nodes; the path exploration terminates when the ant reaches the "environment module node" ( ") or explore 5 steps.

[0049] After each iteration, the pheromone concentration among all nodes is adjusted according to the "global update" strategy, using the following formula: Among them, the pheromone volatile item ( To achieve natural pheromone release in each round, the pheromone increment ( According to the "elite ant strategy", the formula is as follows: ( The pheromone increment constant is 10. For ants The inverse of the "root cause credibility" of the exploration path. , This represents the probability that the path matches the true root cause. ).

[0050] When the maximum number of iterations is reached ( After that, the "cumulative pheromone concentration" of all paths is calculated. The path with the highest cumulative pheromone concentration is selected as the final root cause path of the anomaly. The formula for calculating the cumulative pheromone concentration for each path is as follows: ( (The complete path explored by the ant), if multiple paths exist. If they are the same, then the one containing them will be selected first. The path. When outputting the final root cause chain, mark the degree of abnormality of each node (e.g., " The system clarifies the causal relationship of "environmental anomaly → performance anomaly → behavior anomaly" and this link can be directly linked to the optimization suggestions in the aforementioned comprehensive judgment report. If it includes a "BW insufficient" node, the solution of "expanding the dedicated line bandwidth to ≥100Mbit / s" will be given priority, providing accurate basis for system problem location and repair.

[0051] For example, historical test data, business scenario characteristic data, and system deployment environment data were collected over the past 12 months. The historical test data includes ASR error rate (WER average 7.2%), human intervention rate (HTR average 8.5%), and system version iteration records (V2.0.0 to V2.3.0). In the business scenario characteristic data, peak traffic during promotional periods (TP1) was 8000 times / hour, with an average of 5000 times / hour; peak traffic during normal periods (TP2) was 3000 times / hour, with an average of 1500 times / hour. High-frequency business types accounted for 45% of traffic inquiries, 30% of after-sales complaints, and 25% of order inquiries. The system deployment environment data shows an East China region (RN=2), 5G network (NT=2), cluster server architecture (SA=2), 32 CPU cores, 128GB of memory, and 8 nodes.

[0052] Standardized preprocessing: Remove abnormal data such as WER>10% and CPU core count<4; use linear interpolation to complete one missing average traffic data point for a major promotion period, and use the mode to complete two missing network architecture type data points; unify the value of indicators to two decimal places to form a standardized structured dataset.

[0053] Core Dimension Quantification: The complexity of the business scenario is calculated using the formula SCC=0.4×IC+0.3×DC+0.3×ICF. The current scenario is multi-intent (IC=2), complex multi-turn (DC=2), standard input (ICF=1), and SCC=0.4×2+0.3×2+0.3×1=1.7, corresponding to level 4 complexity. The system deployment environment is labeled "2-2-2" (cluster-5G-East China). The system iteration version is classified as a regular optimized version (SIV=2).

[0054] Test metric configuration: Set the initial weights of five categories of metrics, namely technical module, performance stability, business effect, interaction experience and user feedback, to 30%, 25%, 20%, 15% and 10%, respectively. The thresholds are preset according to the plan (e.g., WER≤8%, system response delay≤2s).

[0055] Particle Swarm Optimization: The "Level 4 Complexity + 2-2-2 Environment + SIV=2 + Indicator Weight Configuration" is mapped to particles. Constraints are set at a total weight of 100% and individual class weights of 5%-40%. Particle positions and velocities are initialized, and the inertia weight is adjusted (from 0.9 to 0.4) using a linear decreasing strategy. The learning factor is adaptively adjusted, and after 80 iterations, the fitness function value stabilizes, outputting the optimal configuration: 25% weight for technical modules, 30% for performance stability, 25% for business effectiveness, 12% for interactive experience, and 8% for user feedback.

[0056] The test case set for the corresponding scenario is invoked according to the optimal configuration. Each test case is executed repeatedly 3 times, with a test cycle of 2 hours (covering 1 hour during peak promotion periods and 1 hour during regular periods). All data, including ASR recognition results, NLP intent data, and system operation logs, are collected. The real-time performance of core indicators is extracted as follows: WER=6.8%, NLP intent recognition accuracy 93.5%, average system response latency 1.8s, average CPU utilization 65%, first-time resolution rate 88%, manual referral rate 7.2%, TTS MOS score 3.8, and user satisfaction score 4.3.

[0057] Overall Assessment: Based on the compliance calculation rules, the technical module compliance is 1.0, performance stability compliance is 1.0, business effect compliance is 1.0, interactive experience compliance is 1.0, and user feedback compliance is 1.0. The overall score S = 0.25×1.0 + 0.3×1.0 + 0.25×1.0 + 0.12×1.0 + 0.08×1.0 = 1.0, which is judged as "Excellent".

[0058] Anomaly Simulation and Localization: A scenario of insufficient dedicated line bandwidth (BW=80Mbit / s, 20% below the threshold) was simulated. The test was re-executed, and the average system response latency increased to 2.8s (HTR increased to 12%). An anomaly root cause localization model was constructed using an ant colony algorithm. Anomaly manifestation nodes (SRL anomaly, HTR anomaly), technical module nodes, performance module nodes, and environmental module nodes were abstracted as algorithm nodes. The pheromone concentration between nodes was calculated. After 50 iterations, the path with the highest cumulative pheromone concentration was selected: Environmental module node (insufficient BW) → Performance module node (network latency 150ms) → Anomaly manifestation node (SRL anomaly → HTR anomaly). This clarified that the root cause was insufficient dedicated line bandwidth leading to increased network latency, which in turn caused increased system response latency and manual intervention rate.

[0059] Integrate data from the entire process to build a traceable evaluation system and output quantitative evaluation results: data credibility 99.2%, system performance level "excellent" (normal scenario) / "qualified" (simulated abnormal scenario). Optimize data feedback to the test configuration stage and generate targeted optimization strategies: expand the dedicated line bandwidth of nodes in East China to ≥100Mbit / s, simultaneously adjust the weight of performance stability indicators to 35%, and increase the weight of network performance testing in high-concurrency scenarios.

[0060] Through the implementation of this solution, the system's ASR error rate was reduced to 6.8% and the rate of transferring to human operators was reduced to 7.2% under normal circumstances, both of which are better than the preset thresholds. Under simulated abnormal scenarios, by accurately locating the root cause and implementing optimization strategies, the system response latency was restored to 1.7s and the rate of transferring to human operators was reduced to 7.0%, which significantly improved the system's ability to adapt to dynamic business scenarios and ensured the stable operation of the system during peak sales periods.

[0061] The processes described above with reference to the flowcharts in the embodiments disclosed in this invention can be implemented as computer software programs. The embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0063] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the stated principles, the implementation of the present invention may have any variations or modifications.

Claims

1. A method for automated testing of a voice customer service system, characterized in that, The method includes: Collect data from multiple sources and perform standardized preprocessing to form a standardized dataset; Based on the standardized dataset, a multi-dimensional quantitative model is constructed to determine the core dimension classification rules and test indicator standards. The optimal test configuration scheme is generated by iterative optimization of the particle swarm intelligent optimization model through multi-dimensional test configuration. Based on the optimal test configuration scheme, test data is collected and core indicators are extracted through full-link testing. The system performance level and optimization direction are generated through a comprehensive evaluation model, forming decision support data. Based on decision support data, an anomaly root cause localization model is constructed to mine the correlation links of indicators and locate the core causes of system anomalies. Integrate data from the entire process to build a traceable evaluation system, and output quantitative evaluation results and optimization data.

2. The method for automated testing of a voice customer service system according to claim 1, characterized in that, The specific process for obtaining the standardized dataset is as follows: Multi-source data encompasses historical test data, business scenario characteristic data, and system deployment environment data; Abnormal data for numerical indicators and data with non-standard formats or mismatched categories are removed according to preset thresholds; For continuous indicators, missing values ​​are filled using linear interpolation, while for discrete indicators, missing values ​​are filled using the mode. Standardized indicator values ​​are retained to two decimal places to ensure data consistency.

3. The method for automated testing of a voice customer service system according to claim 2, characterized in that, The specific process for determining the core dimension classification rules and the test indicator standards is as follows: The core dimensions include business scenario complexity, system deployment environment, and system iteration version. Business scenario complexity is classified into levels by weighted average of intent complexity, dialogue rounds, and input complexity. System deployment environment is labeled by server architecture, network type, and combination of regional nodes. System iteration version is divided into new feature version, regular optimization version, and bug fix version. The test index standard includes five categories of indicators: technical modules, performance stability, business effectiveness, interactive experience, and user feedback. The total weight is 100%, and the weight of each individual indicator is limited to 5%-40%. Each indicator also has a pre-defined threshold range.

4. The method for automated testing of a voice customer service system according to claim 3, characterized in that, The specific process of iterative optimization by the particle swarm intelligent optimization model is as follows: Particle mapping is a candidate scheme for the combination of core dimension features and the configuration of test index weights. The initial position and velocity are generated within the preset constraint boundary. A linear decreasing strategy is used to dynamically adjust the inertia weight, and a sine function is used to adaptively adjust the self-cognition and social cognition learning factors. The particle velocity and position are updated according to a preset formula. The process stops when the preset maximum number of iterations or the change in the global optimal fitness function value is less than a threshold after 5 consecutive iterations, and the optimal test configuration scheme is output.

5. The method for automated testing of a voice customer service system according to claim 4, characterized in that, The specific process of performing the full-link test is as follows: The corresponding test case set is invoked based on the complexity of the business scenario, the system deployment environment, and the system iteration version. Each test case is executed three times to eliminate occasional errors, and the test cycle covers both peak and off-peak business periods. Collect ASR recognition results, NLP intent data, system operation logs, and TTS output audio at a frequency not lower than the specified value.

6. The method for automated testing of a voice customer service system according to claim 5, characterized in that, The specific process of generating system performance levels using the comprehensive evaluation model is as follows: The real-time compliance rate of various indicators is calculated according to preset rules. The minimum value is taken for technical module indicators, and the average value is taken for performance stability, business effect, interaction experience and user feedback indicators. The comprehensive score is calculated based on the weighted average of the indicators, and the scores are divided into four levels: excellent, good, qualified, and unqualified.

7. The method for automated testing of a voice customer service system according to claim 6, characterized in that, The specific process of mining the correlation links of the aforementioned indicators is as follows: The core metrics are abstracted into five categories: abnormal behavior, technology, performance, business, and environment. The pheromone concentration between nodes was calculated by weighting the Pearson correlation coefficient and the frequency of historical anomalies. The ant colony was initialized and the number of ants and the maximum number of iterations were set. All ants start from the abnormal node and choose the next unvisited node according to the roulette wheel method; After each iteration, the pheromone concentration between all nodes is adjusted according to the global update strategy to enhance the pheromone of high-quality paths and evaporate the pheromone of low-quality paths. After iterating to the preset maximum number of times, the path with the highest cumulative pheromone concentration is selected as the root cause path of the anomaly.

8. The method for automated testing of a voice customer service system according to claim 7, characterized in that, The specific process for outputting the quantitative evaluation results is as follows: Integrate standardized datasets, decision support data, anomaly root cause localization results, and raw data from end-to-end testing; The quantitative evaluation results include data credibility, system performance level, and optimization priority. The optimized data is fed back into the test configuration optimization process.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement a method for automated testing of a voice customer service system as described in any one of claims 1-8.

Images